If you enable caching in a Lookup transformation, the Integration Service builds a cache in memory to store lookup data. When the Integration Service builds a lookup cache in memory, it processes the first row of data in the transformation and queries the cache for each row that enters the transformation. If you do not enable caching, the Integration Service queries the lookup source for each input row.
The result of the Lookup query and processing is the same, whether or not you cache the lookup source. However, using a lookup cache can increase session performance. You can optimize performance by caching the lookup source when the source is large.
If the lookup does not change between sessions, you can configure the transformation to use a persistent lookup cache. When you run the session, the Integration Service rebuilds the persistent cache if any cache file is missing or invalid.
The Integration Service creates the following caches for the Lookup transformation:
Data cache.
For a connected Lookup transformation, stores data for the connected output ports, not including ports used in the lookup condition. For an unconnected Lookup transformation, stores data from the return port.Index cache.
Stores data for the columns used in the lookup condition.
The Integration Service creates disk and memory caches based on the lookup caching and partitioning information.
The following table describes the caches that the Integration Service creates based on the cache and partitioning information:
Lookup Conditions | Disk Cache | Memory Cache |
|---|---|---|
| One disk cache for all partitions. | One memory cache for each partition. |
| One disk cache for all partitions. | One memory cache for all partitions. |
| One disk cache for each partition. | One memory cache for each partition. |
When you create multiple partitions in a session with a Lookup transformation and create a hash auto-keys partition point at the Lookup transformation, the Integration Service uses cache partitioning.
When the Integration Service uses cache partitioning, it creates caches for the Lookup transformation when the first row of any partition reaches the Lookup transformation. If you configure the Lookup transformation for concurrent caches, the Integration Service builds all caches for the partitions concurrently.
