Process Flow - Execute a map for all files in a directory
By David Allan on Jul 05, 2007
How to call a map for every file in a directory? Flat file maps have the data file name as a parameter. The map can be called from a flow and the data file name set, the map will be wrapped in a while loop that iterates over every file in a directory. Here is an illustration, it uses the functions getfilecount and getnthfile from the post here.
The flow looks like you'd expect for a basic programming loop;
- an initialization block to get the number of files in a directory, I have the directory as a parameter to the process flow too. The initialization is the GETFILECOUNT activity which uses the function from the blog post linked above.
- the while statement with the condition (iterate for every file in directory using the index)
- get the nth file in the directory using the function GETNTHFILE
- execute the map (LOAD_F2) passing the fully qualified name of the file
- increment the index
- end loop
The process flow uses the following variables and parameters;
In the above image DIR_PATH is a process flow parameter, so the flow can be executed and directory specified upon execution. There are a few variables; one for holding an index for the loop (FILE_INDEX), one for holding the file name returned from getnthfile (FILE_NAME) and another variable for the number of files (NO_OF_FILES).
The while condition for the loop is defined as loop while index is less than the number of files;
The increment index (ASSIGN) activity inside the while loop looks like
This lets us easily execute a single map many times with a group of files. Its also possible to separate the control file/log file location so they can be written to one directory and source data files processed from another (so getnthfile will not process control files etc..).