Welcome back! We hope that you’re enjoying these introductory blogs about Oracle Cloud Infrastructure (OCI) Data Integration. To get the list of previous blogs related to Data Integration, refer to the Data Integration blog. Today, we’re learning about the different file patterns that you can use for selecting and processing one or more data entities in a single task in OCI Data Integration.
File patterns are sets of rules. You can define a file pattern by setting up rules to find matching files. The file pattern is usually based on the file name. File and directory names are compared to patterns to include (or sometimes exclude) them in a task. It’s also used to handle the matched files when found.
Syntax to use file patterns
OCI Data Integration supports glob patterns. You can specify their use in the source operator within Data Flows or the source section of Data Loader Task. Use data asset for your Object Storage type. For more information on the syntax, you can refer to the Syntax to use section (note: Syntax is specified under Source Operator -> Using File Patterns -> Syntax to use) in the documentation.
Implementing file patterns
The source operator in OCI Data Integration data flows now supports processing multiple files from folder paths, a list of files (file sets), and wildcards. The wildcards fully support Linux file globbing capability without needing to use looping constructs in a control flow.
To pick files using a pattern, use the following steps:
- Add the source operator in the canvas.
- Click Select for your data entity.
- Select Browse By Pattern.
- When clicking Browse By Pattern, we can define the required patterns and select the required files from the Object Storage.
In the following screenshot, all the *.csv files are selected from all the partition directories by defining “country=*.”
After clicking Select Pattern, the provided pattern is attached to the data entity of the source operator.
The data entity can also be parameterized, and during the task definition or runtime, different patterns can be assigned accordingly.
File pattern examples
The following table provides the example of different patterns that can be used with OCI Data Integration:
Conclusion
Now, we can use the file patterns and read different files in OCI Data Integration.
Have you seen some of our other blogs? Check out all the blogs related to Oracle Cloud Infrastructure Data Integration. To learn more, check out some Oracle Cloud Infrastructure Data Integration tutorials and documentation.