GoldenGate for Big Data “File Writer Handler” is used to generate the files from the trail files and “Event Handlers” loads these files to the target systems. File Writer Handler creates a file in the local file system and keeps it open till file rolls over. Once an open file is closed and rolled over to necxt one, selected event handler picks that file and loads into the desired target system.
For OCI GoldenGate for Big Data, these local files will not be accesible to end users as there is no compute access in OCI GoldenGate for Big Data.
File roll overs are triggered by a metadata change event and/ or they can be configured based on some rules.
There are 4 cases here:
1. Reaching Max File Size: By default, file sizes are configured to 1gb; but, it can be configured as needed by “gg.handler.name.maxFileSize” property. In this case, File Writer Handler opens the file and keeps it open till Max File Size is reached (assuming there are no metadata changes). When the size is reached, file is closed, and a new file is generated.
For example: gg.handler.filewriter.maxFileSize=1g
2. Reaching File Roll Interval: By default, roll interval is not active; but, it can be configured by “gg.handler.name.fileRollInterval” property. When configured, it starts a timer when file is created. When the interval timing is reached, if the files is still open, it closes the file and rolls over to a new one. Let’s assume that you set the Max File Size as 5gb and File Roll Interval as 30 minutes. At the end of the 30 minutes, if the file still open, it will close and roll over to a new one. If Max File Size is reached earlier, it will be the trigger for roll over action.
For example: gg.handler.filewriter.fileRollInterval=10m
3. Reaching Inactivity Roll Interval: By default, inactivity roll is not active; but, it can be configured by “gg.handler.name.inactivityRollInterval” property. When configured, it starts a timer for tracking the inactivity period. Here, inactivity means there are no operations coming from the source system. In other words, there are no CDC data being written to the file. When set, it starts the count down when the last operation is written to a file. At the end of the count down, if there are no incoming operations, file is closed. It can be used together with the previous 2 properties. For example, you set the file size as 5gb, roll interval as 30 minutes and inactivity roll interval as 5 minutes. In this case, file is closed when one of these rules is fulfilled.
For example: gg.handler.filewriter.inactivityRollInterval=5s
4. Roll on Shutdown: By default, roll on shutdown is set to false; but can be configured by “gg.handler.name.rollOnShutdown” property. When configured, it shuts the open file when replicat process is stopped. By default, file writer handler keeps the file open even if the replicat stops and continues writing to the same file when the replicat restarts. This behaviour may change if any previous roll over rules is set.
For example: gg.handler.filewriter.rollOnShutdown=true
For more details on File Wirter Handler, you can refer to GoldenGate for Big Data Documentation from this link.