Handling those Pesky Dependencies with the GoldenGate 21c for BigData Dependency Downloader

September 28, 2021 | 4 minute read
Sydney Nurse
EMEA AppDev & Security Expert
Text Size 100%:

Handling those Pesky Dependencies with the GoldenGate 21c for BigData Dependency Downloader

Dependency Headaches!

You have them? Me too! Every new install, new version and adapter, updates the list of jars that are needed for the different handlers to work correctly. We don't need this kind of stress in our Data integration projects for Oracle GoldenGate, where things are supposed to be simple & easy, kind of like Autonomous DB.

Do we have a comprehensive list of this version = that set of jars and lib files? 

Managing versions for all the Open Source projects is a difficult task and any provided list would be quickly outdated, which highlights how great the Dependency Downloader in release 21c of GoldenGate for BigData will be.

The What's New section of the Release notes tells us that this is a set of utility scripts that have been added to download 3rd party dependency libraries for various integrations and that all we need to supply for each script is the desired version as an argument.

A small note on the documentation

The release note provides the default location as {OGGBD install}/DependencyDownloader which is true for the Classic architecture installation. i.e. unzip & un-tar the software downloadable from eDelivery. (As of 15.Sept.2021) only classic is currently available).

Those of you leveraging the OCI Marketplace image and the Microservices Architecture (MSA) will find the file here: /u01/app/ogg/opt/DependencyDownloader

About the Dependency Downloader

The Dependency Downloader is not a single tool but a set of shell scripts, one per supported Adapter Handler in fact. The scripts fall into two main categories

  1. Search for artefacts
  2. Use a known Repository 

In either case, Apache Maven is used to search, download, build and package the required dependencies. Maven is one of those tools you will find in CI/CD processes & DevOps organisation, providing consistent builds and deployments of projects.

Running the Dependency Downloader Scripts

Execution is simple

  1. Navigate to the directory for the DependencyDownloader - see the small note above
  2. If a PROXY is required, edit the config_proxy.sh script as per the documentation
  3. Run the script for the BigData handler that you will be implementing, supplying the version* that matches
    1. A supported version
    2. The version being implemented for the project

*version - some handlers can be used without a specific version, like Snowflake

That's it on the OCI Marketplace image and the script created a subdirectory in the ./dependencies/ directory for the script and version that I supplied.

A small note on Proxies and Self-Signed Certificates

During a recent customer project, things were not so simple. A proxy was required which we configured in the config_proxy.sh script but during the execution it reported that the proxy_port=0 and failed to download the required package files.

To work-around this, I commented out the proxy configuration variables in config_proxy.sh script and used the normal session environment variables to set up the proxy:

  • export HTTP_PROXY <proxyHost:proxyPort>
  • export HTTPS_PROXY <proxyHost:proxyPort>

This took us a step further but not ver far, as the script failed again. This time with a warning

 [WARNING] Could not transfer metadata or POM

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This issue, or similar, may occur when an attempt is made to download dependencies from a HTTPS repository that uses self-signed certificates.

This is common and most HTTP client tools provide options to ignore verification and certificate validation. Maven system properties are no different and the default behaviour can be changed. 

There is a simple trick is to identify which script executes the Maven client dependency download process. In each of the handler dependency downloader scripts, the last line shows the location and script that will eventually call the Maven tool.

Just before the very last line there will be a comment like this:

#Execute the script to execute the maven project

Followed by:

. ./internal_scripts/<downloader_execution_script_name>

For my case, we simply made a backup of the supplied script and added these 3 system properties just before the proxy parameters

  • -Dmaven.wagon.http.ssl.insecure=true 
  • -Dmaven.wagon.http.ssl.allowall=true 
  • -Dmaven.wagon.http.ssl.ignore.validity.dates=true

All on a single line, of course :)

In Summary

GoldenGate 21c has made significant strides in simplifying the installation, deployment, configuration and management of GoldenGate. The Dependency Downloader is a great additional, providing a set of utility scripts that help teams implement the BigData handlers.

I am hoping we can have this included for all deployments, like JDBC drivers for 3rd party RDBMs and exposed as well via the Service Manager or Administration Server UI.

That would be great, but for know I am happy to settle with passing a version number to a script via CLI.

Hope you find this information useful and helps speed up your next project.

Regards

Sydney

Sydney Nurse

EMEA AppDev & Security Expert

One thing Golf teaches you, is attention to detail, practice to execution and trust.

These same principals apply to expertise in an given area, and even more so in Software Systems.

 

Show more

Previous Post

Oracle GoldenGate 21c Release Announcement

Mack Bell | 2 min read

Next Post


GoldenGate for Big Data 21.3.0.0.0 is now available

Julien TESTUT | 2 min read
Oracle Chatbot
Disconnected