X

Welcome to All Things Data Integration: Announcements, Insights, Best Practices, Tips & Tricks, and Trend Related...

See How Easily You Can Use Oracle GoldenGate Rest API

Sydney Nurse
Data Integration Architect

You can use the Oracle GoldenGate MicroServices Architecture Restful services too!

Why would you care?

  1. The Microservices Archtecture is Oracle's GoldenGate strategy. 
  2. Classic architectures and GGSCI command line maybe simple but present different risks as OS access is difficult to control and monitor in an active development or operations deployment.
  3. Understanding how to use Rest APIs and JSON is a valuable skill to have

The Oracle GoldenGate (OGG) Microservices Architecture (MSA) APIs provide a current technology set to manage, monitor and migrate OGG deployments; Development to Test or Production to QA

Many organisations are implementing cloud solutions or considering the roadmap of OGG in the cloud or as a cloud service and the path to adopt these future services. MSA and its Restful APIs are the stepping stones in succeeding with those future services and projects.

The Oracle GoldenGate Architecture Revolution

Oracle GoldenGate Microservices Architecture (MSA) was announced back in 2017, introduced the re-architected Oracle GoldenGate (OGG) product, a new Web UI, enhanced database features and that was restful enabled, exposing access to all monitoring, configuration and administration tasks as services.

Many customers still opt for the classic architecture and the ability to reuse custom scripts to automate OGG, even as newer releases communicated the MSA strategy. Existing and new projects driven by implementation partners found it far easier to re-use previous work, copying & pasting between projects than to learn a technology set more suited to Mobile & Integration projects than Data Replication.

The Old or Classic and Microservice Architectures both support all the standard replication use cases and interoperability between versions and platforms, such as Non-Oracle, Oracle, and Big Data. The MSA architecture is only available for Oracle currently with Non-Oracle and Big Data as roadmap items as to today, Summer 2020.

Restful Services and DBAs?

Rest APIs is synonymous with JSON (JavaScript Object Notation) has been in the domain of application integration for some time now and is seen as the successor to SOAP  (Simple Object Access Protocol) and XML (eXtensible Markup Language) which was found to be anything but Simple.

Once Rest and JSON were combined its adoption grew and even found its way into the Oracle Application Express Listener back in release version 10gR2 which required a Java Application Server to function. In today's current release of the database it is embedded with its own set of hosting components direct from the database.

Rest is no longer foreign to the DBA in 2020 as they can both expose data services as well as administer the database. Why should this be noted here, because DBAs tend to be at the center of Data Replication projects and the relationship between the Oracle Database and OGG has been enhanced with a set of deep integrations.

The ART of the Possible with OGG MSA - a Restful Day is a Good Day

Let's start with what you can do with the OGG Rest APIs

The APIs are broken into three main categories, at least the ones I'm interested in:

  1. Overall Service Management
  2. Replication Configuration
  3. Monitoring Performance

 

A deeper understanding of the microservices architecture is required, so reviewing the Getting Started with Oracle GoldenGate is recommended as installation components, deployments and types of services are only touched on here.

Regardless of the type or API, a {version} is always required and lucky for us this is currently v2

Overall Service Management

In the new architecture OGG is installed with a set of shared binaries which are accessed by the different server services (Administration, Distribution, Receiver, Performance Metrics) from a deployment. All of the deployments and services related to the replication process are overseen by the Service Manager.

The structure and use of the Service Manager service are simple:

  • Gather Information about the Installation Deployments !Not so useful!
  • Gather Information about the Deployment Services
  • Manage Deployments & Services (recommend using the Configuration Assistant for these instead)

All of the Service Manager services can be found under the base uri of https://<your_ogg_msa_server>/services/v2/

I'll refer to the host uri as a variable host_uri = https://<your_ogg_msa_server>

examples:

  • To get a list of Deployments use  host_uri + /services/v2/deployments
  • To get details about a specific Deployment use host_uri + /services/v2/deployments/{deployment_name}
  • To get a list of services running in a specific Deployment use host_uri + /services/v2/deployments/{deployment_name}/services

It is important to get the names of the various deployments as they are required to use all other Server services APIs discussed in the next section. For example if I wanted to get all the parameter files from my instance, I would first need to find out how many deployments there are and then reference each deployment in subsequent requests to get all the parameter files.

For Example: 

If my deployment is called ogg19c then I would connect to https: host_uri + /services/v2/deployments/ogg19c/services to list all of the services for that deployment.

In a browser session, connect to the OGG MSA Service Manager home site, login.

Append to your host_uri the Rest API (/services/v2/deployments) to list the instance deployments and this will return a json payload of the deployments available:

The last section of "links" lists the names of the deployments for the instance and provides the "href" to the API for the deployment. Copy the "href" uri and append the Rest API (/services) to list the services for the deployment and this will return a json payload of the services available:

That's how simple it is and you've already used two of the MSA APIs already even if it was only via the browser. What have we demonstrated here?

  1. Authentication is required to use the services, perhaps you missed this as we used the browser but the first thing we did was log into the Service Manager home
  2. Using the host address as the base, we can simply append the APIs

**tip** While using any of the MSA website pages, have a look at the browser address and you will begin to under stand the format and use the APIs.

Let's cover a few more key service groups and then provide a small python sample of building up the service uri and extracting the values from one of those responses.

Replication Service APIs

The End-to-End Replication credentials, configuration, processes, management & logging can be found under the Administrative Server.

The Administrative Server has the more interesting services and covers:

It should be noted that the base uri to invoke the services is not the same as the Service Manager APIs.

Where the Service Manager APIs start from  host_uri + /services/v2/ the other services must refernce both the deployment and the service host_uri + /services/<deployment>/<service>/v2/. For example to list Extracts or Replicats on the Administrative Server:

  • For Extracts - host_uri + /services/<deployment>/<service>/v2/extracts
  • For Replicats - host_uri + /services/<deployment>/<service>/v2/replicats

It is a subtle difference and not clearly stated in the documentation, again I suggest to pay close attention to the address in the browser session to discover the correct uri format.

  • Administraive - host_uri + services/ogg19c/adminsrvr/v2/content/index.html
  • Distribution - host_uri + /services/ogg19c/distsrvr/v2/content/index.html
  • Receiver - host_uri + /services/ogg19c/recvsrvr/v2/content/index.html
  • Performance Metrics - host_uri + /services/ogg19c/pmsrvr/v2/content/index.html

Once you get comfortable with this construct the rest is much easier to understand, now let's see some code examples!

Scripting administration of MSA with Python

I will use Python for my examples and provide some code snippets but any programming language can be used or via command line with the AdminClient. Experience with programming and your language of choice is assumed here, so not all code is shown nor clarified.

Python provides functions to support web service calls and parsing of JSON message bodies, which are built up of arrays, and string key pairs. 

Use of property files, constants and different utility classes is suggested along with the main script.

  • Property Files - Use to store instance related properties, allowing use across different instances
    • Host Name / IP address
    • User Credentials
    • Certificate location
  • Constants - Use to store static values referenced with the main script
    • Names
    • Versions
    • Filter Lists
  • Utility Classes - Used for full implementations of related functionality that can be re-used in different projects and main scripts
    • A Rest utility
    • A File & Director utility
    • A Security Handler utility for specilised implementations

The basis of rest calls in Python are handled by the requests module allowing you to send HTTP requests and accept the HTTP request response. It provides the ability to set/get headers, use different HTTP request methods, set parameters, etc. The code to handle Rest requests should be placed in a Rest utility class or module with a set of functions for each method type and handling of exceptions and return of responses.

The main script handles reading property files, using the constants & various utilities and executing the flow of your script.

#psuedoCode

  • def myFunction(params):
  • set baseRestURI = host_uri + <service_uri> ##depends of the API to be called
  • get responseFlag, response = restUtil.makePostCall(params)
  • if success:
    • parseResponse

In the call to list the deployment services, the entire JSON repsonse is what will be mapped to the responeReturn vairable. The JSON object or document is a set of Key : Value pairs, some are complex or arrays and others simple. Once we have the document we can navigate either by loading it as a JSON object or element to element using normal string comparison & iteration through arrays.

For every 'name' element found under the 'items' complex element (array) we can do something like get the list of services in that deployment by calling the List Services APIs or navigating over to the Administrative servive APIs to download the configuration (parameter) files - host_uri + services/ogg19c/adminsrvr/v2/config/files.  

Note in this snippet that  deploymentsList is set to responseReturn['response'], now we have all of the elements under the 'response' element in our deployments variable. Check the screenshot of the browser response above, using the same API call. In that image, the reponse has items, an array of 1 to many deployments each having links, schema and a name. 

In the code below for each row/element in the array, I get the name and use that in my next set of API calls to the Administrative Services.

The structure of your code will always be the same, a variable for the restURL, make the HTTP method request, parse the response. For complex elements or arrays, iterate over the child elements,for simple ones reference the element to get the value. 

Understanding the JSON response structure is key to using the APIs and the documentation does a decent job but I would highly recommend calling each API via a browser or a program such as Postman.

Good Luck and Have Fun Automating you GG tasks using the MSA APIs

Documentation Links

What is Oracle GoldenGate?

Getting Started with Oracle GoldenGate

About the GoldenGate APIs

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.