In a previous post we discussed how to use Oracle Enterprise Manager (OEM) data with OCI to unlock new insights. In this blog, we dive deep into the topic of using OEM data with OCI to unlock greater insights.  We discuss how to configure OEM to use Cloud Management Gateway as a proxy between OEM Agents and OCI. This approach helps in complex environments where outbound Internet connectivity is permitted only from specific subnets or hosts. In this way, OCI Observability & Management (O&M) capabilities are adopted and used to maintian a strong security posture for on-premises infrastructure.

Let us consider a scenario where multiple Database servers are monitored by OEM in an on-premises setup.  You would like to enable insights to this data by sharing it with Oracle Cloud O&M platform services as discussed in the blog reference above.

OEM to OCI Architecture
Figure 1:  Multiple database servers monitored on-premise

Configure OEM to use Cloud Management Gateway

To avoid OCI outbound to connectivity from each OEM agent, use Cloud Management Gateway as a unique entry point to reach OCI services. To enable the entry point, open outbound HTTPS port to OCI from a single host in your network.  This can be the OEM host or a dedicated VM for Cloud Management Gateway.

This helps address the outbound network connectivity requirements from on-premises hosts to OCI API REST endpoints as below:

https://loganalytics.<OCI_REGION>.oci.oraclecloud.com
https://objectstorage.<OCI_REGION>.oraclecloud.com
https://auth.<OCI_REGION>.oraclecloud.com
https://telemetry-ingestion.<OCI_REGION>.oraclecloud.com
https://certificates.<OCI_REGION>.oci.oraclecloud.com
https://certificatesmanagement.<OCI_REGION>.oci.oraclecloud.com
https://management-agent.<OCI_REGION>.oci.oraclecloud.com

 

The configuration is simple and requires two steps:

  • Install Cloud Management Gateway on a on-premises host of your choice
  • Configure OEM Agent to use Cloud Management Gateway

Install Cloud Management Gateway

The communication between the gateway and OCI requires certificates. Choose to manually create certificates or use automatic method. In the example below, “Automatic Certificate Creation” is used.

Choose a compartment to create the required resources for Cloud Management Gateway and take note of compartment name and OCID:

  • Name: <SAMPLECOMPARTMENT>
  • OCID: <SAMPLECOMPARTMENT_OCID>

Then create the following dynamic groups and policies:

Name = SAMPLECOMPARTMENT_Credential_Dynamic_Group
Definition =
ALL {resource.type=’certificateauthority’, resource.compartment.id=’SAMPLECOMPARTMENT_OCID’}

 

Name = SAMPLECOMPARTMENT_Management_Gateway_Dynamic_Group
Definition =
ALL {resource.type=’managementagent’, resource.compartment.id=’SAMPLECOMPARTMENT_OCID’}

 

Name = SAMPLECOMPARTMENT-Mgmt-Gw-Policy
Description = Policy to allow deploy of Management Gateway on compartment SAMPLECOMPARTMENT
Statements =
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Credential_Dynamic_Group to USE certificate-authority-delegates in compartment SAMPLECOMPARTMENT
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Credential_Dynamic_Group to USE vaults in compartment SAMPLECOMPARTMENT
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Credential_Dynamic_Group to USE keys in compartment SAMPLECOMPARTMENT
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Management_Gateway_Dynamic_Group to READ certificate-authority-bundle in compartment SAMPLECOMPARTMENT
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Management_Gateway_Dynamic_Group to READ leaf-certificate-bundle in compartment SAMPLECOMPARTMENT
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Management_Gateway_Dynamic_Group to MANAGE certificate-authorities in compartment SAMPLECOMPARTMENT where any{request.permission=’CERTIFICATE_AUTHORITY_CREATE’, request.permission=’CERTIFICATE_AUTHORITY_INSPECT’, request.permission=’CERTIFICATE_AUTHORITY_READ’}
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Management_Gateway_Dynamic_Group to MANAGE leaf-certificates in compartment SAMPLECOMPARTMENT where any{request.permission=’CERTIFICATE_CREATE’, request.permission=’CERTIFICATE_INSPECT’, request.permission =’CERTIFICATE_UPDATE’, request.permission=’CERTIFICATE_READ’}
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Management_Gateway_Dynamic_Group to MANAGE vaults in compartment SAMPLECOMPARTMENT where any{request.permission=’VAULT_CREATE’, request.permission=’VAULT_INSPECT’, request.permission=’VAULT_READ’, request.permission=’VAULT_CREATE_KEY’, request.permission=’VAULT_IMPORT_KEY’, request.permission=’VAULT_CREATE_SECRET’}
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Management_Gateway_Dynamic_Group to MANAGE keys in compartment SAMPLECOMPARTMENT where any{request.permission=’KEY_CREATE’, request.permission=’KEY_INSPECT’, request.permission=’KEY_READ’}
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Management_Gateway_Dynamic_Group to USE certificate-authority-delegates in compartment SAMPLECOMPARTMENT
Allow DYNAMIC-GROUP SAMPLECOMPARTMENT_Management_Gateway_Dynamic_Group to USE key-delegate in compartment SAMPLECOMPARTMENT

 

Using ssh, connect to the on-premises gateway host and install Java 8 (if not present). Expected output is similar to the one below:

[root@stlabgw01 ~]# java -version
java version “1.8.0_361”
Java(TM) SE Runtime Environment (build 1.8.0_361-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.361-b09, mixed mode)

 

Download gateway software RPM from OCI Console, navigating to “Observability & Management”, “Management Agents” and “Downloads & Keys”:

Cloud Management Agents Download
Figure 2:  Download agent software

Install the RPM:

[root@stlabgw01 ~]# rpm -ivh oracle.mgmt_gateway.rpm
Verifying…                          ################################# [100%]
Preparing…                          ################################# [100%]
Checking pre-requisites
        Checking if any previous gateway service exists
        Checking if OS has systemd or initd
        Checking available disk space for gateway install
        Checking if /opt/oracle/mgmt_agent directory exists
        Checking if ‘mgmt_agent’ user exists
        Checking Java version
                JAVA_HOME is not set or not readable to root
                Trying default path /usr/bin/java
                Java version: 1.8.0_361 found at /usr/bin/java
        Checking agent version
Updating / installing…
   1:oracle.mgmt_gateway-221214.2318.1################################# [100%]
Executing install
        Unpacking software zip
        Copying files to destination dir (/opt/oracle/mgmt_agent)
        Initializing software from template
        Checking if JavaScript engine is available to use
        Creating ‘mgmt_gateway’ daemon
        Gateway Install Logs: /opt/oracle/mgmt_agent/installer-logs/installer.log.0
        Setup gateway using input response file (run as any user with ‘sudo’ privileges)
        Usage:
                sudo /opt/oracle/mgmt_agent/agent_inst/bin/setupGateway.sh opts=[FULL_PATH_TO_INPUT.RSP]
Gateway install successful
[root@stlabgw01 ~]#

 

Now, the gateway is installed but it is not configured. We need to create a response file to finalize the configuration. See an example below.

[root@stlabgw01 ~]# sudo -u mgmt_agent sh
sh-4.4$
sh-4.4$ cd $HOME
sh-4.4$ pwd
/usr/share/mgmt_agent
sh-4.4$ vi input.rsp

 

sh-4.4$ cat input.rsp
ManagementAgentInstallKey = ***
CredentialWalletPassword = ***
GatewayPort = 4480

 

The install key can be created (and downloaded) from OCI console, navigating to “Observability & Management”, “Management Agents” and “Downloads & Keys”:

Cloud Management Agents Installation Keys
Figure 3:  Install key download from OCI console

 

Check OS firewall configuration on gateway host (port 4480 should be opened in ingress, to accept the connections from OEM agents). In this example, the firewall was disabled (dependent on individual network policies):

[root@stlabgw01 ~]# systemctl stop firewalld
[root@stlabgw01 ~]# systemctl disable firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

 

Next, execute setupGateway.sh command, that automatically creates on OCI the following resources (in the same compartment as Management Gateway):

  1. OCI Vault OCI_MGMT_GATEWAY_VAULT
  2. Certificate CA OCI_MGMT_GATEWAY_ROOT_CA
  3. Certificate ocid1.managementagent.oc1.***

See the execution below:

[root@stlabgw01 ~]# sudo /opt/oracle/mgmt_agent/agent_inst/bin/setupGateway.sh opts=/usr/share/mgmt_agent/input.rsp
/opt/oracle/mgmt_agent/agent_inst/bin/setupAgent.sh opts=/usr/share/mgmt_agent/input.rsp
Executing configure
        Parsing input response file
        Validating install key
        Generating communication wallet
        Generating security artifacts
        Registering Management Gateway
                Found service plugin(s): [GatewayProxy]
Starting gateway…
Gateway started successfully
Starting plugin deployment for: [GatewayProxy]
Deploying service plugin(s)…Done.
        GatewayProxy : Successfully deployed external plugin
Gateway setup completed and the gateway is running.
In the future gateway can be started by directly running: sudo systemctl start mgmt_gateway
Please make sure that you delete /usr/share/mgmt_agent/input.rsp or store it in secure location.
Creating gateway system properties file
Creating properties file
Creating or validating certificates
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Waiting for Management Gateway to create or validate certificates…
Creating wallets
Skipping update of wallets as credentials are empty
Waiting for Management Gateway to start…
Management Gateway Plugin set up successfully.
[root@stlabgw01 ~]#

 

After a few minutes, gateway will be in “Active” state on OCI Console:

Cloud Management Gateway Status
Figure 4:  Gateway in active state

Configure OEM Agents

Once the gateway is correctly configured, use it to forward OCI REST API calls from OEM agents.

For each host, enable the integration with OCI Observability & Management, apply the following properties:

export ORACLE_HOME=/u01/app/oracle/agent_base/agent_13.5.0.0.0
$ORACLE_HOME/bin/emctl setproperty agent -allow_new -name oci_http_proxy_host -value stlabgw01
$ORACLE_HOME/bin/emctl setproperty agent -allow_new -name oci_http_proxy_port -value 4480

 

And restart the agent:

$ORACLE_HOME/bin/emctl stop agent
$ORACLE_HOME/bin/emctl start agent

 

To verify, check emd.properties file:

cat ${AGENT_INSTANCE_DIRECTORY}/sysman/config/emd.properties | grep ^oci

 

Expected output is similar to the following:

oci_http_proxy_host=stlabgw01
oci_http_proxy_port=4480

 

This blog outlines how to enable the OCI Bridge and complete the required setup to use OCI Operations Insights and OCI Logging Analytics for on-premises OEM targets. To understand more on how to get Oracle EM data into OCI Object Storage, see part 2 and part 3 of the series.

Resources

Troubleshooting OCI Operations Insights Service Setup from Enterprise Manager 13c (Doc ID 2913953.1)

EM13c : How To Configure OCI REST Connectivity via HTTP Proxy? (Doc ID 2852014.1)

Oracle Cloud Infrastructure Documentation – Management Gateway