Configure Management Gateways for High Availability with an HAProxy Load Balancer

For many of today’s enterprises, any downtime for business application services, whether planned or unplanned, is to be avoided at all costs. Monitoring these applications becomes vital to delivering these business goals. Management gateway offers a single point of egress and delivering high availability for management gateway is critical. This blog describes gh availability of management gateway clusters provides un-paralleled resiliency for collection
egress aggregation based setup, ensuring a collected data is delivered in case of software or
hardware failures, so if one instance in the gateway fails, another will take over, continuing to collect and deliver full stack monitoring payload to OCI O&M services

For many of today’s enterprises, any downtime for business application services, whether planned or unplanned, is to be avoided at all costs. Monitoring these applications becomes ned or unplanned, is to be avoided at all costs. Monitoring these applications becomes pivotal to delivering these business goals. Management gateway offers a single point of egress and is also could become a single point of failure for monitoring business applications. Hence livering high availability for management gateway is critical. This blog describes configuration support for Management Gateway high-availability deployment High availability of management gateway clusters provides un-paralleled resiliency for collection egress aggregation based setup, ensuring a collected data is delivered in case of software or hardware failures, so if one instance in the gateway fails, another will take over, continuing to
For many of today’s enterprises, any downtime for business application services, whether planned or unplanned, should be avoided at all costs. Monitoring these applications becomes pivotal to delivering business goals. Because Management Gateway offers a single point of egress and, in turn, could become a single point of failure for monitoring business applications, delivering high availability for Management Gateway is critical. High availability of management gateway clusters provides unparalleled resiliency, so if one instance in the gateway fails, another will take over, continuing to collect and deliver full stack monitoring payload to OCI Observability and Management (O&M) services. This document illustrates a reference deployment architecture to take advantage of highly available Management Gateways.  

Management Gateway High Availability Solution

Configuration steps for High Availability Management Gateways 

Prerequisites

• Every Management Gateway installed in a cluster should be in the same compartment
• Management Agents that are routed through the Load balancer to the Management Gateways should also be installed in the same compartment as Management Gateway.
• Load balancer network should allow ingress from the Management Agents.
• Load balancer network should allow egress to Management Gateways.

Configuration Steps

• Install and configure Management Gateway(s)
• Install and configure the load balancer 
• Configure Management Agent to communicate via HA Gateway Cluster
• Validate Management Agent deployment.

Configuration of Management Gateways with an HAProxy Load Balancer details include:

Step 1: Install and Configure Management Gateway(s)

The communication between the Management gateway and OCI requires certificates. Choose to manually create certificates or use the automatic method. In the example below, "Automatic Certificate Creation" is used.

Choose a compartment to create the required resources for Cloud Management Gateway and take note of the compartment name and OCID.  In this example, the compartment is:

• Name: DemoManagementGatewayCompartment
• OCID: ocid1.compartment.oc1..fakeaaaam5kjyeascari3yfpgarkzvxo7lrjlgefakew4y3bx643noeuh4tq

Create policies and dynamic groups for the Management Gateway to allow the automatic creation of certificates as mentioned here

Download the Management Gateway RPM from OCI Console by navigating to “Observability & Management”, “Management Agents” and “Downloads & Keys”.

Install the RPM:

So far, Management Gateway is installed but it is not configured. We need to create a response file to configure the Management Gateway to OCI and also to indicate that this Management Gateway will be part of the cluster.

Create the install key as mentioned here to be provided in the response file.

Create a response file input.rsp with the below contents. Please note we provided the "GatewayGroup" with some identifier (cluster name) in the FreeFormTags option. This is a mandatory field to be specified for installing Gateways in the cluster.

Start the configuration:

Similarly, install multiple gateways on different hosts which would be part of the same cluster. In our case, we installed a total of three Gateways. We provided the free form tag "GatewayGroup" with the same value. This way all Gateways will be part of the same cluster.

Management Gateways will now appear in the UI with information about the associated gateways in the cluster, as shown below:

Step 2: Installation and configuration of a Load Balancer 

A Load balancer should be installed on a host which routes the traffic to multiple Management Gateways. Note:

1. Any type of Load balancer can be used.  In this example, we configure an HAProxy Load Balancer
2. The Load Balancer should be configured for SSL pass-through configuration
3. The traffic mode should be "TCP"
4. The installed Gateways should be backends for the Load Balancer

Follow these instructions to install an HAProxy Load Balancer.

Configure the HAProxy:  its configuration file location is /etc/haproxy/haproxy.cfg

Change the following sections:

defaults - default settings
   defaults
   mode  tcp

frontend - this section determines the load balancer port and connects the backend to the frontend
    frontend  localnodes
        bind *:9091
        bind *:443
        option tcplog
        default_backend app
backend - this section specifies the various backend servers
    backend app
        server  app1  <Gateway Server1>:<Gateway Server Port> check
        server  app2 <Gateway Server2>:<Gateway Server Port> check

To configure a tcp/https HealthCheck for this Load Balancer, modify the backend section to include the httpcheck:

backend - this section specifies the various backend servers
   backend app
        option httpchk GET /healthcheck
        server  app1  <Gateway Server1>:<Gateway Server Port>  check ssl ca-file <location of CA authority certificate chain>
        server  app2 <Gateway Server2>:<Gateway Server Port>  check ssl ca-file <location of CA authority certificate chain>

Step 3: Configuring the Management Agent to communicate via HA Gateway Cluster

Download the Management Gateway RPM from OCI Console by navigating to “Observability & Management”, “Management Agents” and “Downloads & Keys”.

Install the RPM:

Create a response file input.rsp with the below contents. Provide the Load Balancer host and port as the GatewayServerHost and GatewayServerPort. Create the install key in the same compartment as the Management Gateway:

ManagementAgentInstallKey = <Your Key Value>
GatewayServerHost=lbhost.example.com
GatewayServerPort=9091
GatewayServerUser=gatewayuser
GatewayServerPassword=gatewaypass

Configure the Management Agent:

/opt/oracle/mgmt_agent/agent_inst/bin/setup.sh opts=/tmp/input.rsp
 
Executing configure
        Parsing input response file
        Validating install key
        Generating communication wallet
        Generating security artifacts
        Registering Management Agent
               Found service plugin(s): [logan]
        Setting proxy for agent communication
Starting agent...
Agent started successfully
Waiting for credential operation to complete...Done.
Agent setup is completed and the agent is running.
In the future agent can be started by directly running: sudo systemctl start mgmt_agent
Please make sure that you delete /tmp/input.rsp or store it in a secure location.

Now, this Management Agent would transfer all requests to Load Balancer which in turn would route the request to OCI through any of the Management Gateways in the cluster.

To validate the successful configuration of the Management Agent, check the availability status from the OCI Management Agent Console UI. For any configuration failures, check the troubleshooting guide.

References

Management Agent Install Guide

Management Gateway Install Guide

HAProxy Documentation