Redis high availability deployment with HAProxy on Oracle Cloud Infrastructure

April 26, 2021 | 9 minute read
Farooq Nafey
Principal Cloud Architect
Text Size 100%:

HAProxy delivers Redis high availability

How can you achieve Redis high availability deployment? We’ve already documented the detailed process of deploying Redis replication on Oracle Cloud Infrastructure (OCI). The main purpose of this blog is to demonstrate the implementation of HAProxy in front of a Redis master and slave (agent) replication to ensure that client and application connections are always redirected to the master server. High availability for Redis deployment including master and agent implementation is achieved by implementing Redis Sentinel, which continuously performs monitoring, notification, and automatic failover if a master becomes unresponsive.

The challenge comes in when the master stops responding because of the Redis process termination, hardware failure, or network hiccups, and Sentinel automatically fails over the master to one of the remaining agent nodes. In this case, the clients and applications connecting to Redis fail, unless the connection configuration is manually changed to point to the new master. However, this solution might not be acceptable in production environment where you want automatic failover capability.

Here, HAProxy comes to the rescue. This free and reliable solution provides Redis high availability, load balancing, and proxying for TCP- or HTTP-based applications.

HAProxy architecture for Redis

The following diagram shows the high-level deployment architecture:
A diagram depicting the architecture for running HAProxy in Oracle Cloud Infrastructure.Three Redis servers provide the master-agent replication. For Redis high availability, all the three servers are separated to different fault domains. If the region supports multiple availability domains, you can place Redis servers in different availability domains for better high availability. To ease the network provisioning and on-going management, you can also use regional subnets instead of availability-domain specific subnets.

To allow access to all the servers in a private subnet, two HAProxy servers for high availability sit behind a private load balancer and a bastion server in a public subnet. If the connectivity to the on-premises environment is available in a VPN or OCI FastConnect, you can skip the bastion server.

Deployment for Redis high availability

The following table gives the server details, which we refer to in the rest of the blog.

Server name Private or public IP address Private subnet
bastion / No
redis1 Yes
redis2 Yes
redis3 Yes
haproxy1 Yes
haproxy2 Yes
private load balancer Yes

Verify the Redis configuration

[opc@redis1 ~]$ redis-cli -h -p 6379> auth 2822dbbd71fa5d8e03440e78c66dab16dbe085c4
# Replication

A client connects to the server redis2 with IP address and acts as the master. The servers with IP address and are the agents.

[opc@redis1 ~]$ redis-cli -h -p 26379> SENTINEL MASTER mymaster
 1) "name"
 2) "mymaster"
 3) "ip"
 4) ""
 5) "port"
 6) "6379"
 7) "runid"
 8) "5f698427a528c2ca8b529b982a7e62f631903b30"
 9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "369"
19) "last-ping-reply"
20) "369"
21) "down-after-milliseconds"
22) "30000"
23) "info-refresh"
24) "1400"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "5160828"
29) "config-epoch"
30) "3"
31) "num-slaves"
32) "2"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "180000"
39) "parallel-syncs"
40) "1"

The sentinel information shows that the sentinel is configured on all three Redis servers, with redis2 server acting as the master.

After the Redis Sentinel configuration is validated, we can move onto the HAProxy setup.

HAProxy setup for Redis high availability

  1. To install HAProxy on the haproxy1 and haproxy2 server, run the following command:

    [opc@haproxy1 ~]$ sudo yum -y install haproxy

  2. Append the following code in file /etc/haproxy/haproxy.cfg and ensure that you modify the code appropriately:
    listen stats
            bind *:1936
            stats enable
            stats hide-version
            stats refresh 30s
            stats show-node
            stats auth admin:oracle
            stats uri /stats
    # redis block start
    defaults REDIS
            mode tcp
            timeout connect 3s
            timeout server 6s
            timeout client 6s
            option  log-health-checks
    frontend front_redis
            bind name redis
            mode tcp
            default_backend redis_cluster
    backend redis_cluster
            option tcp-check
            tcp-check connect
            tcp-check send AUTH\ 2822dbbd71fa5d8e03440e78c66dab16dbe085c4\r\n
            tcp-check send PING\r\n
            tcp-check expect string +PONG
            tcp-check send info\ replication\r\n
            tcp-check expect string role:master
            tcp-check send QUIT\r\n
            tcp-check expect string +OK
            server redis1 check inter 3s
            server redis2 check inter 3s
            server redis3 check inter 3s
    # redis block end


    The IP address is the private IP address of the haproxy1 server. We use it to connect to the Redis master on port 6379. Modify the IP address accordingly for both the HAProxy servers.

    Specify the IP address of all the Redis servers and specify the interval in seconds. The interval is the duration between the probes to learn about the configuration and determine which server act as primary.

    We use the parameter “tcp-check send AUTH” to take the password for authenticating to the Redis database. If no password authenticating is configured, comment out that line.

    Ensure that Security-Enhanced Linux (SElinux) is either configured to run in permissive mode or is disabled. You can add the following rule to allow HAProxy to connect to all TCP ports. SElinux is disabled by default.
    sudo setsebool -P haproxy_connect_any 1

  3. Enable and start the HAProxy service.

    sudo systemctl enable haproxy --now

  4. Validate that HAProxy service is running.

    sudo systemctl status haproxy

  5. Next, connect using the private IP address of the haproxy1 server.


    [opc@redis1 ~]$ redis-cli -h -p 6379> auth 2822dbbd71fa5d8e03440e78c66dab16dbe085c4
    OK> ROLE
    1) "master"
    2) (integer) 6752924
    3) 1) 1) ""
          2) "6379"
          3) "6752924"
       2) 1) ""
          2) "6379"
          3) "6752924"

    Ensure that either the firewall is disabled or port 6379 is allowed through the firewall. Also ensure that OCI security lists allow the TCP traffic through port 6379.

    Connecting through the HAProxy always connects you to the master server. When the master fails, Sentinel promotes one of the remaining agents as a new master and HAProxy detects the change in configuration and redirects all the application or client connections to the new master server.

  6. Create an SSH tunnel between the haproxy1 server in a private subnet through a bastion server and forward the port 1936. This port was configured during the configuration for accessing the HAProxy statistics report. If you have VPN or FastConnect connectivity, skip this step.

    ssh -L 1936: opc@
  7. To access the HAProxy statistics, open a web browser and type the following URL. When prompted, use the username and password specified in the HAProxy configuration file.

  8. The HAProxy statistics report looks like the following example where redis2 is the master server, and all traffic redirects to it:

    An example statistics report for HAProxy.

High availability of the HAProxy server

The HAProxy server can be a single point of failure and might not be a recommended solution for highly available and visible applications. So, it's important to have at least two HAProxy servers to achieve high availability and load balancing.

You can create a network load balancer with both the HAProxy servers configured as the backend and the listener configured to listen on port 6379 for Redis.

A screenshot of the LB_Redis load balancer information in the Console.

A screenshot of the backends associated with the example.

The IP address that connects to Redis is the IP address of the private load balancer, It routes through one of the HAProxy servers configured as the backend of the load balancer.

[opc@redis1 ~]$ redis-cli -h -p 6379> auth 2822dbbd71fa5d8e03440e78c66dab16dbe085c4
1) "master"
2) (integer) 6361817
3) 1) 1) ""
      2) "6379"
      3) "6361803"
   2) 1) ""
      2) "6379"
      3) "6361803"

HAProxy testing using Redis failover

  1. Manually fail over the configuration from the existing master, redis2, to redis1.> FAILOVER TO 6379
  2. Connect using the private IP of either HAProxy server.

    redis-cli -h> auth 2822dbbd71fa5d8e03440e78c66dab16dbe085c4
    OK> role
    1) "master"
    2) (integer) 262348
    3) 1) 1) ""
          2) "6379"
          3) "262348"
       2) 1) ""
          2) "6379"
          3) "262348"

    The HAProxy server always routes the client or application connections to only the master server in the configuration.

  3. Test the connection to Redis master using the private IP of a load balancer.

    redis-cli -h> auth 2822dbbd71fa5d8e03440e78c66dab16dbe085c4
    OK> role
    1) "master"
    2) (integer) 358968
    3) 1) 1) ""
          2) "6379"
          3) "358968"
       2) 1) ""
          2) "6379"
          3) "358823"

    The load balancer also routes the client connections to the master Redis server. However, the load balancer ensures high availability and loading balancing by routing through both of the HAProxy server. If failure of any HAProxy server occurs, the load balancer only routes the traffic through the remaining HAProxy.

Conclusion: High availability for Redis

In essence, you can achieve Redis high availability and maximum protection against any failure and disaster for Redis replication on OCI by combining the infrastructure services with the software configuration. From an infrastructure perspective, we achieve high availability for Redis replication between master and slave by using availability domains and fault domains. Redis Sentinel monitors and performs the failover of Redis replication from master to slave. You can deploy HAProxy and Oracle Cloud Infrastructure's native load balancer to always automatically route the client or application connections to the master.

Every use case is different. The only way to know if Oracle Cloud Infrastructure is right for you is to try it. You can select either the Oracle Cloud Free Tier or a 30-day free trial, which includes US$300 in credit to get you started with a range of services, including compute, storage, and networking.

Farooq Nafey

Principal Cloud Architect

Principal Cloud Architect at Oracle, focusing on Databases, Cloud-Native Architecture

Previous Post

Not just for Government customers: Gold standard security and controls with Oracle Government Cloud

Robert Huie | 5 min read

Next Post

A simple guide to adding rules to security lists using OCI CLI

Sam Eu | 3 min read