Tuesday Jul 19, 2011

Integrated Load Balancer

I'm not sure how well known it is that Solaris 11 contains a load balancer. The official documentation, starting with the Integrated Load Balancer Overview, does a great job of explaining this feature. In this blog entry my goal is to provide an implementation example.

For starters, I will be using the HALF-NAT operation mode. Basically, HALF-NAT means that the client's IP address is not mapped so that the servers know the real client address. This is usually preferred for server logging (see ILB Operation Modes for more). 

I will load balance traffic across 2 zones, each running the Apache Tomcat server. The load balancer itself will be configured as a multi-homed zone. The configuration will look as follows:

Step 1: Create the VNICs

The first step is to create VNICs for all of these interfaces:

bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 ilb0
bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 ilb1
bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 server1
bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 server2

Step 2: Create the Zones:

If you don't already have a file system for your zones:

bleonard@solaris:~$ sudo zfs create -o mountpoint=/zones rpool/zones

Then create the ILB zones:

bleonard@solaris:~$ sudo zonecfg -z ilb-zone
ilb-zone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:ilb-zone> create
zonecfg:ilb-zone> set zonepath=/zones/ilb-zone
zonecfg:ilb-zone> set ip-type=exclusive
zonecfg:ilb-zone> add net
zonecfg:ilb-zone:net> set physical=ilb0
zonecfg:ilb-zone:net> end
zonecfg:ilb-zone> add net
zonecfg:ilb-zone:net> set physical=ilb1
zonecfg:ilb-zone:net> end
zonecfg:ilb-zone> verify
zonecfg:ilb-zone> exit 

And the server zones (repeat this step for server 2 - changing values where appropriate):

bleonard@solaris:~$ sudo zonecfg -z server1-zone
server1-zone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:server1-zone> create
zonecfg:server1-zone> set zonepath=/zones/server1-zone
zonecfg:server1-zone> set ip-type=exclusive
zonecfg:server1-zone> add net
zonecfg:server1-zone:net> set physical=server1
zonecfg:server1-zone:net> end
zonecfg:server1-zone> verify
zonecfg:server1-zone> exit

Step 3: Install the ILB Zone

Then install the ilb-zone (wait to install the server zones as we will just clone this zone):

bleonard@solaris:~$ sudo zoneadm -z ilb-zone install
A ZFS file system has been created for this zone.
   Publisher: Using solaris (http://pkg.oracle.com/solaris/release/ ).
       Image: Preparing at /zones/ilb-zone/root.
       Cache: Using /var/pkg/download.
Sanity Check: Looking for 'entire' incorporation.
  Installing: Core System (output follows)
------------------------------------------------------------
Package: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.151.0.1:20101104T230646Z
License: usr/src/pkg/license_files/lic_OTN

Oracle Technology Network Developer License Agreement

...

               Packages to install:     1
           Create boot environment:    No
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1         1/1      0.0/0.0

PHASE                                        ACTIONS
Install Phase                                  11/11

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 
               Packages to install:    45
           Create boot environment:    No
               Services to restart:     3
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                45/45 12511/12511    89.1/89.1

PHASE                                        ACTIONS
Install Phase                            17953/17953 

PHASE                                          ITEMS
Package State Update Phase                     45/45 
Image State Update Phase                         2/2 
  Installing: Additional Packages (output follows)
               Packages to install:    46
           Create boot environment:    No
               Services to restart:     2
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                46/46   4498/4498    26.5/26.5

PHASE                                        ACTIONS
Install Phase                              6139/6139 

PHASE                                          ITEMS
Package State Update Phase                     46/46 
Image State Update Phase                         2/2 

        Note: Man pages can be obtained by installing SUNWman
 Postinstall: Copying SMF seed repository ... done.
 Postinstall: Applying workarounds.
        Done: Installation completed in 499.617 seconds.

  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.

I will be using the following sysidcfg file to automate the zone's system configuration. Adjust your values accordingly. The root password is "abc123":

bleonard@solaris:~$ sudo cat /zones/ilb-zone/root/etc/sysidcfg 
system_locale=C
terminal=xterms
network_interface=ilb0 {
    primary
    hostname=ilb-ext
    ip_address=10.0.2.21
    netmask=255.255.255.0
    default_route=10.0.2.2
    protocol_ipv6=no}
network_interface=ilb1 {
    hostname=ilb-int
    ip_address=192.168.1.21
        default_route=NONE
    netmask=255.255.255.0
     protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI

Boot and log into the zone:

bleonard@solaris:~$ sudo zoneadm -z ilb-zone boot
bleonard@solaris:~$ sudo zlogin -C ilb-zone
[Connected to zone 'ilb-zone' console]
100/100
Hostname: ilb-zone
Loading smf(5) service descriptions: 3/3
 network_interface=ilb0 {
ilb0 is not a valid network interface  line 3 position 19
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: ilb0 ilb1.

ilb-ext console login: root
Password: abc123
Jul  1 10:54:37 ilb-ext login: ROOT LOGIN /dev/console
Oracle Corporation      SunOS 5.11      snv_151a        November 2010
root@ilb-ext:~# 

Since our ilb-zone has 2 network interfaces, we also want to make sure a packet arriving on one network interface and addressed to a host on a different network is forwarded to the appropriate interface.

 root@ilb-ext:~# svcadm enable ipv4-forwarding

Step 4: Install the Serve 1 Zone

We'll create the first server zone as a clone of the ilb-zone. We'll then configure the server 1 zone and clone it to server 2.

Shut down ilb-zone so that it can be cloned:

bleonard@solaris:~$ sudo zoneadm -z ilb-zone halt

Then clone ilb-zone:

bleonard@solaris:~$ sudo zoneadm -z server1-zone clone ilb-zone

Here's a sysidcfg file to use with server1-zone:

bleonard@solaris:~$ sudo cat /zones/server1-zone/root/etc/sysidcfg
system_locale=C
terminal=xterms
network_interface=PRIMARY {
	hostname=server1-zone
	ip_address=192.168.1.50
	netmask=255.255.255.0
	default_route=none
	protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI

Then boot and log in to server1-zone:

bleonard@solaris:~$ sudo zoneadm -z server1-zone boot
Password: 
bleonard@solaris:~$ sudo zlogin -C server1-zone
[Connected to zone 'server1-zone' console]
Hostname: server1-zone
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: server1.

server1-zone console login: root
Password: abc123
Jul  1 14:53:20 server1-zone login: ROOT LOGIN /dev/console
Last login: Fri Jul  1 13:54:37 on console
Oracle Corporation      SunOS 5.11      snv_151a        November 2010

Also boot back up the ilb zone:

bleonard@solaris:~$ sudo zoneadm -z ilb-zone boot
Password: 

Step 5: Configure Internet Access

Test if you can ping the outside world from within the ilb zone:

root@ilb-ext:~# ping www.oracle.com
ping: unknown host www.oracle.com

Open another terminal window. The new terminal window should have you in the global zone. Copy the /etc/resolve.conf and /etc/[nsswitch.conf files from the global zone to ilb-zone and server1-zone:

bleonard@solaris:~$ sudo cp /etc/resolv.conf /zones/ilb-zone/root/etc/.
Password: 
bleonard@solaris:~$ sudo cp /etc/nsswitch.conf /zones/ilb-zone/root/etc/.
bleonard@solaris:~$ sudo cp /etc/resolv.conf /zones/server1-zone/root/etc/.
bleonard@solaris:~$ sudo cp /etc/nsswitch.conf /zones/server1-zone/root/etc/. 

Return to the ilb-zone. You should not be able to reach the outside world:

root@ilb-ext:~# ping www.oracle.com
www.oracle.com is alive 

However, server1-zone needs some routing set up before it can reach out as it will route its traffic through the ilb-zone:

root@server1-zone:~# route -p add  default 192.168.1.21
add net default: gateway 192.168.1.21
add persistent net default: gateway 192.168.1.21

root@server1-zone:~# ping www.oracle.com
www.oracle.com is alive 

Step 6: Install Tomcat

Apache Tomcat will be the service we load balance to:

root@server1-zone:~# pkg install tomcat tomcat-examples runtime/java
               Packages to install:     3
           Create boot environment:    No
               Services to restart:     2
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  3/3   1166/1166    38.9/38.9

PHASE                                        ACTIONS
Install Phase                              1504/1504 

PHASE                                          ITEMS
Package State Update Phase                       3/3 
Image State Update Phase                         2/2 
Loading smf(5) service descriptions: 1/1

root@server1-zone:~# svcadm enable http:tomcat6

Step 7: Configure Routing to the Server Zone

From the global zone we need to be able to reach the server. Add the following route (the -p option makes the changes persistent across network restarts):

bleonard@solaris:~$ sudo route -p add 192.168.1.0 10.0.2.21
Password: 
add net 192.168.1.0: gateway 10.0.2.21
add persistent net 192.168.1.0: gateway 10.0.2.21
And now you should be able to reach Tomcat from the global zone (or any client on that subnet):


Step 8: Cloning the Tomcat Server Zone

Now that we have the Tomcat server running in a zone, we can quickly create another instance. First, we need to shut down the server1-zone:

bleonard@solaris:~$ sudo zoneadm -z server1-zone halt

Then clone it:

bleonard@solaris:~$ sudo zoneadm -z server2-zone clone server1-zone

Copy the sysidcfg file you created for server 1 to server 2:

bleonard@solaris:~$ sudo cp /zones/server1-zone/root/etc/sysidcfg /zones/server2-zone/root/etc/sysidcfg

Then change the hostname and ip_address. This time around we'll also set the default router. Once editing is complete, the file should look as follows:

bleonard@solaris:~$ sudo cat /zones/server2-zone/root/etc/sysidcfg
system_locale=C
terminal=xterms
network_interface=PRIMARY {
	hostname=server2-zone
	ip_address=192.168.1.60
	netmask=255.255.255.0
	default_route=192.168.1.21
	protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI

Then boot and log into the server 2 zone:

bleonard@solaris:~$ sudo zoneadm -z server2-zone boot
bleonard@solaris:~$ sudo zlogin -C server2-zone
[Connected to zone 'server2-zone' console]
Hostname: server2-zone
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: server2.

server2-zone console login: root
Password: abc123
Jul  6 10:39:54 server2-zone login: ROOT LOGIN /dev/console
Last login: Fri Jul  1 16:19:45 on console
Oracle Corporation      SunOS 5.11      snv_151a        November 2010
root@server2-zone:~# 

Not only does this 2nd zone install much, much quicker, but Tomcat is already up and running:

It really gives you a feel for how easy it can be to scale using Solaris in a cloud type environment.

Don't forget to boot server1-zone:

bleonard@solaris:~$ sudo zoneadm -z server1-zone boot

Step 9: Configure Load Balancing

OK, that was a lot of setup just to get to the point of this blog. But now that we have 2 servers running at 2 IP address, let's set up a load balancer to scale traffic between them.

In the ILB zone, install the ILB:

root@ilb-ext:~# pkg install ilb
               Packages to install:     1
           Create boot environment:    No
               Services to restart:     1
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1       11/11      0.2/0.2

PHASE                                        ACTIONS
Install Phase                                  38/38 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 
Loading smf(5) service descriptions: 1/1

Then enable the ILB service:

root@ilb-ext:~# svcadm enable ilb

Then define a server group:

root@ilb-ext:~# ilbadm create-servergroup -s servers=192.168.1.50:8080,192.168.1.60:8080 tomcatgroup
root@ilb-ext:~# ilbadm show-servergroup 
SGNAME         SERVERID            MINPORT MAXPORT IP_ADDRESS
tomcatgroup    _tomcatgroup.0      8080    8080    192.168.1.50
tomcatgroup    _tomcatgroup.1      8080    8080    192.168.1.60

Then define a load balancing rule. This is the most complicated part of the process. For starters, I'll try to keep the rule as simple as possible. The rule is enabled (-e), will persist (-p), incoming packets (-i) are matched against destination virtual IP address (vip) and port 10.0.2.20:80. The packet is handled (-m) using round robin (rr). The destination for the packets (-o) is server group tomcatgroup. The rule is called tomcatrule_rr.

root@ilb-ext:~# ilbadm create-rule -e -p -i vip=10.0.2.20,port=80 -m lbalg=rr,type=HALF-NAT -o servergroup=tomcatgroup tomcatrule_rr

You can view the rule as follows:

root@ilb-ext:~# ilbadm show-rule
RULENAME            STATUS LBALG       TYPE    PROTOCOL VIP         PORT
tomcatrule_rr       E      roundrobin  HALF-NAT TCP 10.0.2.20       80
root@ilb-ext:~# ilbadm show-rule -f
       RULENAME: tomcatrule_rr
         STATUS: E
           PORT: 80
       PROTOCOL: TCP
          LBALG: roundrobin
           TYPE: HALF-NAT
      PROXY-SRC: --
          PMASK: /32
        HC-NAME: --
        HC-PORT: --
     CONN-DRAIN: 0
    NAT-TIMEOUT: 120
PERSIST-TIMEOUT: 60
    SERVERGROUP: tomcatgroup
            VIP: 10.0.2.20
        SERVERS: _tomcatgroup.0,_tomcatgroup.1

Finally, we need to tell the outside world that packets destined for our VIP, 10.0.2.20, should be sent to ilb0. First, find the MAC address of ilb0:

root@ilb-ext:~# dladm show-vnic ilb0
LINK         OVER         SPEED  MACADDRESS        MACADDRTYPE         VID
ilb0         ?            1000   2:8:20:bf:2a:d9   random              0

Then resolve the address to interface ilb0 using arp:

root@ilb-ext:~# arp -s 10.0.2.20 2:8:20:bf:2a:d9 pub permanent

You should then be able to ping the VIP:

root@ilb-ext:~# ping 10.0.2.20
10.0.2.20 is alive

Step 10: Load Balance!

You can now point your browser to the virtual IP address and get a result back from one of the Tomcat servers:

Very cool! But from which server was I served? I modified the example snoop.jsp to return the server's hostname and IP Address. Save the snoop.jsp to the /var/tomcat6/webapps/examples/jsp/snp directory in each of your zones.

bleonard@solaris:~$ sudo cp Downloads/snoop.jsp /zones/server1-zone/root/var/tomcat6/webapps/examples/jsp/snp/.
bleonard@solaris:~$ sudo cp Downloads/snoop.jsp /zones/server2-zone/root/var/tomcat6/webapps/examples/jsp/snp/.

I've appended the Server Side IP Address section to the bottom of the page, http://10.0.2.20/examples/jsp/snp/snoop.jsp:


Step 11: Health Checks

To keep things simple on the first go-around, I avoided health checks. However, it's pointless to have a load balancer that continues to feed requests to a dead server.

Health check options include ping probes, TCP probes, UDP probes and a user-defined script. Since I'm concerned about the health of Tomcat, I've created a simple script:

root@ilb-ext:~# cat /var/hc-tomcat 
#!/bin/bash
result=`curl -s http://$2:8080`
if [ ${result:0:5} = "<meta" ]; then
        echo 0
else
        echo -1 
fi

The load balancer provides the following variables to use with your script, of which I'm only using $2:

$1 - VIP (literal IPv4 or IPv6 address)
$2 - Server IP (literal IPv4 or IPv6 address)
$3 - Protocol (UDP, TCP as a string)
$4 - Numeric port range (the user-specified value for hc-port)
$5 - maximum time (in seconds) that the test should wait before returning a failure. If the test runs beyond the specified time, it might be stopped, and the test would be considered failed. This value is user-defined and specified in hc-timeout.

Ensure the script has execute permissions (the ilbd deamon, which runs the health check, is not run as root):

root@ilb-ext:~# chmod +x /var/hc-tomcat

Giving the script a quick test:

root@ilb-ext:~# /var/hc-tomcat n/a 192.168.1.50
0

You then create a health check rule as follows:

root@ilb-ext:~# ilbadm create-healthcheck -h hc-test=/var/hc-tomcat,hc-timeout=2,hc-count=1,hc-interval=10 hc-tomcat

The hc-timeout is how many seconds the health check will wait for a response before giving up. The hc-count is how many times the script will attempt to succeed before claiming the server to be dead. The hc-interval is how often the health-check is performed.

Once created you can view the configured health-checks as follows:

root@ilb-ext:~# ilbadm show-hc
HCNAME        TIMEOUT COUNT   INTERVAL DEF_PING TEST
hc-tomcat     2       1       10       Y        /var/hc-script

Now that we have a health check, we need to add it to our load balancing rule. Unfortunately, ilbadm doesn't have a command to modify an existing load balancing rule, so we have to delete it and create it again:

root@ilb-ext:~# ilbadm delete-rule tomcatrule_rr

We'll create the same rule as before, this time including the health check:

root@ilb-ext:~# ilbadm create-rule -e -p -i vip=10.0.2.20,port=80 -m lbalg=rr,type=HALF-NAT -h hc-name=hc-tomcat -o servergroup=tomcatgroup tomcatrule_rr

Once the new rule is created, the health check goes into effect. You can see the status as follows:

root@ilb-ext:~# ilbadm show-hc-result
RULENAME      HCNAME        SERVERID      STATUS   FAIL LAST     NEXT     RTT
tomcatrule_rr hc-tomcat     _tomcatgroup.0 alive   0    10:39:51 10:40:02 2509
tomcatrule_rr hc-tomcat     _tomcatgroup.1 alive   0    10:39:57 10:40:09 1869

So now, if probe.jsp is showing that you're hitting server 2 and we then disable Tomcat on Server 1:

root@server1-zone:~# svcadm disable tomcat6

When you refresh your browser you will be directed to Server 2. Of course, any state they you may have been maintaining on Server 1 will be lost. You can also see the status as dead using ilbadm show-hc-result:

root@ilb-ext:~# ilbadm show-hc-result
RULENAME      HCNAME        SERVERID      STATUS   FAIL LAST     NEXT     RTT
tomcatrule_rr hc-tomcat     _tomcatgroup.0 dead    4    10:43:36 10:43:45 1102
tomcatrule_rr hc-tomcat     _tomcatgroup.1 alive   0    10:43:42 10:43:53 5919

See Administering Health Checks in ILB for the official documentation.

That was a fair amount of work to configure this environment. Would it be worth providing a VM pre-configured with load balancing for download?

About

The Observatory is a blog for users of Oracle Solaris. Tune in here for tips, tricks and more as we explore the Solaris operating system from Oracle.

Connect with Oracle Solaris:


Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today