Integrated Load Balancer

I'm not sure how well known it is that Solaris 11 contains a load balancer. The official documentation, starting with the Integrated Load Balancer Overview, does a great job of explaining this feature. In this blog entry my goal is to provide an implementation example.

For starters, I will be using the HALF-NAT operation mode. Basically, HALF-NAT means that the client's IP address is not mapped so that the servers know the real client address. This is usually preferred for server logging (see ILB Operation Modes for more). 

I will load balance traffic across 2 zones, each running the Apache Tomcat server. The load balancer itself will be configured as a multi-homed zone. The configuration will look as follows:

Step 1: Create the VNICs

The first step is to create VNICs for all of these interfaces:

bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 ilb0
bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 ilb1
bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 server1
bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 server2

Step 2: Create the Zones:

If you don't already have a file system for your zones:

bleonard@solaris:~$ sudo zfs create -o mountpoint=/zones rpool/zones

Then create the ILB zones:

bleonard@solaris:~$ sudo zonecfg -z ilb-zone
ilb-zone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:ilb-zone> create
zonecfg:ilb-zone> set zonepath=/zones/ilb-zone
zonecfg:ilb-zone> set ip-type=exclusive
zonecfg:ilb-zone> add net
zonecfg:ilb-zone:net> set physical=ilb0
zonecfg:ilb-zone:net> end
zonecfg:ilb-zone> add net
zonecfg:ilb-zone:net> set physical=ilb1
zonecfg:ilb-zone:net> end
zonecfg:ilb-zone> verify
zonecfg:ilb-zone> exit 

And the server zones (repeat this step for server 2 - changing values where appropriate):

bleonard@solaris:~$ sudo zonecfg -z server1-zone
server1-zone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:server1-zone> create
zonecfg:server1-zone> set zonepath=/zones/server1-zone
zonecfg:server1-zone> set ip-type=exclusive
zonecfg:server1-zone> add net
zonecfg:server1-zone:net> set physical=server1
zonecfg:server1-zone:net> end
zonecfg:server1-zone> verify
zonecfg:server1-zone> exit

Step 3: Install the ILB Zone

Then install the ilb-zone (wait to install the server zones as we will just clone this zone):

bleonard@solaris:~$ sudo zoneadm -z ilb-zone install
A ZFS file system has been created for this zone.
   Publisher: Using solaris (http://pkg.oracle.com/solaris/release/ ).
       Image: Preparing at /zones/ilb-zone/root.
       Cache: Using /var/pkg/download.
Sanity Check: Looking for 'entire' incorporation.
  Installing: Core System (output follows)
------------------------------------------------------------
Package: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.151.0.1:20101104T230646Z
License: usr/src/pkg/license_files/lic_OTN

Oracle Technology Network Developer License Agreement

...

               Packages to install:     1
           Create boot environment:    No
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1         1/1      0.0/0.0

PHASE                                        ACTIONS
Install Phase                                  11/11

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 
               Packages to install:    45
           Create boot environment:    No
               Services to restart:     3
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                45/45 12511/12511    89.1/89.1

PHASE                                        ACTIONS
Install Phase                            17953/17953 

PHASE                                          ITEMS
Package State Update Phase                     45/45 
Image State Update Phase                         2/2 
  Installing: Additional Packages (output follows)
               Packages to install:    46
           Create boot environment:    No
               Services to restart:     2
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                46/46   4498/4498    26.5/26.5

PHASE                                        ACTIONS
Install Phase                              6139/6139 

PHASE                                          ITEMS
Package State Update Phase                     46/46 
Image State Update Phase                         2/2 

        Note: Man pages can be obtained by installing SUNWman
 Postinstall: Copying SMF seed repository ... done.
 Postinstall: Applying workarounds.
        Done: Installation completed in 499.617 seconds.

  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.

I will be using the following sysidcfg file to automate the zone's system configuration. Adjust your values accordingly. The root password is "abc123":

bleonard@solaris:~$ sudo cat /zones/ilb-zone/root/etc/sysidcfg 
system_locale=C
terminal=xterms
network_interface=ilb0 {
    primary
    hostname=ilb-ext
    ip_address=10.0.2.21
    netmask=255.255.255.0
    default_route=10.0.2.2
    protocol_ipv6=no}
network_interface=ilb1 {
    hostname=ilb-int
    ip_address=192.168.1.21
        default_route=NONE
    netmask=255.255.255.0
     protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI

Boot and log into the zone:

bleonard@solaris:~$ sudo zoneadm -z ilb-zone boot
bleonard@solaris:~$ sudo zlogin -C ilb-zone
[Connected to zone 'ilb-zone' console]
100/100
Hostname: ilb-zone
Loading smf(5) service descriptions: 3/3
 network_interface=ilb0 {
ilb0 is not a valid network interface  line 3 position 19
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: ilb0 ilb1.

ilb-ext console login: root
Password: abc123
Jul  1 10:54:37 ilb-ext login: ROOT LOGIN /dev/console
Oracle Corporation      SunOS 5.11      snv_151a        November 2010
root@ilb-ext:~# 

Since our ilb-zone has 2 network interfaces, we also want to make sure a packet arriving on one network interface and addressed to a host on a different network is forwarded to the appropriate interface.

 root@ilb-ext:~# svcadm enable ipv4-forwarding

Step 4: Install the Serve 1 Zone

We'll create the first server zone as a clone of the ilb-zone. We'll then configure the server 1 zone and clone it to server 2.

Shut down ilb-zone so that it can be cloned:

bleonard@solaris:~$ sudo zoneadm -z ilb-zone halt

Then clone ilb-zone:

bleonard@solaris:~$ sudo zoneadm -z server1-zone clone ilb-zone

Here's a sysidcfg file to use with server1-zone:

bleonard@solaris:~$ sudo cat /zones/server1-zone/root/etc/sysidcfg
system_locale=C
terminal=xterms
network_interface=PRIMARY {
	hostname=server1-zone
	ip_address=192.168.1.50
	netmask=255.255.255.0
	default_route=none
	protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI

Then boot and log in to server1-zone:

bleonard@solaris:~$ sudo zoneadm -z server1-zone boot
Password: 
bleonard@solaris:~$ sudo zlogin -C server1-zone
[Connected to zone 'server1-zone' console]
Hostname: server1-zone
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: server1.

server1-zone console login: root
Password: abc123
Jul  1 14:53:20 server1-zone login: ROOT LOGIN /dev/console
Last login: Fri Jul  1 13:54:37 on console
Oracle Corporation      SunOS 5.11      snv_151a        November 2010

Also boot back up the ilb zone:

bleonard@solaris:~$ sudo zoneadm -z ilb-zone boot
Password: 

Step 5: Configure Internet Access

Test if you can ping the outside world from within the ilb zone:

root@ilb-ext:~# ping www.oracle.com
ping: unknown host www.oracle.com

Open another terminal window. The new terminal window should have you in the global zone. Copy the /etc/resolve.conf and /etc/[nsswitch.conf files from the global zone to ilb-zone and server1-zone:

bleonard@solaris:~$ sudo cp /etc/resolv.conf /zones/ilb-zone/root/etc/.
Password: 
bleonard@solaris:~$ sudo cp /etc/nsswitch.conf /zones/ilb-zone/root/etc/.
bleonard@solaris:~$ sudo cp /etc/resolv.conf /zones/server1-zone/root/etc/.
bleonard@solaris:~$ sudo cp /etc/nsswitch.conf /zones/server1-zone/root/etc/. 

Return to the ilb-zone. You should not be able to reach the outside world:

root@ilb-ext:~# ping www.oracle.com
www.oracle.com is alive 

However, server1-zone needs some routing set up before it can reach out as it will route its traffic through the ilb-zone:

root@server1-zone:~# route -p add  default 192.168.1.21
add net default: gateway 192.168.1.21
add persistent net default: gateway 192.168.1.21

root@server1-zone:~# ping www.oracle.com
www.oracle.com is alive 

Step 6: Install Tomcat

Apache Tomcat will be the service we load balance to:

root@server1-zone:~# pkg install tomcat tomcat-examples runtime/java
               Packages to install:     3
           Create boot environment:    No
               Services to restart:     2
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  3/3   1166/1166    38.9/38.9

PHASE                                        ACTIONS
Install Phase                              1504/1504 

PHASE                                          ITEMS
Package State Update Phase                       3/3 
Image State Update Phase                         2/2 
Loading smf(5) service descriptions: 1/1

root@server1-zone:~# svcadm enable http:tomcat6

Step 7: Configure Routing to the Server Zone

From the global zone we need to be able to reach the server. Add the following route (the -p option makes the changes persistent across network restarts):

bleonard@solaris:~$ sudo route -p add 192.168.1.0 10.0.2.21
Password: 
add net 192.168.1.0: gateway 10.0.2.21
add persistent net 192.168.1.0: gateway 10.0.2.21
And now you should be able to reach Tomcat from the global zone (or any client on that subnet):


Step 8: Cloning the Tomcat Server Zone

Now that we have the Tomcat server running in a zone, we can quickly create another instance. First, we need to shut down the server1-zone:

bleonard@solaris:~$ sudo zoneadm -z server1-zone halt

Then clone it:

bleonard@solaris:~$ sudo zoneadm -z server2-zone clone server1-zone

Copy the sysidcfg file you created for server 1 to server 2:

bleonard@solaris:~$ sudo cp /zones/server1-zone/root/etc/sysidcfg /zones/server2-zone/root/etc/sysidcfg

Then change the hostname and ip_address. This time around we'll also set the default router. Once editing is complete, the file should look as follows:

bleonard@solaris:~$ sudo cat /zones/server2-zone/root/etc/sysidcfg
system_locale=C
terminal=xterms
network_interface=PRIMARY {
	hostname=server2-zone
	ip_address=192.168.1.60
	netmask=255.255.255.0
	default_route=192.168.1.21
	protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI

Then boot and log into the server 2 zone:

bleonard@solaris:~$ sudo zoneadm -z server2-zone boot
bleonard@solaris:~$ sudo zlogin -C server2-zone
[Connected to zone 'server2-zone' console]
Hostname: server2-zone
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: server2.

server2-zone console login: root
Password: abc123
Jul  6 10:39:54 server2-zone login: ROOT LOGIN /dev/console
Last login: Fri Jul  1 16:19:45 on console
Oracle Corporation      SunOS 5.11      snv_151a        November 2010
root@server2-zone:~# 

Not only does this 2nd zone install much, much quicker, but Tomcat is already up and running:

It really gives you a feel for how easy it can be to scale using Solaris in a cloud type environment.

Don't forget to boot server1-zone:

bleonard@solaris:~$ sudo zoneadm -z server1-zone boot

Step 9: Configure Load Balancing

OK, that was a lot of setup just to get to the point of this blog. But now that we have 2 servers running at 2 IP address, let's set up a load balancer to scale traffic between them.

In the ILB zone, install the ILB:

root@ilb-ext:~# pkg install ilb
               Packages to install:     1
           Create boot environment:    No
               Services to restart:     1
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1       11/11      0.2/0.2

PHASE                                        ACTIONS
Install Phase                                  38/38 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 
Loading smf(5) service descriptions: 1/1

Then enable the ILB service:

root@ilb-ext:~# svcadm enable ilb

Then define a server group:

root@ilb-ext:~# ilbadm create-servergroup -s servers=192.168.1.50:8080,192.168.1.60:8080 tomcatgroup
root@ilb-ext:~# ilbadm show-servergroup 
SGNAME         SERVERID            MINPORT MAXPORT IP_ADDRESS
tomcatgroup    _tomcatgroup.0      8080    8080    192.168.1.50
tomcatgroup    _tomcatgroup.1      8080    8080    192.168.1.60

Then define a load balancing rule. This is the most complicated part of the process. For starters, I'll try to keep the rule as simple as possible. The rule is enabled (-e), will persist (-p), incoming packets (-i) are matched against destination virtual IP address (vip) and port 10.0.2.20:80. The packet is handled (-m) using round robin (rr). The destination for the packets (-o) is server group tomcatgroup. The rule is called tomcatrule_rr.

root@ilb-ext:~# ilbadm create-rule -e -p -i vip=10.0.2.20,port=80 -m lbalg=rr,type=HALF-NAT -o servergroup=tomcatgroup tomcatrule_rr

You can view the rule as follows:

root@ilb-ext:~# ilbadm show-rule
RULENAME            STATUS LBALG       TYPE    PROTOCOL VIP         PORT
tomcatrule_rr       E      roundrobin  HALF-NAT TCP 10.0.2.20       80
root@ilb-ext:~# ilbadm show-rule -f
       RULENAME: tomcatrule_rr
         STATUS: E
           PORT: 80
       PROTOCOL: TCP
          LBALG: roundrobin
           TYPE: HALF-NAT
      PROXY-SRC: --
          PMASK: /32
        HC-NAME: --
        HC-PORT: --
     CONN-DRAIN: 0
    NAT-TIMEOUT: 120
PERSIST-TIMEOUT: 60
    SERVERGROUP: tomcatgroup
            VIP: 10.0.2.20
        SERVERS: _tomcatgroup.0,_tomcatgroup.1

Finally, we need to tell the outside world that packets destined for our VIP, 10.0.2.20, should be sent to ilb0. First, find the MAC address of ilb0:

root@ilb-ext:~# dladm show-vnic ilb0
LINK         OVER         SPEED  MACADDRESS        MACADDRTYPE         VID
ilb0         ?            1000   2:8:20:bf:2a:d9   random              0

Then resolve the address to interface ilb0 using arp:

root@ilb-ext:~# arp -s 10.0.2.20 2:8:20:bf:2a:d9 pub permanent

You should then be able to ping the VIP:

root@ilb-ext:~# ping 10.0.2.20
10.0.2.20 is alive

Step 10: Load Balance!

You can now point your browser to the virtual IP address and get a result back from one of the Tomcat servers:

Very cool! But from which server was I served? I modified the example snoop.jsp to return the server's hostname and IP Address. Save the snoop.jsp to the /var/tomcat6/webapps/examples/jsp/snp directory in each of your zones.

bleonard@solaris:~$ sudo cp Downloads/snoop.jsp /zones/server1-zone/root/var/tomcat6/webapps/examples/jsp/snp/.
bleonard@solaris:~$ sudo cp Downloads/snoop.jsp /zones/server2-zone/root/var/tomcat6/webapps/examples/jsp/snp/.

I've appended the Server Side IP Address section to the bottom of the page, http://10.0.2.20/examples/jsp/snp/snoop.jsp:


Step 11: Health Checks

To keep things simple on the first go-around, I avoided health checks. However, it's pointless to have a load balancer that continues to feed requests to a dead server.

Health check options include ping probes, TCP probes, UDP probes and a user-defined script. Since I'm concerned about the health of Tomcat, I've created a simple script:

root@ilb-ext:~# cat /var/hc-tomcat 
#!/bin/bash
result=`curl -s http://$2:8080`
if [ ${result:0:5} = "<meta" ]; then
        echo 0
else
        echo -1 
fi

The load balancer provides the following variables to use with your script, of which I'm only using $2:

$1 - VIP (literal IPv4 or IPv6 address)
$2 - Server IP (literal IPv4 or IPv6 address)
$3 - Protocol (UDP, TCP as a string)
$4 - Numeric port range (the user-specified value for hc-port)
$5 - maximum time (in seconds) that the test should wait before returning a failure. If the test runs beyond the specified time, it might be stopped, and the test would be considered failed. This value is user-defined and specified in hc-timeout.

Ensure the script has execute permissions (the ilbd deamon, which runs the health check, is not run as root):

root@ilb-ext:~# chmod +x /var/hc-tomcat

Giving the script a quick test:

root@ilb-ext:~# /var/hc-tomcat n/a 192.168.1.50
0

You then create a health check rule as follows:

root@ilb-ext:~# ilbadm create-healthcheck -h hc-test=/var/hc-tomcat,hc-timeout=2,hc-count=1,hc-interval=10 hc-tomcat

The hc-timeout is how many seconds the health check will wait for a response before giving up. The hc-count is how many times the script will attempt to succeed before claiming the server to be dead. The hc-interval is how often the health-check is performed.

Once created you can view the configured health-checks as follows:

root@ilb-ext:~# ilbadm show-hc
HCNAME        TIMEOUT COUNT   INTERVAL DEF_PING TEST
hc-tomcat     2       1       10       Y        /var/hc-script

Now that we have a health check, we need to add it to our load balancing rule. Unfortunately, ilbadm doesn't have a command to modify an existing load balancing rule, so we have to delete it and create it again:

root@ilb-ext:~# ilbadm delete-rule tomcatrule_rr

We'll create the same rule as before, this time including the health check:

root@ilb-ext:~# ilbadm create-rule -e -p -i vip=10.0.2.20,port=80 -m lbalg=rr,type=HALF-NAT -h hc-name=hc-tomcat -o servergroup=tomcatgroup tomcatrule_rr

Once the new rule is created, the health check goes into effect. You can see the status as follows:

root@ilb-ext:~# ilbadm show-hc-result
RULENAME      HCNAME        SERVERID      STATUS   FAIL LAST     NEXT     RTT
tomcatrule_rr hc-tomcat     _tomcatgroup.0 alive   0    10:39:51 10:40:02 2509
tomcatrule_rr hc-tomcat     _tomcatgroup.1 alive   0    10:39:57 10:40:09 1869

So now, if probe.jsp is showing that you're hitting server 2 and we then disable Tomcat on Server 1:

root@server1-zone:~# svcadm disable tomcat6

When you refresh your browser you will be directed to Server 2. Of course, any state they you may have been maintaining on Server 1 will be lost. You can also see the status as dead using ilbadm show-hc-result:

root@ilb-ext:~# ilbadm show-hc-result
RULENAME      HCNAME        SERVERID      STATUS   FAIL LAST     NEXT     RTT
tomcatrule_rr hc-tomcat     _tomcatgroup.0 dead    4    10:43:36 10:43:45 1102
tomcatrule_rr hc-tomcat     _tomcatgroup.1 alive   0    10:43:42 10:43:53 5919

See Administering Health Checks in ILB for the official documentation.

That was a fair amount of work to configure this environment. Would it be worth providing a VM pre-configured with load balancing for download?

Comments:

Thanks for the information its very informative. is there also an option for Load Balance between physical hardware? this would be very helpful in many Load Balancing environments.

Thanks,
-Eli

Posted by Eli Kleinman on July 20, 2011 at 06:40 AM GMT #

Eli, yes, of course. I only used zones so I could demonstrate the concept. The ILB in this example has no idea it's load balancing to zones.

Posted by W Brian Leonard on July 20, 2011 at 07:04 AM GMT #

very interesting post Leonard,
I am pretty sure that most people don't know that this load balancer exists.
I'll add this information to my workshops!

regards,

Frank

--
Oracle Cloud Computing Book
http://www.munzandmore.com/writing/cloudcomputing_book

Posted by Frank on July 20, 2011 at 07:29 AM GMT #

Does ILB has any clustering/failover functionality? It makes little sense to have multiple servers behind a single point of failure.

Posted by Sean on July 20, 2011 at 08:58 AM GMT #

Sean, agreed. Here's a link to the information on the HA configuration: http://download.oracle.com/docs/cd/E19963-01/html/821-1453/gkkky.html .

Posted by W Brian Leonard on July 20, 2011 at 09:25 AM GMT #

Thanks Brian!

Posted by Sean on July 20, 2011 at 10:26 AM GMT #

Will a combination of vrrp and ilb be a valid configuration? if yes I think it would be a great load balancing solution (active stand by) and it would just be missing stat sync.

Posted by Eli Kleinman on July 20, 2011 at 10:58 AM GMT #

I successfully tested ilb with vrrp and it works great.

Just one open question, the documentation notes, The current implementation of ILB does not synchronize primary and standby load balancers, which in my case by Load balance SSH all sessions will drop after a fail-over. will that change in the final Solaris 11 release?

Thanks for all your help,
-Eli

Posted by Eli Kleinman on July 21, 2011 at 06:21 AM GMT #

Hi Eli,

The final Solaris 11 release will still not synchronize the primary and standby load balancers. If there is enough customer demand, we may implement it in a future update.

Regards,
Brian

Posted by W Brian Leonard on July 25, 2011 at 05:33 AM GMT #

Is there a way to have sticky sessions? Apache (httpd) load balancer can sticky on jsessionid with tomcat for example... Can you do that with ILB as well?

Posted by guest on August 02, 2011 at 11:26 PM GMT #

Seems to me this part is worth at least a comment:

network_interface=ilb0 {
ilb0 is not a valid network interface line 3 position 19

Any help?

Posted by guest on August 06, 2011 at 02:19 PM GMT #

You can ignore the invalid network interface message. You can see 3 lines down that the interfaces are configured: "Configuring network interface addresses: ilb0 ilb1."

I did address this comment when I first wrote about creating zones using virtual network interfaces: http://blogs.oracle.com/observatory/entry/zones_and_crossbow1 .

Regards,
Brian

Posted by guest on August 08, 2011 at 03:47 AM GMT #

Thank you Brian. Your previous blog declares it doesn't seem to be a problem.

Unfortunately, I didn't have the same luck. After trying several go's and variations, I still couldn't get ilb0 to come up with its IP address. Some entries on opensolaris.org discuss what seems like either a race condition with vnic definitions and sysidtool -- which doesn't sound right.

Finally I put my ilb1 definition first, got the syntax error on it, but both IPs attached. Sure would like to nail that one down and paint it...

Regards,

Michael

Posted by Michael Ernest on August 08, 2011 at 05:27 AM GMT #

Can the ILB do SSL offload on sparc T4 servers with on processor crypto. I am trying to see if there is a way to use the Solaris 11 ILB + Sparc T4 server to be used as a replacement proposal for F5 load balancer with ssl offload.
the SSL offload should happen at the ILB zone and not at the app tier zones (similar to how the F5s and other appliances work)

integration with Solaris11 KSSL supported?

thanks
Martin

Posted by martin francis k on February 07, 2012 at 12:25 PM GMT #

Hey Brian, looks like this a typo:

sudo route -p add 192.168.1.0 10.0.2.21

should be

sudo route -p add 192.168.1.0 10.0.2.16

Posted by Rafael on January 29, 2014 at 10:42 PM GMT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

The Observatory is a blog for users of Oracle Solaris. Tune in here for tips, tricks and more as we explore the Solaris operating system from Oracle.

Connect with Oracle Solaris:


Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today