Wednesday Sep 23, 2009
Monday Aug 17, 2009
By Nicholas Solter on Aug 17, 2009
At OSCON last month I had the pleasure of discussing OpenSolaris, the OpenSolaris Bible, and Open HA Cluster with “Jack”. It was fun playing the “straight man,” especially since we hadn't planned anything ahead of time so I had no idea where the conversation was going to head. Take a look and let me know what you think.
Wednesday Jun 24, 2009
By Nicholas Solter on Jun 24, 2009
Monday Jun 22, 2009
By Nicholas Solter on Jun 22, 2009
If you want to try out a High Availability Cluster on OpenSolaris, but don't have the physical hardware, you can easily prototype it in VirtualBox. You need only a single physical machine with an AMD or Intel processor and at least 3 GB of RAM. Even laptops work fine; I'm using a Toshiba Tecra M10 laptop.
When using VirtualBox, the "cluster" will be two VirtualBox guests. Because of a new preview feature in Open HA Cluster 2009.06, called "weak membership," you don't need to worry about a quorum device or quorum server. More on that below.
For detailed instructions on running Open HA Cluster 2009.06 in VirtualBox, I highly recommend Thorsten Frueauf's whitepaper (pdf link). This post won't attempt to be a substitute for that document. Instead, I will describe a single, simple configuration to get you up and running. If this piques your interest, please read Thorsten's whitepaper for more details.
Without further ado, here are the instructions for running a two-node Open HA Cluster in VirtualBox.
1. Install OpenSolaris 2009.06
- You first need to install OpenSolaris 2009.06 on the physical machine. I'll assume you already know how to do that. If not, check out the documentation.
- I also suggest you create a separate Boot Environment (BE) for playing with Open HA Cluster. In case you mess up anything beyond repair, it's nice to have a safe BE into which you can boot.
# beadm create cluster
# beadm activate cluster
# init 6
2. Install VirtualBox
- Download VirtualBox from www.virtualbox.org (I'm using 2.2.0, but the latest version should work fine).
- Install VirtualBox:
# gunzip VirtualBox-2.2.0-45846-SunOS.tar.gz
# tar -xf VirtualBox-2.2.0-45846-SunOS.tar
# pkgadd -G -d VirtualBoxKern-2.2.0-SunOS-r45846.pkg
# pkgadd -G -d VirtualBox-2.2.0-SunOS-r45846.pkg
3. Create The First OpenSolaris VirtualBox Guest
- Download the iso image for OpenSolaris 2009.06 from www.opensolaris.com.
- Start VirtualBox:
- Click the “New” button in the GUI interface.
- In the first screen, choose “Solaris” as the Operating System and “OpenSolaris” as the version.
- In the second screen, select 1024 MB RAM:
- In the next screen, select "Create new hard disk". Follow the wizard's instructions, choosing "dynamically-expanding storage" and a 16GB hard drive (the default).
Verify that everything is correct and create the guest:
- Now select the guest that you just created and click the green "Start" arrow. You'll be presented by a "first-run" wizard:
- Select the location of the OpenSolaris 2009.06 iso image you previously downloaded, and click "Next." The guest will now boot into the OpenSolaris Live CD.
- Once it's booted, run the installer to install it into the VirtualBox guest. After it installs, shut it down and edit the settings. Select the "CD/DVD-ROM" option and make sure the "Mount CD/DVD Drive" box is unchecked, so it doesn't boot back into the Live CD.
- Now boot the guest and create a "cluster" boot environment:
# beadm create cluster
# beadm activate cluster
4. Create a Second VirtualBox Guest
- Repeat the above procedure for the second guest (naming it something different, obviously).
4.5.(Optional) Disable Graphical Boot and Login in the GuestsDisabling the graphical login helps reduce memory consumption by the guests. Perform the following procedure in each of the two guests:
- Edit the "cluster" entry in the GRUB menu to remove the "splashimage", "foreground", and "background" lines, and remove ",console=graphics" from the kernel line:
# cp /rpool/boot/grub/menu.lst menu.lst.bak1
# vi /rpool/boot/grub/menu.lst
The diffs should be something like this (though your line numbers may vary depending on what BEs you have in the GRUB menu):
# diff menu.lst menu.bak1
> splashimage /boot/solaris.xpm
> foreground d25f00
> background 115d93
< kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
> kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS,console=graphics
Your BE entry should look something like this:
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
- Disable the graphical-login/gdm service and reboot.
# svcadm disable graphical-login/gdm
# init 6
5. Configure Cluster Publisher
In order to form the cluster you'll later configure "bridged networking" for the VirtualBox guests. But once you do that, the guests won't be able to access the Internet without some additional steps that I won't document here (see Thorsten's whitepaper for details).
Thus, you need to install all the packages you'll need from the repositories before you've configured the networking.
- First, point your web browser at https://pkg.sun.com. Login with your Sun online account (creating a new account if you don't already have one). Then select the "Open HA Cluster 2009.06" repository, accept the license terms, and follow the instructions for downloading and installing the certificate and key on both of your VirtualBox guests.
- Now in each guest, set the publisher of the cluster packages.
# pkg set-publisher -k /var/pkg/ssl/Open_HA_Cluster_2009.06.key.pem -c /var/pkg/ssl/Open_HA_Cluster_2009.06.certificate.pem -O https://pkg.sun.com/opensolaris/ha-cluster ha-cluster
6. Install Packages
# pkg install ha-cluster-full
I also recommend installing the COMSTAR packages now, which you'll need if you want to use iSCSI to create highly available shared storage using the directly attached disks on each node of the cluster. I'll describe this process in detail in a subsequent post.
# pkg install SUNWstmf SUNWiscsi SUNWiscsit
- You can also install applications that you'll need at this point. For example, to install Tomcat and MySQL:
# pkg install SUNWtcat
# pkg install SUNWmysql51
7. Configure Networking on the Physical Host
You can now set up the networking framework to allow the two cluster nodes (the VirtualBox guests) to communicate both with each other and with the physical host.
- First, on the physical host, create an “etherstub”. This is a fake network adapter that will let the guests and host communicate as if they were on their own subnet. The benefit of using an etherstub instead of a physical adapter is that the network communication works whether or not the host is connected to an outside network.
# dladm create-etherstub etherstub0
- Next, create five VNICs on the etherstub. These virtual NICs will be assigned to the two guests and the host. You can use different MAC addresses if you prefer.
# dladm create-vnic -l etherstub0 -m a:b:c:d:1:2 vnic0
# dladm create-vnic -l etherstub0 -m a:b:c:d:1:3 vnic1
# dladm create-vnic -l etherstub0 -m a:b:c:d:1:4 vnic2
# dladm create-vnic -l etherstub0 -m a:b:c:d:1:5 vnic3
# dladm create-vnic -l etherstub0 -m a:b:c:d:1:6 vnic4
- Still on the physical host, disable NWAM and enable network/physical:default.
# svcadm disable nwam
# svcadm enable network/physical:default
- Now you can start assigning IP addresses. This demo uses the 10.0.2.0/24 subnet for the internal communication, and leaves the 192.168.1.0/24 subnet for the external network. Pick three IP addresses: one for the physical host, and one for each of the cluster nodes. I'm using:
These are random choices. Feel free to use any IP addresses within the proper subnet.
- Configure the host's IP address on one of the VNICs. I use vnic0.
# ifconfig vnic0 plumb
# ifconfig vnic0 inet 10.0.2.97/24 up
- Make the configuration persistent across reboots:
# echo "10.0.2.97/24" > /etc/hostname.vnic0
- Add entries to /etc/inet/hosts for the two guests and the host:
Plumb and configure the physical adapter to access an external network with DHCP. This assumes the public adapter is named e1000g0. Run dladm show-link to find the name of the adapter on your system.
# ifconfig e1000g0 plumb
# ifconfig e1000g0 dhcp start
- Make it persistent:
# touch /etc/hostname.e1000g0 /etc/dhcp.e1000g0
Add dns to /etc/nsswitch.conf:
# grep dns /etc/nsswitch.conf
hosts: files dns
8. Configure Networking in the First Guest
As described earlier, you need to use "bridged networking" for the guests, which gives the guests emulated physical adapters that run on VNICs on the host. You need to give each guest two adapters – one for the public network and one for the cluster private network. Note that you can't use VNICs inside the guests because they don't work inside VirtualBox.
- While the guest is shut down, select it, and select the "Settings" button to edit it.
- Select the "Network" settings.
- Select "Adapter 1" and change "Attached to" to "Bridged Networking". Click the little screwdriver icon to the right of the selection box, and select the VNIC to use (I suggest vnic1) and fill in its MAC address. You can use dladm show-vnic on the host in case you forgot the MAC you chose when creating the VNIC.
- Do the same for "Adapter 2", using one of the other free vnics (vnic3 for example). For Adapter 2, you'll first need to check the "Enable Network Adapter" box, as only one adapter is enabled by default.
- Now boot the guest.
- Once booted, disable NWAM:
# svcadm disable network/physical:nwam
# svcadm enable network/physical:default
Configure the IP address you chose earlier for this cluster node on the e1000g0 adapter.
# ifconfig e1000g0 plumb
# ifconfig e1000g0 inet 10.0.2.98/24 up
# echo "10.0.2.98/24" > /etc/hostname.e1000g0
Add entries to /etc/inet/hosts:
Verify that you can connect to the physical host:
chopin# ping 10.0.2.97
On the host, verify you can connect to the guest:
host# ping 10.0.2.98
9. Configure Networking in the Second Guest
- Repeat the above steps for the second guest, but using the IP address for that cluster node on the e1000g0 adapter (for example 10.0.2.99)
# ifconfig e1000g0 inet 10.0.2.99/24 up
# echo "10.0.2.99/24" > /etc/hostname.e1000g0
- Verify that you can connect from each guest to the other and to the physical host, and from the physical host to both guests.
10. Configure the Cluster
- In each guest, open up rpcbind:
# svccfg -s rpc/bind setprop config/local_only = boolean: false
# svcadm refresh rpc/bind
- Now in only one of the guests, run scinstall and follow its prompts. In order to configure the cluster with only one private interconnect, you need to select “custom” mode. Make sure, also, to select “lofi” for the Global Devices file system instead of a /globaldevices partition (even though /globaldevices is the default, it won't work on ZFS root). Here is a sample run of scinstall.
11. Configure Weak Membership
# svccfg -s rpc/bind setprop config/local_only = boolean: false
# svcadm refresh rpc/bind
If you're familiar with HA Clusters, you may notice that you haven't configured a quorum device or quorum server to break a tie and ensure that only one node of the cluster stays up in the case of a network partition. Instead, you can use "weak membership" which is a new preview features in Open HA Cluster 2009.06. Weak membership allows a two-node cluster to run without a quorum device arbitrator. Instead, you use a "ping target" arbitrator, which can be any network device on the same subnet. In the case of node death or a network partition, each node attempts to ping the ping target. If the node can ping it successfully, it stays up. As you might guess, this mechanism is imperfect, and in the worst case can lead to a split-brain scenario of both nodes providing services simultaneously, which can lead to data loss. To configure weak membership in the VirtualBox setup, you can use the physical host as the ping target.
- On one of the cluster nodes, run:
# /usr/cluster/bin/clq set -p multiple_partitions=true -p ping_targets=10.0.2.97 membership
This action might result in data corruption or loss.
Are you sure you want to enable multiple partitions in the cluster to be operational (y/n) [n]?y
# /usr/cluster/bin/clq reset
Thursday Jun 18, 2009
By Nicholas Solter on Jun 18, 2009
Dave and I are pleased to be offering a tutorial on OpenSolaris at the O'Reilly Open Source Convention (OSCON) this summer in San Jose. The three hour tutorial, Becoming an OpenSolaris Power User, will be on Tuesday, July 21 at 1:30 PM in Ballroom A8. As usual, this tutorial will be based strongly on our book, OpenSolaris Bible.
OSCON this year features an amazing lineup in addition to our tutorial, and should be a great conference. You can use this registration code for a 20% discount: os09fos. See you there!
Wednesday Jun 10, 2009
By Nicholas Solter on Jun 10, 2009
I'm happy to report that the first Open HA Cluster Summit last week was a fantastic event.
The summit kicked off with Professor David Cheriton's keynote address, The Network is the Cluster (pdf link). If the audience took away only one thing from the whole summit, I hope it was the point Cheriton made up-front on the second slide, that everyone needs high availability because the alternative is unpredictability and un-dependability. We all take availability of the computer services we use for granted until those services are not there. Here's a photo of Professor Cheriton concluding his talk:
As you can see the room was full and the audience attentive (I'm in the front row on the left):
After Dr. Cheriton's address, we had a discussion featuring panelists from Aster Data Systems, Google, and Sun Microsystems and moderated by Eve Kleinknecht.
One point I took away from the discussion is that high availability isn't for the 364 days a year that everything works right. It's for the one day a year when something goes wrong. Here Thorsten Frueauf (left) and I are watching the panel:
The morning session wrapped up with my talk on a minimal and modular HA cluster for OpenSolaris (pdf link). Here I am at the podium:
Lunch featured a nice centerpiece of an OpenSolaris Bible at each table for one lucky winner from that table.
I happily signed the books for all the winners.
The event concluded with a fun "Casino Night":
First two photos by Thorsten Frueauf. All other photos by Tirthankar Das
Tuesday Jun 09, 2009
Monday Jun 01, 2009
By Nicholas Solter on Jun 01, 2009
As technical lead of the Open HA Cluster 2009.06 release, I've been engrossed in it for over a year, so it with some pride and no small amount of relief that I announce it today. Maybe now I can stop working weekends.
Known as project Colorado, this release had two principal goals:
- Port Solaris Cluster 3.2 to run on OpenSolaris.
- Provide some minimization and modularization features to reduce the perceived and actual complexity of the HA clustering solution.
Rather than repeating details of these points here, I instead will point you to the slides for a presentation that I gave yesterday at the Open HA Cluster Summit in San Francisco. It was recorded, so hopefully the video will be available soon as well.
As with other packages for OpenSolaris, Open HA Cluster is available from a network package repository. In this case the "ha-cluster" repository on pkg.sun.com. Although the product is free to use and to put into production, you must register and accept a license agreement at pkg.sun.com. After accepting the agreement, follow the directions to download the certificate and key to your systems. Then set the ha-cluster publisher like this:
# pkg set-publisher -k /var/pkg/ssl/Open_HA_Cluster_2009.06.key.pem -c /var/pkg/ssl/Open_HA_Cluster_2009.06.certificate.pem -O https://pkg.sun.com/opensolaris/ha-cluster/ ha-cluster
# pkg refresh
Next, install the cluster packages:
# pkg install ha-cluster-full
For more details, you can read the official documentation. If you don't have physical hardware available to form a cluster, check out Thorsten Frueauf's whitepaper on running Open HA Cluster in VirtualBox.
Thursday May 14, 2009
By Nicholas Solter on May 14, 2009
As I mentioned previously, one of the talks I will be giving at CommunityOne West is part of a deep dive track, "Deploying OpenSolaris in your Datacenter." I'm pleased to announce that, contrary to the "official" CommunityOne information you might find elsewhere, this deep dive track is completely free. Just register with the "OSDDT" registration code.
Thursday May 07, 2009
By Nicholas Solter on May 07, 2009
I'm looking forward to the upcoming CommunityOne West conference in San Francisco at the beginning of June. This is a great event at which to learn about OpenSolaris and other FOSS software, and I'm particularly excited about the several opportunities that I'll have in which to share some information about OpenSolaris and Open HA Cluster.
The first talk, Developing on OpenSolaris will provide a tutorial on using OpenSolaris as your development platform, focusing on the features unique to OpenSolaris. This talk, at 10:50 AM, will kick off the "Developing On OpenSolaris" track.
Our second talk of the day, at 11:50 AM is titled, Becoming an OpenSolaris Power User. If you missed us at CommunityOne East, come check this one out, with revised and updated content for the OpenSolaris 2009.06 release. This presentation is part of the "Managing OpenSolaris" track.
Also on Monday, the three of us will do a book signing for OpenSolaris Bible, though the time and location haven't yet been finalized.
On Tuesday, Jerry and I will be part of a “deep dive” tutorial on Deploying OpenSolaris in your Datacenter. I'll cover High Availability, including SMF, FMA, and Open HA Cluster (of course!). Jerry will cover Zones. Additional topics will include ZFS.
Attendance at CommunityOne is completely free on Monday. I'm not sure yet if the deep dive on Tuesday will also be free.
When I'm not speaking, you can probably find me in the pavilion at the Open HA Cluster booth.
I hope to see you at Moscone in June! Also, don't forget about the Open HA Cluster Summit on the preceding Sunday, May 31.
Friday Apr 17, 2009
By Nicholas Solter on Apr 17, 2009
I'm excited that Professor David Cheriton has agreed to give the keynote address at the Open HA Cluster summit on May 31. Dr. Cheriton has an impressive resume in both academia and industry, and his lecture should be quite interesting. I can testify from personal experience that Dr. Cheriton is an entertaining speaker, as I took CS244b (Distributed Systems) from him as a grad student. This was my first exposure to distributed systems, and I've been in the field ever since.
The summit is free and open to anyone. It falls on the Sunday directly before CommunityOne and JavaOne, so if you're planning to attend those conferences, come a day early and check out this one. There will be free food, and the first 10 students to arrive at the conference, bright and early at 8:45 AM, will be able to participate in a drawing for a nano-iPod. Later at the evening reception we will be giving away a Toshiba Portege laptop. You can register on the summit wiki.
Monday Mar 30, 2009
By Nicholas Solter on Mar 30, 2009
Please join us at the first Open HA Cluster summit on May 31, 2009! This event will come almost exactly two years after we formed the HA Clusters community and released the first Open HA Cluster source code, and one year after we released the code for the Sun Cluster Core. I'm looking forward to showing off project Colorado. Here's the official invitation from Jatin, our new community manager:
Make High Availability Work For You
You are invited to participate in the first OpenSolaris Summit for Open HA Cluster.
Open HA Cluster Summit
Sunday, May 31st, 2009
San Francisco Marriott (Next to Moscone Convention Center)
55 Fourth Street
San Francisco, CA 94103 USA
The Open HA Cluster Summit will precede the CommunityOne West and JavaOne Conferences which start on June 1. We will bring together members of the HA Clusters community, technologists, and users of High Availability and Business Continuity software. Not only will experts lead interactive sessions, panel discussions and technical tracks, but there will be ample time for you to also be an active participant.
We invite you to register yourself for this event at your earliest convenience. Email firstname.lastname@example.org if you have any difficulty with the registration. Attendance is free. There will be a reception and Community Marketplace, an informal venue to showcase your products and ideas, in the evening following the technical sessions.
HA Clusters Community Manager
This event is sponsored by Sun Microsystems, Inc. Spread the word.
Tuesday Jan 20, 2009
By Nicholas Solter on Jan 20, 2009
Last year, I wrote a bible. No, I didn’t change my name to Paul or Ezekial. I mean bible in the fourth definition sense of the word. My bible, on the topic of OpenSolaris, will be released next month under the appropriate enough title, OpenSolaris Bible. It’s available now for pre-order from any of the online booksellers.
If you’re interested in OpenSolaris, whether you’re a novice or an experienced user or admin, this book should have something for you. The only prerequisite is some experience with UNIX or Linux; and at close to 1000 pages, we’re able to cover both the basics and many advanced topics. The detailed table of contents (note: PDF link) and index (also PDF link) on the book web site give an idea of the topics and scope of the book.
I’ll have more to say about the contents of the book later, but in this post I’d like to write a little about how and why I wrote it. Four years ago, my first book, Professional C++, was released. I did the project somewhat on a whim, mostly because I was offered the opportunity. I knew I didn’t want to write full-time, but couldn’t turn down the opportunity to try my hand at it. And of course there was the lure of getting the chance to influence thousands of programmers to code the "right" (i.e. "my") way. However, after spending almost one year working every night and weekend on it, I was ready to focus on other things for a while.
Given that I’m still working full-time for Sun, have added a child to my family, and my wife is just starting her own business, why did I decide to again spend almost a year writing every night and weekend? As any author this side of John Grisham can attest, it’s certainly not for the money. Nor, if I’m being realistic about the market for C++ and OpenSolaris books, is it for the fame. However, there are a few reasons other than just seeing my words in print again.
First and foremost, I strongly believe that the OpenSolaris community needs this book. At the time we started writing there were no books available or, as far as we knew, even in the works, on OpenSolaris. In fact, OpenSolaris Bible will be the first English-language book on OpenSolaris. A good tutorial and reference book on OpenSolaris is imperative in order for the technology to gain hold and grow market share in the open source community.
Of course, that doesn’t explain why I wrote the book myself, especially since I’m not a core Solaris engineer directly involved in developing the OpenSolaris distribution. To be frank, one reason is simply that I enjoy taking opportunities that come my way.
More importantly, however, I work for Sun, am involved in the OpenSolaris community, and use OpenSolaris every day. I am quite familiar with the details and intricacies of OpenSolaris and knew that I would be comfortable writing the content of the book. As with Professional C++, I wrote the book that I would want to have as my tutorial and reference. Additionally, having written Professional C++, I had the contacts at Wiley, and knew that I was capable of writing a book of this magnitude.
That said, there was no way I could have written this book by myself. I had a great experience working with Scott on Professional C++, and at first wasn’t sure I could repeat it. However, I was extremely lucky to find two amazing co-authors: Jerry and Dave. They both have a rare combination of exceptional technical knowledge and the ability to explain it clearly in writing. (If you’ve spent much time around technical folks, you’ll know that second quality is in short supply). In particular, Jerry’s understanding of Zones, virtualization, file systems, and a host of other topics, and Dave’s knowledge about the OpenSolaris distribution, IPS, Networking, and pretty much everything else were invaluable. They wrote all the hard chapters, including some material that took significant research and testing.
And in addition to their technical abilities, Dave and Jerry were both a pleasure to work with. Although I don’t think we’ve all three ever been in the same place at the same time, we didn’t just each go off into a locked room and write our chapters. We had weekly phone conversations and innumerable email exchanges about all sorts of subjects from global chapter topics and ordering to detailed questions about a particular technical issue. Additionally, we each reviewed each other’s chapters in detail several times, and all kept our eyes out for OpenSolaris changes that would impact any of our material. I believe that this diligence shows, and that the resultant tome, in my obviously biased opinion, is a well-organized, comprehensive, and cohesive tutorial and reference on OpenSolaris.
But that’s just my opinion. I’m looking forward to hearing yours!
Wednesday Aug 27, 2008
By Nicholas Solter on Aug 27, 2008
I've launched a project page on OpenSolaris.org for project Colorado. As this is quite a large project, there are several efforts occurring in parallel. One of these tasks is to write the requirements for the project. If you're interested, you can read a draft of the requirements and send any comments to email@example.com by September 10.
Another task is basic building and bringup of Sun Cluster / Open HA Cluster on OpenSolaris. You can see some of the status on that effort on the project wiki. We are also starting to investigate what it will take to convert the existing SVR4 packages to IPS packages. There's a lot of work to do! If you're interested in getting involved with the project, please let me know.
Tuesday Jul 15, 2008
By Nicholas Solter on Jul 15, 2008
I've just proposed a new OpenSolaris project to port Open HA Cluster to the OpenSolaris distribution, including the new Image Packaging System. To quote from my proposal email, this distribution of OHAC will provide basic cluster functionality with a low barrier to entry and easy deployment for OpenSolaris. Additional functionality will be provided via optional IPS packages. The intended audience is system administrators needing simple HA and developers needing an HA framework for their applications or software appliances.
This project to me feels like the natural next step from the open sourcing work I've been doing for the past year and 1/2. Now that the code is out there, it's time to get it running on OpenSolaris. I'm particularly excited to get back to hands-on engineering after suffering through the legal and process work of open sourcing.
One note on the project name: Following the state-name precedence of Nevada and Indiana, I naturally chose the state in which I live.
Nick Solter is a software engineer and author living in Colorado.
- Leaving Sun/Oracle
- Project Colorado: A minimal and modular HA cluster for OpenSolaris
- OpenSolaris for Puppets
- Installing Open HA Cluster Screencast
- Running Open HA Cluster in VirtualBox
- OpenSolaris Bible Co-Authors Together
- Become an OpenSolaris Power User at OSCON
- Developing On OpenSolaris
- OpenSolaris Bible Makes Best-Seller List at JavaOne
- Cluster Summit Recap and Photos
No bookmarks in folder
- /Open Source