Friday Mar 20, 2009

Notes on Setting up a Drupal automated test bot on OpenSolaris

On, there is a topic about how to set up an automated test bot.  I'd like us at Sun to contribute to Drupal's automated test farm, so I'm trying to set up a computer running OpenSolaris to be a test server.  Here are notes I've taken based on the work I've done so far to get it working.

I got the test package here.  Unpacked it into /var/local, as per the instructions listed here.

- To get my test bot running on OpenSolaris, I have modified the following files in the directory tree:
First, I created a new profile, called it "opensolaris" (here it is).  Some parameters are different on OpenSolaris than on CentOS or Debian / Ubuntu; re-defined as appropriate for OpenSolaris.  (example: starting, stopping, and re-launching services like Apache.  Use the Service Management Framework (SMF) In OpenSolaris for easy start/stop/restart of services and automatically take care of other services which may need to be managed as dependencies.)

Next, tried to run the "" script to see if everything just worked.  Nope.  Ran into several obstacles; here they are, with the fixes / workarounds to make things work.

Obstacle #1: cvs checkout with "-z6" option seemed to stall.  I think the check-out happened okay and cvs just isn't exiting, but I'm not sure.

Fix #1: added new variable to the profiles: CVS_COMPRESSION.  Then, modified the "install" shell script; whenever it used to say "$CVS -z6", substitute "$CVS $CVS_COMPRESSION".  Now in the platform-specific templates (in the profiles/ directory), look where "CVS" is defined.  Add a line before it, creating a new variable called "CVS_COMPRESSION" and on OpenSolaris, define it as empty (i.e., don't pass the "-z6" flag).  The cvs works fine without the "-z6" option.

Note to self: need to find out why -z6 didn't work.

Obstacle #2: tried running the script, it complained that it can't log into MySQL with root user & password.

Fix #2: I forgot to set up the DB with password.  Went into MySQL and did that.

Obstacle #3: APACHE_INIT didn't work as currently defined.  Needs to accommodate different ways of starting services; in particular, the OpenSolaris svcadm command syntax different from using init.d.

Fix #3: Modify the definition of $APACHE_INIT for OpenSolaris.  Also, modify the "install" script to just invoke "$APACHE_INIT" with no arguments (i.e., no "restart") since it'll now be part of the $APACHE_INIT variable itself.

Obstacle #4: during useradd, complains that usernames are too long. Red herring; it's a warning, not an error.

Fix #3: ignore it.  Ideally, either have OpenSolaris accept usernames longer than 8 characters (really, it doesn't do that?), or shorten the usernames for Drupal test package.

Obstacle #4: during useradd, the use of the "-m" flag caused useradd to fail.  Why does "-m" fail?  It's a conflict with trying to add directory in /home, which is automount-mapped to /export/home.  There's an OpenSolaris bug about this.

Fix #4: Specify a "-b" option to set the base directory to /export/home, i.e., "-b /export/home".  Added a variable to the profiles: "USERADDFLAGS", specified "-b /export/home" in the opensolaris profile.  Then, changed the "install" script to invoke $USERADD with $USERADDFLAGS.

Alternate Fix #4: disable automounting for the test apparatus.  I didn't try that.

Obstacle #5: PHP complains about not having valid timezone set.  Script continues fine, doesn't complain, but I'd like to get rid of the warning complaint message.  I'd rather not modify the php.ini file to set timezone, but setting TZ didn't work either.

Fix #5: Set /etc/php/5.2/php.ini line "date.timezone" to "US/Pacific"; that eliminates the warning complaint message.

After these fixes, the script ran to completion.  Next, launched the browser, tried to visit my virtual test site's virtual domain, which I called "".

Obstacle #6: In my browser, went to http://localhost.  Nothing appeared.

Fix #6: Don't go there; go to instead.  That's the virtual site used for the testing.

Obstacle #7: In my browser, went to  Nothing appeared.  Need to tell my system to resolve "" to "" (i.e., localhost).

Fix #7: added "" to the end of the line in /etc/hosts that contains "".  The entire line now looks like this: vbox-testhost localhost loghost

Obstacle #8: In my browser, went to  Apache told me it couldn't access this site; didn't have permissions to access the drupal subdir in /var/www, where the drupal instance is installed.  As it turns out; the install process creates a "vhosts.conf" file to tell Apache that we're creating a virtual host (; the install process puts this in /var/apache2/2.2/vhosts.conf which is fine, but the default Apache install on OpenSolaris denies access to virtual directories. 

Fix #8: Reverse access policy for the virtual host.  Modify the "install" script, the section that begins and add the following two lines to that directive:

Order allow,deny
Allow from all

(note: no space around the comman separating allow and deny)

Next, restart the web server (go to the GNOME Applications menu and choose "Applications -> Developer Tools -> Web Stack Admin -> Start Apache2/MySQL Servers".

Now, browsing to works; I see the DrupalTestBot page and can log into Drupal.

Note to self: was this the correct way to give permissions to Apache for the drupal directory root /var/www/drupal on the VirtualHost

Okay, so now it's all configured; so far, so good.  What next?

How can I try out my test bot to see if it's configured and working correctly?

How do I make my test bot part of the test server farm?

Powered by ScribeFire.

Tuesday Dec 23, 2008

My Home Media Server on OpenSolaris + ZFS: Part 2

In my previous blog entry, I decided how ZFS will protect my data for a home media server I'm building.  Next: partition the disks on my two larger drives and install OpenSolaris on them.

This would be stupendously easy if my four disk drives were all the same size: I would type "zpool create mediapool raidz <disk1> <disk2> <disk3> <disk4>" and ZFS would give me a ton of storage all nice and protected for me.  But I have two 1TB drives and two 1.5TB drives.   My problem: ZFS wants all the pieces of a "vdev" (a virtual device; in this case, I'm creating a virtual RAID-Z device with four disks in it) to be the same size.  So I have some partitioning work to do.  I'm documenting what I did in case any of you want to use ZFS with different sized drives.

Here is my plan:
  • make sure the 1.5TB drives are the 1st and 2nd drives seen by the computer's BIOS, so that I can install OpenSolaris on one of these bigger drives
  • partition each 1.5TB drive into a 1TB partition and a .5TB partition (I recommend doing the partitioning from the Live CD instead of after installing the OS; it went easier for me this way)
  • install OpenSolaris onto the first 1.5TB drive's .5TB partition; installation will create a ZFS pool called "rpool"
  • put the four 1TB partitions into a ZFS raidz pool I will call "mediapool", my primary storage for our home's stuff
  • attach the remaining .5TB partition (from the second 1.5TB drive) to "rpool", making it a ZFS mirror pool so that the OS is protected against a single disk failure
I suppose I could've just made a single pool for storage, but I still like the idea of being able to separate my media storage from my OS.  Anyway, this is my plan for now.

Recalling that ZFS wants all the devices in a vdev to be the same size, I need to do some disk math to make sure the partition sizes are the same number of bytes.  Here's why (and don't laugh if this is all trivial to you; I'm a manager, okay?  If I don't see headcount or budget somewhere in this, I just get confused):

First of all, fdisk lets me specify partition sizes by either a percentage of the disk or a number of cylinders.  Specifying a percentage doesn't let me get precise enough to match the partition sizes on the the 1.5TB disks and the 1TB disks, so I need to specify partition size in terms of cylinders.  But cylinders aren't the same size on the two different disks.

The fdisk utility reports the following information about the 1.5TB and 1TB disks:

1.5TB Disk geometry:
Total disk size is 60800 cyls
Cylinder size is 48195 512-byte blocks
1TB Disk geometry:
Total disk size is 60800 cyls
Cylinder size is 32130 512-byte blocks

Notice that one cylinder on the 1.5TB drive is 1.5 the size (or 3/2, this way of reckoning comes in handy later) of a cylinder on the 1TB drive (48195 = 3/2 \* 32130).

I want to use as much of the 1TB drive as possible (60800 cylinders) but I can't: 60800 cylinders on the 1TB drive corresponds to 40533.33333 cylinders on the 1.5TB drive; I can't enter a non-integer number into fdisk.  I must find a size that works for both disks.  It needs to be a multiple of 3 cylinders on the 1TB drive (which would be a multiple of 2 cylinders on the 1.5TB drive).  I'll waste a little space (2 cylinders' worth on the 1TB drive or about 32MB), but that's okay given that I'll get RAID-Z error correction in return.

I'll create 1 partition on the 1TB disk, 60798 cylinders (next closest multiple of 3) == 1,953,439,740 blocks.
I'll create two partitions the 1.5TB disk:
  1. 40532 cyls == 1,953,439,740 blocks
  2. 20268 cyls (use this for the OS "rpool")
Now that I know exactly how big each partition needs to be on the four disks, I can use this easy-to-follow example to create the Solaris fdisk partitions.  It's easy; it takes less than five minutes, once I've worked out the math I just laid out here.

Next, it's time to install the OS.  I'm already running OpenSolaris from the Live CD, so I just click on the icon to install and less then 20 minutes later, it's there.

Next: create the media storage pool, using all four disks in a RAID-Z configuration:
drapeau@blackfoot:$ pfexec formatSearching for disks...done

 0. c1t0d0  /pci@0,0/pci108e,534a@7/disk@0,0
 1. c1t1d0  /pci@0,0/pci108e,534a@7/disk@1,0 
 2. c2t0d0  /pci@0,0/pci108e,534a@8/disk@0,0
 3. c2t1d0  /pci@0,0/pci108e,534a@8/disk@1,0
Specify disk (enter its number): \^C

drapeau@blackfoot:$ zpool listNAME SIZE USED AVAIL CAP HEALTH ALTROOT
rpool 444G 13.7G 430G 3% ONLINE -

drapeau@blackfoot:$ pfexec zpool create mediapool raidz c2t1d0p1 c2t0d0p1 c1t1d0p2 c1t0d0p2

Note that I used partition names for these disks, which is important: according to this helpful document, in Solaris disk device names, you'll see four primary partitions (p1-p4) and a "p0" as well which means "the whole disk".  I had to be clear to tell ZFS that I didn't want to use the whole 1TB disks, only the 1st partition on them (c2t1d0p1, c2t0d0p1).  And I told ZFS to use the 2nd partitions on the 1.5TB disks (c1t1d0p2, c1t0d0p2), which are the roughly-1TB partitions.

So, did it work?  Let's see:
drapeau@blackfoot:$ zpool list

mediapool 3.62T 132K 3.62T 0% ONLINE -
rpool 444G 13.7G 430G 3% ONLINE -

So far, so good: two ZFS pools. Let's check status:
drapeau@blackfoot:$ zpool status
 pool: mediapool
state: ONLINE
scrub: none requested

 mediapool ONLINE 0 0 0
 raidz1 ONLINE 0 0 0
 c2t1d0p1 ONLINE 0 0 0
 c2t0d0p1 ONLINE 0 0 0
 c1t1d0p2 ONLINE 0 0 0
 c1t0d0p2 ONLINE 0 0 0

errors: No known data errors

 pool: rpool
state: ONLINE
scrub: none requested

 rpool ONLINE 0 0 0
 c1t0d0s0 ONLINE 0 0 0

errors: No known data errors

drapeau@blackfoot:$ zfs list

mediapool 92.0K 2.67T 26.9K /mediapool
rpool 21.7G 415G 72K /rpool
rpool/ROOT 5.74G 415G 18K legacy
rpool/ROOT/opensolaris 5.74G 415G 5.61G /
rpool/dump 8.00G 415G 8.00G -
rpool/export 634K 415G 19K /export
rpool/export/home 615K 415G 19K /export/home
rpool/export/home/drapeau 596K 415G 596K /export/home/drapeau
rpool/swap 8.00G 423G 16K -

Sweet.  Now I've got a mediapool configured as a four-disk RAID-Z, and I have the rpool but right now it's only using one disk.  I want to mirror it now, using the 2nd 1.5TB disk's extra space.  I'll do that right now, then ask ZFS for status (I'll omit ZFS's status report on the mediapool because we just saw that).  Oh, and I'll make sure that mirrored rpool is bootable; ZFS will remind me to do it, so I'll include my steps here:

drapeau@blackfoot:$ pfexec zpool attach rpool c1t0d0s0 c1t1d0p1
Please be sure to invoke installgrub(1M) to make 'c1t1d0p1' bootable.

drapeau@blackfoot:$ pfexec installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c1t1d0s0

Updating master boot sector destroys existing boot managers (if any).
continue (y/n)?y
stage1 written to partition 0 sector 0 (abs 48195)
stage2 written to partition 0, 267 sectors starting at 50 (abs 48245)
stage1 written to master boot sector

drapeau@blackfoot:$ zpool status
  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h5m, 39.40% done, 0h9m to go
        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c1t0d0s0  ONLINE       0     0     0  84.1M resilvered
            c1t1d0p1  ONLINE       0     0     0  5.41G resilvered

errors: No known data errors
About 8 minutes later, zpool status reported that the 2nd drive in rpool was resilvered and I had a fully-mirrored rpool.  Now if one of the two drives fails, I can still boot the OS and replace the bad disk.  And with the mediapool, I'm protected against any one of the four disks failing.  I'm feeling nice and secure; it's unlikely that two disks will fail at once unless the whole computer goes up in flames.  I'll deal with backup later, maybe by looking into Zmanda or something.

This is great: to this point, I've decided how to set up my storage and protect it, I've installed the OS, and I've created my storage pools.

My next blog entry will describe how I set up the computer to share all that storage with the rest of the house.

Powered by ScribeFire.


The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. What more do you need to know, really?


« July 2016