X

News, tips, partners, and perspectives for the Oracle Solaris operating system

Fast Teamware Bringover with ZFS & RBAC

Darren Moffat
Senior Software Architect
I run a pair (one SPARC, one x86) of reasonable sized build servers for the Solaris security group, a V880 with 16G Ram and 1.6Tb of ZFS disk space and a v40z which NFS mounts that same disk space.

I have a single ZFS pool with all the disks in it. There are two top level filesystems /cube/projects and /cube/builds. /cube/builds is scratch space for people to have what ever workspaces, test results etc etc they need for work in progress. /cube/builds doesn't get backed up (people use wx(1) backup to backup their work). Every user that uses the machine has their own ZFS filesystem in /cube/builds, eg /cube/builds/darrenm.

During the development of ZFS internally at Sun I heard there was a zbringover commmand to do really fast bringovers of the ZFS gate. I never used it but I guessed that it was probably using clones.

Yesterday I decieded to implement a similar thing for the build machine I run. I didn't bother looking at the old zbringover script since I wanted to use this to really understand the relationships between clones snapshots and filesystems in ZFS as they are now.

So I created a new filesystem /cube/builds/onnv-clone and did a teamware bringover of the ON consolidation into it. I then created a snapshot of this using the date in ISO format as the name of the snapshot, ie onnv-clone@2005-12-14. I then created a clone of that to hold the changes for a bug fix I was working on. Great the bringover took about 1 second instead of the previous record for an initial bringover of the full ONNV
gate which is about 15 minutes and normally over the network this could be much longer depending on how close your machine is to the gate machine.

So at this point it was all working wonderful for me, but I wanted everyone who uses the machine to be able to take advanatage of /cube/builds/onnv-clone as well and I wanted it to be really easy to use, so I put this into a script that does the zfs clone followed by a chown -R of the clone workspace, that
chown increases the average time to make the initial workspace out of the
zfs clone from 1 second to about 12-15 seconds still a fantastic improvement,
and not worth IMO trying to get it any faster than that.

#!/bin/ksh -p
# Copyright 2005 Sun Microsystems Inc. All rights reserved.
# Use is subject to license terms.
if [ "$_" != "/usr/bin/pfexec" -a -x /usr/bin/pfexec ]; then
/usr/bin/pfexec $0 $\*
exit $?
fi
NEW_WS=$1
BUILDS_BASE=cube/builds
if [ -z "$NEW_WS" ]; then
echo "zbringover <workspacename>"
echo "\\t<workspacename> is relative to /$BUILDS_BASE/$LOGNAME"
exit 1
fi
if [ -d "/$BUILDS_BASE/$LOGNAME/$NEW_WS" -o -f "/$BUILDS_BASE/$LOGNAME/$NEW_WS" ]; then
echo $BUILDS_BASE/$LOGNAME/$NEW_WS already exists
exit 1
fi
CLONE_FS=$BUILDS_BASE/onnv-clone
CLONE_SNAP=`date +%F`
echo "Creating new $NEW_WS by cloneing ${CLONE_FS}@${CLONE_SNAP}...\\c"
pfexec /sbin/zfs clone ${CLONE_FS}@${CLONE_SNAP} $BUILDS_BASE/$LOGNAME/$NEW_WS
[ $? != 0 ] && exit $?
echo "changing ownership...\\c"
chown -R $LOGNAME /$BUILDS_BASE/$LOGNAME/$NEW_WS 2>&1 | \\
egrep -v "chown: .zfs:|chown: snapshot:"
echo "done.\\n"

That funny little pfexec dance at the top ensures that for all users
that have the RBAC profile then zbringover will be run with pfexec so
that it gets the file_chown privilege. This means users can just type
zbringover rather than pfexec zbringover.

I then added the following line into /etc/security/exec_attr

ZFS Builds:solaris:cmd:::/cube/builds/zbringover:privs=file_chown,basic

and this one into /etc/security/prof_attr

ZFS Builds:::ZFS Builds on Borg:profiles=ZFS File System Management

note that the second one uses the hierarchy capabilities of Solaris RBAC
profiles to include the standard "ZFS File System Management" profile
into my new "ZFS Builds" profile.

I trust all the users on this machine, prior to us having the ability to
build as a normal user all of them had root access anwyay - they don't now because they don't need it. So I'm okay with giving out the ability to manage ZFS file systems. So I want all of the users to get this profile by default. The easiest way to do that is to update /etc/security/policy.conf thus:

PROFS_GRANTED=ZFS Builds,Basic Solaris User

So now when the users login they have that profile by default and can
run zbringover with the necessary privilege to change the permissions of
the clone file system they have just created to themselves.

So I test this a bit with a dummy account and was so proud of myself for
such a quick job (start to finish all of about 30 minutes work including
the testing) that I sent an email off to all the users of the machine annoucing the new functionality.


Spoke to soon! I had another small script to do the nightly bringovers
from the ONNV gate machine into the /cube/builds/onnv-clone file system
and I had that setup to run as me from cron. Well I got in this morning
and found that the teamware bringover had failed. It was getting confused by the things it saw in the .zfs/snapshot directory. Easy fix for that:

    $ pfexec zfs set snapdir=hidden cube/builds/onnv-clone

Note the pfexec and the $, this was run as me since I have the
appropriate RBAC profile from what we did above. Now teamware's bringover
command is happy. The script that does the teamware bringover into
the file system and creates the snapshot is this one:

#!/bin/pfksh
# Copyright 2005 Sun Microsystems Inc. All rights reserved.
# Use is subject to license terms.
PATH=/usr/bin:/usr/sbin:/usr/X11/bin:/usr/dt/bin:/usr/openwin/bin:/opt/onbld/bin:/opt/onbld/bin/sparc:/opt/teamware/bin:/usr/ccs/bin:/usr/sfw/bin
export PATH
CLONE_FS=cube/builds/onnv-clone
CLONE_SNAP=`date +%F`
bringover -p /ws/onnv-clone -w /$CLONE_FS
zfs snapshot ${CLONE_FS}@${CLONE_SNAP}

Remeber I said above that this runs from cron as me not as root, the script
using pfksh rather than ksh so the zfs snapshot automatically runs with
the appropriate privilege since I have the necessary RBAC profile.

All of this is was really simple and pretty obvious to implement once you know RBAC and ZFS. The bit that caught me out though was the .zfs/snapshot
directory, but I was glad that the ZFS team had made that a tunable - I now have a real use case for this.


Thanks ZFS team for a great file system, it makes my life as the admin of these two builds machines not just easier but actually fun!



The scripts and RBAC configuration above are released under the CDDL if you copy them out of this blog you MUST place the CDDL header into them that can be found here.
It is only missing from here to make the article easier to read.





Technorati Tags:

Join the discussion

Comments ( 1 )
  • Chris Gerhard Friday, December 16, 2005

    Should the script not set the PATH for it to be secure?

Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.