An initial encounter with ZFS
By Jonathan Adams-Oracle on Nov 16, 2005
During the initial setup, my intent was to use only the slices I'd already set up to do the transfer. What I did not plan for was the fact that ZFS does not currently allow you to remove a non-redundant slice from a storage pool without destroying the pool; here's what I did, as well as what I should have done:
Before I began, my system layout was fairly simple:
c0t0d0: Total disk cylinders available: 24620 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 root wm 1452 - 7259 8.00GB (5808/0/0) 16779312 1 unassigned wm 0 0 (0/0/0) 0 2 backup wm 0 - 24619 33.92GB (24620/0/0) 71127180 3 swap wu 0 - 1451 2.00GB (1452/0/0) 4194828 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 aux0 wm 7260 - 24619 23.91GB (17360/0/0) 50153040 c0t1d0: Total disk cylinders available: 24620 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 altroot wm 0 - 5807 8.00GB (5808/0/0) 16779312 1 unassigned wm 0 0 (0/0/0) 0 2 backup wm 0 - 24619 33.92GB (24620/0/0) 71127180 3 swap wu 5808 - 7259 2.00GB (1452/0/0) 4194828 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 aux1 wm 7260 - 24619 23.91GB (17360/0/0) 50153040That is, two 34GB hard disks, partitioned identically. There are four slices of major interest:
c0t0d0s0 / 8G root filesystem c0t0d0s7 /aux0 24G data needing preserving c0t1d0s0 /altroot 8G alternate root, currently empty c0t1d0s7 /aux1 24G some data (/opt) which needed preservingMy goal was to create a 24Gig mirrored ZFS pool, using the underlying slices of /aux0 and /aux1. /altroot would be my initial stepping stone.
Without any prior experience setting up a ZFS pool, I did the following steps:
... remove /altroot from /etc/vfstab ... # zpool create mypool c0t1d0s0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c0t1d0s0 contains a ufs filesystem # zpool create -f mypool c0t1d0s0 # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT mypool 7.94G 32.5K 7.94G 0% ONLINE - # zfs set compression=yes mypool # zfs create pool/opt # zfs set mountpoint=/new_opt mypool/opt ... copy data from /aux1/opt to /new_opt, clear out /aux1 ... ... remove /aux1 from vfstab, and remove the /opt symlink ... # zfs set mountpoint=/opt mypool/opt # df -h /opt Filesystem size used avail capacity Mounted on mypool/opt 7.9G 560M 7.4G 6% /opt #I now had all of the data I needed off of /aux1, and I wanted to add it to the storage pool. This is where I made a mistake; zfs, in its initial release, cannot remove a non-redundant device from a pool (this is being worked on). I did:
# zpool add -f mypool c0t1d0s7 (\*MISTAKE\*) # zpool status mypool pool: mypool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM mypool ONLINE 0 0 0 c0t1d0s0 ONLINE 0 0 0 c0t1d0s7 ONLINE 0 0 0 # zfs create mypool/aux0 # zfs create mypool/aux1 # zfs set mountpoint=/new_aux0 mypool/aux0 # zfs set mountpoint=/aux1 mypool/aux1 ... move data from /aux0 to /new_aux0 ... ... remove /aux0 from /etc/vfstab ... # zfs set mountpoint=/aux0 mypool/aux0And now I was stuck; I wanted to end up with a configuration of:
mypool mirror c0t0d0s7 c0t1d0s7but there was no way to get there without removing c0t1d0s0 from the pool, which ZFS doesn't allow you to do directly. I ended up creating a new pool, "pool" with c0t0d0s7 in it, copying all of my data \*again\*, destroying "mypool", then mirroring c0t0d0s7 by doing:
# zpool attach pool c0t0d0s7 c0t1d0s7 #
The right wayIf I'd planned this all better, the right way to build the pool I wanted would have been to do:
# zpool create pool c0t1d0s0 ... create /opt_new, move data to it ... # zpool replace pool c0t1d0s0 c0t1d0s7 ... create /aux0_new, move data to it ... # zpool attach pool c0t1d0s7 c0t0d0s7 ... clean up attributes, wait for sync to complete ... # zpool iostat -v capacity operations bandwidth pool used avail read write read write ------------ ----- ----- ----- ----- ----- ----- pool 15.0G 8.83G 0 1 1.47K 6.99K mirror 15.0G 8.83G 0 1 1.47K 6.99K c0t1d0s7 - - 0 1 2.30K 7.10K c0t0d0s7 - - 0 1 2.21K 7.10K ------------ ----- ----- ----- ----- ----- ----- #The real lesson here is to do a little planning if you have to juggle slices around; missteps can take some work to undo. Read about "zpool replace" and "zpool attach"; they are very useful for this kind of juggling.
Once I got everything set up, everything just works; despite having cut my available storage in /aux0 and /aux1 in half (48GB -> 24GB, due to mirroring), compression is giving me back a substantial fraction of the loss (~70%, give or take, and assuming the ratio holds steady):
# zfs get -r compressratio pool NAME PROPERTY VALUE SOURCE pool compressratio 1.70x - pool/aux0 compressratio 1.54x - pool/aux1 compressratio 2.01x - pool/opt compressratio 1.68x - #and ZFS itself is quite zippy. I hope this is a useful lesson, and that you enjoy ZFS!