An initial encounter with ZFS
By jwadams on Nov 16, 2005
During the initial setup, my intent was to use only the slices I'd already set up to do the transfer. What I did not plan for was the fact that ZFS does not currently allow you to remove a non-redundant slice from a storage pool without destroying the pool; here's what I did, as well as what I should have done:
Before I began, my system layout was fairly simple:
c0t0d0: Total disk cylinders available: 24620 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 root wm 1452 - 7259 8.00GB (5808/0/0) 16779312 1 unassigned wm 0 0 (0/0/0) 0 2 backup wm 0 - 24619 33.92GB (24620/0/0) 71127180 3 swap wu 0 - 1451 2.00GB (1452/0/0) 4194828 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 aux0 wm 7260 - 24619 23.91GB (17360/0/0) 50153040 c0t1d0: Total disk cylinders available: 24620 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 altroot wm 0 - 5807 8.00GB (5808/0/0) 16779312 1 unassigned wm 0 0 (0/0/0) 0 2 backup wm 0 - 24619 33.92GB (24620/0/0) 71127180 3 swap wu 5808 - 7259 2.00GB (1452/0/0) 4194828 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 aux1 wm 7260 - 24619 23.91GB (17360/0/0) 50153040That is, two 34GB hard disks, partitioned identically. There are four slices of major interest:
c0t0d0s0 / 8G root filesystem c0t0d0s7 /aux0 24G data needing preserving c0t1d0s0 /altroot 8G alternate root, currently empty c0t1d0s7 /aux1 24G some data (/opt) which needed preservingMy goal was to create a 24Gig mirrored ZFS pool, using the underlying slices of /aux0 and /aux1. /altroot would be my initial stepping stone.
Without any prior experience setting up a ZFS pool, I did the following steps:
... remove /altroot from /etc/vfstab ... # zpool create mypool c0t1d0s0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c0t1d0s0 contains a ufs filesystem # zpool create -f mypool c0t1d0s0 # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT mypool 7.94G 32.5K 7.94G 0% ONLINE - # zfs set compression=yes mypool # zfs create pool/opt # zfs set mountpoint=/new_opt mypool/opt ... copy data from /aux1/opt to /new_opt, clear out /aux1 ... ... remove /aux1 from vfstab, and remove the /opt symlink ... # zfs set mountpoint=/opt mypool/opt # df -h /opt Filesystem size used avail capacity Mounted on mypool/opt 7.9G 560M 7.4G 6% /opt #I now had all of the data I needed off of /aux1, and I wanted to add it to the storage pool. This is where I made a mistake; zfs, in its initial release, cannot remove a non-redundant device from a pool (this is being worked on). I did:
# zpool add -f mypool c0t1d0s7 (\*MISTAKE\*) # zpool status mypool pool: mypool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM mypool ONLINE 0 0 0 c0t1d0s0 ONLINE 0 0 0 c0t1d0s7 ONLINE 0 0 0 # zfs create mypool/aux0 # zfs create mypool/aux1 # zfs set mountpoint=/new_aux0 mypool/aux0 # zfs set mountpoint=/aux1 mypool/aux1 ... move data from /aux0 to /new_aux0 ... ... remove /aux0 from /etc/vfstab ... # zfs set mountpoint=/aux0 mypool/aux0And now I was stuck; I wanted to end up with a configuration of:
mypool mirror c0t0d0s7 c0t1d0s7but there was no way to get there without removing c0t1d0s0 from the pool, which ZFS doesn't allow you to do directly. I ended up creating a new pool, "pool" with c0t0d0s7 in it, copying all of my data \*again\*, destroying "mypool", then mirroring c0t0d0s7 by doing:
# zpool attach pool c0t0d0s7 c0t1d0s7 #
The right wayIf I'd planned this all better, the right way to build the pool I wanted would have been to do:
# zpool create pool c0t1d0s0 ... create /opt_new, move data to it ... # zpool replace pool c0t1d0s0 c0t1d0s7 ... create /aux0_new, move data to it ... # zpool attach pool c0t1d0s7 c0t0d0s7 ... clean up attributes, wait for sync to complete ... # zpool iostat -v capacity operations bandwidth pool used avail read write read write ------------ ----- ----- ----- ----- ----- ----- pool 15.0G 8.83G 0 1 1.47K 6.99K mirror 15.0G 8.83G 0 1 1.47K 6.99K c0t1d0s7 - - 0 1 2.30K 7.10K c0t0d0s7 - - 0 1 2.21K 7.10K ------------ ----- ----- ----- ----- ----- ----- #The real lesson here is to do a little planning if you have to juggle slices around; missteps can take some work to undo. Read about "zpool replace" and "zpool attach"; they are very useful for this kind of juggling.
Once I got everything set up, everything just works; despite having cut my available storage in /aux0 and /aux1 in half (48GB -> 24GB, due to mirroring), compression is giving me back a substantial fraction of the loss (~70%, give or take, and assuming the ratio holds steady):
# zfs get -r compressratio pool NAME PROPERTY VALUE SOURCE pool compressratio 1.70x - pool/aux0 compressratio 1.54x - pool/aux1 compressratio 2.01x - pool/opt compressratio 1.68x - #and ZFS itself is quite zippy. I hope this is a useful lesson, and that you enjoy ZFS!