Encrypting ZFS pools using lofi crypto

I'm running OpenSolaris 2009.06 on my laptop, soon I'll be running my own development bits of ZFS Crypto but I couldn't do that because OpenSolaris 2009.06 is based on build 111 but the ZFS crypto source is already at build 116.  Once the /dev repository catches up to 116 (or later) then I can put ZFS Crypto onto my laptop and run it "real world" rather than just pounding on it with the test script.

In the mean time I had a need to store some confidential and also some highly personal data on my laptop - not something I generally do as it is mostly used for remote access or the sources I have on it are all open source.

I also really wanted to take advantage of the ZFS auto-snapshot service for this data.  So I really needed ZFS Crypto but I was also constrained to 2009.06.   I could have gone back and used an older build of ZFS crypto that was in sync with 111 but that would have taken me a few days to backport my current sources - not something that I'd consider useful work given the only person that would benefit from this was me and for a short period of time.

It was also only a relatively small amount of data and performance of access wasn't critical.  I should also mention that storing it on a USB mass storage device of any kind was a complete non starter given where I was going with this laptop at the time - lets just say one can't take usb disks there.

I wrote previously about the crypto capability we have in the lofi(7D) driver - this is "block" device based crypto and knows nothing about filesystems.  As I mentioned before it has a number of weaknesses but by running ZFS on top of it some of those are mitigated for me in the situation I outlined above.  Particularly since I'd end up with "ZFS  - lofi - ZFS".

What I decided to do was use lofi as a "shim" and create a ZFS pool on a lofi device which was backed by a ZVOL.    As I mentioned above the reason for using ZFS as the end filesystem was to take advantage of the auto-snapshot feature, using ZFS below lofi means that I can expand the size of the encrypted area by adjusting the size of the ZVOL if necessary.

darrenm$ export PVOL=rpool/export/home/darrenm/pvol
# zfs create -V 1g $PVOL
darrenm$ pktool genkey keystore=pkcs11 label=$PVOL keylen=256 keytype=aes
Enter PIN for Sun Software PKCS#11 softtoken: 
darrenm$ pfexec lofiadm -a /dev/zvol/rdsk/$PVOL -T:::$PVOL -c aes-256-cbc
# zpool create darrenm -O canmount=off  -O checksum=sha256 -O mountpoint=/export/home/darrenm darrenm /dev/lofi/1
# zfs allow darrenm create,destroy,mount darrenm 
darrenm$ zfs create -o canmount=off darrenm/Documents
darrenm$ zfs create darrenm/Documents/Private
That sets things up the first time. This won't be automatically available after a reboot, infact the "darrenm" pool will be known about but will be in an FAULTED state since its devices won't be present.  To make it available again after reboot do this:
darrenm$ pfexec lofiadm -a /dev/zvol/rdsk/$PVOL -T :::$PVOL -c aes-256-cbc
darrenm$ pfexec zpool import -d /dev/lofi darrenm
darrenm$ zfs mount -a

A few notes on the choice of naming.  I deliberately made the pool named after my usename and set the mountpoint of the top level dataset to match where my home dir is mounted (NOT the automounted /home/ entry but the actual underlying one) and set that as canmount=off so that my default home dir is still unencrypted but I can have subdirs below it that are actually ZFS datasets mounted encrypted from the encrypted pool.

The lofi encryption key is stored (encrypted) in my keystore managed by pkcs11_softtoken.  (As it happens my latop has a TPM so I could have stored it in there - expect for the fact that the TPM driver stack isn't fully operational until after build 111).  I chose to name the key using the name of the ZVOL just to keep things simple and to remind me what it is for, the label is arbitary but using the ZVOL name will help when this is scripted.

I'm planning on polishing this off a bit and making some scripts available, but first I want to try out some more complex use cases including having the "guest" pool be mirrored.  I also want to get the appropriate ZFS delegation and RBAC details correct.  For how though the above should be enough to show what is possible here until the "real" ZFS encrypted dataset support appears.  This isn't designed to be a replacement for the ZFS crypto project but a, for me at least, a usable workaround using what we have in 2009.06.


Thanks for the walk-through Darren, it's very interesting. Please keep the posts coming if you have time!

Posted by Keith on June 01, 2009 at 04:47 AM BST #

I do nearly this exact thing on FreeBSD (8.0-CURRENT): I have a 50GB zpool on a GELI encrypted zvol; it works rather well; even performance is not too bad. The base zvol are two external drives connected via FireWire 400. I populate them using zfs send from another (non encrypted) zpool on some internal drives and I get about 35MB/s write performance which is pretty much at the limit of FW400, so I am really happy with this setup currently and have had it stable for about a month now.

Also, both the lower layer zpool and the zpool on top of the zvol on the external drives have zlib compression enabled, but due to the encryption running atop the zvol, it is not worthwile to compress it (it is pretty close to random, uncompressible data) nor is it particularly effective to snapshot the zvol due to the way GELI works.

For grins I have tried some interesting things with this setup that may fall into your 'more complex' cases such as using an entirely separate ssd as zil and l2arc both with and without a GELI layer. Everything has worked without a hitch and performance impacts seem negligible in my testing (though the box has 8 cores).

I think FreeBSD may have a little edge on solaris for this type of voodoo due to the way it allows pretty much anything to be a GEOM provider and how it allows providers to easily layer.

I would also imagine this would work OK on linux too using zfs-fuse and dm-crypt.

Still, I am waiting with baited breath for 'proper' zfs crypto if for nothing else than to finally provide for a (sort of) cross platform implementation of encrypted storage beyond crappy hardware based solutions.

Posted by John Laur on June 01, 2009 at 08:32 PM BST #

Please, \*please\* integrate whatever zfs-crypto functionality currently exists into OpenSolaris ASAP.

I don't care if it's imperfect - I'm just very worried that Oracle will kill or neuter OS, and I've been waiting for years for you to finally add this!

Posted by Rob on June 13, 2009 at 08:22 AM BST #

Post a Comment:
  • HTML Syntax: NOT allowed

Darren Moffat-Oracle


« June 2016