While working on a virtual box appliance for our field engineers, I was confronted with a side effect of the copy-on-write behavior of ZFS and had to find a way to get around it. Here's the story:
The appliance will include a software repository. To have sufficient space for unpacking and building, I created a virtual disk of 50GB in size. As things went along, I had to create and delete a rather large collection of files a few times. Once complete, ZFS told me it had used about 30 GB of space. But when I created the *.vmdk image of the virtual disk, it came out at 50GB. Considering the copy-on-write nature of ZFS, that's not surprising. And since I had enabled lz4 compression on the filesystem, compressing the disk image was useless - all the blocks had been filled with compressed data at one point in time, although 20GB of them contained only deleted data. So how to get around having to upload 20GB of unused space over my poor little ADSL uplink?
ZFS to the rescue! After all, those blocks, while filled with random, compressed data, contained deleted data... So I added a second 50GB virtual drive to the appliance, attached it as a mirror to the Zpool and waited for the resilver to complete. Then I detached the original disk and removed it from the appliance configuration. Exporting this produced a compressed disk image of 30GB - the ZFS resilver operation had not touched those extra 20GB of empty space on the new virtual disk. This saves me quite a bit of time - and might even allow our son to enjoy his online gaming without network lagging while my upload continues ;-)