From time to time I get asked about space reporting in ZFS. There’s a very nice blog entry about it here. Just recently, the question was combined with NFS as served from a ZFS storage array. I had a detailed look at this – here’s a summary.
First of all, let’s have a short recap of the commands available on Solaris to check out filesystem space:
- du
Simply put, du (think “Disk Usage”) will tell you how much storage your files use. The man page says:
The du utility writes to standard output the size of the file space
allocated to, and the size of the file space allocated to each subdi-
rectory of, the file hierarchy rooted in each of the specified files.
The size of the file space allocated to a file of type directory is
defined as the sum total of space allocated to all files in the file
hierarchy rooted in the directory plus the space allocated to the
directory itself. This sum will include the space allocated to any
extended attributes encountered. -
df
Focused on filesystems, this command tells us how much space is available. Think of “disk free”. The man page says:The df utility displays the amount of disk space occupied by mounted or
unmounted file systems, the amount of used and available space, and how
much of the file system's total capacity has been used. The file system
is specified by device, or by referring to a file or directory on the
specified file system. -
ls
Very likely the first command anyone types on any Unix. In combination with the “-l” flag, it will tell us what it sees as the size of a file. Which, technically, is the size of a file as it is registered in the directory file for that file.
In this article, I’ll try to address these questions:
- How can I tell how much storage a file uses?
- How does that change with compression or snapshots?
- How can I tell how much space is still available in a filesystem?
- If my filesystem is NFS living on a ZFS storage array (or a Solaris system running NFS on ZFS), are the numbers shown by NFS and ZFS consistent? How do I tell?
The majority of this article is actually just a long example session, showing the variations one can encounter. Before I go into that, here’s a short summary for those just looking to refresh their minds on how to interpret the output of the above commands:
root@mars:~# ls -lh /mnt/*
-rw------- 1 nobody nobody 25G Sep 30 01:48 /mnt/25g.file
-rw------- 1 nobody nobody 25G Sep 30 01:55 /mnt/25g.file2
-rw-r--r-- 1 nobody nobody 148M Sep 30 01:58 /mnt/usr.bin.tar
root@mars:~# du -sh /mnt/*
25G /mnt/25g.file
0K /mnt/25g.file2
49M /mnt/usr.bin.tar
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
23T 25G 23T 1% /mnt
In the above example, you can see the following:
- Looking at the output of “ls”, you can see:
- Both files “25g.file” and “25g.file2” have a size of 25 GB. This is the real size of the file, so you can read 25 GB worth of data from it.
- The size of the file “usr.bin.tar” is 148 MB.
- du reports how much space is actually used for these files:
- “25g.file” truly uses 25 GB of space.
- “25g.file2” needs only 0k – nothing really. That suggests that the storage subsystem has compressed that file very very well… The file size is still 25GB, it just doesn’t need any space right now. Should the data in that file change to something that doesn’t compress as well, the amount of storage needed will grow.
- “usr.bin.tar” is also compressed, but not as radically as the other file. With a size of 148MB, it needs 49MB of storage.
- Looking at the overall filesystem with df:
- The filesystem size is currently 23TB
- Out of that, 25GB (well, 25.049GB) are used.
- This leads to a used capacity of 1% and available space of 23TB.
- Since we’re using the “-h” flag, all of those numbers are rounded, which explains why we don’t see the impact of those 48MB.
This should get you started. For more details, follow along the rest of this article. Sorry, it is a little long…
Let’s start with logging into a ZFS appliance and creating a filesystem. I’m showing the commandline here, but this all works in the BUI just the same.
zfs-array:shares stefan> filesystem quotademo
zfs-array:shares stefan/quotademo (uncommitted)> commit
zfs-array:shares stefan> show
Properties:
aclinherit = restricted
aclmode = discard
atime = true
checksum = fletcher4
compression = off
dedup = false
compressratio = 100
copies = 1
creation = Tue Jul 19 2016 13:36:09 GMT+0000 (UTC)
logbias = latency
mountpoint = /export
quota = 0
readonly = false
recordsize = 128K
reservation = 0
rstchown = true
secondarycache = all
nbmand = false
sharesmb = off
sharenfs = on
snapdir = hidden
vscan = false
defaultuserquota = 0
defaultgroupquota = 0
encryption = off
snaplabel =
sharedav = off
shareftp = off
sharesftp = off
sharetftp = off
pool = ssc
canonical_name = ssc/local/stefan
default_group = other
default_permissions = 700
default_sparse = false
default_user = nobody
default_volblocksize = 8K
default_volsize = 0
exported = true
nodestroy = false
maxblocksize = 1M
space_data = 31K
space_unused_res = 0
space_unused_res_shares = 0
space_snapshots = 0
space_available = 22.6T
space_total = 31K
origin =
Shares:
Filesystems:
NAME SIZE ENCRYPTED MOUNTPOINT
quotademo 31K off /export/quotademo
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 31K
space_available = 22.6T
space_snapshots = 0
quota = 0
compressratio = 100
You can see that the ZFS array shows pretty much the same metadata that you would also see on a regular zfs filesystem. Right now, it shows just 31K of used space and available capacity of 22.6 TB.
Let’s mount it on a NFS client and check it from there:
root@mars:~# mount 192.168.30.1:/export/quotademo /mnt
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
23T 31K 23T 1% /mnt
“df” reports the very same information:
- The size corresponds to the available space, which is 22.6 TB (or 23 TB rounded up).
- Used space is 31K – just that one metadata block.
- And of course, since we don’t have any quotas or other restrictions, available space is the same as the filesystem size: 22.6 TB.
Now, let’s create a first simple file and see how the numbers change:
root@mars:~# mkfile 25g /mnt/25g.file
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
23T 25G 23T 1% /mnt
root@mars:~# du -sh /mnt/*
25G /mnt/25g.file
We can see that df now reports 25GB used. Available space doesn’t change, because 25g isn’t enough to matter here. “du” also truthfully reports that our file really does use 25 GB of space, just as we would expect. Let’s have a look on the ZFS array:
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 25.0G
space_available = 22.6T
space_snapshots = 0
quota = 0
compressratio = 100
As expected, the numbers are consistent. Now let’s start compression on the ZFS array to see how NFS and the filesystem tools deal with this confusion.
Of course, the existing file isn’t compressed, so we’ll need a new one. This example will also show how well files created with mkfile compress…zfs-array:shares stefan/quotademo> set compression=gzip
compression = gzip (uncommitted)
zfs-array:shares stefan/quotademo> commit
root@mars:~# mkfile 25g /mnt/25g.file2
root@mars:~# tar cf /mnt/usr.bin.tar /usr/bin
root@mars:~# ls -lh /mnt/*
-rw------- 1 nobody nobody 25G Sep 30 01:48 /mnt/25g.file
-rw------- 1 nobody nobody 25G Sep 30 01:55 /mnt/25g.file2
-rw-r--r-- 1 nobody nobody 148M Sep 30 01:58 /mnt/usr.bin.tar
root@mars:~# du -sh /mnt/*
25G /mnt/25g.file
0K /mnt/25g.file2
49M /mnt/usr.bin.tar
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
23T 25G 23T 1% /mnt
Above, we can see that “ls” reports that our “25g.file2” is actually 25 GB in size, and usr.bin.tar is 148 MB. That’s true – these files actually have these sizes. But do they also require that amount of space? Not according to “du”, which reports 0k for that 25g.file2 and just 49M for the tar file. This means that the compression done by ZFS in the storage array is also visible to NFS and the filesystem tools running on it. They report the correct file sizes, but also the correct amount of space used by each file. Which, in the case of the file created by mkfile, isn’t very much.
To make the numbers a bit more meaningful, let’s create another, larger file and then check the numbers against what the ZFS array reports.
root@mars:~# cp /tmp/bigfile /mnt
root@mars:~# ls -lh /mnt/*
-rw------- 1 nobody nobody 25G Sep 30 01:48 /mnt/25g.file
-rw------- 1 nobody nobody 25G Sep 30 01:55 /mnt/25g.file2
-rw-r--r-- 1 nobody nobody 7.2G Sep 30 02:02 /mnt/bigfile
-rw-r--r-- 1 nobody nobody 148M Sep 30 01:58 /mnt/usr.bin.tar
root@mars:~# du -sh /mnt/*
25G /mnt/25g.file
0K /mnt/25g.file2
2.4G /mnt/bigfile
49M /mnt/usr.bin.tar
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
23T 27G 23T 1% /mnt
So now our 7.2GB file is compressed to 2.4 GB.
Filesizes add up to about 57.3 GB.
But because of compression, we only use 27.4 GB. Let’s check the storage array:
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 27.4G
space_available = 22.6T
space_snapshots = 0
quota = 0
compressratio = 116
And of course – the numbers match.
Let’s start over with a fresh dataset to have a look at quotas:
On the server, we can again see the same numbers. Let’s add the 25gb file again.zfs-array:shares stefan/quotademo> destroy
This will destroy all data in "quotademo"! Are you sure? (Y/N)
zfs-array:shares stefan> filesystem quotademo
zfs-array:shares stefan/quotademo (uncommitted)> commit
zfs-array:shares stefan> select quotademo
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 31K
space_available = 22.6T
space_snapshots = 0
quota = 0
compressratio = 100
root@mars:~# mount 192.168.30.1:/export/quotademo /mnt
root@mars:~# mkfile 25g /mnt/25g.file
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
23T 25G 23T 1% /mnt
root@mars:~# du -h /mnt/*
25G /mnt/25g.file
root@mars:~# ls -lh /mnt/*
-rw------- 1 nobody nobody 25G Sep 30 02:11 /mnt/25g.file
That’s no surprise. Now let’s add a 100gb quota for this filesystem and see how NFS reacts.
zfs-array:shares stefan/quotademo> set quota=100g
quota = 100G (uncommitted)
zfs-array:shares stefan/quotademo> commit
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 25.0G
space_available = 75.0G
space_snapshots = 0
quota = 100G
compressratio = 100
After adding the quota, the ZFS array reports 25GB used and the remaining 75GB as available space. The quota also immediately shows on the client and is displayed as the filesystem size. This makes sense – for NFS, it doesn’t make a difference if the size is limited because of available space or because of a quota.root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
100G 25G 75G 26% /mnt
Reservations, btw, don’t show on the NFS client, because NFS doesn’t understand the notion of reservations or overprovisioning or thin provisioning, which are all storage related concepts available in the storage pool, but not to the filesystem.
So let’s see if it works:
root@mars:~# mkfile 70g /mnt/70g.file
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
100G 95G 5.0G 96% /mnt
root@mars:~# du -h /mnt/*
25G /mnt/25g.file
70G /mnt/70g.file
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 95.0G
space_available = 4.98G
space_snapshots = 0
quota = 100G
compressratio = 100
Ok, so now we have just under 5 GB space available. Still all consistent.
Let’s hit the quota wall:
root@mars:~# mkfile 10g /mnt/10g.file
/mnt/10g.file: initialized 3668647936 of 2147483648 bytes: Disc quota exceeded
root@mars:~# ls -lh /mnt/*
-rw------- 1 nobody nobody 10G Sep 30 02:29 /mnt/10g.file
-rw------- 1 nobody nobody 25G Sep 30 02:11 /mnt/25g.file
-rw------- 1 nobody nobody 70G Sep 30 02:25 /mnt/70g.file
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
102G 102G 0K 100% /mnt
root@mars:~# du -h /mnt/*
7.4G /mnt/10g.file
25G /mnt/25g.file
70G /mnt/70g.file
Ok, so we’re 2g above quota, but it did produce the expected error message and the file was not completely written. It shows as “10g” in ls output, because that’s how it was created. That is expected behavior and not a bug. However, it’s only 7.4 GB, so “truncated” in a way because mkfile ran out of space.
Again, the ZFS array shows the same numbers:
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 102G
space_available = 0
space_snapshots = 0
quota = 100G
compressratio = 100
Next, let’s remove the bad file, turn on compression and add “bigfile” again to see how the numbers change.
root@mars:~# rm /mnt/10g.file
root@mars:~# du -h /mnt/*
25G /mnt/25g.file
70G /mnt/70g.file
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
100G 95G 5.0G 96% /mnt
root@mars:~# cp /tmp/verybigfile /mnt
root@mars:~# ls -lh /mnt/*
-rw------- 1 nobody nobody 70G Sep 30 02:25 /mnt/70g.file
-rw-r--r-- 1 nobody nobody 3.6G Sep 30 02:38 /mnt/verybigfile
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
100G 71G 29G 72% /mnt
root@mars:~# du -h /mnt/*
70G /mnt/70g.file
1.2G /mnt/verybigfile
So, while our file is really 3.6 GB in size, it only uses 1.2 GB because of compression. This is consistent with what the ZFS array reports:
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 71.2G
space_available = 28.8G
space_snapshots = 0
quota = 100G
compressratio = 103
So now let’s turn off compression. We should then still be able to create a 28.8 GB file. Let’s see…
zfs-array:shares stefan/quotademo> set compression=off
compression = off (uncommitted)
zfs-array:shares stefan/quotademo> commit
So while nominally already over the quota (sum of file sizes is now 101.6 gb) we are only accounted for the compressed space used, which is 99.2 gb.root@mars:~# mkfile 28g /mnt/28g.file
root@mars:~# du -h /mnt/*
28G /mnt/28g.file
70G /mnt/70g.file
1.2G /mnt/manytarfiles.tar
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
100G 99G 808M 100% /mnt
root@mars:~# ls -lh /mnt/*
-rw------- 1 nobody nobody 28G Sep 30 02:42 /mnt/28g.file
-rw------- 1 nobody nobody 70G Sep 30 02:25 /mnt/70g.file
-rw-r--r-- 1 nobody nobody 3.6G Sep 30 02:38 /mnt/manytarfiles.ta
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 99.2G
space_available = 808M
space_snapshots = 0
quota = 100G
compressratio = 102
Let’s hit the quota wall again:
root@mars:~# mkfile 2g /mnt/2g.file
/mnt/2g.file: initialized 1510924288 of 2147483648 bytes: Disc quota exceeded
root@mars:~# ls -lh /mnt/*
-rw------- 1 nobody nobody 28G Sep 30 02:42 /mnt/28g.file
-rw------- 1 nobody nobody 2.0G Sep 30 02:48 /mnt/2g.file
-rw------- 1 nobody nobody 70G Sep 30 02:25 /mnt/70g.file
-rw-r--r-- 1 nobody nobody 3.6G Sep 30 02:38 /mnt/manytarfiles.tar
root@mars:~# du -h /mnt/*
28G /mnt/28g.file
1.4G /mnt/2g.file
70G /mnt/70g.file
1.2G /mnt/manytarfiles.tar
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
101G 101G 0K 100% /mnt
Ok, we went a little over quota again, but we were stopped pretty quickly. Just as expected.
Let’s clean up and see how snapshots change all of this…
zfs-array:shares stefan/quotademo> destroy
This will destroy all data in "quotademo"! Are you sure? (Y/N)
zfs-array:shares stefan> filesystem quotademo
zfs-array:shares stefan/quotademo (uncommitted)> commit
zfs-array:shares stefan> select quotademo
zfs-array:shares stefan/quotademo> set quota=100g
quota = 100G (uncommitted)
zfs-array:shares stefan/quotademo> commit
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 31K
space_available = 100G
space_snapshots = 0
quota = 100G
compressratio = 100
Just like before, we’re starting with a 100GB quota on the filesystem. There’s no more compression – it would just complicate things since we’re looking for easily understandable numbers. To start, let’s create that 25GB file again:
root@mars:~# mkfile 25g /mnt/25g.file
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
100G 25G 75G 26% /mnt
Nothing new. Let’s create a snapshot of this situation on the array:
zfs-array:shares stefan/quotademo snapshots> snapshot one25gfile
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 25.0G
space_available = 75.0G
space_snapshots = 0
quota = 100G
compressratio = 100
That’s expected, we’ve not changed anything on the filesystem, so the snapshot doesn’t need any space.
Let’s create another file:
root@mars:~# mkfile 25g /mnt/new.25g.file
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
100G 50G 50G 51% /mnt
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 50.0G
space_available = 50.0G
space_snapshots = 19K
quota = 100G
compressratio = 100
Ok, so the snapshot now contains the metadata that changed. Fine.
Let’s delete the original file.
root@mars:~# rm /mnt/25g.file
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/quotademo
75G 25G 50G 34% /mnt
The available space didn’t change!!! Rather, the size of our filesystem seems to have shrunk by the amount of data we deleted. Rather unexpected behavior, but of course we can explain that with the data still being in the snapshot:
zfs-array:shares stefan/quotademo> get space_total space_available
space_snapshots quota compressratio
space_total = 50.0G
space_available = 50.0G
space_snapshots = 25.0G
quota = 100G
compressratio = 100
Again, because NFS doesn’t know about snapshots, it has no other way of telling us. And as long as we are at a 100g quota (which NFS doesn’t know and shows as the size of the filesystem), the only way of telling us how much space we can use, and be consistent, is by reducing the “size” of the filesystem. Confusing, perhaps, but logical nontheless.
Let’s go and roll back that snapshot to undo our changes:
zfs-array:shares stefan/quotademo@one25gfile> rollback
Rolling back will revert data to snapshot, destroying newer data. Active
initiators will be disconnected.
Continue? (Y/N)
root@mars:~# du -h /mnt/*
25G /mnt/25g.file
So of course, we are back at 100GB size, and the second file is gone. Just as one would expect.
Now, as a last exercise, let’s create a clone of this filesystem.
zfs-array:shares stefan/quotademo@one25gfile> clone stefan clonedemo
zfs-array:shares stefan/clonedemo (uncommitted clone)> commit
zfs-array:shares stefan> list filesystem
Filesystems:
NAME SIZE ENCRYPTED MOUNTPOINT
quotademo 25.0G off /export/quotademo
clonedemo 18K off /export/clonedemo
zfs-array:shares stefan> select clonedemo
zfs-array:shares stefan/clonedemo> get space_total space_available
space_snapshots quota compressratio
space_total = 18K
space_available = 100G
space_snapshots = 0
quota = 100G
compressratio = 100
We can see that the new filesystem “clonedemo” doesn’t use any space (except 18k of metadata) and has a quota of 100g.
Let’s mount and use it:
root@mars:~# umount /mnt
root@mars:~# mount 192.168.30.1:/export/clonedemo /mnt
root@mars:~# df -h /mnt
Filesystem Size Used Available Capacity Mounted on
192.168.30.1:/export/clonedemo
125G 25G 100G 21% /mnt
root@mars:~# du -sh /mnt/*
25G /mnt/25g.file
Surprise! Our filesystem is 125GB in size! That’s because as a clone, it comes with a “view” of the 25GB of data that was in the parent filesystem. But because it has a quota of 100GB, which means we’re allowed to add 100GB of new data to it, it actually provides us with data worth 125GB.
I hope these examples help clear up some of the confusion that sometimes come with the combination of compression, snapshots, clones and the normal Solaris reporting commands.