ZFS the perfect file system for audit trails

Those of you who have to deal with audit trails from busy systems know that they can get really big, and when you need to store them for a couple of years they consume a considerable amount of disk.

To minimize the disk usage that you can compress the audit trails, and it works really well (I've adjusted the output to right-align the size):

root@warlord# ls -l
total 19447638
-rw-r--r--   1 root     other    1353214369 Aug  9 09:27 20070728065900.20070730163539.warlord
-rw-------   1 root     other      62209268 Aug  7 08:25 20070728065900.20070730163539.warlord.gz
-rw-r--r--   1 root     other    1965073391 Aug  7 09:35 20070728065900.20070730163539.warlord.txt
-rw-r--r--   1 root     other      71460194 Aug  7 09:35 20070728065900.20070730163539.warlord.txt.gz

As you can see the compression ratio is really good (over 90%), but one problem still remains: if you need to work with the files you have to uncompress them before you can run your scripts to go an find who edited /etc/passwd. Uncompressing one file doesn't take that long, but when you don't know exactly in which file the audit records you are looking for are, things start to take time.

Enter ZFS: with on-the-fly disk compression it is the perfect file system to store audit trails. First of all you have to enable compression:

root@warlord# zfs set compression on pool/audit

That only makes future writes compressed, so the files you already have needs to be rewritten to be compressed. After having done that it is time to look at the compression ratio:

root@warlord# ls -l
total 19447638
-rw-r--r--   1 root     other    1353214369 Aug  9 09:27 20070728065900.20070730163539.warlord
-rw-r--r--   1 root     other    1965073391 Aug  7 09:35 20070728065900.20070730163539.warlord.txt

Wait a minute! That didn't compress anything, or did it? ls -l shows the size of the uncompressed file, so you have to use du to see the compressed size:

root@warlord# du -k
554067  20070728065900.20070730163539.warlord
732399  20070728065900.20070730163539.warlord.txt

Much better! (Note that it displays the size in kilo bytes while ls -l displays it in bytes) But it still is not as good as when I ran gzip on them. Why? ZFS uses the LZJB compression algorithm which isn't as space effecient as the gzip algorithm, but it is much faster. If I had been running Nevada I could have used:

root@warlord# zfs set compression gzip pool/audit

And have gotten the same compression ratio as when I "hand compress" my audit trails. This thanks to Adam who integrated gzip support in ZFS in build 62 of Nevada.

[Technorati Tags: ]

Comments:

Post a Comment:
Comments are closed for this entry.
About

martin

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today