Saturday Mar 28, 2009

snapshot on unlink?

This thread on OpenSolaris made me wonder how hard it would be to take a snapshot before any file is deleted. It turns out that using dtrace it is not hard at all. Using dtrace to monitor unlink and unlinkat calls and a short script to take the snapshots:

#!/bin/ksh93

function snapshot
{
	eval $(print x=$2)

	until [[ "$x" == "/" || -d "$x/.zfs/snapshot" ]]
	do
		x="${x%/\*}"
	done
	if [[ "$x" == "/" || "$x" == "/tmp" ]]
	then
		return
	fi
	if [[ -d "$x/.zfs/snapshot" ]]
	then
		print mkdir "$x/.zfs/snapshot/unlink_$1"
		pfexec mkdir "$x/.zfs/snapshot/unlink_$1"
	fi
}
function parse
{
	eval $(print x=$4)
	
	if [[ "${x%%/\*}" == "" ]]
	then
		snapshot $1 "$2$4"
	else
		snapshot $1 "$2$3/$4"
	fi
}
pfexec dtrace -wqn 'syscall::fsat:entry /pid != '$$' && uid > 100 && arg0 == 5/ {
	printf("%d %d \\"%s\\" \\"%s\\" \\"%s\\"\\n",
	pid, walltimestamp, root, cwd, copyinstr(arg2)); stop()
}
syscall::unlink:entry /pid != '$$' && uid > 100 / {
	printf("%d %d \\"%s\\" \\"%s\\" \\"%s\\"\\n",
	pid, walltimestamp, root, cwd, copyinstr(arg0)); stop()
}' | while read pid timestamp root cwd file
do
	print prun $pid
	parse $timestamp $root $cwd $file
	pfexec prun $pid
done

Now this is just a Saturday night proof of concept and it should be noted it has a significant performance impact and single threads all calls to unlink.

Also you end up with lots of snapshots:

cjg@brompton:~$ zfs list -t snapshot -o name,used | grep unlink

rpool/export/home/cjg@unlink_1238270760978466613                           11.9M

rpool/export/home/cjg@unlink_1238275070771981963                             59K

rpool/export/home/cjg@unlink_1238275074501904526                             59K

rpool/export/home/cjg@unlink_1238275145860458143                             34K

rpool/export/home/cjg@unlink_1238275168440000379                            197K

rpool/export/home/cjg@unlink_1238275233978665556                            197K

rpool/export/home/cjg@unlink_1238275295387410635                            197K

rpool/export/home/cjg@unlink_1238275362536035217                            197K

rpool/export/home/cjg@unlink_1238275429554657197                            136K

rpool/export/home/cjg@unlink_1238275446884300017                            350K

rpool/export/home/cjg@unlink_1238275491543380576                            197K

rpool/export/home/cjg@unlink_1238275553842097361                            197K

rpool/export/home/cjg@unlink_1238275643490236001                             63K

rpool/export/home/cjg@unlink_1238275644670212158                             63K

rpool/export/home/cjg@unlink_1238275646030183268                               0

rpool/export/home/cjg@unlink_1238275647010165407                               0

rpool/export/home/cjg@unlink_1238275648040143427                             54K

rpool/export/home/cjg@unlink_1238275649030124929                             54K

rpool/export/home/cjg@unlink_1238275675679613928                            197K

rpool/export/home/cjg@unlink_1238275738608457151                            198K

rpool/export/home/cjg@unlink_1238275800827304353                           57.5K

rpool/export/home/cjg@unlink_1238275853116324001                           32.5K

rpool/export/home/cjg@unlink_1238275854186304490                           53.5K

rpool/export/home/cjg@unlink_1238275862146153573                            196K

rpool/export/home/cjg@unlink_1238275923255007891                           55.5K

rpool/export/home/cjg@unlink_1238275962114286151                           35.5K

rpool/export/home/cjg@unlink_1238275962994267852                           56.5K

rpool/export/home/cjg@unlink_1238275984723865944                           55.5K

rpool/export/home/cjg@unlink_1238275986483834569                             29K

rpool/export/home/cjg@unlink_1238276004103500867                             49K

rpool/export/home/cjg@unlink_1238276005213479906                             49K

rpool/export/home/cjg@unlink_1238276024853115037                           50.5K

rpool/export/home/cjg@unlink_1238276026423085669                           52.5K

rpool/export/home/cjg@unlink_1238276041792798946                           50.5K

rpool/export/home/cjg@unlink_1238276046332707732                           55.5K

rpool/export/home/cjg@unlink_1238276098621721894                             66K

rpool/export/home/cjg@unlink_1238276108811528303                           69.5K

rpool/export/home/cjg@unlink_1238276132861080236                             56K

rpool/export/home/cjg@unlink_1238276166070438484                             49K

rpool/export/home/cjg@unlink_1238276167190417567                             49K

rpool/export/home/cjg@unlink_1238276170930350786                             57K

rpool/export/home/cjg@unlink_1238276206569700134                           30.5K

rpool/export/home/cjg@unlink_1238276208519665843                           58.5K

rpool/export/home/cjg@unlink_1238276476484690821                             54K

rpool/export/home/cjg@unlink_1238276477974663478                             54K

rpool/export/home/cjg@unlink_1238276511584038137                           60.5K

rpool/export/home/cjg@unlink_1238276519053902818                             71K

rpool/export/home/cjg@unlink_1238276528213727766                             62K

rpool/export/home/cjg@unlink_1238276529883699491                             47K

rpool/export/home/cjg@unlink_1238276531683666535                           3.33M

rpool/export/home/cjg@unlink_1238276558063169299                           35.5K

rpool/export/home/cjg@unlink_1238276559223149116                           62.5K

rpool/export/home/cjg@unlink_1238276573552877191                           35.5K

rpool/export/home/cjg@unlink_1238276584602668975                           35.5K

rpool/export/home/cjg@unlink_1238276586002642752                             53K

rpool/export/home/cjg@unlink_1238276586522633206                             51K

rpool/export/home/cjg@unlink_1238276808718681998                            216K

rpool/export/home/cjg@unlink_1238276820958471430                           77.5K

rpool/export/home/cjg@unlink_1238276826718371992                             51K

rpool/export/home/cjg@unlink_1238276827908352138                             51K

rpool/export/home/cjg@unlink_1238276883227391747                            198K

rpool/export/home/cjg@unlink_1238276945366305295                           58.5K

rpool/export/home/cjg@unlink_1238276954766149887                           32.5K

rpool/export/home/cjg@unlink_1238276955946126421                           54.5K

rpool/export/home/cjg@unlink_1238276968985903108                           52.5K

rpool/export/home/cjg@unlink_1238276988865560952                             31K

rpool/export/home/cjg@unlink_1238277006915250722                           57.5K

rpool/export/home/cjg@unlink_1238277029624856958                             51K

rpool/export/home/cjg@unlink_1238277030754835625                             51K

rpool/export/home/cjg@unlink_1238277042004634457                           51.5K

rpool/export/home/cjg@unlink_1238277043934600972                             52K

rpool/export/home/cjg@unlink_1238277045124580763                             51K

rpool/export/home/cjg@unlink_1238277056554381122                             51K

rpool/export/home/cjg@unlink_1238277058274350998                             51K

rpool/export/home/cjg@unlink_1238277068944163541                             59K

rpool/export/home/cjg@unlink_1238277121423241127                           32.5K

rpool/export/home/cjg@unlink_1238277123353210283                           53.5K

rpool/export/home/cjg@unlink_1238277136532970668                           52.5K

rpool/export/home/cjg@unlink_1238277152942678490                               0

rpool/export/home/cjg@unlink_1238277173482320586                               0

rpool/export/home/cjg@unlink_1238277187222067194                             49K

rpool/export/home/cjg@unlink_1238277188902043005                             49K

rpool/export/home/cjg@unlink_1238277190362010483                             56K

rpool/export/home/cjg@unlink_1238277228691306147                           30.5K

rpool/export/home/cjg@unlink_1238277230021281988                           51.5K

rpool/export/home/cjg@unlink_1238277251960874811                             57K

rpool/export/home/cjg@unlink_1238277300159980679                           30.5K

rpool/export/home/cjg@unlink_1238277301769961639                             50K

rpool/export/home/cjg@unlink_1238277302279948212                             49K

rpool/export/home/cjg@unlink_1238277310639840621                             28K

rpool/export/home/cjg@unlink_1238277314109790784                           55.5K

rpool/export/home/cjg@unlink_1238277324429653135                             49K

rpool/export/home/cjg@unlink_1238277325639636996                             49K

rpool/export/home/cjg@unlink_1238277360029166691                            356K

rpool/export/home/cjg@unlink_1238277375738948709                           55.5K

rpool/export/home/cjg@unlink_1238277376798933629                             29K

rpool/export/home/cjg@unlink_1238277378458911557                             50K

rpool/export/home/cjg@unlink_1238277380098888676                             49K

rpool/export/home/cjg@unlink_1238277397738633771                             48K

rpool/export/home/cjg@unlink_1238277415098386055                             49K

rpool/export/home/cjg@unlink_1238277416258362893                             49K

rpool/export/home/cjg@unlink_1238277438388037804                             57K

rpool/export/home/cjg@unlink_1238277443337969269                           30.5K

rpool/export/home/cjg@unlink_1238277445587936426                           51.5K

rpool/export/home/cjg@unlink_1238277454527801430                           50.5K

rpool/export/home/cjg@unlink_1238277500967098623                            196K

rpool/export/home/cjg@unlink_1238277562866135282                           55.5K

rpool/export/home/cjg@unlink_1238277607205456578                             49K

rpool/export/home/cjg@unlink_1238277608135443640                             49K

rpool/export/home/cjg@unlink_1238277624875209357                             57K

rpool/export/home/cjg@unlink_1238277682774484369                           30.5K

rpool/export/home/cjg@unlink_1238277684324464523                             50K

rpool/export/home/cjg@unlink_1238277685634444004                             49K

rpool/export/home/cjg@unlink_1238277686834429223                           75.5K

rpool/export/home/cjg@unlink_1238277700074256500                             48K

rpool/export/home/cjg@unlink_1238277701924235244                             48K

rpool/export/home/cjg@unlink_1238277736473759068                           49.5K

rpool/export/home/cjg@unlink_1238277748313594650                           55.5K

rpool/export/home/cjg@unlink_1238277748413593612                             28K

rpool/export/home/cjg@unlink_1238277750343571890                             48K

rpool/export/home/cjg@unlink_1238277767513347930                           49.5K

rpool/export/home/cjg@unlink_1238277769183322087                             50K

rpool/export/home/cjg@unlink_1238277770343306935                             48K

rpool/export/home/cjg@unlink_1238277786193093885                             48K

rpool/export/home/cjg@unlink_1238277787293079433                             48K

rpool/export/home/cjg@unlink_1238277805362825259                           49.5K

rpool/export/home/cjg@unlink_1238277810602750426                            195K

rpool/export/home/cjg@unlink_1238277872911814531                            195K

rpool/export/home/cjg@unlink_1238277934680920214                            195K

rpool/export/home/cjg@unlink_1238277997220016825                            195K

rpool/export/home/cjg@unlink_1238278063868871589                           54.5K

rpool/export/home/cjg@unlink_1238278094728323253                             61K

rpool/export/home/cjg@unlink_1238278096268295499                             63K

rpool/export/home/cjg@unlink_1238278098518260168                             52K

rpool/export/home/cjg@unlink_1238278099658242516                             56K

rpool/export/home/cjg@unlink_1238278103948159937                             57K

rpool/export/home/cjg@unlink_1238278107688091854                             54K

rpool/export/home/cjg@unlink_1238278113907980286                             62K

rpool/export/home/cjg@unlink_1238278116267937390                             64K

rpool/export/home/cjg@unlink_1238278125757769238                            196K

rpool/export/home/cjg@unlink_1238278155387248061                            136K

rpool/export/home/cjg@unlink_1238278160547156524                            229K

rpool/export/home/cjg@unlink_1238278165047079863                            351K

rpool/export/home/cjg@unlink_1238278166797050407                            197K

rpool/export/home/cjg@unlink_1238278168907009714                             55K

rpool/export/home/cjg@unlink_1238278170666980686                            341K

rpool/export/home/cjg@unlink_1238278171616960684                           54.5K

rpool/export/home/cjg@unlink_1238278190336630319                            777K

rpool/export/home/cjg@unlink_1238278253245490904                            329K

rpool/export/home/cjg@unlink_1238278262235340449                            362K

rpool/export/home/cjg@unlink_1238278262915331213                            362K

rpool/export/home/cjg@unlink_1238278264915299508                            285K

rpool/export/home/cjg@unlink_1238278310694590970                             87K

rpool/export/home/cjg@unlink_1238278313294552482                             66K

rpool/export/home/cjg@unlink_1238278315014520386                             31K

rpool/export/home/cjg@unlink_1238278371773568934                            258K

rpool/export/home/cjg@unlink_1238278375673503109                            198K

rpool/export/home/cjg@unlink_1238278440802320314                            138K

rpool/export/home/cjg@unlink_1238278442492291542                           55.5K

rpool/export/home/cjg@unlink_1238278445312240229                           2.38M

rpool/export/home/cjg@unlink_1238278453582077088                            198K

rpool/export/home/cjg@unlink_1238278502461070222                            256K

rpool/export/home/cjg@unlink_1238278564359805760                            256K

rpool/export/home/cjg@unlink_1238278625738732194                           63.5K

rpool/export/home/cjg@unlink_1238278633428599541                           61.5K

rpool/export/home/cjg@unlink_1238278634568579678                            137K

rpool/export/home/cjg@unlink_1238278657838186760                            288K

rpool/export/home/cjg@unlink_1238278659768151784                            223K

rpool/export/home/cjg@unlink_1238278661518121640                            159K

rpool/export/home/cjg@unlink_1238278664378073421                            136K

rpool/export/home/cjg@unlink_1238278665908048641                            138K

rpool/export/home/cjg@unlink_1238278666968033048                            136K

rpool/export/home/cjg@unlink_1238278668887996115                            281K

rpool/export/home/cjg@unlink_1238278670307970765                            227K

rpool/export/home/cjg@unlink_1238278671897943665                            162K

rpool/export/home/cjg@unlink_1238278673197921775                            164K

rpool/export/home/cjg@unlink_1238278674027906895                            164K

rpool/export/home/cjg@unlink_1238278674657900961                            165K

rpool/export/home/cjg@unlink_1238278675657885128                            165K

rpool/export/home/cjg@unlink_1238278676647871187                            241K

rpool/export/home/cjg@unlink_1238278678347837775                            136K

rpool/export/home/cjg@unlink_1238278679597811093                            199K

rpool/export/home/cjg@unlink_1238278687297679327                            197K

rpool/export/home/cjg@unlink_1238278749616679679                            197K

rpool/export/home/cjg@unlink_1238278811875554411                           56.5K

cjg@brompton:~$ 

Good job that snapshots are cheap. I'm not going to be doing this all the time but it makes you think what could be done.

Saturday Jan 03, 2009

Making the zfs snapshot service run faster

I've not been using Tim's auto-snapshot service on my home server as once I configured it so that it would work on my server I noticed it had a large impact on the system:

: pearson FSS 15 $; time /lib/svc/method/zfs-auto-snapshot \\
         svc:/system/filesystem/zfs/auto-snapshot:frequent

real    1m22.28s
user    0m9.88s
sys     0m33.75s
: pearson FSS 16 $;

The reason is two fold. First reading all the properties from the pool takes time and second it destroys the unneeded snapshots as it takes new ones. Something the service I used cheats with and does only very late at night. Looking at the script there are plenty of things that could be made faster and so I wrote a python version that could replace the cron job and the results , while and improvement were disappointing:

: pearson FSS 16 $; time ./zfs.py \\
         svc:/system/filesystem/zfs/auto-snapshot:frequent

real    0m47.19s
user    0m9.45s
sys     0m31.54s
: pearson FSS 17 $; 

still too slow to actually use. The time was dominated by cases where the script could not use a recursive option to delete the snapshots. The problem being that there is no way to list all the snapshots of a filesystem or volume but not it's decendents.


Consider this structure:

# zfs list -r -o name,com.sun:auto-snapshot tank
NAME                                  COM.SUN:AUTO-SNAPSHOT
tank                                  true
tank/backup                           false
tank/dump                             false
tank/fs                               true
tank/squid                            false
tank/tmp                              false

The problem here is that the script wants to snapshots and clean up “tank” but can't use recustion without backing up all the other file systems that have the false flag set and set for very good reason. Howeve If I did not bother to snapshot “tank” then tank/fs could be managed recusively and there would be no need for special handling. The above list does not reflect all the file systems I have but you get the picture. The results of making this change brings the timing for the service

: pearson FSS 21 $; time ./zfs.py \\
         svc:/system/filesystem/zfs/auto-snapshot:frequent

real    0m9.27s
user    0m2.43s
sys     0m4.66s
: pearson FSS 22 $; time /lib/svc/method/zfs-auto-snapshot \\
         svc:/system/filesystem/zfs/auto-snapshot:frequent

real    0m12.85s
user    0m2.10s
sys     0m5.42s
: pearson FSS 23 $; 

While the python module still gets better results than the korn shell script the korn shell script does not do so badly. However it still seems worthwhile spending the time to get the python script to be able to handle all the features of the korn shell script. More later.

Wednesday Jun 13, 2007

Home server back to build 65

My home server is taking a bit of a battering of late. I keep tripping over bug 6566921 which I can work around by not running my zfs_backup script locally. I have an updated version which will send the snapshots over an ssh pipe to a remote system which in my case is my laptop. Obviously this just moves the panic from my server to the laptop but that is a very much better state of affairs. I'm currently building a fixed zfs module which I will test later.

However the final straw that has had me revert to build 65 is that smbd keeps core dumping. Having no reliable access to their data caused the family more distress than you would expect. This turns out to be bug 6563383 which should be fixed in build 67.

Thursday Jun 07, 2007

zfs_versions of a file

This post reminds me that I should have posted my zfs_versions script a while back. I did not as it had a theoretical bug where if you could move a new file into place which had the same age as the old file it would not see this as a new version. I've fixed that now at the expense of some performance and the script is called zfs_versions.

The script lists all the different versions of a file that live in the snapshots.

Here is the output:

: pearson FSS 124 $; /home/cjg/bin/sh/zfs_versions  ~/.profile 
/tank/fs/users/cjg/.zfs/snapshot/month_09/.profile
/tank/fs/users/cjg/.zfs/snapshot/smb2007-03-27-15:54/.profile
/tank/fs/users/cjg/.zfs/snapshot/minute_2007-06-06-17:30/.profile
: pearson FSS 125 $; 

Compare this to the number of snapshots:


: pearson FSS 128 $; ls -1 ~/.zfs/snapshot/\*/.profile | wc -l
     705
: pearson FSS 129 $; 

So I have 705 snapshots that contain my .profile file but actually only three of them contain different versions.

Update

The addition of the check to fix the theoretical bug slows down the script enough that the programmer in me could not let it lie. Hence I now have the same thing in TCL.

Update 2

See http://blogs.sun.com/chrisg/entry/new_wish_and_tickle

Friday Jun 01, 2007

Rolling incremental backups

Why do you take back ups?

  • User error

  • Hardware failure

  • Disaster Recovery

  • Admin Error

With ZFS using redundant storage and plenty of snapshots my data should be safe from the first two. However that still leaves two ways all my data could be lost if I don't take some sort of back up.

Given the constraints I have my solution is to use my external USB disk containing a standalone zpool and then use zfs send and receive via this script to send all the file systems I really care about to the external drive.

To make this easier I have put all the filesystems into another “container” filesystem which has the “nomount” option set so it is hidden from the users. I can then recursively send that file system to the external drive. Also to stop the users getting confused by extra file systems appearing and disappearing I have set the mount point on the external disk to “none”.

The script only uses the snapshots that are prefixed “day” (you can change that with the -p (prefix) option) so that it reduces the amount of work that the script does. Backing up the snapshots that happen every 10 minutes on this system does not seem worth while for a job I will run once a day or so.

The really cool part of this is that once I had the initial snapshot on the external drive every back up from now on will be incremental. A rolling incremental backup. How cool is that.

# time ./zfs_backup tank/fs/safe removable/safe      



real    12m10.49s

user    0m11.87s

sys     0m12.32s

# 
 zfs list  tank/fs/safe removable/safe      

NAME             USED  AVAIL  REFER  MOUNTPOINT

removable/safe  78.6G  66.0G    18K  none

tank/fs/safe    81.8G  49.7G    18K  /tank/fs

# 

The performance is slightly disappointing due to the number of transport errors that are reported by the usb2scsa layer but the data is really on the disk so I am happy.

Currently I don't have the script clearing out the old snapshots but will get that going later. The idea of doing this over ssh to a remote site is compelling when I can find a suitable remote site.

Saturday Apr 07, 2007

recovering my laptop using zfs send and receive

First I owe Windows and apology. It was not making itself the active partition, grub was due to me copying the wrong entry in there after Solaris deleted it.

However before I went on holiday grub decided it would cease working. It could not find the menu.lst at all on this laptop. After a bit of investigation and failing to recover it (due I think to the laptop having two Solaris fdisk partitions, one had the zpool in it and the other the boot partitions, Installgrub and grub itself did not like this, though what pushed it over the edge I'm not sure. Perhaps some change in build 60.)

Anyway I decided to reinstall solaris without the extra fdsik partition which was a hangover from when the system had Linux on it as well. It should be simple. ufsdump the current boot environment (BE) and then install the system and let the zpool resilver from the external USB disk and ufsrestore the BE. The install would only need to be an end user install as I was going to restore from backup anyway. Strictly I did not need to do the install at all. All would have been fine had I not decided to detach the internal disk from the zpool (after scrubing the pool) prior to the install but the install would sort out grub for me without me having to do to much fiddling.

Once I had reinstalled the system I could not attach the new partition to the pool as it was to small. This was all thanks to my “thinking ahead” when I created the USB partition. Since eventually the partition will grow to be 30Gb that was how big the external disk partition was. As soon as I detached the smaller partition it “grew” to fill the partition it had. Doh.

So now I had to fit a 30Gb pool into a 10Gb partition. Next time I won't detach the mirror first! Being in a hurry, to go on holiday, I just knocked together a short script that would take a source file system and using zfs send and zfs receive copy all of it's snapshots to the target file system. So first doing a recursive snapshot of the pool I then ran the script which and copied the file systems into the new pool I created on the laptop. I then had to fix up a few of the attributes of the file systems that were copied. I'm not immediately sure how to handle this in a script since some attributes are obvious (compression, sharenfs etc) but others are less so (mountpoint). Even with attributes like compression you have a problem in that the zfs receive creates the file system with inherited attributes so there is no way to set them before the file system is populated unless the inherited file system attribute is correct. When I say no way, I mean, no simple way. There clearly are ways by using temporary container filesystems which you create with the right options for the file system to inherit and then use zfs rename to move the file system to the correct location. However that was not required by me and would cease to be a simple short script.

Tags:

Thursday Nov 23, 2006

A faster ZFS snapshot massacre

I moved the zfs snapshot script into the office and started running it on our build system. Being a cautious type when it comes to other people's data I ran the clean up script in “do nothing” mode so I could be sure it was not cleaning snapshots that it should not. After a while running like this we had over 150,000 snapshots of 114 file systems which meant that zfs list was now taking a long time to run.

So long in fact that the clean up script was not actually making forward progress against snapshots being created every 10 minutes. So I now have a new clean up script. This is functionally identical to the old one but a lot faster. Unfortunately I have now cleaned out the snapshots so the times are not what they were, zfs list was taking 14 minutes, however the difference is still easy to see.

When run with the option to do nothing the old script:

# time /root/zfs_snap_clean > /tmp/zfsd2

real    2m23.32s
user    0m21.79s
sys     1m1.58s
#

And the new:

# time ./zfs_cleanup -n > /tmp/zfsd

real    0m7.88s
user    0m2.40s
sys     0m4.75s
#

which is a result.


As you can see the new script is mostly a nawk script and more importantly only calls the zfs command once to get all the information about the snapshots:


#!/bin/ksh -p
#
# Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License, Version 1.0 only
# (the "License").  You may not use this file except in compliance
# with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END
#
#	Script to clean up snapshots created by the script from this blog
#	entry:
#
#	http://blogs.sun.com/chrisg/entry/cleaning_up_zfs_snapshots
#
#	or using the command given in this entry to create snapshots when
#	users mount a file system using SAMBA:
#
#	http://blogs.sun.com/chrisg/entry/samba_meets_zfs
#
#	Chris.Gerhard@sun.com 23/11/2006
#

PATH=$PATH:$(dirname $0)

while getopts n c
do
	case $c in
	n) DO_NOTHING=1 ;;
	\\?) echo "$0 [-n] [filesystems]"
		exit 1 ;;
	esac
done
shift $(($OPTIND - 1))
if (( $# == 0))
then
	set - $(zpool list -Ho name)
fi


export NUMBER_OF_SNAPSHOTS_boot=${NUMBER_OF_SNAPSHOTS:-10}
export DAYS_TO_KEEP_boot=${DAYS_TO_KEEP:-365}

export NUMBER_OF_SNAPSHOTS_smb=${NUMBER_OF_SNAPSHOTS:-100}
export DAYS_TO_KEEP_smb=${DAYS_TO_KEEP:-14}

export NUMBER_OF_SNAPSHOTS_month=${NUMBER_OF_SNAPSHOTS:-24}
export DAYS_TO_KEEP_month=365

export NUMBER_OF_SNAPSHOTS_day=${NUMBER_OF_SNAPSHOTS:-$((28 \* 2))}
export DAYS_TO_KEEP_day=${DAYS_TO_KEEP:-28}

export NUMBER_OF_SNAPSHOTS_hour=$((7 \* 24 \* 2))
export DAYS_TO_KEEP_hour=$((7))

export NUMBER_OF_SNAPSHOTS_minute=$((100))
export DAYS_TO_KEEP_minute=$((1))


zfs get -Hrpo name,value creation $@ | sort -r -n -k 2 |\\
	nawk -v now=$(convert2secs $(date)) -v do_nothing=${DO_NOTHING:-0} '
function ttg(time)
{
	return (now - (time \* 24 \* 60 \* 60));
}
BEGIN {
	time_to_go["smb"]=ttg(ENVIRON["DAYS_TO_KEEP_smb"]);
	time_to_go["boot"]=ttg(ENVIRON["DAYS_TO_KEEP_boot"]);
	time_to_go["minute"]=ttg(ENVIRON["DAYS_TO_KEEP_minute"]);
	time_to_go["hour"]=ttg(ENVIRON["DAYS_TO_KEEP_hour"]);
	time_to_go["day"]=ttg(ENVIRON["DAYS_TO_KEEP_day"]);
	time_to_go["month"]=ttg(ENVIRON["DAYS_TO_KEEP_month"]);
	number_of_snapshots["smb"]=ENVIRON["NUMBER_OF_SNAPSHOTS_smb"];
	number_of_snapshots["boot"]=ENVIRON["NUMBER_OF_SNAPSHOTS_boot"];
	number_of_snapshots["minute"]=ENVIRON["NUMBER_OF_SNAPSHOTS_minute"];
	number_of_snapshots["hour"]=ENVIRON["NUMBER_OF_SNAPSHOTS_hour"];
	number_of_snapshots["day"]=ENVIRON["NUMBER_OF_SNAPSHOTS_day"];
	number_of_snapshots["month"]=ENVIRON["NUMBER_OF_SNAPSHOTS_month"];
} 
/.\*@.\*/ { 
	split($1, a, "@");
	split(a[2], b, "_");
	if (number_of_snapshots[b[1]] != 0 &&
		++snap_count[a[1], b[1]] > number_of_snapshots[b[1]] &&
		time_to_go[b[1]] > $2) {
		str= sprintf("zfs destroy %s\\n", $1);
		printf(str);
		if (do_nothing == 0) {
			system(str);
		}
	}
}'

Tags:

Saturday Nov 11, 2006

ZFS snapshot massacre.

As the number of snapshots grows I started wondering how much space they are really taking up on the home server. This is pretty much also shows how much data gets modified after being initially created. I would guess not much as the majority of the data on the server would be:

  1. Solaris install images. Essentially read only.

  2. Photographs.

  3. Music mostly in the form of iTunes directories.

Running this command line get the result:

zfs get -rHp used $(zpool list -H -o name ) |\\
nawk '/@/ && $2 == "used" { tot++; total_space+=$3 ;\\
        if ( $3 == 0 ) { empty++ }} \\
END { printf("%d snapshots\\n%d empty snapshots\\n%2.2f G in %d snapshots\\n", tot, \\
        empty, total_space/(1024\^3), tot - empty ) }'
68239 snapshots
63414 empty snapshots
2.13 G in 4825 snapshots'
: pearson TS 15 $; zfs get used $(zpool list -H -o name )
NAME  PROPERTY  VALUE  SOURCE
tank  used      91.2G  -
: pearson TS 16 $;

So I only have 2.13G of data saved in snapshots out of 91.2 G of data. Not really a surprising result. The biggest user of space for snapshots is one file system. The one that contains planetcycling.org. As the planet gets updated every 30 minutes and the data is only indirectly controlled by me I'm not shocked by this. I would expect the amount to stabilize over time as the system and to that end I will note the current usage:


zfs get -rHp used tank/fs/web|\\
nawk '/@/ && $2 == "used" { tot++; total_space+=$3 ;\\
        if ( $3 == 0 ) { empty++ }} \\
END { printf("%d snapshots\\n%d empty snapshots\\n%2.2f G in %d snapshots\\n", tot,
        empty, total_space/(1024\^3), tot - empty ) }'
1436 snapshots
789 empty snapshots
0.98 G in 647 snapshots

All this caused me to look a bit harder at the zfs_snapshot_clean script I have as it appeared to be keeping some really old snapshots from some of the classes that I did not expect. Now while the 68000 snapshots were having no negative impact on the running of the system it it was not right. There were two issues. First it was sorting the list of snapshots using the snapshot creation time, which was correct, but it was sorting in reverse order which was not. Secondly I was keeping a lot more of the hourly snapshots than I intended.


After fixing this and running the script (you can download it from here) there was a bit of a snapshot massacre leading to a lot less snapshots:


zfs get -rHp used $(zpool list -H -o name ) |\\
nawk '/@/ && $2 == "used" { tot++; total_space+=$3 ;\\
        if ( $3 == 0 ) { empty++ }} \\
END { printf("%d snapshots\\n%d empty snapshots\\n%2.2f G in %d snapshots\\n", tot, \\
        empty, total_space/(1024\^3), tot - empty ) }'
25512 snapshots
23445 empty snapshots
2.20 G in 2067 snapshots

Only 25000 snapshots, much better, most of them remain empty.

Tags:

Saturday Oct 21, 2006

Shared samba directories

The samba set up on the new server for users has been flawless, but the shared directories slightly less so. I had a problem where if one of the family created a directory then the rest of the family could not add to that directory. Looking on the Solaris side it was clear the problem, the directory was created mode 755. Typing this I realize just how bad that is. 755 could not possibly mean anything to anyone who was not up to their armpits into UNIX computing and the explication would fill pages and indeed it does.


The permissions I want to force for directories are "read, write and execute for group" as well as the owner. Ie mode 775. It would also be nice if I could stop one user deleting the other users work so setting the sticky bit would also be good giving mode 1755.


Trundling through the smb.conf manual page tells me that there is an option, "force directory mode" that does exactly what it implies and what I want. I'm sure I could achieve the same with an ACL and will do that later so that SMB and NFS give the same results. However for now smb.conf serves this purpose.


So the new entry in the smb.conf for the shared area where we keep pictures looks like this:

[pics]
   comment = Pictures
   path = /tank/shared/pics
   public = yes
   writable = yes
   printable = no
   write list = @staff
   force directory mode = 1775
   force create mode = 0444
   root preexec = ksh -c '/usr/sbin/zfs snapshot tank/shared/pics@smb$(/tank/local/smbdate)'

Now everyone can add to the file system but can't delete others photos, plus I get a snapshot every time someone starts to access the file system.


Tags:

Thursday Sep 07, 2006

How many snapshots?

Having a non laptop system that is on all the time running zfs with automatic snapshots you start to build up the snapshots at an alarming rate.

# for i in $(zfs list -Ho name -t snapshot )
do
zfs get -pH used $i
done | nawk '{ count++;
        if ($3 > 0 ) {
                nonzero++
        };
        total+=$3
}
END {
        print count, nonzero, total/(1024\^2)
}'
7071 188 83.939
#

So after one week I have 7071 snapshots of which only 188 currently contain data taking just 85 megabytes with the total pool taking 42.8G.


No downsides have been seen so far so while the numbers appear alarming I see no reason not to continue.


Tags:



Tuesday Sep 05, 2006

Cleaning up zfs snapshots

Thank you to the anonymous comments about samba and ZFS and the clean up script.

A days worth of samba snapshots look like this:

tank/users/cjg@smb1157437900  37.5K      -  21.1G  -
tank/users/cjg@smb1157441840      0      -  21.1G  -
tank/users/cjg@smb1157441861      0      -  21.1G  -
tank/users/cjg@smb1157000000  40.5K      -  21.1G  -
tank/users/cjg@smb1157445557  40.5K      -  21.1G  -
tank/users/cjg@smb2006-09-05-12:03  40.5K      -  21.1G  -
tank/users/cjg@smb2006-09-05-18:27      0      -  21.1G  -
tank/users/keg@smb2006-09-05-18:29      0      -   465M  -
tank/users/rlg@smb1157441373      0      -   673M  -
tank/users/rlg@smb1157446766      0      -   675M  -
tank/users/rlg@smb1157449795    21K      -   675M  -
tank/users/rlg@smb2006-09-05-17:14      0      -   675M  -
tank/users/rlg@smb2006-09-05-17:54      0      -   675M  -
tank/users/rlg@smb2006-09-05-18:07      0      -   675M  -
tank/users/stc@smb1157437923      0      -   294M  -
tank/users/stc@smb1157446971      0      -   294M  -
tank/users/stc@smb2006-09-05-15:34      0      -   294M  -
tank/users/stc@smb2006-09-05-17:47      0      -   294M  -
tank/users/stc@smb2006-09-05-20:27      0      -   294M  -

from which you can see I experimented with naming them with the seconds from the epoch to make the clean up script simpler. However after a few minutes I realized there was a better way.


I now have a clean up script that uses the zfs file system creation time to do all the sorting. Getting this to work quickly requires a script to convert the time stamp into seconds from the epoch:


#!/usr/sfw/bin/tclsh8.3
puts [clock scan $argv ]

Call the script “convert2secs” and then the rest of script is simple;

#!/bin/ksh -p
#
#       Quick scipt to clean up the snapshots created by each samma login.
#       See: http://blogs.sun.com/chrisg/entry/samba_meets_zfs
#
#       It is clear that this could be much more generic. Espeically if you
#       could add a property to the snapshot to say when it should be deleted.
#
PATH=$PATH:/tank/local/bin
ALL_TYPES="smb minute hour day month boot"

NUMBER_OF_SNAPSHOTS_boot=${NUMBER_OF_SNAPSHOTS:-10}
DAYS_TO_KEEP_boot=${DAYS_TO_KEEP:-365}

NUMBER_OF_SNAPSHOTS_smb=${NUMBER_OF_SNAPSHOTS:-100}
DAYS_TO_KEEP_smb=${DAYS_TO_KEEP:-14}

NUMBER_OF_SNAPSHOTS_month=${NUMBER_OF_SNAPSHOTS:-24}
DAYS_TO_KEEP_month=365

NUMBER_OF_SNAPSHOTS_day=${NUMBER_OF_SNAPSHOTS:-$((28 \* 2))}
DAYS_TO_KEEP_day=${DAYS_TO_KEEP:-28}

NUMBER_OF_SNAPSHOTS_hour=$((7 \* 24 \* 2))
DAYS_TO_KEEP_hour=$((7 \* 24))

NUMBER_OF_SNAPSHOTS_minute=$((60\*24))
DAYS_TO_KEEP_minute=$((1))


today=$(convert2secs $(date))

function do_fs
{
        typeset fd
        typeset -i count=0
        typeset -i seconds2keep
        typeset -i time2go
        typeset -i number_of_snapshots
        typeset type=$2
        #
        # days2keep and number_of_snaphots should come from
        # file system properties. Until then the are fed from the
        # global entities.
        #
        days2keep=$(eval echo \\${DAYS_TO_KEEP_${type}})
        number_of_snapshots=$(eval echo \\${NUMBER_OF_SNAPSHOTS_${type}})

        seconds2keep=$(( days2keep \* 24 \* 60 \* 60 ))
        time2go=$((today - seconds2keep))

        for fs in $(zfs list -r -t snapshot -o name $1 | grep $type | sort -r -t @ -k 1)
        do
                snap_time=$(convert2secs $(/usr/sbin/zfs list -H -o creation ${fs}))

                if (( count > number_of_snapshots )) && \\
                        (( snap_time < time2go ))
                then
                        zfs destroy $fs
                else
                        : echo $fs is kept for $((snap_time - time2go)) seconds
                fi
                let count=count+1
        done
}

for type in ${ALL_TYPES}
do
        for i in $(zfs list -H -t snapshot -r $@ | sort | nawk -F '@' '/'$type'/ { print $1 }' | uniq)
        do
                do_fs $i $type
        done
done

When zfs has user defined options all the configuration can be kept in the file system but until then the configuration variables will do.

The script allows me to have different classes of snapshot: smb, minute, hour, day, month and boot. This allows the same script to clean up both the snapshots taken by samba and the ones taken via cron and boot.


The script errs on the side of not destroying snapshots so for each class I'm keeping all snapshots less than a certain number of days old and also keeping a minimun number of snapshots.


Class

Minimum number of snapshots

Number of days to keep snapshots

boot

10

365

smb

100

14

months

24

365

day

28 \* 2

28

hour

7 \* 24 \* 2

7 \* 24

minute

60 \* 24

1



The advantage is that I can now both keep the snapshots longer and also give them more user friendly names. The new snapshot cron job script is here. I'm sure the number of snapshots generated is overkill but while I have the disk space why not?


Now if I can stop smb mangling the names all would be perfect.


Tags:

Monday Sep 04, 2006

Samba meets ZFS

There has been much progress on the new server at home which I will write up later. Today I'll dig into what I have done to make samba and ZFS play well together. As I mentioned getting Samba running was easy. However there is a bit more that you can do to make ZFS even and Samba even better together.

Why not have zfs take a snapshot whenever you login to the PC? So in addition to the regular snapshots I also get one of the home directory of each user when they login.


Just add this line to the [homes] section in the smb.conf:

root preexec = ksh -c '/usr/sbin/zfs snapshot tank/users/%u@smb$(/tank/local/smbdate)'

Then you need the smbdate script to print a good date. You can't just use the date command directly as Samba expands the % entries before they are passed to date. Hence I wrap it in a script:


#!/bin/ksh -p
exec date +%F-%R

This results in snapshots like this each time a user logins on the PC


# zfs list | grep smb
tank/users/cjg@smb2006-09-04-22:53      0      -  21.1G  -
#

At some point a script to clean up the old snapshots will be needed.


Tags

Friday Sep 01, 2006

Home server progress

Progress on the new home server:

All the user accounts have been created each with their own ZFS file system for their home directory.

I've installed my ZFS snapshot script and have crontab entries like this:

10,20,30,40,50 \* \* \* \* /tank/local/snapshot minute
0 \* \* \* \* /tank/local/snapshot hour
1 1 \* \* \* /tank/local/snapshot day
2 1 1 \* \* /tank/local/snapshot month

I think there is some scope for improvement here which would mean keeping the snapshots for longer. When the proposal to have user defined properties becomes real I will define the snapshot rules in the file systems and have the script use those rather than a one size fits all.


I have samba running from SMF thanks to Trevor's manifest and script. I did the changes to the script suggested in the comments. This all worked perfectly and now the PC flies and that is before I install gigabit Ethernet in it. Already you can see the snapshot directories under .zfs in each of the samba file systems on the PC which is just about as cool as it can get.


Finally I have solaris serving dhcp and have turned off the server on the Qube. Most uncharacteristically I used the GUI tool to configure dhcp and apart from having to create another ZFS file system to put the data in the GUI did it all. Very slick. Plus by some magic it managed to hand out the same IP addresses as the Qube used to to each computer. I suspect I should have done DNS before DHCP since the DHCP server can update the DNS records so this may have to be done again.


Tags:

Tuesday Mar 07, 2006

When you have lost your data.....

Today was not the first time, and won't be the last time that someone contacted me after loosing a load of data. Previous incidents have included when my partner was writing a letter on my Ultra1 at home and my daughter decided that the system would run much better without power.

On rebooting the file was zero length. This letter had taken days to craft so simply starting again was really low on the list of priorities. The file system on which it had lived was UFS and I could access the system over the network (I was on a business trip at the time), there were no backups...

The first thing about any situation like this is not to panic. You need to get the partition that used to contain the data quiesed so that no further damage is done. Unmount it if you can. Then take a backup of it using dd:

dd if=/dev/rdsk/c0t0d0s4 of=copy_of_slice bs=128k

Now you use the copy_of _slice to go looking for the blocks. If the file is a text file then you can use strings and grep to search for blocks that may contain your data. Specifically:


strings -a -t d < copy_of_slice | grep “text in document” 

This outputs the strings that contain “text in document” and their byte offsets you use these offsets to read the blocks.


73152472 moz-abmdbdirectory://abook.mab
136142157 fc9roaming.default.files.abook.mab
136151743 7moz-abmdbdirectory://abook.mab
136151779 !6fb-moz-abmdbdirectory://abook.mab


I use a shell function like this for a file system with an 8K block size:


function readblock
{
        dd bs=8k count=1 iseek=$(($1/ 8192)) < slice7
}


Since the file in this case was called slice7 to get the blocks.


$ readblock 73152472


then you have to use your imagination and skill to put the blocks back together. In my case the letter was recovered, sent and had the disired outcome.


Todays example is not looking so good. Firstly the victim had actually run suninstall over the drive and had no backup (stop giggling at the back) which had relabled the drive and then run newfs on the partition. Then when the dd was run the output file was wirtten onto the same disk so if the label did not match more damage was done. I might suggest that he run over the drive and then throw it into the pond just to make live interesting. It's a pity as since only the super blocks would have been written the chances of recovery where not that bad.


So to recap. Don't get in this situation. Backup everything. Use ZFS, use snapshots, lots of them.


However if you have lost your data and want to stand any chance of getting it back:

  1. Don't Panic.

  2. Quiese the file system. Powering off the system may well be your best option.

  3. Get a bit for bit copy of the disk that had the data. All slices. Do this while booted of release media.

  4. Hope you are lucky.


Tags:

Thursday Dec 15, 2005

Letting users create ZFS file systems

Darren has just posted his fast bringover script that solves some of my desire to be able to have a file system per workspace. I'm not commenting on the script since it manages to trip one of my shell script peeves that of calling a program and then calling exit $?. What is wrong with exec? I'll keep taking the tablets.

However it does not solve my wanting to be able to let users be able to create their own ZFS file systems below a file system that they own.

Like I said in the email this can mostly be done via an RBAC script, well here it is:

#!/bin/ksh -p

PATH=/usr/bin:/usr/sbin

if [ "$_" != "/usr/bin/pfexec" -a -x /usr/bin/pfexec ]; then
        exec /usr/bin/pfexec $0 $@
fi

function get_owner
{
	echo $(ls -dln ${PARENT} | nawk '{ print $3 }')
}

function create_file_system
{
	typeset mpt name

	zfs list -H -t filesystem -o mountpoint,name,quota | \\
		 while read mpt name quota
	do
		if [[ $mpt == $PARENT ]]
		then
			zfs create ${DIR#/} && chown $uid $DIR && \\
				zfs set quota=${quota} ${DIR#/}
			exit $?
		fi
	done
	echo no zfs file system $PARENT >&2
	exit 1
}

function check_quota
{
	typeset -i count
	typeset mpt name
	count=0

	zfs list -H -t filesystem -o mountpoint,name | while read mpt name
	do
		if [[ $(get_owner $name) == $uid ]]
		then
			let count=count+1
		fi
	done
	echo $count
}

MAX_FILE_SYSTEMS_PER_USER=10

test -f /etc/default/zfs_user_create && . /etc/default/zfs_user_create

if [[ $# -ne 1 ]]
then
	echo "Usage: $1 filesystem" >&2
	exit 1
fi

DIR=$1
PARENT=${1%/\*}

if ! [[ -d $PARENT ]]
then
	echo "$0: Failed to make directory \\"$1\\"; No such file or directory" >&2
	exit 1
fi

uid=$(id | sed -e s/uid=// -e 's/(.\*//')
owner=$(get_owner $1)

if [[ $uid != $owner ]]
then
	echo "$0: $1 not owner" >&2
	exit 1
fi

if [[ $(check_quota) -gt ${MAX_FILE_SYSTEMS_PER_USER} ]]
then
	echo "too many file systems"
	exit 1
fi

create_file_system

It has a hack in it to limit the number of file systems that a user can create just to stop them being silly. Then you just need the line in /etc/security/exec_attr:


All:suser:cmd:::/usr/local/share/sh/zfs_create:euid=0

Now any user can create a file system under a file system they already own. The file systems don't share a single quota which would be nice but for my purposes this will do.


Next trick to let them destroy them and take snapshots of them. The snapshots being the real reason I want all of this.

Tags:

Thursday Dec 01, 2005

ZFS snapshots meet automounter and have a happy time together

There is a thread going over on zfs-interest where the following question was posed by biker (I live in hope that this is really “cyclist”):

How can I make a snapshot of home in zfs containing the data including the stuff within the user homes (home/ann, home/bob) - like a recursive snapshot.

The only way so far I could think of was
 - copy the directory structure (home/ann, home/bob) to /snap
 - initiate a snapshot of every dataset (home/ann, home/bob)
 - mount each snapshot to the counterpart under /snap
 - run the backup
 - remove the mounts
 - release the snapshots
 - clear /snap

If there is something like a recursive snapshot or user and group quota in the classical sense, the efford needed could be minimized, ...

It got me thinking that in the absence of a real solution this should be doable with a script. For the recursive backup script we have:

#!/bin/ksh -p

for filesystem in $(zfs list -H -o name -t filesystem)
do
        zfs snapshot $filesystem@$1
done

No prizes there but what biker wanted was a copy of the file system structure. The problem is that all those snapshots are each under the individual file systems .zfs/snapshot directory so are spread about.


If only we could mount all of them under one tree? By adding this line to /etc/auto_master:


/backup /root/auto_snapshot

and then this script as /root/auto_snapshot:

#!/bin/ksh -p

PATH=/usr/sbin:/sbin:$PATH

for filesystem in $(zfs list -H -o name -t filesystem)
do
if [[ -d /$filesystem/.zfs/snapshot/$1 ]]
then
        fs=${filesystem#\*/}
        if [[ ${fs} = ${filesystem} ]]
        then
                fs=""
        fi
        ANS="${ANS:-}${ANS:+ }/$fs localhost:/$filesystem/.zfs/snapshot/$1"
fi
done
echo $ANS

Suddenly I can do this:


1071 # ./backup fullbackup
1072 # (cd /backup/fullbackup ; tar cf /mypool/root/backup.tar . )
1073 # df -h
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c0d0s0        7.3G   6.7G   574M    93%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   829M   704K   829M     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
/usr/lib/libc/libc_hwcap1.so.1
                       7.3G   6.7G   574M    93%    /lib/libc.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                   829M   136K   829M     1%    /tmp
swap                   829M    44K   829M     1%    /var/run
mypool                 9.5G   5.8G   2.9G    67%    /mypool
mypool/jfg             9.5G     8K   2.9G     1%    /mypool/jfg
mypool/keg             9.5G    16M   2.9G     1%    /mypool/keg
mypool/rlg             9.5G     8K   2.9G     1%    /mypool/rlg
mypool/stc             9.5G    14M   2.9G     1%    /mypool/stc
/mypool/cg13442        8.8G   5.8G   2.9G    67%    /home/cg13442
/mypool/.zfs/snapshot/fullbackup
                       8.8G   3.9G   4.8G    45%    /backup/fullbackup
/mypool/rlg/.zfs/snapshot/fullbackup
                       4.8G     8K   4.8G     1%    /backup/fullbackup/rlg
/mypool/jfg/.zfs/snapshot/fullbackup
                       4.8G     8K   4.8G     1%    /backup/fullbackup/jfg
/mypool/stc/.zfs/snapshot/fullbackup
                       4.8G    14M   4.8G     1%    /backup/fullbackup/stc
1074 #

The tar backup file now contains the whole of the “fullback snapshot” and apart from the snapshot not really being atomic, since each file system is snapped in sequence this pretty much does what is wanted.


If you were really brave/foolish you could have the automounter executable maps generate the snapshots for you but that would be a receipe for filling the pool with snapshots. Deleting the snapshots is also a snip:

#!/bin/ksh -p

for filesystem in $(zfs list -H -o name -t filesystem)
do
        zfs destroy $filesystem@$1
done


Tags:

Sunday Nov 20, 2005

Snapping every minute

I'm not sure when ZFS was being designed it's target was going to be laptops. However on mine it is looking great. So good I upgraded my Dell this afternoon.

A short script and I have a snapshot every minute which I can keep for an hour and then the hourly snaphshots kept for a day. The daily snapshots kept for a month and the monthly snaphosts kept for a year.

It will be ineteresting to see how the disk space usage works out over the longer term. This all comes from this script:

#!/bin/ksh -p

function take_snap
{
        if zfs list -H -o name $1 >/dev/null 2>&1
        then
                zfs destroy $1
        fi
        zfs snapshot ${1}
}

case ${1:-boot} in
        "boot")
                snap=$(date '+%F-%T')
                ;;
        "minute")
                snap=minute_$(date +%M)
                ;;
        "hour")
                snap=hour_$(date +%H)
                ;;
        "day")
                snap=day_$(date +%e)
                ;;
        "month")
                snap=month_$(date +%m)
                ;;
esac

for fs in $(zfs list -H -o name -t filesystem)
do
        take_snap ${fs}@${snap}
done

And this entry in cron:

\* \* \* \* \* /export/home/local/snapshot minute
0 \* \* \* \* /export/home/local/snapshot hour
1 1 \* \* \* /export/home/local/snapshot day
2 1 1 \* \* /export/home/local/snapshot month

All those snapshots:

: sigma IA 9 $; ls ~/.zfs/snapshot/
2005-11-20-17:00:26  minute_20            minute_46
hour_21              minute_34            minute_47
hour_22              minute_35            minute_48
minute_00            minute_36            minute_49
minute_11            minute_37            minute_50
minute_12            minute_38            minute_51
minute_13            minute_39            minute_52
minute_14            minute_40            minute_53
minute_15            minute_41            minute_54
minute_16            minute_42            minute_55
minute_17            minute_43            minute_56
minute_18            minute_44            minute_57
minute_19            minute_45
: sigma IA 10 $;

listing out the snapshots it really does seem that they are free:

# zfs list | grep cjg
home/cjg              33.0M  8.52G  30.5M  /export/home/cjg
home/cjg@2005-11-20-17:00:26  30.5K      -  29.4M  -
home/cjg@hour_21       296K      -  30.6M  -
home/cjg@minute_34    42.0K      -  31.0M  -
home/cjg@minute_35    22.0K      -  31.0M  -
home/cjg@minute_36    50.5K      -  31.0M  -
home/cjg@minute_37    37.0K      -  31.0M  -
home/cjg@minute_38        0      -  31.0M  -
home/cjg@minute_39        0      -  31.0M  -
home/cjg@minute_40        0      -  31.0M  -
home/cjg@minute_41        0      -  31.0M  -
home/cjg@minute_42    6.00K      -  31.0M  -
home/cjg@minute_43    68.0K      -  31.0M  -
home/cjg@minute_44    31.5K      -  31.0M  -
home/cjg@minute_45    37.5K      -  31.0M  -
home/cjg@minute_46        0      -  31.0M  -
home/cjg@minute_47        0      -  31.0M  -
home/cjg@minute_48        0      -  31.0M  -
home/cjg@minute_49        0      -  31.0M  -
home/cjg@minute_50        0      -  30.3M  -
home/cjg@minute_51        0      -  30.3M  -
home/cjg@minute_52        0      -  30.3M  -
home/cjg@minute_53        0      -  30.3M  -
home/cjg@minute_54        0      -  30.3M  -
home/cjg@minute_55        0      -  30.3M  -
home/cjg@minute_56        0      -  30.3M  -
home/cjg@minute_57        0      -  30.3M  -
home/cjg@hour_22          0      -  30.4M  -
home/cjg@minute_00        0      -  30.4M  -
home/cjg@minute_11        0      -  30.5M  -
home/cjg@minute_12        0      -  30.5M  -
home/cjg@minute_13        0      -  30.5M  -
home/cjg@minute_14        0      -  30.5M  -
home/cjg@minute_15        0      -  30.5M  -
home/cjg@minute_16        0      -  30.5M  -
home/cjg@minute_17        0      -  30.5M  -
home/cjg@minute_18        0      -  30.5M  -
home/cjg@minute_19        0      -  30.5M  -
home/cjg@minute_20        0      -  30.5M  -
home/cjg@minute_21        0      -  30.5M  -
home/cjg@minute_22        0      -  30.5M  -
home/cjg@minute_23        0      -  30.5M  -
#

I can feel an open letter to the admin of my home directory server comming along the lines of:

I want a snapshot every minute and I want it now.

Tags:

Friday Nov 18, 2005

ZFS snapshot on boot

ZFS hits my Toshiba laptop and so I have moved the home directories onto a zfs file system.

The big advantage for me is that I can now have the system snapshot the filesystem each time it boots. First this script does the work:

#!/bin/ksh -p
date=$(date '+%F-%T')

for fs in $(zfs list -H -o name -t filesystem)
do
        zfs snapshot ${fs}@${date}
done

And this manifest gets it to be run:

<?xml version="1.0"?>

<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">

<service_bundle type='manifest' name='snapshot'>

<service
        name='system/filesystem/snapshot'
        type='service'
        version='1'>

        <create_default_instance enabled='true' />

        <dependency
                name='filesystem'
                grouping='require_all'
                restart_on='none'
                type='service'>
                <service_fmri value='svc:/system/filesystem/local' />
        </dependency>

        <exec_method
                type='method'
                name='start'
                exec='/mypool/root/snapshot'
                timeout_seconds='10' />

        <exec_method
                type='method'
                name='stop'
                exec=':true'
                timeout_seconds='3' />

        <property_group name='startd' type='framework'>
                <propval name='duration' type='astring' value='transient'/>
        </property_group>

</service>
</service_bundle>

Now import the manifest into smf:

 # svccfg import  snapshot.xml

Now I get a snapshot of the home directories everytime I reboot. These are mounted under .zfs/snapshot so users can pull back anything they need when they want to. Just what you want from a file system.

Tags: , , ,

About

This is the old blog of Chris Gerhard. It has mostly moved to http://chrisgerhard.wordpress.com

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today