Wednesday Jun 25, 2008

Why check the digest of files you copy and what to do when they don't match

I'm always copying data from home to work and less often from work to home. Mostly these are disk images. I always check the md5 sum just out of paranoia. It turns out you can't be paranoid enough! The thing to remember if the check sums don't match is not to copy the file again but use rsync. It will bring over just the blocks that are corrupt.

: enoexec.eu FSS 43 $; scp thegerhards.com:/tank/tmp/diskimage.fat.bz2 .
diskimage.fat.bz2    100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  1825 MB 11:10:31    
: enoexec.eu FSS 44 $; digest -a md5 diskimage.fat.bz2    
674f69eec065da2b4d3da4bf45c7ae5f
: enoexec.eu FSS 45 $; ssh thegerhards.com digest -a md5 /tank/tmp/diskimage.fat.bz2
191f26762d5b48e0010a575b54746e80
: enoexec.eu FSS 46 $; ls -l diskimage.fat.bz2
-rw-r-----   1 cg13442  staff    1913779931 Jun 25 08:56 diskimage.fat.bz2
: enoexec.eu FSS 47 $; rsync thegerhards.com:/tank/tmp/diskimage.fat.bz2 diskimage.fat.bz2            
: enoexec.eu FSS 48 $; digest -a md5 diskimage.fat.bz2                        
191f26762d5b48e0010a575b54746e80
: enoexec.eu FSS 49 $; 

Since my home directory is now on ZFS and I snapshot every time my card gets inserted into the Sun Ray I can now take a look at what went wrong. Using my zfs_versions script I can get a list of the different versions of the file from all the snapshots:


: enoexec.eu FSS 56 $; digest -a md5 $( zfs_versions diskimage.fat.bz2 | nawk '{ print $NF }')
(/home/cg13442/.zfs/snapshot/user_snap_2008-06-25-05:51:57/diskimage.fat.bz2) = 0a193e0e80dbf83beabca12de09702a0
(/home/cg13442/.zfs/snapshot/user_snap_2008-06-25-05:54:44/diskimage.fat.bz2) = 7aa78dba6a7556fe10115aa5fc345bad
(/home/cg13442/.zfs/snapshot/user_snap_2008-06-25-07:05:34/diskimage.fat.bz2) = c6a77429920f258dfca1dbbd5018a69c
(/home/cg13442/.zfs/snapshot/user_snap_2008-06-25-09:06:39/diskimage.fat.bz2) = 674f69eec065da2b4d3da4bf45c7ae5f
(/home/cg13442/.zfs/snapshot/user_snap_2008-06-25-09:38:22/diskimage.fat.bz2) = 191f26762d5b48e0010a575b54746e80
: enoexec.eu FSS 57 $;


So the last two files in the list represent the corrupted file and the good file:

: enoexec.eu FSS 57 $; cmp -l /home/cg13442/.zfs/snapshot/user_snap_2008-06-2>
cmp -l /home/cg13442/.zfs/snapshot/user_snap_2008-06-25-09:06:39/diskimage.fat.bz2 /home/cg13442/.zfs/snapshot/user_snap_2008-06-25-09:38:22/diskimage.fat.bz2 | head -10                 
84262913   0 360
84262914   0  14
84262915   0 237
84262916   0  25
84262917   0 342
84262918   0 304
84262919   0  41
84262920   0  12
84262921   0 372
84262922   0  20
: enoexec.eu FSS 58 $;

and there appear to be blocks of zeros.

: enoexec.eu FSS 58 $; cmp -l /home/cg13442/.zfs/snapshot/user_snap_2008-06-2>
cmp -l /home/cg13442/.zfs/snapshot/user_snap_2008-06-25-09:06:39/diskimage.fat.bz2 /home/cg13442/.zfs/snapshot/user_snap_2008-06-25-09:38:22/diskimage.fat.bz2 | nawk '$2 != 0 { print $0 } $2 == 0 { count++ } END { printf("%x\\n", count ) }'
23d8c
: enoexec.eu FSS 58 $;

or at least 0x23d8c bytes were zero that should not have been. Need to see if I can reproduce this.

Anyway the moral is always check the md5 digest and if it is wrong use rsync to correct it.

Tuesday Mar 11, 2008

zone copy, aka zcp.

After messing around with zones for a few minutes it became clear that it would be really useful if there was a zcp command that worked just like scp(1) but used zlogin as the transport rather than using ssh. For those cases when you are root and don't want to mess with ssh authorizations since you know you can zlogin without a password anyway.

Specifically I wanted to be able to do:

# zcp  /etc/resolv.conf bookable-129-156-208-37.uk:/etc

Well it turns out that this is really easy to do. The trick is to let scp(1) do the heavy lifting for you and use zlogin(1) act as your transport. So I knocked together this script. You need to install it on your path called “zcp” and then make a hard link in the same directory called “zsh”. For example:

# /usr/sfw/bin/wget --quiet http://blogs.sun.com/chrisg/resource/zcp.sh
# cp zcp.sh /usr/local/bin/zcp 
# ln /usr/local/bin/zcp /usr/local/bin/zsh
# chmod 755  /usr/local/bin/zsh

Now the glorious simplicity of zcp, I'll even trhow in recursvice copy for free:

# zcp -r /etc/inet bookable-129-156-208-37.uk:/tmp
ipqosconf.1.sample   100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  2503       00:00    
config.sample        100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  3204       00:00    
wanboot.conf.sample  100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  3312       00:00    
hosts                100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   286       00:00    
ipnodes              100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   286       00:00    
netmasks             100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   384       00:00    
networks             100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   372       00:00    
inetd.conf           100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  1519       00:00    
sock2path            100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   566       00:00    
protocols            100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  1901       00:00    
services             100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  4201       00:00    
mipagent.conf-sample 100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  6274       00:00    
mipagent.conf.fa-sam 100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  6232       00:00    
mipagent.conf.ha-sam 100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  5378       00:00    
ntp.client           100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   291       00:02    
ntp.server           100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  2809       00:00    
slp.conf.example     100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  5750       00:00    
ntp.conf             100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   155       00:00    
ntp.keys             100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   253       00:00    
inetd.conf.orig      100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  6961       00:00    
ntp.drift            100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|     6       00:00    
ipsecalgs            100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   920       00:00    
ike.preshared        100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   308       00:00    
ipseckeys.sample     100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   510       00:00    
datemsk.ndpd         100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|    22       00:00    
ipsecinit.sample     100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  2380       00:00    
ipaddrsel.conf       100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   545       00:00    
inetd.conf.preupgrad 100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  6563       00:00    
hosts.premerge       100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   112       00:00    
ipnodes.premerge     100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|    61       00:00    
hosts.postmerge      100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|   286       00:00    
ipqosconf.2.sample   100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  3115       00:00    
ipqosconf.3.sample   100% |\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|  1097       00:00    
# 

I'll file and RFE for this to go into Solaris and update this entry when I have the number.

Update: The Bug ID is 6673792. The script now also supports zsync and zdist although niether of those have been tested yet.

About

This is the old blog of Chris Gerhard. It has mostly moved to http://chrisgerhard.wordpress.com

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today