Wednesday Jan 30, 2008

The ZFS Device Shuffle

I shot this video/screencast demo thing nine or ten months ago. The audio from the datacenter was terrible. I have been sitting on it and some voice overs that I wasn't pleased with for about 4 months. Today I decided that it was time finally get it published, did another voice over and while it isn't perfect its good enough.

This video was inspired by CSI: Munich I watched it and thought, wait I have an x4500!

Link to ZFS Device Shuffle on Mediacast

Here instead we will try an object embedded quicktime found at Bye Bye Embed, the video above is being served from the all new and improved

Edit: I have uploaded an amplified version, it appears that unless you are using good isolation earbuds or have relatively good quality speakers you can't really hear the audio portion. The trial copy of "Sound Studio" I downloaded did a wonderful job of letting me amplify the audio. The audio and video are the same, only the link has changed (It does not appear possible to replace the original content )

While I was at it we just released a new version of (I could never get my login to the old one working). For some reason smugmug is taking 23MB of video and reprocessing it to 94MB. I can download that in about 3 minutes but the size is ridiculous. (No wonder my wife said it took forever to play)

ZFS Device Shuffle Video
(This is the smugmug video, I still like the interface it just isn't quite doing it for me otherwise)

 Edit: A bit of clarity I should go to bed at 02:30 instead of messing with my blog

Edit: SmugMug fixed the encoding issue a long time ago

Tuesday Dec 11, 2007

Cat Proof External Storage

What happens when you disconnect the drive on the fly?
(e.g. set up a tripping hazard for your cat)

You get this message
The device you removed was not properly put away.Data might have been lost or damaged. Before youunplug your device, you must first select its icon inthe Finder and choose Eject from the File menu.
What happens when you do the same thing to a ZFS pool? This is no different
than what you get on Solaris, except it is on my Mac Book Pro.
  pool: p1
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
    the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
 scrub: scrub completed with 0 errors on Sat Dec  8 01:44:00 2007

    p1           DEGRADED     0     0     0
      mirror     DEGRADED     0     0     0
        disk7s1  UNAVAIL      0   114     0  cannot open
        disk0s3  ONLINE       0     0     0

Fortunately for me, my system keeps merrily chugging along.

Unfortunately (sort of) and this is much more of an issue on my Mac than on
any other system I am adding and removing devices much more frequently.

When I last connected the external disk and imported the pool I had 6
more "disks" visible to the system (really a usb thumb drive) another FW
device and a few SW RAID devices. This caused my nice portable external
FW disk to appear as disk7.
  diskutil list             
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        \*149.1 Gi   disk0
   1:                        EFI                         200.0 Mi   disk0s1
   2:                  Apple_HFS noroute                 88.9 Gi    disk0s2
   3:                  Apple_HFS                         57.0 Gi    disk0s3
   4:                  Apple_HFS noroute_2_1_2           2.6 Gi     disk0s4
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        \*55.9 Gi    disk1
   1:                  Apple_HFS                         55.9 Gi    disk1s1


The difference between the zpool status and diskutil list output
is the device configured for the pool is disk7s1 not the currently visible

I don't know how to recover from this situation live. If the device has not
renumbered you just zpool online p1 disk7s1 to reconnect on a different
device number I export and import the pool again.

To state that another way, I don't know how to effectively do:
zpool online p1 <old disk location> <new disk location>
similar to
zpool replace p1 <old disk location> <new disk location>
(the replace, even with -f, failed telling me that <new disk> was busy)

I have just requested a pre-release tarball that is reported to fix some bugs
in the current beta. We will see how it goes.

Other Observations:

Once you load the RW kext, it appears that anyone can do anything to the ZFS pools
and filesystems. On my single user system this is OK. I am the only user and guest
accounts are disabled. e.g. Initiate a scrub as a non-admin guest account or create a
snpashot as nobody (sudo -u nobody zfs snapshot p1/Music@nobody)

You CANNOT download music from the iTunes store in the current beta. you get a
permission error. I can however add music from local files.


Edit: Cleaned up the block quotes a bit, still not really rendering as one might hope.

Sunday Dec 09, 2007

iTunes, ZFS and Time Machine

I have been blogging about ZFS on OS X recently
RW ZFS and Leopard first thoughts
How to get experimental rw ZFS support AFTER upgrading to 10.5.1

As a way to start using ZFS on OS X on a daily basis I have migrated the contents of
my Music folder to a ZFS mirrored pool composed of an external disk and a matching
internal partition.

After a slightly rocky start including reinstalling my laptop so I could do the repartitioning
I wanted and a fairly high number of initial crashes things currently seems to be going well.

One of the things that I have noticed is that my iTunes Music folder and Library are polluted.
Now embedded in my library with the same name as a number of songs I have '.htm' files.
(./path/to/foo.mp3 also has a matching ./path/to/foo.htm in a large number of cases)
The files are part of some O'Reilly books I have on CD, as such I know "where" they came from, sort of.
(My library from my pre-experimentation CCC backup also has the same corruption)

A restore from my Time Machine backup is in order to get things back to a known good state. Or a recovery
from my slightly older CCC backup which is also not corrupted.

I am not particularly impressed with the Time Machine interface. It is novel and seems relatively functional
for finding the most recent backup by using the timeline on the right.

[Time Passes]

The Time Machine restore failed after ~8GB on a couldn't read/write a file error aborting the whole restore.
I do have to admit that the backups are working I can browse the backup structure. The use of hard links and
metadata is cool. Restoring via rsync does the trick, possibly faster as well since it doesn't copy unchanged files.

Even better I don't see a log, I am feeling somewhat less than impressed with the feedback from and utility
of Time Machine from a administrator's point of view. Am I asking for too much? A log of the restore with an
indication of the failure?

I have a feeling that the restore is really just a copy in Finder. An automator type action passing the
selection to Finder and executing a copy.

Again what finally worked for me was a combination of methods. I used the Time Machine interface to find the
backup I wanted and CMD-I to see the path to the directory (although would have been just as good).
Once I found the backup I wanted rsync worked quite well restoring 48GB of Music in 12,273 files.
(rsync --archive -P /Volumes/TimeMachine/.../Music/iTunes ~sferry/Music/iTunes)

Thursday Dec 06, 2007

RW ZFS and Leopard, my first impressions

Not long ago I posted about installing the RW ZFS beta seed on Leopard after updating to 10.5.1.

As I said after the end of that post, It does work.  On the one hand I haven't had this many system
panics on a mac ever. On the other hand I have repartitioned my laptop and now have ZFS pools
mirrored to an external USB stick and a firewire drive.

Since the simple tests below I have re-installed to get more partitioning flexibility. I couldn't do a live
repartition to get the space I wanted.  Every attempt to repartition after the first was informing me that
there wasn't enough space.

first hint: I recommend turning off spotlight indexing on the ZFS volume it seems SIGNIFICANTLY more stable.

[Read More]



« February 2017
Sun Managed Operations