Saturday Jul 09, 2011

VirtualBox 4.1 Beta 2 is now available for testing

The second beta release of VirtualBox 4.1 is now available for download and testing. You can find binaries for your host platform at http://download.virtualbox.org/virtualbox/4.1.0_BETA2. This release contains several new features which may not be quite complete. Please do not use this on any production systems - it is only for early evaluation and testing.

VirtualBox 4.1 will be a new major release and includes the following new features.

  • Support for cloning virtual machines
  • Enhanced wizard for creating new virtual disks
  • A new wizard for copying virtual disks
  • Increased the guest memory limit for 64-bit hosts to 1TB
  • Experimental WDDM graphics driver for Windows guests
  • Modules and features in the guest additions now are represented as facilities to have a common interface for frontends.
  • A new networking mode (UDP tunnel) which will allow easy and transparent communication between guests running on different hosts
In addition to the new features, many bugs in the existing release have been fixed. Please see the VirtualBox 4.1 Beta 1 Announcement for a complete list.

Some notable fixes since Beta 1 include

  • VMM: fixed a Beta 1 regression which prevented certain guests from booting
  • VMM: better out-of-memory error reporting under certain circumstances
  • GUI: display CPU cap in the VM details page
  • Storage: introduced non-rotational flag for hard disks for optimizing the performance with modern guests (e.g. Windows 7 will not automatically defrag the disk)
  • Floppy: make unmounting a host floppy disk work
  • Solaris hosts: Host-only/Bridged networking fixes
  • Solaris guests: fixed automounting of shared folders
  • Linux installer: fixed compilation of the vboxpci module if DKMS is not installed
  • Windows Installer: fixed automated VBox Python API installation.
  • Experimental support for PCI passthrough (Linux hosts only)

    Please report any problems with the beta release at the VirtualBox Beta Forum. The development team is particularly interested in problems or regressions since the last release (4.0.10).



    Technocrati Tags: <script type="text/javascript"> var sc_project=1193495; var sc_invisible=1; var sc_security="a46f6831"; </script> <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>

    vBulletin analytics
  • Wednesday Jul 06, 2011

    VirtualBox 4.1 Beta 1 is now available for testing

    The first beta release of VirtualBox 4.1 is now available for download and testing. You can find binaries for your host platform at http://download.virtualbox.org/virtualbox/4.1.0_BETA1. This release contains several new features which may not be quite complete. Please do not use this on any production systems - it is only for early evaluation and testing.

    VirtualBox 4.1 will be a new major release and includes the following new features.

    • Support for cloning virtual machines
    • Enhanced wizard for creating new virtual disks
    • A new wizard for copying virtual disks
    • Increased the guest memory limit for 64-bit hosts to 1TB
    • Experimental WDDM graphics driver for Windows guests
    • Modules and features in the guest additions now are represented as facilities to have a common interface for frontends.
    • A new networking mode (UDP tunnel) which will allow easy and transparent communication between guests running on different hosts
    In addition to the new features, many bugs in the existing release have been fixed. Please see the VirtualBox 4.1 Beta 1 Announcement for a complete list.

    Please report any problems with the beta release at the VirtualBox Beta Forum. The development team is particularly interested in problems or regressions since the last release (4.0.10).

    A big thanks to Frank Mehnert for the heads up on this new Beta release.

    Technocrati Tags: <script type="text/javascript"> var sc_project=1193495; var sc_invisible=1; var sc_security="a46f6831"; </script> <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>

    vBulletin analytics

    Tuesday Jun 28, 2011

    VirtualBox 4.0.10 is now available for download



    VirtualBox 4.0.10 has been released and is now available for download. You can get binaries for Windows, OS X (Intel Mac), Linux and Solaris hosts at
    http://www.virtualbox.org/wiki/Downloads

    The full changelog can be found here. The high points for the 4.0.10 maintenance release include ....

  • GUI: fixed disappearing settings widgets on KDE hosts (bug #6809)
  • Storage: fixed hang under rare circumstances with flat VMDK images
  • Storage: a saved VM could not be restored under certain circumstances after the host kernel was updated
  • Storage: refuse to create a medium with an invalid variant
  • Snapshots: none of the hard disk attachments must be attached to another VM in normal mode when creating a snapshot
  • USB: fixed occasional VM hangs with SMP guests
  • USB: proper device detection on RHEL/OEL/CentOS 5 guests
  • ACPI: force the ACPI timer to return monotonic values for improve behavior with SMP Linux guests
  • RDP: fixed screen corruption under rare circumstances
  • rdesktop-vrdp: updated to version 1.7.0
  • OVF: under rare circumstances some data at the end of a VMDK file was not written during export
  • Mac OS X hosts: Lion fixes
  • Mac OS X hosts: GNOME 3 fix
  • Linux hosts: fixed VT-x detection on Linux 3.0 hosts
  • Linux hosts: fixed Python 2.7 bindings in the universal Linux binaries
  • Windows hosts: fixed leak of thread and process handles
  • Windows Additions: fixed bug when determining the extended version of the Guest Additions
  • Solaris Additions: fixed installation to 64-bit Solaris 10u9 guests
  • Linux Additions: RHEL6.1/OL6.1 compile fix
  • Linux Additions: fixed a memory leak during VBoxManage guestcontrol execute

    Technocrati Tags: <script type="text/javascript"> var sc_project=1193495; var sc_invisible=1; var sc_security="a46f6831"; </script> <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>
  • Monday Feb 15, 2010

    VirtualBox 3.1.4 has been released



    VirtualBox 3.1.4 has been released and is now available for download. You can get binaries for Windows, OS X (Intel Mac), Linux and Solaris hosts at http://www.virtualbox.org/wiki/Downloads

    Version 3.1.4 is a maintenance release and contains the following updates and fixes.
    • SMP stability fixes
    • activate NXE for PAE enabled guests (VT-x and AMD-V on 32 bits hosts only)
    • turn on nested paging by default for new VMs (if available; VT-x and AMD-V only)
    • turn on VPID by default for new VMs (if available; VT-x only)
    • perform strict CPUID compatibility checks when teleporting; to get the old behavior set "VBoxInternal/CPUM/StrictCpuIdChecks" to 0
    • VBoxManage: add restorecurrent operation to snapshots command
    • fixed status LEDs for storage devices
    • workaround for disabling the seamless mode on KDE hosts (KWin bug)
    • disable seamless mode when guest changes screen resolution
    • really fixed host DVD loading "passthrough" setting from config file
    • Solaris hosts: several USB fixes (including support for Apple iPod)
    • Solaris installer: fixed USB module removal and Solaris 10 "id" binary incompatibility
    • XFree86 Additions: accept video mode hints for the initial mode again

    You can also see a complete list of bugs fixed in this new release.


    Technocrati Tags:

    Sunday Nov 22, 2009

    Taking ZFS deduplication for a test drive

    Now that I have a working OpenSolaris build 128 system, I just had to take ZFS deduplication for a spin, to see if it was worth all of the hype.

    Here is my test case: I have 2 directories of photos, totaling about 90MB each. And here's the trick - they are almost complete duplicates of each other. I downloaded all of the photos from the same camera on 2 different days. How many of you do that ? Yeah, me too.

    Let's see what ZFS can figure out about all of this. If it is super smart we should end up with a total of 90MB of used space. That's what I'm hoping for.

    The first step is to create the pool and turn on deduplication from the beginning.
    # zpool create -f scooby -O dedup=on c2t2d0s2
    
    This will use sha256 for determining if 2 blocks are the same. Since sha256 has such a low collision probability (something like 1x10\^-77), we will not turn on automatic verification. If we were using an algorithm like fletcher4 which has a higher collision rate we should also perform a complete block compare before allowing the block removal (dedup=fletcher4,verify)

    Now copy the first 180MB (remember, this is 2 sets of 90MB which are nearly identical sets of photos).
    # zfs create scooby/doo
    # cp -r /pix/Alaska\* /scooby/doo
    
    And the second set.
    # zfs create scooby/snack
    # cp -r /pix/Alaska\* /scooby/snack
    
    And finally the third set.
    # zfs create scooby/dooby
    # cp -r /pix/Alaska\* /scooby/dooby
    
    Let's make sure there are in fact three copies of the photos.
    # df -k | grep scooby
    scooby               74230572      25 73706399     1%    /scooby
    scooby/doo           74230572  174626 73706399     1%    /scooby/doo
    scooby/snack         74230572  174626 73706399     1%    /scooby/snack
    scooby/dooby         74230572  174625 73706399     1%    /scooby/dooby
    


    OK, so far so good. But I can't quite tell if the deduplication is actually doing anything. With all that free space, it's sort of hard to see. Let's look at the pool properties.
    # zpool get all scooby
    NAME    PROPERTY       VALUE       SOURCE
    scooby  size           71.5G       -
    scooby  capacity       0%          -
    scooby  altroot        -           default
    scooby  health         ONLINE      -
    scooby  guid           5341682982744598523  default
    scooby  version        22          default
    scooby  bootfs         -           default
    scooby  delegation     on          default
    scooby  autoreplace    off         default
    scooby  cachefile      -           default
    scooby  failmode       wait        default
    scooby  listsnapshots  off         default
    scooby  autoexpand     off         default
    scooby  dedupratio     5.98x       -
    scooby  free           71.4G       -
    scooby  allocated      86.8M       -
    
    Now this is telling us something.

    First notice the allocated space. Just shy of 90MB. But there's 522MB of data (174MB x 3). But only 87MB used out of the pool. That's a good start.

    Now take a look at the dedupratio. Almost 6. And that's exactly what we would expect, if ZFS is as good as we are lead to believe. 3 sets of 2 duplicate directories is 6 total copies of the same set of photos. And ZFS caught every one of them.

    So if you want to do this yourself, point your OpenSolaris package manager at the dev repository and wait for build 128 packages to show up. If you need instructions on using the OpenSolaris dev repository, point the browser of your choice at http://pkg.opensolaris.org/dev/en/index.shtml. And if you can't wait for the packages to show up, you can always .

    Technocrati Tags:
    <script type="text/javascript"> var sc_project=1193495; var sc_invisible=1; var sc_security="a46f6831"; </script> <script type="text/javascript" src="http://www.statcounter.com/counter/counter.js"></script>

    Friday Oct 09, 2009

    What's New in Solaris 10 10/09


    Solaris 10 10/09 (u8) is now available for download at http://sun.com/solaris/get.jsp. DVD ISO images (full and segments that can be reassembled after download) are available for both SPARC and x86.

    Here are a few of the new features in this release that caught my attention.

    Packaging and Patching

    Improved performance of SVR4 package commands: Improvements have been made in the SVR4 package commands (pkgadd, pkgrm, pkginfo et al). The impact of these can be seen in drastically reduced zone installation time. How much of an improvement you ask (and you know I have to answer with some data, right) ?
    # cat /etc/release; uname -a
    
                            Solaris 10 5/09 s10x_u7wos_08 X86
               Copyright 2009 Sun Microsystems, Inc.  All Rights Reserved.
                            Use is subject to license terms.
                                 Assembled 30 March 2009
    SunOS chapterhouse 5.10 Generic_141415-09 i86pc i386 i86pc
    
    # time zoneadm -z zone1 install
    Preparing to install zone .
    Creating list of files to copy from the global zone.
    Copying <2905> files to the zone.
    Initializing zone product registry.
    Determining zone package initialization order.
    Preparing to initialize <1453> packages on the zone.
    Initialized <1453> packages on zone.
    Zone  is initialized.
    Installation of these packages generated errors: 
    The file  contains a log of the zone installation.
    
    real    5m48.476s
    user    0m45.538s
    sys     2m9.222s
    
    #  cat /etc/release; uname -a
    
                           Solaris 10 10/09 s10x_u8wos_08a X86
               Copyright 2009 Sun Microsystems, Inc.  All Rights Reserved.
                            Use is subject to license terms.
                               Assembled 16 September 2009
    SunOS corrin 5.10 Generic_141445-09 i86pc i386 i86pc
    
    # time zoneadm -z zone1 install
    Preparing to install zone .
    Creating list of files to copy from the global zone.
    Copying <2915> files to the zone.
    Initializing zone product registry.
    Determining zone package initialization order.
    Preparing to initialize <1432> packages on the zone.
    Initialized <1432> packages on zone.
    Zone  is initialized.
    Installation of these packages generated errors: 
    The file  contains a log of the zone installation.
    
    real    3m4.677s
    user    0m44.593s
    sys     0m48.003s
    
    OK, that's pretty impressive. A zone installation on Solaris 10 10/09 takes about half of the time as it does on Solaris 10 5/09. It is also worth noting the rather large reduction in the amount of system time (48 seconds vs 129 seconds) too.

    Zones parallel patching: Before Solaris 10 10/09 the patching process was single threaded which could lead to prolonged patching time on a system with several nonglobal zones. Starting with this update you can specify the number of threads to be used to patch a system with zones. Enable this feature by assigning a value to num_proc in /etc/patch/pdo.conf. The maximum value is capped at 1.5 times the number of on-line CPUs, but can be limited by a lower value of num_proc.

    This feature is also available by applying Solaris patches 119254-66 (SPARC) or 119255-66 (x86).

    For more information on the effects of zone parallel patching, see Container Guru Jeff Victor's excellent Patching Zones Goes Zoom.

    ZFS Enhancements

    Flash archive install into a ZFS root filesystem: ZFS support for the root file system was introduced in Solaris 10 10/08 but the install tools did not work with flash archives. Solaris 10 10/09 provides the ability to install a flash archive created from an existing ZFS root system. This capability is also provided by patches 119534-15 + 124630-26 (SPARC) or 119535-15 + 124631-27 (x86) that can be applied to a Solaris 10 10/08 or later system. There are still a few limitations such as the the flash source must be from a ZFS root system and you cannot use differential archives. More information can be found in Installing a ZFS Root File System (Flash Archive Installation).

    Set ZFS properties on the initial zpool file system: Prior to Solaris 10 10/09, ZFS file system properties could only be set once the initial file system was created. This would make it impossible to create a pool with same name as an existing mounted file system or to be able to have replication or compression from the time the pool is created. In Solaris 10 10/09 you can specify any ZFS file system property using zpool -O.
     zpool create -O mountpoint=/data,copies=3,compression=on datapool c1t1d0 c1t2d0
    
    ZFS Read Cache (L2ARC): You now have the ability to add persistent read ahead caches to a ZFS zpool. This can improve the read performance of ZFS as well as reducing the ZFS memory footprint.

    L2ARC devices are added as cache vdevs to a pool. In the following example we will create a pool of 2 mirrored devices, 2 cache devices and a spare.
     
    # zpool create datapool mirror c1t1d0 c1t2d0 cache c1t3d0 c1t4d0 spare c1t5d0
    
    # zpool status datapool
      pool: datapool
     state: ONLINE
     scrub: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            datapool    ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                c1t1d0  ONLINE       0     0     0
                c1t2d0  ONLINE       0     0     0
            cache
              c1t3d0    ONLINE       0     0     0
              c1t4d0    ONLINE       0     0     0
            spares
              c1t5d0    AVAIL
    
    errors: No known data errors
    
    So what do ZFS cache devices do ? Rather than go into a lengthy explanation of the L2ARC, I would rather refer you to Fishworks developer Brendan Gregg's excellent treatment of the subject.

    Unlike the intent log (ZIL), L2ARC cache devices can be added and removed dynamically.
    # zpool remove datapool c1t3d0
    # zpool remove datapool c1t4d0
    
    # zpool status datapool
      pool: datapool
     state: ONLINE
     scrub: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            datapool    ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                c1t1d0  ONLINE       0     0     0
                c1t2d0  ONLINE       0     0     0
            spares
              c1t5d0    AVAIL
    
    errors: No known data errors
    
    
    # zpool add datapool cache c1t3d0
    
    # zpool status datapool
      pool: datapool
     state: ONLINE
     scrub: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            datapool    ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                c1t1d0  ONLINE       0     0     0
                c1t2d0  ONLINE       0     0     0
            cache
              c1t3d0    ONLINE       0     0     0
            spares
              c1t5d0    AVAIL
    
    errors: No known data errors
    
    New cache control properties: Two new ZFS properties are introduced with Solaris 10 10/09. These control what what is stored (nothing, data + metadata, or metadata only) in the ARC (memory) and L2ARC (external) caches. These new properties are
    • primarycache - controls what is stored in the memory resident ARC cache
    • secondarycache - controls what is stored in the L2ARC
    and they can take the values
    • none - the caches are not used
    • metadata - only file system metadata is cached
    • all - both file system data and the metadata is stored in the associated cache
    # zpool create -O primarycache=metadata -O secondarycache=all datapool c1t1d0 c1t2d0 cache c1t3d0 
    
    There are workloads such as databases that perform better or make more efficient use of memory if the system is not competing with the caches that the applications are maintaining themselves.

    User and group quotas:ZFS has always had quotas and reservations but they were applied at the file system level. To achieve user or group quotas would require creating additional file systems which might make administration more complex. Starting with Solaris 10 10/09 you can apply both user and group quotas to a file system much like you would with UFS. The ZFS file system must be at version 15 or later and the zpool must be at version 4 or later.

    Let's create a file system and see if we are at the proper versions to set quotas.
    # zfs create rpool/newdata
    # chown bobn:local /rpool/newdata
    
    # zpool get version rpool
    NAME   PROPERTY  VALUE    SOURCE
    rpool  version   18       default
    
    
    # zpool upgrade -v
    This system is currently running ZFS pool version 18.
    
    The following versions are supported:
    
    VER  DESCRIPTION
    ---  --------------------------------------------------------
     1   Initial ZFS version
     2   Ditto blocks (replicated metadata)
     3   Hot spares and double parity RAID-Z
     4   zpool history
     5   Compression using the gzip algorithm
     6   bootfs pool property
     7   Separate intent log devices
     8   Delegated administration
     9   refquota and refreservation properties
     10  Cache devices
     11  Improved scrub performance
     12  Snapshot properties
     13  snapused property
     14  passthrough-x aclinherit
     15  user/group space accounting
     16  stmf property support
     17  Triple-parity RAID-Z
     18  snapshot user holds
    For more information on a particular version, including supported releases, see:
    
    http://www.opensolaris.org/os/community/zfs/version/N
    
    Where 'N' is the version number.
    
    
    # zfs get version rpool/newdata
    NAME           PROPERTY  VALUE    SOURCE
    rpool/newdata  version   4 
    
    # zfs upgrade -v
    The following filesystem versions are supported:
    
    VER  DESCRIPTION
    ---  --------------------------------------------------------
     1   Initial ZFS filesystem version
     2   Enhanced directory entries
     3   Case insensitive and File system unique identifier (FUID)
     4   userquota, groupquota properties
    
    For more information on a particular version, including supported releases, see:
    
    http://www.opensolaris.org/os/community/zfs/version/zpl/N
    
    Where 'N' is the version number.
    
    Excellent. Now let's set a user and group quota and see what happens. We'll set a group quota of 1GB and a user quota at 2GB.
    # zfs set groupquota@local=1g rpool/newdata
    # zfs set userquota@bobn=2g rpool/newdata
    
    # su - bobn
    
    % mkfile 500M /rpool/newdata/file1
    % mkfile 500M /rpool/newdata/file2
    % mkfile 500M /rpool/newdata/file3
    file3: initialized 40370176 of 524288000 bytes: Disc quota exceeded
    
    
    As expected, we have exceeded our group quota. Let's change the group of the existing files and see if we can proceed to our user quota.
    % rm /rpool/newdata/file3
    % chgrp sales /rpool/newdata/file1 /rpool/newdata/file2
    % mkfile 500m /rpool/newdata/file3
    Could not open /rpool/newdata/disk3: Disc quota exceeded
    
    
    Whoa! What's going on here ? Relax - ZFS does things asynchronously unless told otherwise. And we should have noticed this when the mkfile for file3 actually started. ZFS wasn't quite caught up with the current usage. A good sync should do the trick.
    % sync
    % mkfile 500M /rpool/newdata/file3
    % mkfile 500M /rpool/newdata/file4
    % mkfile 500M /rpool/newdata/file5
    /rpool/newdata/disk5: initialized 140247040 of 524288000 bytes: Disc quota exceeded
    
    
    Great. We now have user and group quotas. How can I find out what I have used against my quota ? There are two new ZFS properties, userused and groupused that will show what the group or user is currently consuming.
    % zfs get userquota@bobn,userused@bobn rpool/newdata
    NAME           PROPERTY        VALUE           SOURCE
    rpool/newdata  userquota@bobn  2G              local
    rpool/newdata  userused@bobn   1.95G           local
    
    % zfs get groupquota@local,groupused@local rpool/newdata
    NAME           PROPERTY          VALUE             SOURCE
    rpool/newdata  groupquota@local  1G                local
    rpool/newdata  groupused@local   1000M             local
    
    % zfs get groupquota@sales,groupused@sales rpool/newdata
    NAME           PROPERTY          VALUE             SOURCE
    rpool/newdata  groupquota@sales  none              local
    rpool/newdata  groupused@sales   1000M             local
    
    % zfs get groupquota@scooby,groupused@scooby rpool/newdata
    NAME           PROPERTY           VALUE              SOURCE
    rpool/newdata  groupquota@scooby  -                  -
    rpool/newdata  groupused@scooby   -   
    
    New space usage properties: Four new usage properties have been added to ZFS file systems.
    • usedbychildren (usedchild) - this is the amount of space that is used by all of the children of the specified dataset
    • usedbydataset (usedds) - this is the total amount of space that would be freed if this dataset and it's snapshots and reservations were destroyed
    • usedbyrefreservation (usedrefreserv) - this is the amount of space that would be freed if the dataset's reservations were to be removed
    • usertbysnapshots (usedsnap) - the total amount of space that would be freed if all of the snapshots of this dataset were deleted.
    # zfs get all datapool | grep used
    datapool  used                  5.39G                  -
    datapool  usedbysnapshots       19K                    -
    datapool  usedbydataset         26K                    -
    datapool  usedbychildren        5.39G                  -
    datapool  usedbyrefreservation  0                      -
    


    These new properties can also be viewed in a nice tabular form using zfs list -o space.
    # zfs list -r -o space datapool
    NAME           AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
    datapool        480M  5.39G       19K     26K              0      5.39G
    datapool@now       -    19K         -       -              -          -
    datapool/fs1    480M   400M         0    400M              0          0
    datapool/fs2   1.47G  1.00G         0   1.00G              0          0
    datapool/fs3    480M    21K         0     21K              0          0
    datapool/fs4   2.47G      0         0       0              0          0
    datapool/vol1  1.47G     1G         0     16K          1024M          0
    

    Miscellaneous

    Support for 2TB boot disks: Solaris 10 10/09 supports a disk Volume Table of Contents (VTOC) of up to 2TB in size. The previous maximum VTOC size was 1TB. On x86 systems you must be running Solaris with a 64bit kernel and have at least 1GB of memory to use a VTOC larger that 1TB.

    pcitool: A new command for Solaris that can assign interrupts to specific threads or display the current interrupt routing. This command is available for both SPARC and x86.

    New iSCSI initiator SMF service: svc:/network/iscsi/initiator:default is a new Service Management Facility (SMF) service to control discovery and enumeration of iSCSI devices early in the boot process. Other boot services that may require iSCSI services can add dependencies to insure that the devices are available before being needed.

    Device Drivers

    The following device drivers are either new to Solaris or have had some new features or chipsets added.
    • MPxIO support for the LSI 6180 Controller
    • LSI MPT 2.0 SAS 2.0 controllers (mpt_sas)
    • Broadcom NetXTreme II gigabit Ethernet (bcm5716c and bcm5716s) controllers
    • Interrupt remapping for Intel VT-x enabled processors
    • Support for SATA AHCI tape
    • Sun StorageTek 6Gb/s SAS RAID controller and LSI MegaRAID 92xx (mt_sas)
    • Intel 82598 and 82599 10Gb/s PCIe Ethernet controller

    Open Source Software Updates

    The following open source packages have been updated for Solaris 10 10/09.
    • NTP 4.2.5
    • PostgreSQL versions 8.1.17, 8.2.13 and 8.3.7
    • Samba 3.0.35

    For more information

    A complete list of new features and changes can be found in the Solaris 10 10/09 Release Notes and the What's New in Solaris 10 10/09 documentation at docs.sun.com.

    Technocrati Tags:

    Friday Mar 27, 2009

    VirtualBox 2.2 Beta 2 now available for testing


    Virtualbox 2.2 Beta 2 is now available for testing. In addition to the feature list from the Beta 1 announcement, Beta 2 contains the following fixes.

  • Raised the memory limit for VMs on 64-bit hosts to 16GB
  • Many fixes for OVF import/export
  • VMM: properly emulate RDMSR from the TSC MSR, should fix some NetBSD guests
  • IDE: fixed hard disk upgrade from XML-1.2 settings (bug #1518)
  • Hard disks: refuse to start the VM if a disk image is not writable
  • USB: Fixed BSOD on the host with certain USB devices (Windows hosts only; bug #1654)
  • E1000: properly handle cable disconnects (bug #3421)
  • X11 guests: prevented setting the locale in vboxmouse, as this caused problems with Turkish locales (bug #3563)
  • Linux additions: fixed typo when detecting Xorg 1.6 (bug #3555)
  • Windows guests: bind the VBoxMouse.sys filter driver to the correct guest pointing device (bug #1324)
  • Windows hosts: fixed BSOD when starting a VM with enabled host interface (bug #3414)
  • Linux hosts: do not leave zombies of VBoxSysInfo.sh (bug #3586)
  • VBoxManage: controlvm dvdattach did not work if the image was attached before
  • VBoxManage showvminfo: don't spam the release log if the additions don't support statistics information (bug #3457)
  • GUI: fail with an appropriate error message when trying to boot a read-only disk image (bug #1745)
  • LsiLogic: fixed problems with Solaris guests

    See the announcement in the VirtualBox Forum for more information. The beta can be downloaded from http://download.virtualbox.org/virtualbox/2.2.0_BETA2.

    As with Beta 1, this release is for testing and evaluation purposes, so please do not run this on a production system. I really don't need to remind you of this - you will get a nice graphical reminder every time you start the control program.

    Point your browser at the VirtualBox Beta Feedback Forum for more information about the beta program.

    Technocrati Tags:
  • Monday Mar 23, 2009

    VirtualBox 2.2 Beta 1 now available for testing


    Virtualbox 2.2 Beta 1 is now available for testing. It is a major release and contains many new features, including

  • OVF (Open Virtualization Format) appliance import and export
  • Host-only networking mode
  • Hypervisor optimizations with significant performance gains for high context switching rates
  • VT-x/AMD-V are enabled by default for newly created virtual machines
  • USB (OHCI & EHCI) is enabled by default for newly created virtual machines (Qt GUI only)
  • Experimental USB support for OpenSolaris hosts
  • Shared folders for Solaris and OpenSolaris guests
  • OpenGL 3d acceleration for Linux guests
  • Experimental support for OS X 10.6 (Snow Leopard) hosts running both the 64-bit and the 32-bit kernel

    as well as numerous bug fixes. See the announcement in the VirtualBox Forum for more information. The beta can be downloaded from http://download.virtualbox.org/virtualbox/2.2.0_BETA1.

    This release is for testing and evaluation purposes, so please do not run this on a production system. I really don't need to remind you of this - you will get a nice graphical reminder every time you start the control program.

    Since there are some configuration file changes in the new release, the beta program will also upgrade your configurations. Please back up your XML files before running it for the first time (or if you are like me, a zfs snapshot -r to your xvm datasets will do the trick).

    Point your browser at the VirtualBox Beta Feedback Forum for more information about the beta program.

    Technocrati Tags:
  • Monday Mar 02, 2009

    Alaska and Oregon Solaris Boot Camps

    A big thanks to all who attended the Solaris Boot Camps in Juneau, Fairbanks, Portland and Salem. I hope that you found the information useful. And thanks for all of the good questions and discussion.

    Here are the materials that were used during the bootcamp.

    Please send me email if you have any questions or want to follow up on any of the discussions.

    Thanks again for your attendance and continued support for Solaris.

    Technocrati Tags:

    Monday Feb 18, 2008

    ZFS and FMA - Two great tastes .....

    Our good friend Isaac Rozenfeld talks about the Multiplicity of Solaris. When talking about Solaris I will use the phrase "The Vastness of Solaris". If you have attended a Solaris Boot Camp or Tech Day in the last few years you get an idea of what we are talking about - when we go on about Solaris hour after hour after hour.

    But the key point in Isaac's multiplicity discussion is how the cornucopia of Solaris features work together to do some pretty spectacular (and competitively differentiating) things. In the past we've looked at combinations such as ZFS and Zones or Service Management, Role Based Access Control (RBAC) and Least Privilege. Based on a conversation last week in St. Louis, let's consider how ZFS and Solaris Fault Management (FMA) play together.

    Preparation

    Let's begin by creating some fake devices that we can play with. I don't have enough disks on this particular system, but I'm not going to let that slow me down. If you have sufficient real hot swappable disks, feel free to use them instead.
    # mkfile 1g /dev/disk1
    # mkfile 1g /dev/disk2
    # mkfile 512m /dev/disk3
    # mkfile 512m /dev/disk4
    # mkfile 1g /dev/disk5
    
    
    Now let's create a couple of zpools using the fake devices. pool1 will be a 1GB mirrored pool using disk1 and disk2. pool2 will be a 512MB mirrored pool using disk3 and disk4. Device spare1 will spare both pools in case of a problem - which we are about to inflict upon the pools.
    # zpool create pool1 mirror disk1 disk2 spare spare1
    # zpool create pool2 mirror disk3 disk4 spare spare1
    # zpool status
      pool: pool1
     state: ONLINE
     scrub: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            pool1       ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                disk1   ONLINE       0     0     0
                disk2   ONLINE       0     0     0
            spares
              spare1    AVAIL   
    
    errors: No known data errors
    
      pool: pool2
     state: ONLINE
     scrub: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            pool2       ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                disk3   ONLINE       0     0     0
                disk4   ONLINE       0     0     0
            spares
              spare1    AVAIL   
    
    errors: No known data errors
    
    
    So far so good. If we were to run a scrub on either pool, it will complete immediately. Remember that unlike hardware RAID disk replacement, ZFS scrubbing and resilvering only touches blocks that contain actual data. Since there is no data in these pools (yet), there is little for the scrubbing process to do.
    # zpool scrub pool1
    # zpool scrub pool2
    # zpool status
      pool: pool1
     state: ONLINE
     scrub: scrub completed with 0 errors on Mon Feb 18 09:24:16 2008
    config:
    
            NAME        STATE     READ WRITE CKSUM
            pool1       ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                disk1   ONLINE       0     0     0
                disk2   ONLINE       0     0     0
            spares
              spare1    AVAIL   
    
    errors: No known data errors
    
      pool: pool2
     state: ONLINE
     scrub: scrub completed with 0 errors on Mon Feb 18 09:24:17 2008
    config:
    
            NAME        STATE     READ WRITE CKSUM
            pool2       ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                disk3   ONLINE       0     0     0
                disk4   ONLINE       0     0     0
            spares
              spare1    AVAIL   
    
    errors: No known data errors
    
    
    Let's populate both pools with some data. I happen to have a directory of scenic images that I use as screen backgrounds - that will work nicely.
    
    # cd /export/pub/pix>
    # find scenic -print | cpio -pdum /pool1
    # find scenic -print | cpio -pdum /pool2
    
    # df -k | grep pool
    pool1                1007616  248925  758539    25%    /pool1
    pool2                 483328  248921  234204    52%    /pool2
    
    
    And yes, cp -r would have been just as good.

    Problem 1: Simple data corruption

    Time to inflict some harm upon the pool. First, some simple corruption. Writing some zeros over half of the mirror should do quite nicely.
    # dd if=/dev/zero of=/dev/dsk/disk1 bs=8192 count=10000 conv=notrunc
    10000+0 records in
    10000+0 records out 
    
    
    At this point we are unaware that anything has happened to our data. So let's try accessing some of the data to see if we can observe ZFS self healing in action. If your system has plenty of memory and is relatively idle, accessing the data may not be sufficient. If you still end up with no errors after the cpio, try a zpool scrub - that will catch all errors in the data.
    # cd /pool1
    # find . -print | cpio -ov > /dev/null
    416027 blocks
    
    
    Let's ask our friend fmstat(1m) if anything is wrong ?
    # fmstat
    module             ev_recv ev_acpt wait  svc_t  %w  %b  open solve  memsz  bufsz
    cpumem-retire            0       0  0.0    0.1   0   0     0     0      0      0
    disk-transport           0       0  0.0  366.5   0   0     0     0    32b      0
    eft                      0       0  0.0    2.6   0   0     0     0   1.4M      0
    fmd-self-diagnosis       1       0  0.0    0.2   0   0     0     0      0      0
    io-retire                0       0  0.0    1.1   0   0     0     0      0      0
    snmp-trapgen             1       0  0.0   16.0   0   0     0     0    32b      0
    sysevent-transport       0       0  0.0  620.3   0   0     0     0      0      0
    syslog-msgs              1       0  0.0    9.7   0   0     0     0      0      0
    zfs-diagnosis          162     162  0.0    1.5   0   0     1     0   168b   140b
    zfs-retire               1       1  0.0  112.3   0   0     0     0      0      0
    
    
    As the guys in the Guinness commercial say, "Brilliant!" The important thing to note here is that the zfs-diagnosis engine has run several times indicating that there is a problem somewhere in one of my pools. I'm also running this on Nevada so the zfs-retire engine has also run, kicking in a hot spare due to excessive errors.

    So which pool is having the problems ? We continue our FMA investigation to find out.
    # fmadm faulty
    --------------- ------------------------------------  -------------- ---------
    TIME            EVENT-ID                              MSG-ID         SEVERITY
    --------------- ------------------------------------  -------------- ---------
    Feb 18 09:56:24 d82d1716-c920-6243-e899-b7ddd386902e  ZFS-8000-GH    Major    
    
    Fault class : fault.fs.zfs.vdev.checksum
    
    Description : The number of checksum errors associated with a ZFS device
                  exceeded acceptable levels.  Refer to
                  http://sun.com/msg/ZFS-8000-GH for more information.
    
    Response    : The device has been marked as degraded.  An attempt
                  will be made to activate a hot spare if available.
    
    Impact      : Fault tolerance of the pool may be compromised.
    
    Action      : Run 'zpool status -x' and replace the bad device.
    
    
    # zpool status -x
      pool: pool1
     state: DEGRADED
    status: One or more devices has experienced an unrecoverable error.  An
            attempt was made to correct the error.  Applications are unaffected.
    action: Determine if the device needs to be replaced, and clear the errors
            using 'zpool clear' or replace the device with 'zpool replace'.
       see: http://www.sun.com/msg/ZFS-8000-9P
     scrub: resilver in progress, 44.83% done, 0h0m to go
    config:
    
            NAME          STATE     READ WRITE CKSUM
            pool1         DEGRADED     0     0     0
              mirror      DEGRADED     0     0     0
                spare     DEGRADED     0     0     0
                  disk1   DEGRADED     0     0   162  too many errors
                  spare1  ONLINE       0     0     0
                disk2     ONLINE       0     0     0
            spares
              spare1      INUSE     currently in use
    
    errors: No known data errors
    
    
    This tells us all that we need to know. The device disk1 was found to have quite a few checksum errors - so many in fact that it was replaced automatically by a hot spare. The spare was resilvering and a full complement of data replicas would be available soon. The entire process was automatic and completely observable.

    Since we inflicted harm upon the (fake) disk device ourself, we know that it is in fact quite healthy. So we can restore our pool to its original configuration rather simply - by detaching the spare and clearing the error. We should also clear the FMA counters and repair the ZFS vdev so that we can tell if anything else is misbehaving in either this or another pool.
    # zpool detach pool1 spare1
    # zpool clear pool
    # zpool status pool1
      pool: pool1
     state: ONLINE
     scrub: resilver completed with 0 errors on Mon Feb 18 10:25:26 2008
    config:
    
            NAME        STATE     READ WRITE CKSUM
            pool1       ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                disk1   ONLINE       0     0     0
                disk2   ONLINE       0     0     0
            spares
              spare1    AVAIL   
    
    errors: No known data errors
    
    
    # fmadm reset zfs-diagnosis
    # fmadm reset zfs-retire
    # fmstat
    module             ev_recv ev_acpt wait  svc_t  %w  %b  open solve  memsz  bufsz
    cpumem-retire            0       0  0.0    0.5   0   0     0     0      0      0
    disk-transport           0       0  0.0  223.5   0   0     0     0    32b      0
    eft                      1       0  0.0    4.6   0   0     0     0   1.4M      0
    fmd-self-diagnosis       4       0  0.0    0.6   0   0     0     0      0      0
    io-retire                1       0  0.0    1.1   0   0     0     0      0      0
    snmp-trapgen             4       0  0.0    8.8   0   0     0     0    32b      0
    sysevent-transport       0       0  0.0  372.7   0   0     0     0      0      0
    syslog-msgs              4       0  0.0    5.4   0   0     0     0      0      0
    zfs-diagnosis            0       0  0.0    1.4   0   0     0     0      0      0
    zfs-retire               0       0  0.0    0.0   0   0     0     0      0      0
    
    
    # fmdump -v -u d82d1716-c920-6243-e899-b7ddd386902e
    TIME                 UUID                                 SUNW-MSG-ID
    Feb 18 09:51:49.3025 d82d1716-c920-6243-e899-b7ddd386902e ZFS-8000-GH
      100%  fault.fs.zfs.vdev.checksum
    
            Problem in: 
               Affects: zfs://pool=pool1/vdev=449a3328bc444732
                   FRU: -
              Location: -
    
    # fmadm repair zfs://pool=pool1/vdev=449a3328bc444732
    fmadm: recorded repair to zfs://pool=pool1/vdev=449a3328bc444732
    
    # fmadm faulty
    

    Problem 2: Device failure

    Time to do a little more harm. In this case I will simulate the failure of a device by removing the fake device. Again we will access the pool and then consult fmstat to see what is happening (are you noticing a pattern here????).
    # rm -f /dev/dsk/disk2
    # cd /pool1
    # find . -print | cpio -oc > /dev/null
    416027 blocks
    
    # fmstat
    module             ev_recv ev_acpt wait  svc_t  %w  %b  open solve  memsz  bufsz
    cpumem-retire            0       0  0.0    0.5   0   0     0     0      0      0
    disk-transport           0       0  0.0  214.2   0   0     0     0    32b      0
    eft                      1       0  0.0    4.6   0   0     0     0   1.4M      0
    fmd-self-diagnosis       4       0  0.0    0.6   0   0     0     0      0      0
    io-retire                1       0  0.0    1.1   0   0     0     0      0      0
    snmp-trapgen             4       0  0.0    8.8   0   0     0     0    32b      0
    sysevent-transport       0       0  0.0  372.7   0   0     0     0      0      0
    syslog-msgs              4       0  0.0    5.4   0   0     0     0      0      0
    zfs-diagnosis            0       0  0.0    1.4   0   0     0     0      0      0
    zfs-retire               0       0  0.0    0.0   0   0     0     0      0      0
    
    
    Rats, the find ran totally out of cache from the last example. As before, should this happen,proceed directly to zpool scrub.
    # zpool scrub pool1
    # fmstat
    module             ev_recv ev_acpt wait  svc_t  %w  %b  open solve  memsz  bufsz
    cpumem-retire            0       0  0.0    0.5   0   0     0     0      0      0
    disk-transport           0       0  0.0  190.5   0   0     0     0    32b      0
    eft                      1       0  0.0    4.1   0   0     0     0   1.4M      0
    fmd-self-diagnosis       5       0  0.0    0.5   0   0     0     0      0      0
    io-retire                1       0  0.0    1.0   0   0     0     0      0      0
    snmp-trapgen             6       0  0.0    7.4   0   0     0     0    32b      0
    sysevent-transport       0       0  0.0  329.0   0   0     0     0      0      0
    syslog-msgs              6       0  0.0    4.6   0   0     0     0      0      0
    zfs-diagnosis           16       1  0.0   70.3   0   0     1     1   168b   140b
    zfs-retire               1       0  0.0  509.8   0   0     0     0      0      0
    
    
    Again, hot sparing has kicked in automatically. The evidence of this is the zfs-retire engine running.
    # fmadm faulty
    --------------- ------------------------------------  -------------- ---------
    TIME            EVENT-ID                              MSG-ID         SEVERITY
    --------------- ------------------------------------  -------------- ---------
    Feb 18 11:07:29 50ea07a0-2cd9-6bfb-ff9e-e219740052d5  ZFS-8000-D3    Major    
    Feb 18 11:16:43 06bfe323-2570-46e8-f1a2-e00d8970ed0d
    
    Fault class : fault.fs.zfs.device
    
    Description : A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for
                  more information.
    
    Response    : No automated response will occur.
    
    Impact      : Fault tolerance of the pool may be compromised.
    
    Action      : Run 'zpool status -x' and replace the bad device.
    
    # zpool status -x
      pool: pool1
     state: DEGRADED
    status: One or more devices could not be opened.  Sufficient replicas exist for
            the pool to continue functioning in a degraded state.
    action: Attach the missing device and online it using 'zpool online'.
       see: http://www.sun.com/msg/ZFS-8000-2Q
     scrub: resilver in progress, 4.94% done, 0h0m to go
    config:
    
            NAME          STATE     READ WRITE CKSUM
            pool1         DEGRADED     0     0     0
              mirror      DEGRADED     0     0     0
                disk1     ONLINE       0     0     0
                spare     DEGRADED     0     0     0
                  disk2   UNAVAIL      0     0     0  cannot open
                  spare1  ONLINE       0     0     0
            spares
              spare1      INUSE     currently in use
    
    errors: No known data errors
    
    
    As before, this tells us all that we need to know. A device (disk2) has failed and is no longer in operation. Sufficient spares existed and one was automatically attached to the damaged pool. Resilvering completed successfully and the data is once again fully mirrored.

    But here's the magic. Let's repair the device - again simulated with our fake device.
    # mkfile 1g /dev/dsk/disk2
    # zpool repair pool1 disk2
    # zpool status pool1 
      pool: pool1
     state: DEGRADED
    status: One or more devices is currently being resilvered.  The pool will
            continue to function, possibly in a degraded state.
    action: Wait for the resilver to complete.
     scrub: resilver in progress, 4.86% done, 0h1m to go
    config:
    
            NAME               STATE     READ WRITE CKSUM
            pool1              DEGRADED     0     0     0
              mirror           DEGRADED     0     0     0
                disk1          ONLINE       0     0     0
                spare          DEGRADED     0     0     0
                  replacing    DEGRADED     0     0     0
                    disk2/old  UNAVAIL      0     0     0  cannot open
                    disk2      ONLINE       0     0     0
                  spare1       ONLINE       0     0     0
            spares
              spare1           INUSE     currently in use
    
    errors: No known data errors
    
    
    Get a cup of coffee while the resilvering process runs.
    # zpool status
      pool: pool1
     state: ONLINE
     scrub: resilver completed with 0 errors on Mon Feb 18 11:23:13 2008
    config:
    
            NAME        STATE     READ WRITE CKSUM
            pool1       ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                disk1   ONLINE       0     0     0
                disk2   ONLINE       0     0     0
            spares
              spare1    AVAIL   
    
    
    # fmadm faulty
    
    
    Notice the nice integration with FMA. Not only was the new device resilvered, but the hot spare was detached and the FMA fault was cleared. The fmstat counters still show that there was a problem and the fault report still existes in the fault log for later interrogation.
    # fmstat
    module             ev_recv ev_acpt wait  svc_t  %w  %b  open solve  memsz  bufsz
    cpumem-retire            0       0  0.0    0.5   0   0     0     0      0      0
    disk-transport           0       0  0.0  171.5   0   0     0     0    32b      0
    eft                      1       0  0.0    3.6   0   0     0     0   1.4M      0
    fmd-self-diagnosis       6       0  0.0    0.6   0   0     0     0      0      0
    io-retire                1       0  0.0    0.9   0   0     0     0      0      0
    snmp-trapgen             6       0  0.0    6.8   0   0     0     0    32b      0
    sysevent-transport       0       0  0.0  294.3   0   0     0     0      0      0
    syslog-msgs              6       0  0.0    4.2   0   0     0     0      0      0
    zfs-diagnosis           36       1  0.0   51.6   0   0     0     1      0      0
    zfs-retire               1       0  0.0  170.0   0   0     0     0      0      0
    
    # fmdump
    TIME                 UUID                                 SUNW-MSG-ID
    Feb 16 11:38:16.0976 48935791-ff83-e622-fbe1-d54c20385afc ZFS-8000-GH
    Feb 16 11:38:30.8519 9f7f288c-fea8-e5dd-bf23-c0c9c4e07233 ZFS-8000-GH
    Feb 18 09:51:49.3025 2ac4568f-4040-cb5d-f3b8-ae3d69e7d713 ZFS-8000-GH
    Feb 18 09:56:24.8029 d82d1716-c920-6243-e899-b7ddd386902e ZFS-8000-GH
    Feb 18 10:23:07.2228 7c04a6f7-d22a-e467-c44d-80810f27b711 ZFS-8000-GH
    Feb 18 10:25:14.6429 faca0639-b82b-c8e8-c8d4-fc085bc03caa ZFS-8000-GH
    Feb 18 11:07:29.5195 50ea07a0-2cd9-6bfb-ff9e-e219740052d5 ZFS-8000-D3
    Feb 18 11:16:44.2497 06bfe323-2570-46e8-f1a2-e00d8970ed0d ZFS-8000-D3
    
    
    # fmdump -V -u 50ea07a0-2cd9-6bfb-ff9e-e219740052d5
    TIME                 UUID                                 SUNW-MSG-ID
    Feb 18 11:07:29.5195 50ea07a0-2cd9-6bfb-ff9e-e219740052d5 ZFS-8000-D3
    
      TIME                 CLASS                                 ENA
      Feb 18 11:07:27.8476 ereport.fs.zfs.vdev.open_failed       0xb22406c635500401
    
    nvlist version: 0
            version = 0x0
            class = list.suspect
            uuid = 50ea07a0-2cd9-6bfb-ff9e-e219740052d5
            code = ZFS-8000-D3
            diag-time = 1203354449 236999
            de = (embedded nvlist)
            nvlist version: 0
                    version = 0x0
                    scheme = fmd
                    authority = (embedded nvlist)
                    nvlist version: 0
                            version = 0x0
                            product-id = Dimension XPS                
                            chassis-id = 7XQPV21
                            server-id = arrakis
                    (end authority)
    
                    mod-name = zfs-diagnosis
                    mod-version = 1.0
            (end de)
    
            fault-list-sz = 0x1
            fault-list = (array of embedded nvlists)
            (start fault-list[0])
            nvlist version: 0
                    version = 0x0
                    class = fault.fs.zfs.device
                    certainty = 0x64
                    asru = (embedded nvlist)
                    nvlist version: 0
                            version = 0x0
                            scheme = zfs
                            pool = 0x3a2ca6bebd96cfe3
                            vdev = 0xedef914b5d9eae8d
                    (end asru)
    
                    resource = (embedded nvlist)
                    nvlist version: 0
                            version = 0x0
                            scheme = zfs
                            pool = 0x3a2ca6bebd96cfe3
                            vdev = 0xedef914b5d9eae8d
                    (end resource)
    
            (end fault-list[0])
    
            fault-status = 0x3
            __ttl = 0x1
            __tod = 0x47b9bb51 0x1ef7b430
    
    # fmadm reset zfs-diagnosis
    fmadm: zfs-diagnosis module has been reset
    
    # fmadm reset zfs-retire
    fmadm: zfs-retire module has been reset
    
    

    Problem 3: Unrecoverable corruption

    For those of you that have attended one of my Boot Camps or Solaris Best Practices training classes know, House is one of my favorite TV shows - the only one that I watch regularly. And this next example would make a perfect episode. Is it likely to happen ? No, but it is so cool when it does :-)

    Remember our second pool, pool2. It has the same contents as pool1. Now, let's do the unthinkable - let's corrupt both halves of the mirror. Surely data loss will follow, but the fact that Solaris stays up and running and can report what happened is pretty spectacular. But it gets so much better than that.
    # dd if=/dev/zero of=/dev/dsk/disk3 bs=8192 count=10000 conv=notrunc
    # dd if=/dev/zero of=/dev/dsk/disk4 bs=8192 count=10000 conv=notrunc
    # zpool scrub pool2
    
    # fmstat
    module             ev_recv ev_acpt wait  svc_t  %w  %b  open solve  memsz  bufsz
    cpumem-retire            0       0  0.0    0.5   0   0     0     0      0      0
    disk-transport           0       0  0.0  166.0   0   0     0     0    32b      0
    eft                      1       0  0.0    3.6   0   0     0     0   1.4M      0
    fmd-self-diagnosis       6       0  0.0    0.6   0   0     0     0      0      0
    io-retire                1       0  0.0    0.9   0   0     0     0      0      0
    snmp-trapgen             8       0  0.0    6.3   0   0     0     0    32b      0
    sysevent-transport       0       0  0.0  294.3   0   0     0     0      0      0
    syslog-msgs              8       0  0.0    3.9   0   0     0     0      0      0
    zfs-diagnosis         1032    1028  0.6   39.7   0   0    93     2    15K    13K
    zfs-retire               2       0  0.0  158.5   0   0     0     0      0      0
    
    
    As before, lots of zfs-diagnosis activity. And two hits to zfs-retire. But we only have one spare - this should be interesting. Let's see what is happenening.
    # fmadm faulty
    --------------- ------------------------------------  -------------- ---------
    TIME            EVENT-ID                              MSG-ID         SEVERITY
    --------------- ------------------------------------  -------------- ---------
    Feb 18 09:56:24 d82d1716-c920-6243-e899-b7ddd386902e  ZFS-8000-GH    Major    
    Feb 18 13:18:42 c3889bf1-8551-6956-acd4-914474093cd7
    
    Fault class : fault.fs.zfs.vdev.checksum
    
    Description : The number of checksum errors associated with a ZFS device
                  exceeded acceptable levels.  Refer to
                  http://sun.com/msg/ZFS-8000-GH for more information.
    
    Response    : The device has been marked as degraded.  An attempt
                  will be made to activate a hot spare if available.
    
    Impact      : Fault tolerance of the pool may be compromised.
    
    Action      : Run 'zpool status -x' and replace the bad device.
    
    --------------- ------------------------------------  -------------- ---------
    TIME            EVENT-ID                              MSG-ID         SEVERITY
    --------------- ------------------------------------  -------------- ---------
    Feb 16 11:38:30 9f7f288c-fea8-e5dd-bf23-c0c9c4e07233  ZFS-8000-GH    Major    
    Feb 18 09:51:49 2ac4568f-4040-cb5d-f3b8-ae3d69e7d713
    Feb 18 10:23:07 7c04a6f7-d22a-e467-c44d-80810f27b711
    Feb 18 13:18:42 0a1bf156-6968-4956-d015-cc121a866790
    
    Fault class : fault.fs.zfs.vdev.checksum
    
    Description : The number of checksum errors associated with a ZFS device
                  exceeded acceptable levels.  Refer to
                  http://sun.com/msg/ZFS-8000-GH for more information.
    
    Response    : The device has been marked as degraded.  An attempt
                  will be made to activate a hot spare if available.
    
    Impact      : Fault tolerance of the pool may be compromised.
    
    Action      : Run 'zpool status -x' and replace the bad device.
    
    # zpool status -x
      pool: pool2
     state: DEGRADED
    status: One or more devices has experienced an error resulting in data
            corruption.  Applications may be affected.
    action: Restore the file in question if possible.  Otherwise restore the
            entire pool from backup.
       see: http://www.sun.com/msg/ZFS-8000-8A
     scrub: resilver completed with 602 errors on Mon Feb 18 13:20:14 2008
    config:
    
            NAME          STATE     READ WRITE CKSUM
            pool2         DEGRADED     0     0 2.60K
              mirror      DEGRADED     0     0 2.60K
                spare     DEGRADED     0     0 2.43K
                  disk3   DEGRADED     0     0 5.19K  too many errors
                  spare1  DEGRADED     0     0 2.43K  too many errors
                disk4     DEGRADED     0     0 5.19K  too many errors
            spares
              spare1      INUSE     currently in use
    
    errors: 247 data errors, use '-v' for a list
    
    
    So ZFS tried to bring in a hot spare, but there were insufficient replicas to be able to reconstruct all of the data. But here is where is gets interesting. Let's see what zpool status -v says about things.
    zpool status -v
      pool: pool1
     state: ONLINE
     scrub: resilver completed with 0 errors on Mon Feb 18 11:23:13 2008
    config:
    
            NAME        STATE     READ WRITE CKSUM
            pool1       ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                disk1   ONLINE       0     0     0
                disk2   ONLINE       0     0     0
            spares
              spare1    INUSE     in use by pool 'pool2'
    
    errors: No known data errors
    
      pool: pool2
     state: DEGRADED
    status: One or more devices has experienced an error resulting in data
            corruption.  Applications may be affected.
    action: Restore the file in question if possible.  Otherwise restore the
            entire pool from backup.
       see: http://www.sun.com/msg/ZFS-8000-8A
     scrub: resilver completed with 602 errors on Mon Feb 18 13:20:14 2008
    config:
    
            NAME          STATE     READ WRITE CKSUM
            pool2         DEGRADED     0     0 2.60K
              mirror      DEGRADED     0     0 2.60K
                spare     DEGRADED     0     0 2.43K
                  disk3   DEGRADED     0     0 5.19K  too many errors
                  spare1  DEGRADED     0     0 2.43K  too many errors
                disk4     DEGRADED     0     0 5.19K  too many errors
            spares
              spare1      INUSE     currently in use
    
    errors: Permanent errors have been detected in the following files:
    
            /pool2/scenic/cider mill crowds.jpg
            /pool2/scenic/Cleywindmill.jpg
            /pool2/scenic/csg_Landscapes001_GrandTetonNationalPark,Wyoming.jpg
            /pool2/scenic/csg_Landscapes002_ElowahFalls,Oregon.jpg
            /pool2/scenic/csg_Landscapes003_MonoLake,California.jpg
            /pool2/scenic/csg_Landscapes005_TurretArch,Utah.jpg
            /pool2/scenic/csg_Landscapes004_Wildflowers_MountRainer,Washington.jpg
            /pool2/scenic/csg_Landscapes!idx011.jpg
            /pool2/scenic/csg_Landscapes127_GreatSmokeyMountains-NorthCarolina.jpg
            /pool2/scenic/csg_Landscapes129_AcadiaNationalPark-Maine.jpg
            /pool2/scenic/csg_Landscapes130_GettysburgNationalPark-Pennsylvania.jpg
            /pool2/scenic/csg_Landscapes131_DeadHorseMill,CrystalRiver-Colorado.jpg
            /pool2/scenic/csg_Landscapes132_GladeCreekGristmill,BabcockStatePark-WestVirginia.jpg
            /pool2/scenic/csg_Landscapes133_BlackwaterFallsStatePark-WestVirginia.jpg
            /pool2/scenic/csg_Landscapes134_GrandCanyonNationalPark-Arizona.jpg
            /pool2/scenic/decisions decisions.jpg
            /pool2/scenic/csg_Landscapes135_BigSur-California.jpg
            /pool2/scenic/csg_Landscapes151_WataugaCounty-NorthCarolina.jpg
            /pool2/scenic/csg_Landscapes150_LakeInTheMedicineBowMountains-Wyoming.jpg
            /pool2/scenic/csg_Landscapes152_WinterPassage,PondMountain-Tennessee.jpg
            /pool2/scenic/csg_Landscapes154_StormAftermath,OconeeCounty-Georgia.jpg
            /pool2/scenic/Brig_Of_Dee.gif
            /pool2/scenic/pvnature14.gif
            /pool2/scenic/pvnature22.gif
            /pool2/scenic/pvnature7.gif
            /pool2/scenic/guadalupe.jpg
            /pool2/scenic/ernst-tinaja.jpg
            /pool2/scenic/pipes.gif
            /pool2/scenic/boat.jpg
            /pool2/scenic/pvhawaii.gif
            /pool2/scenic/cribgoch.jpg
            /pool2/scenic/sun1.gif
            /pool2/scenic/sun1.jpg
            /pool2/scenic/sun2.jpg
            /pool2/scenic/andes.jpg
            /pool2/scenic/treesky.gif
            /pool2/scenic/sailboatm.gif
            /pool2/scenic/Arizona1.jpg
            /pool2/scenic/Arizona2.jpg
            /pool2/scenic/Fence.jpg
            /pool2/scenic/Rockwood.jpg
            /pool2/scenic/sawtooth.jpg
            /pool2/scenic/pvaptr04.gif
            /pool2/scenic/pvaptr07.gif
            /pool2/scenic/pvaptr11.gif
            /pool2/scenic/pvntrr01.jpg
            /pool2/scenic/Millport.jpg
            /pool2/scenic/bryce2.jpg
            /pool2/scenic/bryce3.jpg
            /pool2/scenic/monument.jpg
            /pool2/scenic/rainier1.gif
            /pool2/scenic/arch.gif
            /pool2/scenic/pv-anzab.gif
            /pool2/scenic/pvnatr15.gif
            /pool2/scenic/pvocean3.gif
            /pool2/scenic/pvorngwv.gif
            /pool2/scenic/pvrmp001.gif
            /pool2/scenic/pvscen07.gif
            /pool2/scenic/pvsltd04.gif
            /pool2/scenic/banhall28600-04.JPG
            /pool2/scenic/pvwlnd01.gif
            /pool2/scenic/pvnature08.gif
            /pool2/scenic/pvnature13.gif
            /pool2/scenic/nokomis.jpg
            /pool2/scenic/lighthouse1.gif
            /pool2/scenic/lush.gif
            /pool2/scenic/oldmill.gif
            /pool2/scenic/gc1.jpg
            /pool2/scenic/gc2.jpg
            /pool2/scenic/canoe.gif
            /pool2/scenic/Donaldson-River.jpg
            /pool2/scenic/beach.gif
            /pool2/scenic/janloop.jpg
            /pool2/scenic/grobacro.jpg
            /pool2/scenic/fnlgld.jpg
            /pool2/scenic/bells.gif
            /pool2/scenic/Eilean_Donan.gif
            /pool2/scenic/Kilchurn_Castle.gif
            /pool2/scenic/Plockton.gif
            /pool2/scenic/Tantallon_Castle.gif
            /pool2/scenic/SouthStockholm.jpg
            /pool2/scenic/BlackRock_Cottage.jpg
            /pool2/scenic/seward.jpg
            /pool2/scenic/canadian_rockies_csg110_EmeraldBay.jpg
            /pool2/scenic/canadian_rockies_csg111_RedRockCanyon.jpg
            /pool2/scenic/canadian_rockies_csg112_WatertonNationalPark.jpg
            /pool2/scenic/canadian_rockies_csg113_WatertonLakes.jpg
            /pool2/scenic/canadian_rockies_csg114_PrinceOfWalesHotel.jpg
            /pool2/scenic/canadian_rockies_csg116_CameronLake.jpg
            /pool2/scenic/Castilla_Spain.jpg
            /pool2/scenic/Central-Park-Walk.jpg
            /pool2/scenic/CHANNEL.JPG
    
    
    
    
    In my best Hugh Laurie voice trying to sound very Northeastern American, that is so cool! But we're not even done yet. Let's take this list of files and restore them - in this case, from pool1. Operationally this would be from a back up tape or nearline backup cache, but for our purposes, the contents in pool1 will do nicely.

    First, let's clear the zpool error counters and return the spare disk. We want to make sure that our restore works as desired. Oh, and clear the FMA stats while we're at it.
    # zpool clear
    # zpool detach pool2 spare1
    
    # fmadm reset zfs-diagnosis
    fmadm: zfs-diagnosis module has been reset
    
    # fmadm reset zfs-retire   
    fmadm: zfs-retire module has been reset
    
    
    Now individually restore the files that have errors in them and check again. You can even export and reimport the pool and you will find a very nice, happy, and thoroughly error free ZFS pool. Some rather unpleasant gnashing of zpool status -v output with awk has been omitted for sanity sake.
    # zpool scrub pool2
    # zpool status pool2
      pool: pool2
     state: ONLINE
     scrub: scrub completed with 0 errors on Mon Feb 18 14:04:56 2008
    config:
    
            NAME        STATE     READ WRITE CKSUM
            pool2       ONLINE       0     0     0
              mirror    ONLINE       0     0     0
                disk3   ONLINE       0     0     0
                disk4   ONLINE       0     0     0
            spares
              spare1    AVAIL   
    
    errors: No known data errors
    
    # zpool export pool2
    # zpool import pool2
    # dircmp -s /pool1 /pool2
    
    

    Conclusions and Review

    So what have we learned ? ZFS and FMA are two great tastes that taste great together. No, that's chocolate and peanut butter, but you get this idea. One more great example of Isaac's Multiplicity of Solaris.

    That, and I have finally found a good lab exercise for the FMA training materials. Ever since Christine Tran put the FMA workshop together, we have been looking for some good FMA lab exercises. The materials reference a synthetic fault generator that is not available in public (for obvious reasons). I haven't explored the FMA test harness enough to know if there is anything in there that would make a good lab. But this exercise that we have just explored seems to tie a number of key pieces together.

    And of course, one more reason why Roxy says, "You should run Solaris."

    Technocrati Tags:
    About

    Bob Netherton is a Principal Sales Consultant for the North American Commercial Hardware group, specializing in Solaris, Virtualization and Engineered Systems. Bob is also a contributing author of Solaris 10 Virtualization Essentials.

    This blog will contain information about all three, but primarily focused on topics for Solaris system administrators.

    Please follow me on Twitter Facebook or send me email

    Search

    Archives
    « April 2014
    SunMonTueWedThuFriSat
      
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
       
           
    Today