A Hands on Introduction to ZFS Pools; Part 2: Mirrors, Disk Failures and Spare Disks

A Hands on Introduction to ZFS Pools; Part 2: Mirrors, Disk Failures and Spare Disks

The small hands on tutorial shows how to manage disks through ZFS with very limited resources. The tutorial is based on the following components

  • One 7 port USB 2.0 hub

  • Six 1GB USB 2.0 memory sticks

It has been released in multiple parts:

  1. Getting started, pool creation import and export of pools 
  2. Mirrors, managing disk failures and spare disks
  3. RAIDZ2, resilvering and scrubbing

Within this tutorial you will learn

  • How to create a mirror in ZFS

  • How to expand it

  • How to replace broken components

As already stated in the first part; get some extra graphical tools like iobar from the freely available Tools CD. This will visualize nicely what's happening on your disks (see below):

Getting Started

Non Disclaimer: Please be cautious with your system. All commands apply significant changes to your system. Data corruption may easily occur. Have a good backup of your system. The system is not warranted to work since it depends on the USB sticks, the hub and how how Solaris get's informed about status changes. ZFS hasn't been optimized for USB sticks and hick ups may occur at any time. Do not use such a configuration for important work. This is a low cost self learning sand box!

As already stated in the first part:

  • Become super user (su)

  • Clean up your /dev/dsk device list (devfsadm -C)

Plug in your first 2 memory sticks:

The rmformat reveils the following mappin:

  • stick 1: c4t0d0p0

  • stick 2: c5t0d0p0

Creating a mirrored pool

Create a mirrored pool with the pool name mpool by typing:

jack@opensolaris:~# zpool create mpool mirror c4t0d0p0 c5t0d0p0

Check the status of our mirror with:

jack@opensolaris:~# zpool status mpool
  pool: mpool
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        mpool         ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c4t0d0p0  ONLINE       0     0     0
            c5t0d0p0  ONLINE       0     0     0

errors: No known data errors

The mirror got mounted automaticallyand it is available through the directory /mpool.

Keep the file system under stress. Create a tar file from the /usr file tree in the background:

jack@opensolaris: cd /mpool
jack@opensolaris: tar -cf usr.tar /usr  &

Enlarging a mirrored pool

The next step is to add two more disks to enhance the file system.

Plug in memory stick 3 and 4:

rmformat will reveil that the following mapping

  • stick 3: c6t0d0p0

  • stick 4: c7t0d0p0

Keep in mind. This will vary system by system.

Enlarge the mirror by typing:

jack@opensolaris:~# zpool add mpool mirror c6t0d0p0 c7t0d0p0

Validate the operation with zpool status:

jack@opensolaris:~# zpool status mpool
  pool: mpool
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        mpool         ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c4t0d0p0  ONLINE       0     0     0
            c5t0d0p0  ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c6t0d0p0  ONLINE       0     0     0
            c7t0d0p0  ONLINE       0     0     0

errors: No known data errors

Your system has now twice the capacity. The IO operation is still going on in the back ground. Use your last 2 memory sticks to configure spare disks for potential failures:.

Plug in stick 5 and 6:

The configuration is now complete. The picture documents as well that stick 4 is currently busy.

rmformat documents the following mapping

  • stick 5: c8t0d0p0

  • stick 6: c9t0d0p0

Add your two disks as spares with the following command:

jack@opensolaris:~# zpool add mpool spare c8t0d0p0 c9t0d0p0

  pool: mpool
 state: ONLINE
 scrub: none requested
config:
        NAME          STATE     READ WRITE CKSUM
        mpool         ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c4t0d0p0  ONLINE       0     0     0
            c5t0d0p0  ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c6t0d0p0  ONLINE       0     0     0
            c7t0d0p0  ONLINE       0     0     0
        spares
          c8t0d0p0    AVAIL   
          c9t0d0p0    AVAIL   

errors: No known data errors

Now we're able to start with the real fun. Disk pulling! Pull stick 3 as showed on the picture below:

Now check with zpool status on what's going on:

ack@opensolaris:~# zpool status -v mpool

  pool: mpool
 state: DEGRADED
 scrub: resilver completed after 0h0m with 0 errors on Thu Jul  3
05:02:06 2008
config:
        NAME          STATE     READ WRITE CKSUM
        mpool         DEGRADED     0     0     0
          mirror      DEGRADED     0     0     0
            c4t0d0p0  REMOVED      0     0     0
            c5t0d0p0  ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c6t0d0p0  ONLINE       0     0     0
            c7t0d0p0  ONLINE       0     0     0
        spares
          c8t0d0p0    AVAIL   
          c9t0d0p0    AVAIL   
errors: No known data errors

Controller 4 is recognized as gone and there is no data corruption. Everything is still operational but the redundancy is gone. We now have to replace the memory stick gone with a spare one. This happens through:

jack@opensolaris:~# zpool replace mpool c4t0d0p0 c8t0d0p0

The spare disks is now getting resilvered. The progress may be checked with the zpool status command. The case below documents a resilvering in progress:

jack@opensolaris:~# zpool status -v mpool
  pool: mpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The
pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 11.57% done, 0h3m to go
config:
        NAME            STATE     READ WRITE CKSUM
        mpool           DEGRADED     0     0     0
          mirror        DEGRADED     0     0     0
            spare       DEGRADED     0     0     0
              c4t0d0p0  REMOVED      0     0     0
              c8t0d0p0  ONLINE       0     0     0
            c5t0d0p0    ONLINE       0     0     0
          mirror        ONLINE       0     0     0
            c6t0d0p0    ONLINE       0     0     0
            c7t0d0p0    ONLINE       0     0     0
        spares
          c8t0d0p0      INUSE     currently in use
          c9t0d0p0      AVAIL   
errors: No known data errors

The spare disk c8t0d0p0 is now in use. It's getting worse now. We pull out stick 4 as well:

c7t0d0p0 now needs to be replace with the remaining spare disk c9t0d0p0. zpool status shows the fix after the resilvering:

jack@opensolaris:~# zpool replace mpool c7t0d0p0 c9t0d0p0
  pool: mpool
 state: DEGRADED
 scrub: resilver completed after 0h9m with 0 errors on Thu Jul  3
05:32:50 2008
config:

        NAME            STATE     READ WRITE CKSUM
        mpool           DEGRADED     0     0     0
          mirror        DEGRADED     0     0     0
            spare       DEGRADED     0     0     0
              c4t0d0p0  REMOVED      0     0     0
              c8t0d0p0  ONLINE       0     0     0
            c5t0d0p0    ONLINE       0     0     0
          mirror        DEGRADED     0     0     0
            c6t0d0p0    ONLINE       0     0     0
            spare       DEGRADED     0     0     0
              c7t0d0p0  REMOVED      0     0     0
              c9t0d0p0  ONLINE       0     0     0
        spares
          c8t0d0p0      INUSE     currently in use
          c9t0d0p0      INUSE     currently in use

errors: No known data errors

Plugging the remaining sticks back in lead to the following:

The remaining disks will be made online and resilvered:

jack@opensolaris:~# zpool status -v mpool
  pool: mpool
 state: ONLINE
 scrub: resilver completed after 0h2m with 0 errors on Thu Jul  3
06:18:46 2008
config:

        NAME          STATE     READ WRITE CKSUM
        mpool         ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c4t0d0p0  ONLINE       0     0     0
            c5t0d0p0  ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c6t0d0p0  ONLINE       0     0     0
            c7t0d0p0  ONLINE       0     0     0
        spares
          c8t0d0p0    AVAIL   
          c9t0d0p0    AVAIL   

errors: No known data errors

This concludes the second part of the ZFS hands an exercises.

You should have learned:

  • How to create a mirror

  • How to expand it

  • How to replace broken components

Released up to now are:

  1. Getting started, pool creation import and export of pools

  2. Mirrors, managing disk failures and spare disks

  3. RAIDZ2, resilvering and scrubbing
  4.  


Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

Application tuning, sizing, monitoring, porting on Solaris 11

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
5
6
8
9
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today