HOWTO: Replace a bad disk in a ZFS mirror pool

Every so often, once upon a MTBF, a disk will fail. Disks are a miracle of engineering, but moving parts are their enemy, so ZFS allows you to easily replace a bad disk with a new one and it even handles all the reconfiguration automatically.

Let's start by creating a pool. I'll simulate disks here with files. (Bold indicates user input).

# mkfile 64m /tmp/a /tmp/b /tmp/c
# zpool create tank mirror /tmp/a /tmp/b
# zpool status
  pool: tank
 state: ONLINE
 scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    /tmp/a  ONLINE       0     0     0
	    /tmp/b  ONLINE       0     0     0

errors: No known data errors

Next, we'll pretend that /tmp/b failed, so we want to replace it.

# zpool replace tank /tmp/b /tmp/c
# zpool status
  pool: tank
 state: ONLINE
 scrub: resilver completed with 0 errors on Fri Jun  1 08:55:47 2007
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    /tmp/a  ONLINE       0     0     0
	    /tmp/c  ONLINE       0     0     0

errors: No known data errors

What if that's not good enough? Let's say having a single failure makes you nervous - after all, what if the good drive, /tmp/a, fails before the resilvering completes? You now decide you want a three-way mirror. Let's create a new file, /tmp/d, and add it to the mirror:

# mkfile 64m /tmp/d
# zpool attach tank /tmp/a /tmp/d
# zpool status
  pool: tank
 state: ONLINE
 scrub: resilver completed with 0 errors on Fri Jun  1 08:58:15 2007
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    /tmp/a  ONLINE       0     0     0
	    /tmp/c  ONLINE       0     0     0
	    /tmp/d  ONLINE       0     0     0

errors: No known data errors

Note the syntax of the attach command. It didn't specify a mirror - it just specified a disk to attach to. The example shows /tmp/a, but it could easily have been the other disk in the mirror, /tmp/c. Extending this concept, we can use it to create redundant storage where there once wasn't.

For example, let's create a pool with a single disk:

# zpool destroy tank
# zpool create tank /tmp/a
# zpool status
  pool: tank
 state: ONLINE
 scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  /tmp/a    ONLINE       0     0     0

errors: No known data errors

The pool no longer has the 'mirror' keyword below the pool name. The pool's vdev is not a mirror because there's only one disk in. Now let's try attaching and see what happens:

# zpool attach tank /tmp/a /tmp/b
# zpool status
  pool: tank
 state: ONLINE
 scrub: resilver completed with 0 errors on Fri Jun  1 09:03:05 2007
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    /tmp/a  ONLINE       0     0     0
	    /tmp/b  ONLINE       0     0     0

errors: No known data errors

And now we've got a mirror. This can be done even in a striped mirror configuration, if a disk in a mirror breaks. For example, let's add another mirror to the pool, using 'zpool add':

# zpool add tank mirror /tmp/c /tmp/d
# zpool status
  pool: tank
 state: ONLINE
 scrub: resilver completed with 0 errors on Fri Jun  1 09:03:05 2007
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    /tmp/a  ONLINE       0     0     0
	    /tmp/b  ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    /tmp/c  ONLINE       0     0     0
	    /tmp/d  ONLINE       0     0     0

errors: No known data errors

And now if /tmp/d breaks, we can detach it:

# zpool detach tank /tmp/d
# zpool status
  pool: tank
 state: ONLINE
 scrub: resilver completed with 0 errors on Fri Jun  1 09:03:05 2007
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    /tmp/a  ONLINE       0     0     0
	    /tmp/b  ONLINE       0     0     0
	  /tmp/c    ONLINE       0     0     0

errors: No known data errors

The indentation of /tmp/c shows us that it is not in the a/b mirror. Instead it is a single 'disk' that's part of the tank stripe. Now we'll add back /tmp/d:

# zpool attach tank /tmp/c /tmp/d
# zpool status
  pool: tank
 state: ONLINE
 scrub: resilver completed with 0 errors on Fri Jun  1 09:08:42 2007
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    /tmp/a  ONLINE       0     0     0
	    /tmp/b  ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    /tmp/c  ONLINE       0     0     0
	    /tmp/d  ONLINE       0     0     0

errors: No known data errors

And we're back to a stripe of mirrors.

Comments:

Post a Comment:
Comments are closed for this entry.
About

Known throughout Sun as a man of infinite wit, of jovial attitude, and of making things up about himself at the slightest whim.

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today