Oracle Solaris ZFS Device Removal

At long last, we provide the ability to remove a top-level VDEV from a ZFS storage pool in the upcoming Solaris 11.4 Beta refresh release.

For many years, our recommendation was to create a pool based on current capacity requirements and then grow the pool to meet increasing capacity needs by adding VDEVs or by replacing smaller LUNs with larger LUNs. It is trivial to add capacity or replace smaller LUNs with larger LUNs, sometimes with just one simple command.

The simplicity of ZFS is one of its great strengths!

I still recommend the practice of creating a pool that meets current capacity requirements and then adding capacity when needed. If you need to repurpose pool devices in an over-provisioned pool or if you accidentally misconfigure a pool device, you now have the flexibility to resolve these scenarios.

Review the following practical considerations when using this new feature, which should be used as an exception rather than the rule for pool configuration on production systems:

• An virtual (pseudo device) is created to move the data off the (removed) pool devices so the pool must have enough space to absorb the creation of the pseudo device
• Only top-level VDEVs can be removed from mirrored or RAIDZ pools
• Individual devices can be removed from striped pools
• Pool device misconfigurations can be corrected

A few implementation details in case you were wondering:

• No additional steps are needed to remap the removed devices
• Data from the removed devices are allocated to the remaining devices but this is not a way to rebalance all data on pool devices
• Reads of the reallocated data are done from the pseudo device until those blocks are freed
• Some levels of indirection are needed to support this operation but they should not impact performance nor increase memory requirements

See the examples below.

Repurpose Pool Devices

The following pool, tank, has low space consumption so one VDEV is removed.

# zpool list tank
NAME  SIZE  ALLOC  FREE  CAP  DEDUP  HEALTH  ALTROOT
tank  928G  28.1G  900G   3%  1.00x  ONLINE  
# zpool status tank
  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE      READ WRITE CKSUM
        tank        ONLINE        0     0     0
          mirror-0  ONLINE        0     0     0
            c3t2d0  ONLINE        0     0     0
            c4t2d0  ONLINE        0     0     0
          mirror-1  ONLINE        0     0     0
            c1t7d0  ONLINE        0     0     0
            c5t3d0  ONLINE        0     0     0

errors: No known data errors
-
# zpool remove tank mirror-1
# zpool status tank
  pool: tank
 state: ONLINE
status: One or more devices are being removed.
action: Wait for the resilver to complete.
        Run 'zpool status -v' to see device specific details.
  scan: resilver in progress since Sun Apr 15 20:58:45 2018
    28.1G scanned
    3.07G resilvered at 40.9M/s, 21.83% done, 4m35s to go
config:

        NAME                       STATE      READ WRITE CKSUM
        tank                       ONLINE        0     0     0
          mirror-0                 ONLINE        0     0     0
            c3t2d0                 ONLINE        0     0     0
            c4t2d0                 ONLINE        0     0     0
          mirror-1                 REMOVING      0     0     0
            c1t7d0                 REMOVING      0     0     0
            c5t3d0                 REMOVING      0     0     0

errors: No known data errors

Run the zpool iostat command to verify that data is being written to the remaining VDEV.

# zpool iostat -v tank 5
                              capacity     operations    bandwidth
pool                       alloc   free   read  write   read  write
-------------------------  -----  -----  -----  -----  -----  -----
tank                       28.1G   900G      9    182   932K  21.3M
  mirror-0                 14.1G   450G      1    182  7.90K  21.3M
    c3t2d0                     -      -      0     28  4.79K  21.3M
    c4t2d0                     -      -      0     28  3.92K  21.3M
  mirror-1                     -      -      8    179   924K  21.2M
    c1t7d0                     -      -      1     28   495K  21.2M
    c5t3d0                     -      -      1     28   431K  21.2M
-------------------------  -----  -----  -----  -----  -----  -----

                              capacity     operations    bandwidth
pool                       alloc   free   read  write   read  write
-------------------------  -----  -----  -----  -----  -----  -----
tank                       28.1G   900G      0    967      0  60.0M
  mirror-0                 14.1G   450G      0    967      0  60.0M
    c3t2d0                     -      -      0     67      0  60.0M
    c4t2d0                     -      -      0     68      0  60.4M
  mirror-1                     -      -      0      0      0      0
    c1t7d0                     -      -      0      0      0      0
    c5t3d0                     -      -      0      0      0      0
-------------------------  -----  -----  -----  -----  -----  -----

Misconfigured Pool Device

In this case, a device was intended to be added as a cache device but was added a single device. The problem is identified and resolved.

# zpool status rzpool
  pool: rzpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE      READ WRITE CKSUM
        rzpool      ONLINE        0     0     0
          raidz1-0  ONLINE        0     0     0
            c5t7d0  ONLINE        0     0     0
            c2t3d0  ONLINE        0     0     0
            c1t7d0  ONLINE        0     0     0
            c5t3d0  ONLINE        0     0     0

errors: No known data errors
# zpool add rzpool c3t3d0
vdev verification failed: use -f to override the following errors:
mismatched replication level: pool uses raidz and new vdev is disk
Unable to build pool from specified devices: invalid vdev configuration
# zpool add -f rzpool c3t3d0
# zpool status rzpool
  pool: rzpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE      READ WRITE CKSUM
        rzpool      ONLINE        0     0     0
          raidz1-0  ONLINE        0     0     0
            c5t7d0  ONLINE        0     0     0
            c2t3d0  ONLINE        0     0     0
            c1t7d0  ONLINE        0     0     0
            c5t3d0  ONLINE        0     0     0
          c3t3d0    ONLINE        0     0     0

errors: No known data errors
# zpool remove rzpool c3t3d0
# zpool add rzpool cache c3t3d0
# zpool status rzpool
  pool: rzpool
 state: ONLINE
  scan: resilvered 0 in 1s with 0 errors on Sun Apr 15 21:09:35 2018

config:

        NAME        STATE      READ WRITE CKSUM
        rzpool      ONLINE        0     0     0
          raidz1-0  ONLINE        0     0     0
            c5t7d0  ONLINE        0     0     0
            c2t3d0  ONLINE        0     0     0
            c1t7d0  ONLINE        0     0     0
            c5t3d0  ONLINE        0     0     0
        cache
          c3t3d0    ONLINE        0     0     0

In summary, Solaris 11.4 includes a handy new option for repurposing pool devices and resolving pool misconfiguration errors.