At long last, we provide the ability to remove a top-level VDEV from a ZFS storage pool in the upcoming Solaris 11.4 Beta refresh release.
For many years, our recommendation was to create a pool based on current capacity requirements and then grow the pool to meet increasing capacity needs by adding VDEVs or by replacing smaller LUNs with larger LUNs. It is trivial to add capacity or replace smaller LUNs with larger LUNs, sometimes with just one simple command.
The simplicity of ZFS is one of its great strengths!
I still recommend the practice of creating a pool that meets current capacity requirements and then adding capacity when needed. If you need to repurpose pool devices in an over-provisioned pool or if you accidentally misconfigure a pool device, you now have the flexibility to resolve these scenarios.
Review the following practical considerations when using this new feature, which should be used as an exception rather than the rule for pool configuration on production systems:
A few implementation details in case you were wondering:
See the examples below.
The following pool, tank, has low space consumption so one VDEV is removed.
# zpool list tank
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
tank 928G 28.1G 900G 3% 1.00x ONLINE
# zpool status tank
pool: tank
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c4t2d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
errors: No known data errors
-
# zpool remove tank mirror-1
# zpool status tank
pool: tank
state: ONLINE
status: One or more devices are being removed.
action: Wait for the resilver to complete.
Run 'zpool status -v' to see device specific details.
scan: resilver in progress since Sun Apr 15 20:58:45 2018
28.1G scanned
3.07G resilvered at 40.9M/s, 21.83% done, 4m35s to go
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c4t2d0 ONLINE 0 0 0
mirror-1 REMOVING 0 0 0
c1t7d0 REMOVING 0 0 0
c5t3d0 REMOVING 0 0 0
errors: No known data errors
Run the zpool iostat command to verify that data is being written to the remaining VDEV.
# zpool iostat -v tank 5
capacity operations bandwidth
pool alloc free read write read write
------------------------- ----- ----- ----- ----- ----- -----
tank 28.1G 900G 9 182 932K 21.3M
mirror-0 14.1G 450G 1 182 7.90K 21.3M
c3t2d0 - - 0 28 4.79K 21.3M
c4t2d0 - - 0 28 3.92K 21.3M
mirror-1 - - 8 179 924K 21.2M
c1t7d0 - - 1 28 495K 21.2M
c5t3d0 - - 1 28 431K 21.2M
------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
------------------------- ----- ----- ----- ----- ----- -----
tank 28.1G 900G 0 967 0 60.0M
mirror-0 14.1G 450G 0 967 0 60.0M
c3t2d0 - - 0 67 0 60.0M
c4t2d0 - - 0 68 0 60.4M
mirror-1 - - 0 0 0 0
c1t7d0 - - 0 0 0 0
c5t3d0 - - 0 0 0 0
------------------------- ----- ----- ----- ----- ----- -----
In this case, a device was intended to be added as a cache device but was added a single device. The problem is identified and resolved.
# zpool status rzpool
pool: rzpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rzpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
errors: No known data errors
# zpool add rzpool c3t3d0
vdev verification failed: use -f to override the following errors:
mismatched replication level: pool uses raidz and new vdev is disk
Unable to build pool from specified devices: invalid vdev configuration
# zpool add -f rzpool c3t3d0
# zpool status rzpool
pool: rzpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rzpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
errors: No known data errors
# zpool remove rzpool c3t3d0
# zpool add rzpool cache c3t3d0
# zpool status rzpool
pool: rzpool
state: ONLINE
scan: resilvered 0 in 1s with 0 errors on Sun Apr 15 21:09:35 2018
config:
NAME STATE READ WRITE CKSUM
rzpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
cache
c3t3d0 ONLINE 0 0 0
In summary, Solaris 11.4 includes a handy new option for repurposing pool devices and resolving pool misconfiguration errors.