Talking to customers this week about the Oracle Solaris ZFS device removal feature brought up a few key points that I would like to emphasize as a follow-on to this previous blog:

• Mirrored pool configurations have always had the most device flexibility because you can add vdevs, detach devices, and replace smaller devices or LUNs with larger devices.
• You could even have a mirrored pool configuration with a uneven number of devices and even mirror across different sized disks. The mirrored capacity would be equivalent to the smallest size disk when mirroring across different sized disks.
• Devices can be detached if you need to temporarily borrow a pool device for another task but its risky to remove redundancy from an active pool on a running system.

Customers who have leveraged this flexibility for years now are asking what exactly is the difference between detaching a device or removing device? This difference is best illustrated with a mirrored storage pool. The key difference is that detaching a device from a mirrored pool does not reduce the pool capacity. Removing a mirrored vdev in the Solaris 11.4 release does reduce the pool capacity. See the example below.

Also illustrating my point that ZFS always had device flexibility was a comment from a customer a few years ago, who said  about ZFS device removal, "Forget it...I don't need it any more!"

I have 2 mirrored pools, tank and tank-clone, roughly the equivalent amount of data and pool capacity:

# zpool list tank tank-clone
NAME        SIZE  ALLOC  FREE  CAP  DEDUP  HEALTH  ALTROOT
tank        928G  56.6G  871G   6%  2.00x  ONLINE  -
tank-clone  928G  56.3G  872G   6%  1.00x  ONLINE  -
# zpool status tank tank-clone
  pool: tank
 state: ONLINE
  scan: resilvered 19.8M in 12s with 0 errors on Mon Apr 16 14:41:28 2018

config:

        NAME                      STATE      READ WRITE CKSUM
        tank                      ONLINE        0     0     0
          mirror-0                ONLINE        0     0     0
            c3t2d0                ONLINE        0     0     0
            c4t2d0                ONLINE        0     0     0
          mirror-3                ONLINE        0     0     0
            c4t4d0                ONLINE        0     0     0
            c3t4d0                ONLINE        0     0     0

errors: No known data errors

  pool: tank-clone
 state: ONLINE
  scan: none requested
config:

        NAME        STATE      READ WRITE CKSUM
        tank-clone  ONLINE        0     0     0
          mirror-0  ONLINE        0     0     0
            c3t0d0  ONLINE        0     0     0
            c6t0d0  ONLINE        0     0     0
          mirror-1  ONLINE        0     0     0
            c5t0d0  ONLINE        0     0     0
            c2t0d0  ONLINE        0     0     0

errors: No known data errors

Detach devices from tank. Removing pool redundancy is not recommended but to illustrate a point that detaching a device from a mirrored configuration doesn't change the pool's overall capacity:

# zpool detach tank c4t2d0
# zpool detach tank c3t4d0
# zpool list tank
NAME  SIZE  ALLOC  FREE  CAP  DEDUP  HEALTH  ALTROOT
tank  928G  56.6G  871G   6%  2.00x  ONLINE  -
# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 19.8M in 12s with 0 errors on Mon Apr 16 14:41:28 2018

config:

        NAME                      STATE      READ WRITE CKSUM
        tank                      ONLINE        0     0     0
          c3t2d0                  ONLINE        0     0     0
          c4t4d0                  ONLINE        0     0     0

errors: No known data errors

Next, remove a mirrored vdev from tank-clone and review the pool capacity change:

# zpool remove tank-clone mirror-1
# zpool status tank-clone
  pool: tank-clone
 state: ONLINE
  scan: resilvered 28.1G in 13m17s with 0 errors on Wed Apr 25 09:36:35 2018

config:

        NAME                      STATE      READ WRITE CKSUM
        tank-clone                ONLINE        0     0     0
          mirror-0                ONLINE        0     0     0
            c3t0d0                ONLINE        0     0     0
            c6t0d0                ONLINE        0     0     0

errors: No known data errors
# zpool list tank tank-clone
NAME        SIZE  ALLOC  FREE  CAP  DEDUP  HEALTH  ALTROOT
tank        928G  56.6G  871G   6%  2.00x  ONLINE  -
tank-clone  464G  56.7G  407G  12%  1.00x  ONLINE  -

In summary, Solaris 11.4  device removal brings additional flexibility for managing your data and devices with ZFS.