Error injection using dm-dust

Introduction

dm-dust is a Linux kernel module which can be used to simulate the bad  blocks behavior on a physical disk. This is done by creating a device mapper target device. The dm-dust kernel module acts as a target driver for the Device-Mapper framework in the Linux kernel stack. The target device created using dm-dust, emulates the behavior of bad sectors on the disk at arbitrary locations, and the ability to enable the emulation of the failures at an arbitrary time. At a given time, the user can send a message to the target device to start failing read requests on specific blocks (to emulate the behavior of a hard disk drive with bad sectors). Once the badblocks are added, dm-dust provides the commands to enable and disable the read failures on the added badblocks. If the read failures on these configured badblocks are not enabled, the specific blocks are treated as good. If it is enabled, then these blocks are treated as bad.

Implementation

The dm-dust kernel module uses an RB tree (Red Black tree) to maintain the badblocks list. Whenever the module receives the addbadblock command, the particular block number is inserted into the RB tree. Whenever a read request is received for any block, the RB tree is checked first to see if the block number exists in the RB tree. If the block number is not found in the list, then the read is forwarded as normal to the next layer. If the block number exists in the RB tree, then a check is made to see if the fail_read_on_bad_block is enabled or not. If it is enabled a read I/O error is returned i.e EIO.

Read behavior on bad blocks

Once the bad blocks are added and the failure behavior is enabled, the reads of blocks in the “bad block list” will fail with EIO (“Input/output error”).

Write behavior on bad blocks

Writes to the disk blocks in the “bad block” list will result in the following:

1. Remove the block from the “bad block list”
2. Successfully complete the write

This behavior of removing the block from the badblock list and successfully completing the write is done to emulate the behavior of re-mapped sector on a hard disk and is transparent to the rest of the system.

dm-dust in the Kernel Stack

Commands in the dm-dust module used with dmsetup utility

1. addbadblock: This command is used to add a specific block to the list of bad blocks
2. removebadblock: Can be used to remove a specific block from the list of bad blocks
3. queryblock: Used to see if a particular block is in the list of bad blocks
4. disable: Disables the read failures on specified bad blocks, subsequent reads on these blocks will complete successfully
5. enable: Enables the read failures on specified bad blocks, subsequent reads on these blocks will return an I/O error
6. countbadblocks: This will give the number of bad blocks in the RB tree
7. clearbadblocks: This will clear all bad blocks from the RB tree. All subsequent reads will complete successfully
8. listbadblocks: This command will give the list of all the bad blocks in the RB tree
9. quiet: Enable quiet mode which suppresses log messages from getting logged into /var/log/messages for addbackblock and removebadblock commands. Quiet mode will still log messages for countbadblocks and queryblock commands

Use Cases

By emulating read I/O failures, dm-dust can be used to test specific I/O error code paths that get exercised when I/O errors are encountered on certain LBA ranges.

For instance we nocited that some NVME flash cards were returning corrupted data or emitting media errors. dm-dust was utilised in an effort to ensure better robustness by testing the error code paths used when these errors were returned.

Similarly we also encountered a scenario where some NVME flash drives used by Exadata were experiencing an I/O hang for up to 30 minutes. These hangs were reported after I/O errors and aborts were seen, dm-dust was used to help reproduce these errors and ultimately further improve code robustness.

UEK Versions

The dm-dust kernel module has been integrated to UEK5, UEK6 and UEK7

Examples

1. Get the size of the device

# blockdev --getsz /dev/sdb
O/P: 33552384

2. Create the dm-dust device

# dmsetup create dust1 --table '0 33552384 dust /dev/sdb 0 512'
# lsblk
 NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
 sdb       8:16   0  500M  0 disk
  dust1 252:0    0  500M  0 dm
 sda       8:0    0 46.6G  0 disk
  sda2    8:2    0    8G  0 part [SWAP]
  sda3    8:3    0 38.4G  0 part /
  sda1    8:1    0  200M  0 part /boot/efi

3. Check the status of the device

# dmsetup status dust1

O/P dmesg:
 0 33552384 dust 252:17 bypass

4. Test reads on the device

# dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=128 oflag=direct

O/P:
128+0 records in
128+0 records ou

5. Test writes on the device

  # dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=128 oflag=direct

O/P:
128+0 records in
128+0 records out

6. Add bad blocks

# dmsetup message dust1 0 addbadblock 90
# dmsetup message dust1 0 addbadblock 80
# dmsetup message dust1 0 addbadblock 55
# dmsetup message dust1 0 addbadblock 75

O/P in dmesg:
[89684.543718] device-mapper: dust: dust_add_block: badblock added at block 90
[89684.573391] device-mapper: dust: dust_add_block: badblock added at block 80
[89684.602177] device-mapper: dust: dust_add_block: badblock added at block 55
[89684.630031] device-mapper: dust: dust_add_block: badblock added at block 76

7. Enable “fail_read_on_bad_block”

# dmsetup message dust1 0 enable

O/P in dmesg:
kernel: device-mapper: dust: enabling read failures on bad sectors

8. Reads on the badblocks

# dd if=/dev/mapper/dust1 of=/dev/null bs=512 count=100 iflag=direct
dd: error reading ‘/dev/mapper/dust1’: Input/output error
55+0 records in
55+0 records out
28160 bytes (28 kB) copied, 0.0431255 s, 653 kB/s

9. Write to badblocks

# dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=100 oflag=direct
100+0 records in
100+0 records out
51200 bytes (51 kB) copied, 0.403964 s, 127 kB/s

O/P dmesg:
[90251.095584] device-mapper: dust: block 55 removed from badblocklist by write
[90251.252730] device-mapper: dust: block 76 removed from badblocklist by write
[90251.303287] device-mapper: dust: block 80 removed from badblocklist by write
[90251.369163] device-mapper: dust: block 90 removed from badblocklist by write

10. Remove badblocks

# dmsetup message dust1 0 removebadblock 90
# dmsetup message dust1 0 removebadblock 80
# dmsetup message dust1 0 removebadblock 55
# dmsetup message dust1 0 removebadblock 76

O/P dmesg:
  [154826.777999] device-mapper: dust: dust_remove_block: badblock removed at block 90
  [154835.161356] device-mapper: dust: dust_remove_block: badblock removed at block 80
  [154839.143604] device-mapper: dust: dust_remove_block: badblock removed at block 55
  [154844.424102] device-mapper: dust: dust_remove_block: badblock removed at block 76

11. Number of badblocks on the device

# dmsetup message dust1 0 countbadblocks

O/P dmesg:
 [155543.511012] device-mapper: dust: countbadblocks: 4 badblock(s) found

12. Query badblocks

# dmsetup message dust1 0 queryblock 90
# dmsetup message dust1 0 queryblock 45

O/P dmesg:
 [155770.624834] device-mapper: dust: dust_query_block: block 90 found in badblocklist
 [155776.975519] device-mapper: dust: dust_query_block: block 45 not found in badblocklist

13. Clear badblocks

# dmsetup message dust1 0 clearbadblocks

O/P dmesg:
 [156008.800436] device-mapper: dust: dust_clear_badblocks: badblocks cleared

14. Quiet mode

# dmsetup message dust1 0 quiet
# dmsetup status dust1

O/P dmesg:
 0 1024000 dust 8:16 fail_read_on_bad_block quiet

15. Switch off quiet mode

# dmsetup message dust1 0 quiet
# dmsetup status dust1
 0 1024000 dust 8:16 fail_read_on_bad_block verbose

References

1. https://git.0l.de/Mirrors/linux/commit/e4f3fabd67480bf
2. https://www.kernel.org/doc/html/v5.5/admin-guide/device-mapper/dm-dust.html