dm-dust is a Linux kernel module which can be used to simulate the bad blocks behavior on a physical disk. This is done by creating a device mapper target device. The dm-dust kernel module acts as a target driver for the Device-Mapper framework in the Linux kernel stack. The target device created using dm-dust, emulates the behavior of bad sectors on the disk at arbitrary locations, and the ability to enable the emulation of the failures at an arbitrary time. At a given time, the user can send a message to the target device to start failing read requests on specific blocks (to emulate the behavior of a hard disk drive with bad sectors). Once the badblocks are added, dm-dust provides the commands to enable and disable the read failures on the added badblocks. If the read failures on these configured badblocks are not enabled, the specific blocks are treated as good. If it is enabled, then these blocks are treated as bad.
The dm-dust kernel module uses an RB tree (Red Black tree) to maintain the badblocks list. Whenever the module receives the addbadblock
command, the particular block number is inserted into the RB tree. Whenever a read request is received for any block, the RB tree is checked first to see if the block number exists in the RB tree. If the block number is not found in the list, then the read is forwarded as normal to the next layer. If the block number exists in the RB tree, then a check is made to see if the fail_read_on_bad_block
is enabled or not. If it is enabled a read I/O error is returned i.e EIO.
Once the bad blocks are added and the failure behavior is enabled, the reads of blocks in the “bad block list” will fail with EIO (“Input/output error”).
Writes to the disk blocks in the “bad block” list will result in the following:
This behavior of removing the block from the badblock list and successfully completing the write is done to emulate the behavior of re-mapped sector on a hard disk and is transparent to the rest of the system.
dmsetup
utilityaddbackblock
and removebadblock
commands. Quiet mode will still log messages for countbadblocks
and queryblock
commandsBy emulating read I/O failures, dm-dust can be used to test specific I/O error code paths that get exercised when I/O errors are encountered on certain LBA ranges.
For instance we nocited that some NVME flash cards were returning corrupted data or emitting media errors. dm-dust was utilised in an effort to ensure better robustness by testing the error code paths used when these errors were returned.
Similarly we also encountered a scenario where some NVME flash drives used by Exadata were experiencing an I/O hang for up to 30 minutes. These hangs were reported after I/O errors and aborts were seen, dm-dust was used to help reproduce these errors and ultimately further improve code robustness.
The dm-dust kernel module has been integrated to UEK5, UEK6 and UEK7
# blockdev --getsz /dev/sdb O/P: 33552384
# dmsetup create dust1 --table '0 33552384 dust /dev/sdb 0 512' # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb 8:16 0 500M 0 disk dust1 252:0 0 500M 0 dm sda 8:0 0 46.6G 0 disk sda2 8:2 0 8G 0 part [SWAP] sda3 8:3 0 38.4G 0 part / sda1 8:1 0 200M 0 part /boot/efi
# dmsetup status dust1 O/P dmesg: 0 33552384 dust 252:17 bypass
# dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=128 oflag=direct O/P: 128+0 records in 128+0 records ou
# dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=128 oflag=direct O/P: 128+0 records in 128+0 records out
# dmsetup message dust1 0 addbadblock 90 # dmsetup message dust1 0 addbadblock 80 # dmsetup message dust1 0 addbadblock 55 # dmsetup message dust1 0 addbadblock 75 O/P in dmesg: [89684.543718] device-mapper: dust: dust_add_block: badblock added at block 90 [89684.573391] device-mapper: dust: dust_add_block: badblock added at block 80 [89684.602177] device-mapper: dust: dust_add_block: badblock added at block 55 [89684.630031] device-mapper: dust: dust_add_block: badblock added at block 76
# dmsetup message dust1 0 enable O/P in dmesg: kernel: device-mapper: dust: enabling read failures on bad sectors
# dd if=/dev/mapper/dust1 of=/dev/null bs=512 count=100 iflag=direct dd: error reading ‘/dev/mapper/dust1’: Input/output error 55+0 records in 55+0 records out 28160 bytes (28 kB) copied, 0.0431255 s, 653 kB/s
# dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=100 oflag=direct 100+0 records in 100+0 records out 51200 bytes (51 kB) copied, 0.403964 s, 127 kB/s O/P dmesg: [90251.095584] device-mapper: dust: block 55 removed from badblocklist by write [90251.252730] device-mapper: dust: block 76 removed from badblocklist by write [90251.303287] device-mapper: dust: block 80 removed from badblocklist by write [90251.369163] device-mapper: dust: block 90 removed from badblocklist by write
# dmsetup message dust1 0 removebadblock 90 # dmsetup message dust1 0 removebadblock 80 # dmsetup message dust1 0 removebadblock 55 # dmsetup message dust1 0 removebadblock 76 O/P dmesg: [154826.777999] device-mapper: dust: dust_remove_block: badblock removed at block 90 [154835.161356] device-mapper: dust: dust_remove_block: badblock removed at block 80 [154839.143604] device-mapper: dust: dust_remove_block: badblock removed at block 55 [154844.424102] device-mapper: dust: dust_remove_block: badblock removed at block 76
# dmsetup message dust1 0 countbadblocks O/P dmesg: [155543.511012] device-mapper: dust: countbadblocks: 4 badblock(s) found
# dmsetup message dust1 0 queryblock 90 # dmsetup message dust1 0 queryblock 45 O/P dmesg: [155770.624834] device-mapper: dust: dust_query_block: block 90 found in badblocklist [155776.975519] device-mapper: dust: dust_query_block: block 45 not found in badblocklist
# dmsetup message dust1 0 clearbadblocks O/P dmesg: [156008.800436] device-mapper: dust: dust_clear_badblocks: badblocks cleared
# dmsetup message dust1 0 quiet # dmsetup status dust1 O/P dmesg: 0 1024000 dust 8:16 fail_read_on_bad_block quiet
# dmsetup message dust1 0 quiet # dmsetup status dust1 0 1024000 dust 8:16 fail_read_on_bad_block verbose
Previous Post