By Lisa Noordergraaf on Jan 10, 2010
In some previous postings I've talked a bit about the impact of IO size and alignments on the performance of flash devices. These parameters are easily controlled in a microbenchmarking environment, but can be quite difficult to determine in more complex application environments.
A lot of folks have asked me about using iostat in this context. The iostat utility is great for telling you the total number of reads and writes, but it doesn't convey information about individual sizes and alignments (start addresses) of those IOs. You can easily figure out the \*average\* IO size from iostat, but that may or may not correlate well to actual size of individual reads and writes depending on the workload.
One way of getting more visibility into IO sizes and alignments is to use
dtrace to gather the information of interest. In this post, I'll share
with you a script that has been used internally for this purpose.
This should be viewed as a "unsupported, evolving" script, and I will
post updates as we tweak it. Nonetheless, it seems to be a fairly useful
bit of code and worth sharing. A link to the full script is provided at the bottom of this post.
A few bits of info about this script...
The script instruments a function in the sd driver, and basically just histograms the sizes of IO packets. It turns out that one of the difficult parts about doing this is not getting the packet size, but gathering the information for only the devices you are interested in.
One option is to restrict by using major/minor device numbers, but that is not the most intuitive thing from a user standpoint. These also tend to vary quite a bit, so one would always be hacking up the script. We needed a better way...
Paul Riethmuller came up with the swell idea of using the product id string that is stored when a inquiry command is done. For example:
# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t1d0
/pci@0,0/pci108e,5354@1f,2/disk@1,0 1. c0t2d0 /pci@0,0/pci108e,5354@1f,2/disk@2,0 2. c0t3d0 /pci@0,0/pci108e,5354@1f,2/disk@3,0 3. c1t0d0 /pci@0,0/pci8086,3408@1/pci1000,1000@0/sd@0,0 4. c1t1d0 /pci@0,0/pci8086,3408@1/pci1000,1000@0/sd@1,0 5. c1t2d0 /pci@0,0/pci8086,3408@1/pci1000,1000@0/sd@2,0 6. c1t3d0 /pci@0,0/pci8086,3408@1/pci1000,1000@0/sd@3,0 Specify disk (enter its number): 6 selecting c1t3d0 [disk formatted] FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk format - format and analyze the disk fdisk - run the fdisk program repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels verify - read and display labels save - save new disk/partition definitions inquiry - show vendor, product and revision volname - set 8-character volume name ! - execute , then return quit format> inquiry Vendor: ATA Product: MARVELL SD88SA02 Revision: D20R
The script as currently presented below tracks the number of total IOs issued for all devices, but only keeps detailed statistics for those devices that have a Product ID starting with "MARVEL". (This happens to be the Product ID returned if you run the inquiry command on a Sun F5100 or F20 flash device. ;-)).
Also note that currently the script produces limited output every 5 seconds, but the full histogram of IO sizes is only printed when one hits \^C. Obviously this can be changed to suit. Also note I'm not much of a dtrace hacker- you can find much better in other places at Sun. ;-)
Acknowledgements: Thank you, Paul Riethmuller, for all the good bits and ideas. Any broken bits are due solely to me.
Some sample output:
# ./align.d 4.966478729 seconds elapsed 2 IOs issued 5.000003686 seconds elapsed 2465 IOs issued 2437 Marvell IOs issued 5 RDs 2432 WRs 0 Marvell IOs misaligned start 5 Marvell IOs non-multiple of 4KB size 0 Percent non-4k Marvell IOs \^C 11.461851773 seconds elapsed 2774 Tot IOs issued 2740 Marvell IOs issued 0 Marvell IOs misaligned start addr 5 Marvell IOs non-multiple of 4KB 0 Percent non-4k Marvell IOs MARVELL WRITE IO IO_size Count 36 0 512 0 0 2 131072 2733 MARVELL READ IO IO_size Count 0 0 131072 0 36 2 512 3 MARVELL ALL IO IO_size Count 0 2 36 2 512 3 131072 2733
Updated 2/14/2010, minor bugfixes (errors in comments):