Where can I/O queue up in sd/ssd
By timatworkhomeandinbetween on Mar 16, 2006
Using the opensolaris source at opensolaris.org you can see the source code to sd.c, which builds the sd and ssd scsi/fibre target drivers.
The driver keeps a count of i/o's it has accepted in the un->un_ncmds_in_driver element of the un structure which is the softstate for that disk lun. It keeps the number of i/o's it has passed down to the HBA to transport in the un->un_ncmds_in_transport variable.
New I/O's passed into sd/ssd to the transport are passed straight down to the HBA until un_ncmds_in_transport exceeds the un_throttle for this lun or during error processing when we retry commands sequentually until one succeeeds.
When we have to queue the bufs we sort them onto a linked list anchored in un->un_waitq_headp and un->un_waitq_tailp.
But there is another place that i/o can be queued before it gets to the waitq and throttles - in the xbuf layer. un->un_xbuf_attr points to the xbuf control block which is of type __ddi_xbuf_attr . This has an unsorted waitq anchored in xa_headp and xa_tailp, by default the xbuf layer only allows 256 commands through to the normal sd waitq and throttles, the rest are queued in the xbuf and un_ncmds_in_driver is incremented. So it is possible when a large number of commands to be dropped on a sd lun, un_ncmds_in_driver is very large but the waitq has less than 256 commands on it - the rest are lurking on the xbuf's queue
These xbuf queued i/o are invisible to iostat's wait column - maybe they shouldn't be?