Tuesday Jun 14, 2005

scsa1394 "symbios" workaround

From a driver writer's perspective, CPUs are amazing devices: they manage to stay extremely reliable while implementing extremely complex logic. Neither is true of most peripheral devices. It is usually not practical to put huge effort into design and testing of cheap, short-lived devices. Which to some degree explains device drivers' reputation for being too obscure, overcomplicated and continuously changing: they reflect imperfections of their hardware... and sometimes of their programmers, but that is an altogether different subject - this blog is about a bug that exists in some IEEE 1394 (FireWire) mass storage devices and how it's been worked around in the scsa1394(7D) driver.

1394 mass storage protocol is based on the Serial Bus Protocol 2. SBP-2 allows to encapsulate arbitrary command sets on buses compliant with IEEE 1212. In case of scsa1394, the IEEE 1212 compliant bus is IEEE 1394 and it transports SCSI commands submitted by SCSI target drivers, such as sd(7D) and st(7D), or via uscsi(7I) interface.

For SCSI commands with data phase, scsa1394 has to map data buffers for DMA transfer. When a buffer cannot be mapped into contiguous I/O memory, the kernel attempts to map it into non-contiguous chunks described by the ddi_dma_cookie(9S) structure. Each cookie contains a DMA address of the memory chunk and its size. Cookies are then programmed into device's DMA engine. This type of DMA transfers is known as scatter-gather.

SBP-2 provides a facility for DMA scatter-gather called page tables. A page table is an array of entries of the following format 1:

 31                                                                0
|         segment_length          |        segment_base_hi          |
|                           segment_base_lo                         |

Page table itself must be located in contiguous I/O space. Once the driver prepares an ORB 2 and attaches a valid page table to it, it signals the device to read them in and perform data transfer. SBP-2 segments fit neatly into Solaris DMA cookie concept: segment length corresponds to cookie size and segment base to cookie address.

SBP-2 does not put any restrictions, other than bit width, neither on segment length, nor the total length of segments in a page table. However, some devices would not function correctly unless page tables meet two requirements:

  • segment length == 4K;
  • total length < 128K;

Evidently, not all vendors use dedicated test software for their devices, some limit themselves to compliance testing on a few target OS/CPU. As far as we know, this bug only exists in devices based on SYM13FW\* chip series, hence known as the "symbios bug".

Solaris driver had to provide a workaround. Satisfying symbios page table limits requires that we divide cookies into 4K segments, which breaks the neat 1:1 mapping between DMA cookies and SBP-2 segments. So I had to introduce an additional mapping layer described by struct scsa1394_cmd_seg:

typedef struct scsa1394_cmd_seg {
	size_t			ss_len;
	uint64_t		ss_daddr;
	uint64_t		ss_baddr;
	t1394_addr_handle_t	ss_addr_hdl;
} scsa1394_cmd_seg_t;

ss_daddr here refers to segment's DMA address and ss_baddr to its 1394 bus address. scsa1394_cmd_dmac2seg() function is responsible for turning cookies into segments. Page table is then created from the segment array in scsa1394_sbp2_seg2pt().

If you ever used Solaris DDI DMA routines, you might think: why not just use ddi_dma_attr(9S) structure to limit cookie size by setting dma_attr_seg to 0xFFF? This is because dma_attr_seg cannot be less that base page size, so 4K would not work on SPARC (or x86 should the page size ever increase).

As if that wasn't enough, there seems to be no way to distinguish between devices with the symbios bug and devices without it. No combination of configuration ROM3 keys provides reliable indicator of a buggy device. This led me to a difficult decision to enable the symbios workaround by default, sacrificing performance for data integrity. Then I thought: we can't blacklist bad devices, but maybe we could whitelist the good ones. That is how the white list was invented:

scsa1394_bw_list_t scsa1394_sbp2_symbios_whitelist[] = {
        { SCSA1394_BW_ONE, 0x0a27 },		/\* Apple \*/
        { SCSA1394_BW_ONE, 0xd04b }		/\* LaCie \*/

It contains vendor IDs of those companies that have never, to the extent of our knowledge, produced devices based on the buggy chip. We'll be adding more to the list in the future. In order to test a device for the symbios bug, the workaround can be disabled by setting scsa1394_wrka_symbios variable to 0:

temporarily in mdb:

	# echo scsa1394_wrka_symbios/W 0 | mdb -kw

or permanently in /etc/system:

	set scsa1394:scsa1394_wrka_symbios = 0

1 These are so called unrestricted page tables. SBP-2 also defines normalized page tables, but at the time of this writing they are not used in scsa1394.

2 ORB: Operation Request Block, prepared by the initiator in I/O memory and read by the target. ORB is a polymorphic data structure: see usr/src/uts/common/sys/sbp2/defs.h for its multiple incarnations.

3 Every 1394 device contains a special address region called configuration ROM. It's a hierarchical key-value structure that describes various device attributes such as device class, vendor/device ID, etc. For those familiar with PCI, it is similar in purpose to PCI configuration space: it is basically what makes these devices self-identifying.

Technorati Tag:
Technorati Tag:
Technorati Tag:
Technorati Tag:




Top Tags
« July 2016