Understanding FC Adapter HBA Drivers Under Linux SCSI Subsystem using qla2xxx (QLogic) HBA driver

Overview

This document offers a general overview of the SCSI subsystem, including the SCSI protocol, Fibre Channel (FC) and Host Bus Adapter (HBA) drivers. The purpose is to build a foundational understanding of these components and their key roles. Additionally, it aims to map the interfaces of these components to illustrate their interactions and functional flow. As part of this discussion, we will also revisit SCSI naming and addressing in the context of Fibre Channel environments.

Introduction

SCSI (Small Computer System Interface) (pronounced “scuzzy”) does indeed sound like an old, rough character, and in many ways, it is. Despite being a decades-old technology, SCSI remains widely used because it is an exceptionally stable, reliable, and versatile protocol.

Originally developed in the 1980s, basic SCSI is a block-level protocol that connects disks to a Host Bus Adapter (HBA) using flat cables. However, this version of SCSI was not suitable for Storage Area Networks (SANs) due to limitations on its ability to operate over distance. To address these limitations new protocols were introduced – Fibre Channel (FC) and iSCSI as mechanisms that would extend SCSI over longer distances and through networked environments.

• Fibre Channel (FC): Provides a high-speed, low-latency transport mechanism for SCSI commands over Fiber optic cables.

• iSCSI: Transports SCSI commands over standard TCP/IP networks, making it more cost-effective.

SCSI has evolved through multiple iterations, from parallel SCSI to modern SAS (Serial Attached SCSI), SRP, SATA and iSCSI (Internet SCSI). You can read the history of SCSI Interfaces here. These advancements have kept SCSI relevant, allowing it to maintain backward compatibility while offering faster speeds, improved data integrity, and robust error handling.

Fibre Channel Protocol

Fibre Channel Protocol (FCP) is the transport mechanism responsible for carrying SCSI commands over Fibre Channel networks. It enables high-speed, reliable communication between servers and storage devices by serving as the transport layer for SCSI-based operations.

FCP was standardized by the T11 technical committee, which is responsible for developing and maintaining Fibre Channel standards under the International Committee for Information Technology Standards (INCITS). In FCP, SCSI commands are encapsulated within Fibre Channel frames, which are the smallest and primary unit of data transmission and routing in an FC network. Fibre Channel (FC) Frame Encapsulation“RFC 3643”

Each Fibre Channel frame consists of:

• A Start of Frame (SOF)
• An End of Frame (EOF)
• Headers used for addressing and control
• A Payload, which carries the encapsulated SCSI command and data.
• A Cyclic Redundancy Check (CRC) error detection

FCP transactions involve a sequence of FC frames that carry the SCSI commands, associated data, and response messages. By encapsulating SCSI operations in this manner, FCP ensures high-speed, reliable, and efficient communication between hosts and storage devices across a Fibre Channel fabric. This capability makes FCP a foundational protocol for enterprise-class Storage Area Networks (SANs).

The Linux kernel supports FCP through several drivers, each tailored for specific hardware platforms:

• lpfc: A driver for Emulex (Emulex LPeXXX) Fibre Channel Host Bus Adapters (HBAs), providing connectivity between the host and Fibre Channel storage targets.
• qla2xxx: A driver for Marvell (Formerly QLogic) (QLogic QLEXXX) Fibre Channel Host Bus Adapters (HBAs), supporting communication between servers and SAN-attached storage devices using FCP.
• fcp: A driver used on IBM z/System platforms to support FCP communication with attached storage devices.
• FCoE and fcoeadm: Fibre Channel over Ethernet (FCoE) encapsulates FCP over Ethernet networks. The fcoe kernel module and the fcoeadm userspace utility provide support and management for FCoE devices.

Within the Linux kernel, the SCSI subsystem depends on FCP to deliver commands and manage device interactions. Key components involved in this process include:

• scsi_ioctl: Handles all IOCTL operations for SCSI devices.
• scsi_cmnd: Represents an individual SCSI command within the kernel.

By abstracting the underlying transport layer, FCP enables the Linux SCSI subsystem to efficiently support Fibre Channel-based storage solutions across a wide range of hardware.

Fibre channel Port and addressing

It is important to understand Fibre Channel (FC) ports and addressing before delving into the SCSI system and HBA driver interface, as this foundation helps visualize the flow and interaction between the FC and SCSI subsystems.

A Host Bus Adapter (HBA) or Storage device includes a hardware interface commonly referred to as a port. Each port acts as a communication endpoint and is uniquely identified to establish and manage connections within the network. A basic topology and connection types are illustrated in Figure 1. There are three general types of Fibre Channel ports:

N_Port (Node Port): Node Port (on a device like HBA or storage controller)

F_Port (Fabric Port): Fabric Port (on a switch that connects to an N_Port)

E_Port (Expansion Port): Expansion Port (used for switch-to-switch connection in an inter-switch link)

E_Port, F_Port, and N_Port are three common types of Fibre Channel (FC) ports. However, Fibre Channel defines several other port types as well. For a complete list and details, refer to “RFC 4172” and Fibre Channel Wikipedia page

Figure 1 represents a simple Storage Area Network, where a dual port HBA has one port (N_Port1) connected to switchA (F_Port1), and second port (N_Port2) connected to SwitchB(F_Port1) to create a failsafe path to one port (N_Port1) of the Target Storage Array.

 

To be able to send SCSI commands over Fibre Channel, we need to answer some basic questions:

• How are devices identified and addressed in Fibre Channel networks?

Each HBA port (initiator) and storage port (target) is assigned a WWN (World Wide Name) These WWNs are like a device’s permanent serial number. on Fabric side When a device logs into the Fibre Channel (FC) fabric, it is dynamically assigned a 24-bit FC Address known as the destination address [D_ID], which is used for routing within the fabric.

WWNs: Every node has two types of globally unique 64-bit identifiers. Together, the WWPN and WWNN play a critical role in identifying devices and managing persistent paths in Fibre Channel-based storage networks.

So, what are these terms—WWPN and WWNN? World Wide Port Name – Wikipedia

WWNN (World-Wide Node Name): is a globally unique 64-bit identifier equipped with the entire device or node. All ports belonging to the same device typically share the same WWNN. It’s like the “device name,” while the WWPNs are the “port addresses.”

WWPN (World-Wide Port Name): is a globally unique 64-bit identifier equipped with a specific port on a Fibre Channel device (like a host bus adapter or a storage port). Each physical port has its own WWPN, which is used during login and communication in the FC fabric.

A 64-bit address is logically and uniquely divided into three major parts:

- An Organizationally Unique Identifier (OUI)   – 24 bits
- Domain or Identifier Type             – 8 bits
- Vendor-Specific Identifier            – 32 bits

Let’s look at an example WWPN: 21:00:24:FF:11:44:33:B2

• OUI (Vendor ID): 00:24:FF - Belongs to QLogic Corporation
• Identifier Type: 21 - Node or port type
• Vendor-Specific ID: 11:44:33:B2 - Unique to the vendor’s device/port

Fibre channel Port addressing in a switch fabric:

Eventually, a distinct 24-bit address format is used for port addressing within a switched Fibre Channel fabric. This address, known as the N_Port_ID, is separate from the World-Wide Name (WWN) and is primarily used for routing frames within the fabric. 24-bit FC Address (or N_Port_ID)

Domain_ID (8 bits) : Area_ID (8 bits) : Port_ID (8 bits)

Example address: 0x070B1F

Domain ID    8 bits    07    Identifies fabric domain.
Area   ID    8 bits    0B    Logical grouping or area on a switch.
Port   ID    8 bits    1F    Identifies the specific port on the switch.

• How are SCSI Commands routed in Fibre Channel? Every initiator (HBA port) and target (storage port) has a WWPN (World Wide Port Name), a 64-bit globally unique identifier. The SCSI initiator sends commands addressed to a specific target WWPN. SCSI commands are encapsulated in FCP frames and routed using the 24-bit FC address (D_ID) of the target device. The initiator sends a FCP_CMND frame to the D_ID of the target.

SCSI Subsystem

The SCSI subsystem is a well-defined component of the Linux kernel, established since Linux version 2.4.

It provides a standardized framework for communication between peripheral devices (such as storage devices) and the host system. Basic concepts of the Linux SCSI subsystem, refer to the SCSI-2.4-HOWTO

Under Linux, SCSI devices follow a four-level hierarchical addressing scheme, which helps uniquely identify devices on the system:

1. SCSI adapter number [host] – The SCSI host adapter (HBA or initiator) number.
2. Channel number [bus] – A legacy value, often 0 or not used.
3. Id number [target] – The ID of the remote target (typically derived from the FC WWPN).
4. LUN [lun] – An individual I/O device is called a LU(logical unit) within the target (like a volume or partition), Each LU comes with an address within a target called a Logical Unit Number (LUN).

According to the Linux kernel documentation, the SCSI subsystem follows a three-layer architecture, consisting of the following layers:

1. Upper Layer: Handles the SCSI command set and provides interfaces for filesystems and applications to interact with SCSI devices.
2. Middle Layer: Acts as a bridge between the upper and lower layers, managing SCSI commands and handling error recovery.
3. Lower Layer: Communicates directly with hardware through low-level drivers for specific SCSI devices (like Host Bus Adapters and HBA drivers).

In SCSI terminology a bus that carries SCSI commands is referred to as a “transport”, and a controller that connects to such a bus called a Host Bus Adapter (HBA).

 

Let us provide an overview of how SCSI and Fibre channel works.

The Host Bus Adapter (HBA) driver’s basic purpose is to manage the physical connect between the system and SCSI devices and translate SCSI commands into hardware specific instructions.

A Physical HBA enables a server to have a Fibre Channel (FC) SAN attachment.

The Host Channel Adapter (HCA), unlike HBA, HCA serves a similar purpose but is primarily associated with InfiniBand (IB) networks.

While HBA and HCA are both types of network adapters, they serve different purposes, and operate using different technologies:

• HBA (Host Bus Adapter):
  • Connects servers to storage devices in a SAN (Storage Area Network).
  • Typically uses SCSI, Fibre Channel, or iSCSI protocols.
  • Manages data transfer between the host and storage devices, handling tasks like data encapsulation and command processing.
• HCA (Host Channel Adapter):
  • Connects servers to InfiniBand networks, which are high-performance computing networks designed for low latency and high throughput.
  • Supports RDMA (Remote Direct Memory Access) to enable direct memory access between servers without CPU intervention, reducing latency and CPU overhead.
  • Provides features like data integrity and congestion management, crucial for HPC (High-Performance Computing) and data-centres.

HCA driver

Provides a high-speed interface to InfiniBand fabrics, enabling high-throughput and low-latency data transfers. Supports RDMA (Remote Direct Memory Access), allowing direct memory access between servers without CPU involvement.

HCAs are focused on high-performance networking like InfiniBand (IB) and RDMA.

More About MLX Driver Here and mlx4 code

HBA driver

A “SCSI Host Bus Adapter driver” in Linux refers to a low-level driver that manages the communication between a computer’s internal bus and a SCSI (Small Computer System Interface) host adapter card, allowing the system to interact with attached SCSI devices like hard drives or tape drives, Essentially, it acts as the bridge between the computer and the SCSI devices by handling data transfer and command execution on the SCSI bus.

The primary role is to translate SCSI commands from the Linux SCSI mid-level driver into hardware-specific operations on the HBA card, managing data transfer, interrupt handling, and device addressing.

• The qla2xxx driver is a widely used HBA driver for QLogic Fibre Channel adapters.
• It provides a complete stack for interacting with FC devices, handling SCSI commands, and managing data transfers.
• The driver interfaces with the SCSI mid-layer and the FC transport layer, enabling the system to communicate with SAN devices.

Data Structures in SCSI Subsystem

The SCSI subsystem in the Linux kernel relies on four key data structures to manage the interaction between the mid-layer and lower-level drivers (LLDs). These structures are:

• struct scsi_host_template
• struct Scsi_Host
• struct scsi_device
• struct scsi_cmnd

Template creation

The scsi_host_template is a structure that defines how the SCSI mid-layer communicates with a lower-level driver (LLD). The qla2xxx driver registers itself with the SCSI mid-layer using this static instance of this structure, This template defines a set of function pointers and attributes that instruct the SCSI mid-layer on how to interact with the qla2xxx driver, specifically how to send commands, handle errors, and manage connected devices.

One of the most critical functions defined in this template is “queuecommand”, which is responsible for submitting SCSI commands to the hardware.

An example qla2xxx_template (defined in qla_os.c)

struct scsi_host_template qla2xxx_driver_template = {
        .module                         = THIS_MODULE,
        .name                           = QLA2XXX_DRIVER_NAME,
        .queuecommand               = qla2xxx_queuecommand,
        .eh_timed_out                   = fc_eh_timed_out,
        .eh_abort_handler               = qla2xxx_eh_abort,
        .eh_should_retry_cmd            = fc_eh_should_retry_cmd,
        .eh_device_reset_handler        = qla2xxx_eh_device_reset,
        .sdev_configure                 = qla2xxx_sdev_configure,
        .sdev_init                      = qla2xxx_sdev_init,
        .sdev_destroy                   = qla2xxx_sdev_destroy,
        .scan_finished                  = qla2xxx_scan_finished,
        .scan_start                     = qla2xxx_scan_start,
        .change_queue_depth             = scsi_change_queue_depth,
        .map_queues                     = qla2xxx_map_queues,
        .this_id                        = -1,
        .cmd_per_lun                    = 3,
        .sg_tablesize                   = SG_ALL,
        .max_sectors                    = 0xFFFF,
}

Instances of this structure are used to deliver SCSI commands to the lower-level driver (LLD) and to return responses back to the SCSI mid-layer. The SCSI mid-layer prepares a scsi_cmnd structure when it needs to issue a command to a device. This structure encapsulates all the necessary information for the command’s execution and is presented below for reference. The mid-layer also ensures that the number of queued commands does not exceed the limit specified by the driver.

struct scsi_cmnd {
    struct scsi_device *device;     // Pointer to the associated SCSI device
    unsigned char cmd_len;      // Length of the command in bytes
    enum dma_data_direction sc_data_direction;
    unsigned char cmnd[32];     // The actual SCSI command SCSI CBD
    struct scsi_data_buffer sdb;    // define the operation to perform
    struct scsi_data_buffer *prot_sdb;  // define the operation to perform
    unsigned underflow;         // Minimum bytes expected to be transferred
    unsigned transfersize;          // Size of each transfer segment
    unsigned resid_len;             // residual count
    unsigned sense_len;
    unsigned char *sense_buffer;        // Buffer for SCSI sense data
    unsigned long state;            // Command completion state
    void *host_scribble;            // Scratchpad for host adapter use
    int result;                 // Status code from lower level driver
}

QLA2XXX Driver (QLogic FC HBA):

The qla2xxx driver is a hardware-specific Fibre Channel Host Bus Adapter (HBA) driver designed for QLogic adapters, responsible for directly managing interactions with the HBA hardware.

It handles the end-to-end process of sending SCSI commands by translating them into Fibre Channel frames for transmission, processes hardware-generated interrupts to manage events such as command completions and errors, communicates with the adapter’s firmware for initialization, configuration, and diagnostics, and manages various HBA events – including link status changes, port logins/logouts, and error recovery – to ensure stable and efficient Fibre Channel operations, when it begins scanning for devices on this host adapter.

Integration with the HBA and SCSI FC Transport Layers

scsi_transport_fc.c: A Generic Transport Layer for Fibre Channel and is part of the SCSI Transport Class infrastructure, it is bridge between FC devices to the SCSI mid-layer. The SCSI mid-layer (aka SCSI core) which provides standardized SCSI command and transport interfaces.

This is the Fibre Channel (FC) SCSI transport layer code in the Linux kernel. It provides a common FC transport interface for SCSI devices over Fibre Channel.

This layer handles generic FC protocol operations, device discovery, link management, and SCSI command transport semantics over FC.

The driver registers with the SCSI mid-layer and the scsi_transport_fc framework as a Fibre Channel Host Bus Adapter (HBA) driver.

It provides a transport template that integrates with the Fibre Channel transport class, enabling the driver to utilize the standardized interfaces offered by scsi_transport_fc for managing device discovery, addressability, and SCSI command transport over Fibre Channel.

This layered architecture promotes hardware abstraction and facilitates seamless support for various HBA vendors within the unified SCSI framework.

The qla2xxx driver, designed specifically for QLogic Fibre Channel (FC) Host Bus Adapters (HBAs), manages hardware-specific tasks such as command transmission to the HBA, interrupt handling, firmware operations, and hardware event processing.

It registers itself as a Fibre Channel transport driver with both the SCSI mid-layer and the FC transport layer, where scsi_transport_fc.c offers a standardized FC transport interface shared across all FC HBA drivers. During system initialization, the qla2xxx driver installs its transport template into the FC transport stack (scsi_transport_fc) and utilizes this layer to process SCSI commands over Fibre Channel, relying on the generic transport mechanisms implemented in scsi_transport_fc.c.

Function Flow Between SCSI Core and FC HBA Driver

The typical initialization flow for the qla2xxx driver triggers when the driver is loaded during the PCI probe (pci_device_id entry match in qla2xxx_pci_tbl). Within the qla2xxx_probe_one() Probe, the driver allocates a SCSI host structure using scsi_host_alloc(&qla2xxx_driver_template, size).

Similarly, the lpfc low-level driver registers its interface with the SCSI mid-layer using the lpfc_template for physical ports and lpfc_vport_template for virtual ports. The lpfc_queuecommand function is functionally analogous to qla2xxx_queuecommand, as both are responsible for submitting SCSI commands to the underlying Fibre Channel hardware.

 

 

In the first step, the host is registered with the SCSI mid-layer by calling scsi_add_host(shost, &pdev->dev). Following this, target discovery is initiated. The function qla2x00_do_dpc() begins the discovery process of remote Fibre Channel ports (targets). The SCSI mid-layer then scans for LUNs by issuing REPORT LUNS and INQUIRY SCSI commands.

As shown in the Figure 3, qla2xxx_scan_start() is defined in the QLogic driver (qla2xxx) and is specific to that hardware.

It is not a SCSI core function; it is provided by the HBA driver. This means the SCSI core will call qla2xxx_scan_start().

Initialization phase (drivers/scsi/qla2xxx/qla_init.c), here it will prepare hardware or firmware for device discovery and set internal flags to track scan stat.

while discovering remote targets,qla2xxx driver fills in struct fc_rport_identifiers, which includes rport_ids.node_name and rport_ids.port_name. This sequence illustrates how the qla2xxx driver interfaces with both the Fibre Channel (FC) transport layer and the SCSI mid-layer to register newly discovered remote FC ports, enabling them for I/O operations.

The ultimate goal is to get to fc_remote_port_add(), which links the remote port into the SCSI subsystem.

During the discussion on Fibre Channel (FC) addressing, we established that FC addressing is used at the transport/network layer, while SCSI addressing serves the command/protocol layer. SCSI commands are routed over Fibre Channel using these low-level addressing implementations.

When a new remote port (a target device) is discovered on the fabric, the HBA driver qla2xxx creates an fc_rport (Fibre Channel remote port) by invoking the function fc_remote_port_add(). This fc_rport encapsulates the FC identity of the remote device (such as port_id, port_name, and node_name) and enables the upper SCSI layers to initiate and manage communication with that remote port using SCSI commands.

struct fc_rport {
   u64 node_name;  // WWNN of the remote port
   u64 port_name;  // WWPN of the remote port
   u32 port_id;    // Fibre Channel address of the remote port
}

Once addressing is established using WWPNs and WWNNs, SCSI commands are encapsulated into Fibre Channel frames and routed across the network. They are primarily managed and exposed through Fibre Channel (FC) transport classes.

For each detected target, the driver calls scsi_scan_host(shost) to begin probing for LUNs. This process triggers qla2x00_update_fcport(ha) and internally, the SCSI core invokes scsi_add_device() to register each discovered device.

The complete flow inside the SCSI mid-layer follows the path defined in drivers/scsi/scsi_scan.c, where the function scsi_scan_host() leads to scsi_scan_target(), which in turn calls scsi_add_device() to finalize the addition of each SCSI device.

 

In the final step, The queuecommand function is defined in the qla2xxx low-level driver (LLD). When a command needs to be issued, scsi_dispatch_cmd() invokes queuecommand, and once the command is completed, the driver calls scsi_done() to notify the SCSI mid-layer.

References:

• https://en.wikipedia.org/wiki/Fibre_Channel
• https://github.com/torvalds/linux/tree/master/drivers/scsi