Using Cfgadm with InfiniBand

Using Cfgadm with InfiniBand Welcome to OpenSolaris, and the wonders of Solaris 10.

This provides a brief introduction to using InifniBand Device management with cfgadm(1m).

Introduction

An InfiniBand (IB) device is enumerated by the IB nexus driver, ib(7D), based on the interfaces provided by IB Device M anager (IBDM). The IB nexus driver creates and initializes five types of device nodes:
    • IB Port devices
    • IB HCA service (HCA_SVC) devices
    • IB Virtual Physical Point of Attachment (VPPA) devices
    • I/O Controller (IOC)
    • IB Pseudo devices
See ib(7d), ibdm(7d), ibtl(7d) and ib(4) for details on InifiniBand nexus driv er, Device Manager respectively.

Attachment Point format

InfiniBand cfgadm plugin supports above five device nodes as 'dynamic' attachmen t points:

Device Type
Attachment Type format
Port Devices
ib::PORT_GUID,0,service-name
HCA_SVC devices
ib::HCA_GUID,0,servicename
VPPA devices
ib::PORT_GUID,P_Key,service-name
IOC devices
ib::IOC-GUID
Pseudo devices
ib::driver_name,unit-address

where
servicename    is name of the communication service
P_Key               is the Partition Key


In addition, two 'static' attachment points are supported
Static attachment
Attachment type format
IB Fabric
ib
Host Channel Adapter(s)
hca:HCA-GUID

See example below that shows all  InfiniBand devices
# cfgadm -a ib hca:2C90109764440 
Ap_Id                          Type         Receptacle   Occupant     Condition
hca:2C90109764440              IB-HCA       connected    configured   ok
ib                             IB-Fabric    connected    configured   ok
ib::2C90109764440,0,svch       IB-HCA_SVC   connected    unconfigured unknown
ib::2C90109764441,0,psvc       IB-PORT      connected    unconfigured unknown
ib::2C90109764441,ffff,ipib    IB-VPPA      connected    unconfigured unknown
ib::2C90109764442,0,psvc       IB-PORT      connected    unconfigured unknown
ib::2C90109764442,ffff,ipib    IB-VPPA      connected    unconfigured unknown
ib::daplt,0                    IB-PSEUDO    connected    configured   ok
ib::rpcib,0                    IB-PSEUDO    connected    configured   ok
# 

The example below shows how to list all kernel clients of a given InfiniBand Host Channel Adapter.
# cfgadm -x list_clients  hca:2C90109764440
Ap_Id                          IB Client                 Alternate HCA
ib::daplt,0                    daplt                     no
ib::rpcib,0                    nfs/ib                    no
-                              ibmf                      no
-                              ibdm                      no
# 

Configure/Unconfigure a IB device

Using cfgadm commands the five devices listed above could be configured for operation or unconfigured. Example below is shown for IB Pseudo device but applies for any device with the appropriate attachment point supplied as the command line argument:

Configuring a device:

# cfgadm -a ib::daplt,0             
Ap_Id                          Type         Receptacle   Occupant     Condition
ib::daplt,0                    IB-PSEUDO    connected    unconfigured unknown
# cfgadm -yc configure ib::daplt,0  
# cfgadm -a ib::daplt,0           
Ap_Id                          Type         Receptacle   Occupant     Condition
ib::daplt,0                    IB-PSEUDO    connected    configured   ok
# 
Unconfiguring a device:
# cfgadm -a ib::daplt,0
Ap_Id                          Type         Receptacle   Occupant     Condition
ib::daplt,0                    IB-PSEUDO    connected    configured   ok
# cfgadm -yc unconfigure ib::daplt,0
# cfgadm -a ib::daplt,0             
Ap_Id                          Type         Receptacle   Occupant     Condition
ib::daplt,0                    IB-PSEUDO    connected    unconfigured unknown
The example below shows how to unconfigure all kernel clients of a give InfiniBand Host Channel Adapter.
# cfgadm -x unconfig_clients hca:2C90109764440 
Unconfigure Clients of HCA /devices/ib:2C90109764440
This operation will unconfigure IB clients of this HCA
Continue (yes/no)? yes <<<<<<< 
# 

Communication Service Commands

InfiniBand Port/HCA-SVC/VPPA devices use communication services. There are certain operations allowed for communication services like adding a service, removing a service or listing known services. See examples below:

Add a communication service:
# cfgadm -x list_services ib   
PORT communication services:
                psvc

VPPA communication services:
                ipib
HCA communication services:
                svch
# cfgadm -o comm=port,service=srp -x add_service ib
# cfgadm -x list_services ib                       
PORT communication services:
                srp <<<<<<<<<<<<<<
                psvc

VPPA communication services:
                ipib
HCA communication services:
                svch
#
Delete a communication service:
# cfgadm -x list_services ib                       
PORT communication services:
                srp
                psvc

VPPA communication services:
                ipib
HCA communication services:
                svch
# cfgadm -o comm=port,service=srp -x delete_service ib
# cfgadm -x list_services ib                          
PORT communication services:
                psvc

VPPA communication services:
                ipib
HCA communication services:
                svch
# 
Note that the examples are shown only for Port Devices but are applicable to all three device types.

Other useful Commands

Two more useful commands provided by InfiniBand cfgadm plugin are:
  • update_pkey_tbls
    It updates the P_Key information inside ibtl(7d) i.e. The InifniBand Transport Layer module. ibtl(7d) reads the P_Key tables for all ports of all the HCAs seen by the host.
  • update_ioc_conf
    It updates the properties of all IOC devices if ib static attachment is supplied.If an IOC attachment point is supplied then only that IOC's properties are updated.Properties updated are:
    port-list, port-entries,  service-id, and service-name

Reference:

1. Ted Kim's Blog
2. InfiniBand Trade Association

Technorati Tag: Technorati Tag:
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

anish

Search

Top Tags
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today