News, tips, partners, and perspectives for the Oracle Linux operating system and upstream Linux kernel work

Using Tracepoints to Debug iSCSI

Using Tracepoints to Debug iSCSI Modules

Oracle Linux kernel developer Fred Herard offered this blog post on how to use tracepoints with iSCSI kernel modules.

The scsi_transport_iscsi, libiscsi, libiscsi_tcp, and iscsi_tcp modules have been modified to leverage Linux Kernel Tracepoints to capture debug messages. Before this modification, debug messages for these modules were simply directed to syslog when enabled. This enhancement gives users the option to use Tracepoint facility to dump enabled events (debug messages) into ftrace ring buffer.

The following tracepoint events are available:

# perf list 'iscsi:*'

List of pre-defined events (to be used in -e):

  iscsi:iscsi_dbg_conn                               [Tracepoint event]
  iscsi:iscsi_dbg_eh                                 [Tracepoint event]
  iscsi:iscsi_dbg_session                            [Tracepoint event]
  iscsi:iscsi_dbg_sw_tcp                             [Tracepoint event]
  iscsi:iscsi_dbg_tcp                                [Tracepoint event]
  iscsi:iscsi_dbg_trans_conn                         [Tracepoint event]
  iscsi:iscsi_dbg_trans_session                      [Tracepoint event]

Here's a simple diagram depicting the tracepoint enhancement:

These tracepoint events can be enabled on the fly to aid in debugging iscsi issues. Here's a sample output of tracing iscsi:iscsi_dbg_eh tracepoint event using the perf utility:

# /usr/bin/perf trace --no-syscalls --event="iscsi:iscsi_dbg_eh"
     0.000 iscsi:iscsi_dbg_eh:session25: iscsi_eh_target_reset tgt Reset [sc ffff883fee609500 tgt iqn.1986-03.com.sun:02:fa41d51f-45a5-cea4-d661-a854dd13cf07])
     0.009 iscsi:iscsi_dbg_eh:session25: iscsi_exec_task_mgmt_fn tmf set timeout)
     3.214 iscsi:iscsi_dbg_eh:session25: iscsi_eh_target_reset tgt iqn.1986-03.com.sun:02:fa41d51f-45a5-cea4-d661-a854dd13cf07 reset result = SUCCESS)

Tracepoint events that have been insterted into ftrace ring buffer can be extracted using e.g. crash utility version 7.1.6 or higher:

crash> extend ./extensions/trace.so
./extensions/trace.so: shared object loaded
crash> trace show
           <...>-18646 [023] 20618.810958: iscsi_dbg_eh:         session4: iscsi_eh_target_reset tgt Reset [sc ffff883fead741c0 tgt iqn.2019-10.com.example:storage]
           <...>-18646 [023] 20618.810968: iscsi_dbg_eh:         session4: iscsi_exec_task_mgmt_fn tmf set timeout
           <...>-18570 [016] 20848.578257: iscsi_dbg_trans_session: session4: iscsi_session_event Completed handling event 105 rc 0
           <...>-18570 [016] 20848.578260: iscsi_dbg_trans_session: session4: __iscsi_unbind_session Completed target removal

This enhancement can be found in Oracle Linux UEK-qu7 and newer releases.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.