iSCSI enhancements in 2009.Q3

The 2009.Q3 software update provides a new iSCSI software stack via the COMSTAR framework. COMSTAR is a Solaris framework which centralizes and simplifies the deployment of SCSI targets. COMSTAR supports many different SCSI protocols, including iSCSI, fibre channel, iSER, with support for more protocols under development. In 2009.Q3, the COMSTAR iSCSI port provider has replaced the older iSCSI target daemon. This new COMSTAR iSCSI provider replaces the entire iSCSI stack, from the management of iSCSI targets and initiators down to the interactions with the underlying ZFS volume. COMSTAR provides improved performance, as well as a more flexible management model. In this blog entry, I'll explain the new management model as well as talk about the advantages of using this new framework.

With the iSCSI target daemon in the 2009.Q2 update, every LUN was advertised within its own iSCSI target. As a result, initiators could hit built-in limits which prevent discovery of all necessary targets. Moreover, managing these targets did not scale well as the number of LUNs on the system increased. Access to a LUN was controlled by that LUN's ACL, a list of initiators which were able to access that LUN.

The administrative model used with the COMSTAR framework has some key differences and advantages. This model breaks this one-to-one mapping of LUN to target, and instead allows an administrator to advertise multiple LUNs behind a single target. Each LUN participates in a target group, a set of iSCSI targets which all advertise that LUN. In addition, each LUN has its own initiator group, a list of initiators not unlike the previous software's initiator ACL. This group defines which initiators may access LUNs bound to that initiator group, as well as the CHAP parameters to expect if CHAP authentication is enabled.

Target and initiator groups define sets of targets and initiators which are associated with LUNs. Besides any user-created target and initiator groups, LUNs may be associated with the default target and initiator group. These default groups contain all targets and all initiators, respectively. While using the default target and initiator group can be useful for evaluation purposes, their use is discouraged since their use may expose the LUN to unwanted hosts.

In addition to the flexible mapping of LUNs to targets, the COMSTAR iSCSI stack also allows targets to be bound to network interfaces:

This binding allows fine separation of iSCSI traffic1 from other data protocols. In addition, iSCSI traffic can be routed over the fastest interfaces, leaving other slow interfaces available management traffic.

Best Practices

When upgrading to 2009.Q3, you'll need to rebuild your iSCSI configuration in this new model. With that reconfiguration in mind, here are some best practices for deploying COMSTAR iSCSI. Keep in mind that IP and SCSI have similar abstractions, and you may find yourself able to solve a problem at the IP level (e.g., creating an LACP aggregation and binding a target to that aggregation's interface) or the SCSI level (e.g., creating a target for each of several interfaces and adding those targets to the same target group).
  • Use a single iSCSI target per target group. As the example above indicates, you may also combine targets into a single target group, but I've found it simplest to create a LACP aggregation for a single target.
  • Don't use the default initiator group -- that group grants every initiator access to a LUN. A renegade or misconfigured initiator may discover an incorrect LUN; explicitly using an initiator group prevents any damage.
  • Avoid using the default target group beyond evaluation.
  • If you know a LUN must have LU number 0, create that LUN first. LUNs which are given automatically-assigned LU numbers can be given number zero.
  • Don't enable the write cache setting on a LUN unless you completely understand the ramifications of doing so. Consult your application's documentation for information about using this setting.

Upgrading to 2009.Q3

Once you've applied the 2009.Q3 software update, any iSCSI configuration from 2009.Q2 and earlier is set aside and the SAN environment must be rebuilt. All LUNs are still intact and available; however, you'll need to define new iSCSI targets and (optionally) initiators to access those LUNs. Here's a quick and dirty set of steps to get up and running with 2009.Q3:
  1. Create one or more iSCSI targets bound to the proper interfaces. This configuration is available in the new Configuration > SAN section.
  2. Assign the iSCSI targets you created in the first step to a target group.
  3. For each LUN, change its target group to the group created above.
  4. (Optional) Create initiator definitions, assign those initiators to initiators groups, then bind LUNs to those initiator groups.
  5. (Optional) Configure any CHAP or RADIUS authentication.
  6. Once you've validated with the configuration, apply the COMSTAR deferred update to allow complete PGR support.

Additional Information


[1] In some circumstances, traffic to or from and iSCSI target on one interface may travel on a different interface. With certain network configurations (like multiple default routes on the system), the routing configuration will dictate that traffic must use certain interfaces which conflict with the iSCSI configuration.

Comments:

This change to the iSCSI layer has dramatically reduced the number of target IQNs you can present from a system. With the old model there was a target IQN for each LUN created. With COMSTAR it seems you can only add 32 targets to a system.

Posted by Iain Watson on September 24, 2009 at 02:04 AM PDT #

@Iain That's correct; the new model only allows for 32 iSCSI targets. Is there some configuration which is problematic using this new model?

Posted by Bill Pijewski on September 24, 2009 at 03:23 AM PDT #

For Sun VDI we have systems with 1000s of LUNs (each representing a virtual disk of a VM) on AR 2009.Q2. How do I do that with Q3?

And more on the practical side of things: I tried your "quick and dirty set of steps" and while the IQN can be seen from the outside after step 1 already, I cannot connect to it after I executed the steps 1,2,3,6 with either the Windows or the sol10u7 initiator. Do I miss something?

Posted by Thomas Pfohe on September 25, 2009 at 02:43 AM PDT #

I created more iSCSI LUNs in a previous Software Release then the amount of iSCSI targets allowed in 2009.Q3.

After applying the new Software, such a iSCSI LUN is no longer exported as an iSCSI Target!

Note, that iSCSI LUNs are continued to be managed on the "Shares" screen.

To export a iSCSI LUN, you'll need to create an iSCSI Target Group on "Configuration - SAN" screen and associate this Group with one or more iSCSI LUNs.
For this purpose, choose an iSCSI LUN under "Shares" screen and select the iSCSI Target Group on the "Sharing Options" screen that you'll find under "Protocols".

Posted by cnd on October 12, 2009 at 09:03 PM PDT #

Post a Comment:
Comments are closed for this entry.
About

The blog of Bill Pijewski, a member of the Fishworks engineering team.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today