By William Pijewski on Sep 28, 2010
I'll be setting up a new blog: billpijewski.com/blog, so join me there!
I'll be setting up a new blog: billpijewski.com/blog, so join me there!
The 2009.Q3 software update provides a new iSCSI software stack via the COMSTAR framework. COMSTAR is a Solaris framework which centralizes and simplifies the deployment of SCSI targets. COMSTAR supports many different SCSI protocols, including iSCSI, fibre channel, iSER, with support for more protocols under development. In 2009.Q3, the COMSTAR iSCSI port provider has replaced the older iSCSI target daemon. This new COMSTAR iSCSI provider replaces the entire iSCSI stack, from the management of iSCSI targets and initiators down to the interactions with the underlying ZFS volume. COMSTAR provides improved performance, as well as a more flexible management model. In this blog entry, I'll explain the new management model as well as talk about the advantages of using this new framework.
With the iSCSI target daemon in the 2009.Q2 update, every LUN was advertised within its own iSCSI target. As a result, initiators could hit built-in limits which prevent discovery of all necessary targets. Moreover, managing these targets did not scale well as the number of LUNs on the system increased. Access to a LUN was controlled by that LUN's ACL, a list of initiators which were able to access that LUN.
The administrative model used with the COMSTAR framework has some key differences and advantages. This model breaks this one-to-one mapping of LUN to target, and instead allows an administrator to advertise multiple LUNs behind a single target. Each LUN participates in a target group, a set of iSCSI targets which all advertise that LUN. In addition, each LUN has its own initiator group, a list of initiators not unlike the previous software's initiator ACL. This group defines which initiators may access LUNs bound to that initiator group, as well as the CHAP parameters to expect if CHAP authentication is enabled.
Target and initiator groups define sets of targets and initiators which are associated with LUNs. Besides any user-created target and initiator groups, LUNs may be associated with the default target and initiator group. These default groups contain all targets and all initiators, respectively. While using the default target and initiator group can be useful for evaluation purposes, their use is discouraged since their use may expose the LUN to unwanted hosts.
In addition to the flexible mapping of LUNs to targets, the COMSTAR iSCSI stack also allows targets to be bound to network interfaces:
This binding allows fine separation of iSCSI traffic1 from other data protocols. In addition, iSCSI traffic can be routed over the fastest interfaces, leaving other slow interfaces available management traffic.
 In some circumstances, traffic to or from and iSCSI target on one interface may travel on a different interface. With certain network configurations (like multiple default routes on the system), the routing configuration will dictate that traffic must use certain interfaces which conflict with the iSCSI configuration.
As discussed in other blog entries, many of the problems we solved in building the Sun Storage 7000 series were not specific to the NAS market. Moreover, the appliance kit framework we built solves many problems associated with an appliance in any domain -- that is, in addition to providing new innovative features, the appliance kit exposes and clarifies existing Solaris abstractions. In that vein, we needed a system to manage which users function as administrators of the appliance. We wanted the capability to pull users from an existing directory service, such as NIS or LDAP. Moreover, administrators need their own properties which were specific to our framework, such as whether they required a session annotation, or were a kiosk user. Finally, administrators have several user preferences which customize their environment to their liking, such as their initial login screen or character locale.
The 7000 series functions as a NIS and LDAP client1 and queries those directory services for information about users. These directory services consolidate the management of users in that single place, and allow a user's password and other properties (UID, home directory, etc.) to remain consistent throughout an entire organization. Adding a administrator from a directory service allows that user to login to the 7000 series and administer the appliance. Attributes and passwords for these directory users are stored in a directory service, and the appliance authenticates those administrators through a directory service. Administrators can also be added locally, in which case passwords for those users are stored locally on the appliance. These local users must be added to each appliance separately.
When our team was discussing how users should administer the appliance, we decided that the best practice was to add all administrators from a directory service and forego adding any local users. We established this best practice for the following reasons:
No UID conflicts. Because local users are given UIDs starting from 2000000000 in the order in which they're added, two local users on different appliances can have different UIDs, or even worse, the same UID can represent two different users. Even though these local users should not be used to access data, local users are no different than any other user on a Solaris system -- they can be used in establishing permissions on the root directory of a filesystem. If a particular user requires the same UID across several machines, shouldn't that user be stored in a directory service?
We've had some beta customers request the ability to prevent any local users on an appliance and restrict all administrators (other than root, of course) to be stored in a directory service. We'll work on this feature for the next software update. The appliance kit also has a rich set of fine-grained access controls for administrators which closely resembles RBAC in its design -- I'll cover this feature in a later entry.
 The 7000 series can also join an Active Directory domain and communicate with a domain controller for the purposes of authenticating CIFS clients. However, support for pulling administrators from Active Directory is not currently supported and will be coming a future software update.
As I described in my earlier blog entry on kiosk users, our team has been in contact with evaluation and beta customers to understand the problems they face as storage administrators. These administrators are responsible for the security of the storage infrastructure, and must understand how and why changes are made to the storage configuration. Many IT departments use a centralized ticket management system to manage open issues and requests. When a storage consumer reports a problem and requests some configuration change, that user is given a unique identifier to track the issue in this database. The ticket database contains all the details of the request, like the person requesting the change, the date and time of the request, which administrator will make the change, and the current status of the change.
One of our larger beta customers requested the ability to tag the audit log with a user-defined string which annotates that particular session. Certain users, upon logging in, must provide a session annotation:
Then, that annotation is saved with every audit record generated in that session. Users who are required to indicate this annotation may change their annotations in the middle of the session, but can never provide an empty session annotation:
Customers who have a centralized ticket management system can have their administrators provide the ticket identifier upon logging in. With those annotations, the auditors who examine the audit log gain a richer understanding of those actions, as each audit record can be correlated with the information in the database.
While the Fishworks team was developing our new storage server, we enlisted the help of several dozen experts in the enterprise storage field to help us define useful and necessary features. In addition to meeting the basic needs of storage administrators, we also wanted to provide additional features which administrators would find convenient. Across many enterprise domains, we continually heard one common complaint: storage servers were being blamed for performance problems, when the real performance problem lay somewhere else in the IT infrastructure.
At large companies, the storage infrastructure is consolidated into a single team of administrators who manage storage for the entire company. This team of administrators has service agreements which stipulate certain capacity, performance, and uptime requirements for other groups, and they must address problems if those service levels are not maintained. Storage administrators complained that they often spent time debugging performance problems with a particular group's application, only to find the storage was performing exactly as prescribed. Storage administrators could not gain any insight into a particular server's operation; instead, they wasted hours proving to application developers that the storage was not the source of the problem.
Our revolutionary analytics interface helps storage administrators understand how the box is performing. The dashboard page provides a summary view of key metrics, including a weather metaphor for each statistic. While this information is useful to storage administrators, it can also be useful to application developers who are actually using the storage. With this idea in mind, we created a new kind of administrator: a kiosk user. Kiosk users are created using the normal user dialog:
Storage administrators can add the application developers as kiosk users, or create a kiosk account for an entire team. By setting the screen to which kiosk users are restricted, administrators control the level of access application developers have to analytics data. For example, a kiosk user may be able to only view the dashboard, but not more specific worksheets. Likewise, a different kiosk user may be restricted to a worksheet for a particular project instead of having access to analytics data across all projects.
We anticipate that our customers will find this kiosk feature most useful for granting access to the dashboard and analytics data, though there are many other useful scenarios:
I hope our customers are able to find many and varied uses for kiosk users; please share your experiences with them in the comments.
Mike Shapiro and Bryan Cantrill have given a great overview of our new storage appliance, the Sun Storage Server 7xxx Series. I'm going to dive right in and give a summary of how our appliance fits into existing Windows environments, and explain what features we've created to help Windows administrators. Because our Sun Storage Server 7xxx series is built on top of Solaris, we're able to leverage many innovative Solaris technologies. The recent addition of a native Solaris CIFS server provides a new CIFS service tightly integrated with the Solaris kernel. This tight integration allows a richer set of functionality for CIFS clients, including seamless integration with NFSv4 clients.
Our storage appliance can join an Active Directory domain, which involves creating an account for that computer in the requisite location of the Active Directory database. Once that account has been established and the appliance has performed the necessary authentication, the appliance can query the Active Directory database for information about Windows users, groups, and other important objects. Because the appliance can join the Active Directory domain in this manner, no changes are needed to a customer's environment to support CIFS clients.
In addition to delegating client authentication to an Active Directory domain controller, the CIFS service can also operate in workgroup mode. A Windows workgroup is a collection of computers which authenticate users locally instead of storing user information in a centralized database like Active Directory or LDAP. We expect that workgroup mode will be used mainly for evaluation purposes, as any environment with more than about a dozen users would find workgroup mode extremely unwieldy.
One of the goals of the Solaris CIFS project was to simplify how NFSv4 and CIFS clients access files concurrently. Because NFSv4 and CIFS clients use different tokens to represent the credentials of a user or group (UID/GIDs and SIDs, respectively), the underlying filesystem must have knowledge of both types of credentials to accurately allow and deny access. Existing solutions involve storing one type of credential, and then performing a translation of that credential to the other form. While this approach solves the problem of the disjoint sets of identities, it is brittle and has a large administrative burden.
When we set out to solve the problem of credential management and mapping more cleanly, Mike developed a plan to unify UID/GID and SID credentials on the ZFS filesystem. The resulting plan changed how access tokens were stored in the kernel and in ZFS, and allowed the identity mapping service to translate Windows credentials to Unix credentials and vice versa.
The identity mapping service maps Windows identities to Unix identities to allow CIFS and NFSv4 clients to share the same identity. If only CIFS or NFS clients are accessing a particular share, or if CIFS and NFS clients are accessing disjoint directories within a share, then no identity mapping configuration is necessary. If NFS and CIFS clients must share the same identity (i.e. if one employee wants to access her home directory over both protocols), then the identity mapping service needs to know how to correlate those two identities. When the identity mapping service sees a request for a user or group which is not already mapped, there are three methods of resolving that mapping: directory-based mapping, name-based mapping, and ephemeral mapping.
Directory-based identity mapping allows annotation of a user's object in both an LDAP and Active Directory database. The LDAP object is annotated with attributes about that user's Windows identity, and the AD object is annotated with information about that user's Unix identity. When the identity mapping service is configured to use directory-based mapping, the service will lookup these additional attributes in the corresponding directory, and create a mapping based on those attributes. The identity mapping service can be configured to use directory-based mapping in either direction: mapping from Windows to Unix ("AD-only"), Unix to Windows ("LDAP-only"), or both directions ("mixed"). This approach to identity mapping is the most scalable, as these attributes will only need to be created once per identity, and are stored in only one place for each mapping direction.
Name-based identity mapping uses various identity mapping rules which map users to other users. These identity mapping rules can either map a particular Windows user or groups to a particular Unix user or group, or can create more general mappings which map across all users or groups in an Active Directory domain. Again, this name-based mapping approach is only needed when NFS and CIFS clients share the same identity.
Finally, the identity mapping service falls back on ephemeral mapping if a mapping cannot be handled through directory-based or name-based mapping. An ephemeral mapping is a temporary mapping for a Windows user which is not persistent across reboots, but is stored persistently on disk. That is, when an appliance reboots, the same Windows user may map to a different ephemeral UID (UIDs above 2\^31 are reserved for ephemeral UIDs.), but that Windows user will have the same access to files as before the reboot. Because these ephemeral UIDs are transient and not static, NFS clients cannot use these same UIDs. However, ephemeral UIDs and GIDs are perfectly valid when defining permissions on shares, and you may see them occasionally in a share's ACL if the appliance cannot communicate with an Active Directory server.
I hope you've received a good overview of some of Windows features and how identities are managed to create a truly native interoperable environment. Once the identity mapping scheme has been established, NFSv4 and CIFS clients can access common files with no restrictions or limitations. In the future, I'll blog about some other interesting Windows features, including:
The blog of Bill Pijewski, a member of the Fishworks engineering team.