Recent Posts


iSCSI enhancements in 2009.Q3

The 2009.Q3software update provides a new iSCSI software stack via the COMSTAR framework.COMSTAR is a Solaris framework which centralizes and simplifies the deploymentof SCSI targets. COMSTAR supports many different SCSI protocols, includingiSCSI, fibre channel, iSER, with support for more protocols under development.In 2009.Q3, the COMSTAR iSCSI port provider has replaced the older iSCSI targetdaemon. This new COMSTAR iSCSI provider replaces the entire iSCSI stack, fromthe management of iSCSI targets and initiators down to the interactions with theunderlying ZFS volume. COMSTAR provides improved performance,as well as a more flexible management model. In this blog entry, I'll explainthe new managementmodel as well as talk about the advantages of using this new framework. With the iSCSI target daemon in the 2009.Q2 update, every LUN was advertisedwithin its own iSCSI target. As a result, initiators could hit built-in limitswhich prevent discovery of all necessary targets. Moreover, managing thesetargets did not scale well as the number of LUNs on the system increased.Access to a LUN was controlled by that LUN's ACL, a list of initiators whichwere able to access that LUN.The administrative model used with the COMSTAR framework has some keydifferences and advantages. This model breaks this one-to-one mapping of LUN totarget, and instead allows an administrator to advertise multiple LUNs behind asingle target. Each LUN participates in a target group, a set of iSCSI targetswhich all advertise that LUN. In addition, each LUN has its own initiatorgroup, a list of initiators not unlike the previous software's initiator ACL.This group defines which initiators may access LUNs bound to that initiatorgroup, as well as the CHAP parameters to expect if CHAP authentication isenabled. Target and initiator groups define sets of targets and initiatorswhich are associated with LUNs. Besides any user-created target and initiatorgroups, LUNs may be associated with the default target and initiator group.These default groups contain all targets and all initiators, respectively.While using the default target and initiator group can be useful for evaluationpurposes, their use is discouraged since their use may expose the LUN tounwanted hosts. In addition to the flexible mapping of LUNs to targets, the COMSTAR iSCSI stackalso allows targets to be bound to network interfaces: This binding allows fine separation of iSCSI traffic1 from other dataprotocols. In addition, iSCSI traffic can be routed over the fastestinterfaces, leaving other slow interfaces available management traffic.Best PracticesWhen upgrading to 2009.Q3, you'll need to rebuild your iSCSI configuration inthis new model. With that reconfiguration in mind, here are some best practicesfor deploying COMSTAR iSCSI. Keep in mind that IP and SCSI have similarabstractions, and you may find yourself able to solve a problem at the IP level(e.g., creating an LACP aggregation and binding a target to that aggregation'sinterface) or the SCSI level (e.g., creating a target for each of severalinterfaces and adding those targets to the same target group).Use a single iSCSI target per target group. As the example above indicates, youmay also combine targets into a single target group, but I've found it simplestto create a LACP aggregation for a single target.Don't use the default initiator group -- that group grants every initiatoraccess to a LUN. A renegade or misconfigured initiator may discover anincorrect LUN; explicitly using an initiator group prevents any damage.Avoid using the default target group beyond evaluation.If you know a LUN must have LU number 0, create that LUN first. LUNs which aregiven automatically-assigned LU numbers can be given number zero.Don't enable the write cache setting on a LUN unless you completely understandthe ramifications of doing so. Consult your application's documentation forinformation about using this setting.Upgrading to 2009.Q3Once you've applied the 2009.Q3 software update, any iSCSI configuration from2009.Q2 and earlier is set aside and the SAN environment must be rebuilt. AllLUNs are still intact and available; however, you'll need to define new iSCSItargets and (optionally) initiators to access those LUNs. Here's a quick anddirty set of steps to get up and running with 2009.Q3:Create one or more iSCSI targets bound to the proper interfaces. Thisconfiguration is available in the new Configuration > SAN section.Assign the iSCSI targets you created in the first step to a target group.For each LUN, change its target group to the group created above.(Optional) Create initiator definitions, assign those initiators to initiatorsgroups, then bind LUNs to those initiator groups.(Optional) Configure any CHAP or RADIUS authentication.Once you've validated with the configuration, apply the COMSTAR deferred updateto allow complete PGR support.Additional InformationRoch explains the performanceadvantages of the COMSTAR stack.COMSTAR project onOpenSolaris.Consult the documenation bundled with the 2009.Q3 update for details ondeploying iSCSI LUNs with COMSTAR.[1] In some circumstances, traffic to or from and iSCSI target on one interfacemay travel on a different interface. With certain network configurations (likemultiple default routes on the system), the routing configuration will dictatethat traffic must use certain interfaces which conflict with the iSCSIconfiguration.

The href='http://blogs.sun.com/fishworks/entry/sun_storage_7000_2009_q3'>2009.Q3 software update provides a new iSCSI software stack via the href='http://opensolaris.org/os/project/comstar/'>COMSTARfr...


Administrators in the Fishworks Appliance Kit

As discussed in other blogentries, many of the problems we solved in building the Sun Storage 7000series were notspecific to the NAS market. Moreover, the appliance kit framework we builtsolves many problems associated with an appliance in any domain -- that is, inaddition to providing new innovative features, the appliance kit exposes andclarifies existing Solaris abstractions.In that vein, we needed a system to manage which users function asadministrators of the appliance. We wanted the capability to pull users from anexisting directory service, such asNIS orLDAP. Moreover, administratorsneed their own properties which were specific to our framework, such aswhether they required a sessionannotation, or were akiosk user. Finally, administrators have several user preferences whichcustomize their environment to their liking, such as their initial login screenor character locale.The 7000series functions as a NIS and LDAP client1 andqueries those directoryservices for information about users. These directory services consolidate themanagement of users in that single place, and allow a user's password and otherproperties (UID, home directory, etc.) to remain consistent throughout an entireorganization. Adding a administrator from a directory service allows that userto login to the 7000 series and administer the appliance. Attributes andpasswords for these directory users are stored in a directory service, and the appliance authenticates those administrators througha directory service. Administrators can also be added locally, in whichcase passwords for those users are stored locally on the appliance. Theselocal users must be added to each appliance separately.Best PracticesWhen our team was discussing how users should administer the appliance, wedecided that the best practice was to add all administrators from a directoryservice and forego adding any local users. We established this best practicefor the following reasons:Simplified management of users. If an employee changes hispassword, that change will take effect on all machines.No UID conflicts. Because local users are given UIDsstarting from 2000000000in the order in which they're added, two local users on different appliancescan have different UIDs, or even worse, the same UID can represent twodifferent users. Even though these local users should not be used to accessdata, local users are no different than any other user on a Solaris system --they can be used in establishing permissions on the root directory of afilesystem. If a particular user requires the same UID across severalmachines, shouldn't that user be stored in a directory service?No orphaned accounts.When an employee leaves a company, only the user account in the directoryservice must be deleted. The administrative accounts added to the appliancewill not function if that user cannot be resolved in the directory service.These users are called degraded users, and while they should be cleanedup as part of normal housekeeping, they pose no security risk since theycannot login.We've had some beta customers request the ability to prevent any localusers on an appliance and restrict all administrators (other than root, ofcourse) to be stored in a directory service. We'll work on this feature for thenext software update.The appliance kit also has a rich set of fine-grained access controls foradministrators which closelyresembles RBACin its design -- I'll cover this feature in a later entry.[1] The 7000 series can also join an ActiveDirectory domain and communicatewith a domain controller for the purposes of authenticating CIFS clients.However, support for pulling administrators from Active Directory isnot currently supported and will be coming a future software update.

As discussed in href="http://blogs.sun.com/bmc/entry/fishworks_now_it_can_be">other blog entries, many of the problems we solved in building the href="http://www.sun.com/storage/disk_systems/unified_st...


Session Annotations

As I described in my earlier blog entry on kiosk users, our team has been in contact with evaluation and beta customers to understand the problems they faceas storage administrators. These administrators are responsible for the security of the storage infrastructure, and must understand how and why changesare made to the storage configuration. Many IT departments use a centralizedticket management system to manage open issues and requests. When astorage consumer reports a problem and requests some configuration change, thatuser is given a unique identifier to track the issue in this database. The ticket database contains all the details of the request, like the person requesting the change, the date and time of the request, which administratorwill make the change, and the current status of the change.One of our larger beta customers requested the ability to tag the audit logwith a user-defined string which annotates that particular session. Certainusers, upon logging in, must provide a session annotation: Then, thatannotation is saved with every audit record generated in that session. Users who are required to indicate this annotation may change their annotationsin the middle of the session, but can never provide an empty session annotation:

As I described in my earlier blog entry on kiosk users, our team has been in contact with evaluation and beta customers to understand the problems they faceas storage administrators. These...


Kiosk Users: Observability for Everyone

While the Fishworks team was developing our new storage server, we enlisted the help of several dozen experts in the enterprise storage field to help us define useful and necessary features. In addition to meeting the basic needs of storage administrators, we also wanted to provide additional features which administrators would find convenient. Across many enterprise domains, we continually heard one common complaint: storage servers were being blamed for performance problems, when the real performance problem lay somewhere else in the IT infrastructure.At large companies, the storage infrastructure is consolidated into a singleteam of administrators who manage storage for the entire company. This team ofadministrators has service agreements which stipulate certain capacity,performance, and uptime requirements for other groups, and they must addressproblems if those service levels are not maintained. Storage administrators complained that they often spent time debugging performance problems with a particular group's application, only to find the storage was performing exactly as prescribed. Storage administrators could not gain any insight into a particular server's operation; instead, they wasted hours proving to application developers that the storage was not the source of the problem.Our revolutionary analytics interface helps storage administrators understand how the box is performing. The dashboard page provides a summary view of key metrics, including a weather metaphor for each statistic. While this information is useful to storage administrators, it can also be useful to application developers who are actually using the storage. With this idea in mind, we created a new kind of administrator: a kiosk user. Kiosk users are created using the normal user dialog: Storage administrators can add the application developers as kiosk users, orcreate a kiosk account for an entire team. By setting the screen to which kioskusers are restricted, administrators control the level of access application developers have to analytics data. For example, a kiosk user may be able to only view the dashboard, but not more specific worksheets.Likewise, a different kiosk user may be restricted to a worksheet for aparticular project instead of having access to analytics data across allprojects. We anticipate that our customers will find this kiosk feature most useful forgranting access to the dashboard and analytics data, though there are many otheruseful scenarios:Clients who have connectivity problems can use the services pages to check thestate of system and data services.By viewing the audit logs, auditors can understand who accessed the appliancewhat changes those users have made.Network administrators can inspect the networking and routing configuration tounderstand and troubleshoot any problems.One important note about kiosk users: even though they are restricted to viewing a certain screen, a malicious Javascript client can still make XML-RPC calls. A kiosk user cannot navigate to any other screen in the UI; however, that user will be able to see the results from raw XML-RPC calls which are notassociated with that screen. Because of the appliance's fine-grained accesscontrol, a kiosk user with no roles or authorizations will not be able tochange any configuration. Do not make the mistake of assuming a kiosk user will not be able to view the current shares, users, or other configuration parameters even if their kiosk screen would not normally allow them to access that data.I hope our customers are able to find many and varied uses for kiosk users; please share your experiences with them in the comments.

While the Fishworks team was developing our new storage server, we enlisted the help of several dozen experts in the enterprise storage field to help us define useful and necessary features....


Windows Features and Interoperability

Mike Shapiro and Bryan Cantrill have given a great overview of our new storage appliance, the Sun Storage Server 7xxx Series. I'm going to dive right in and give a summary of how our appliance fits into existing Windows environments, and explain what features we've created to helpWindows administrators.Because our Sun Storage Server 7xxx series is built on top ofSolaris, we're able to leverage many innovative Solaris technologies.The recent addition of a native SolarisCIFS server provides a new CIFS service tightly integrated withthe Solaris kernel. This tight integration allows a richer set offunctionality for CIFS clients, including seamless integration withNFSv4 clients.Active Directory integrationOur storage appliance can join an Active Directory domain, whichinvolves creating an account for that computer in the requisitelocation of the Active Directory database. Once that account has beenestablished and the appliance has performed the necessaryauthentication, the appliance can query the Active Directory databasefor information about Windows users, groups, and other importantobjects. Because the appliance can join the Active Directory domainin this manner, no changes are needed to a customer's environment tosupport CIFS clients.In addition to delegating client authentication to an ActiveDirectory domain controller, the CIFS service can also operate inworkgroup mode. A Windows workgroup is a collection of computers whichauthenticate users locally instead of storing user information in acentralized database like Active Directory or LDAP. We expect thatworkgroup mode will be used mainly for evaluation purposes, as anyenvironment with more than about a dozen users would find workgroupmode extremely unwieldy.Credential managementOne of the goals of the Solaris CIFS project was to simplify howNFSv4 and CIFS clients access files concurrently. Because NFSv4 andCIFS clients use different tokens to represent the credentials of auser or group (UID/GIDs and SIDs, respectively), the underlyingfilesystem must have knowledge of both types of credentials toaccurately allow and deny access. Existing solutions involve storingone type of credential, and then performing a translation of thatcredential to the other form. While this approach solves the problemof the disjoint sets of identities, it is brittle and has a largeadministrative burden.When we set out to solve the problem of credential management andmapping more cleanly, Mike developed a plan to unify UID/GID andSID credentials on the ZFS filesystem. The resultingplan changed how access tokens were stored in the kernel and inZFS, and allowed the identity mapping service totranslate Windows credentials to Unix credentials and vice versa.Theidentity mapping service maps Windows identities to Unix identitiesto allow CIFS and NFSv4 clients to share the same identity. If onlyCIFS or NFS clients are accessing a particular share, or if CIFS andNFS clients are accessing disjoint directories within a share, thenno identity mapping configuration is necessary. If NFS and CIFSclients must share the same identity (i.e. if one employee wants toaccess her home directory over both protocols), then the identitymapping service needs to know how to correlate those two identities.When the identity mapping service sees a request for a user or groupwhich is not already mapped, there are three methods of resolvingthat mapping: directory-based mapping, name-based mapping, andephemeral mapping.Directory-basedidentity mapping allows annotation of a user's object in both an LDAPand Active Directory database. The LDAP object is annotated withattributes about that user's Windows identity, and the AD object isannotated with information about that user's Unix identity. When theidentity mapping service is configured to use directory-basedmapping, the service will lookup these additional attributes in thecorresponding directory, and create a mapping based on thoseattributes. The identity mapping service can be configured to use directory-basedmapping in either direction: mapping from Windows to Unix ("AD-only"),Unix to Windows ("LDAP-only"), or both directions ("mixed").This approach to identity mapping is the most scalable,as these attributes will only need to be created once per identity, andare stored in only one place for each mapping direction.Name-basedidentity mapping uses various identity mapping rules which map usersto other users. These identity mapping rules can either map aparticular Windows user or groups to a particular Unix user or group,or can create more general mappings which map across all users orgroups in an Active Directory domain. Again, this name-based mappingapproach is only needed when NFS and CIFS clients share the sameidentity.Finally,the identity mapping service falls back on ephemeral mapping if amapping cannot be handled through directory-based or name-basedmapping. An ephemeral mapping is a temporary mapping for a Windowsuser which is not persistent across reboots, but is storedpersistently on disk. That is, when an appliance reboots, the sameWindows user may map to a different ephemeral UID (UIDs above 2\^31are reserved for ephemeral UIDs.), but that Windows user will havethe same access to files as before the reboot. Because theseephemeral UIDs are transient and not static, NFS clients cannot usethese same UIDs. However, ephemeral UIDs and GIDs are perfectly validwhen defining permissions on shares, and you may see themoccasionally in a share's ACL if the appliance cannot communicatewith an Active Directory server.Ihope you've received a good overview of some of Windows features andhow identities are managed to create a truly native interoperableenvironment. Once the identity mapping scheme has been established,NFSv4 and CIFS clients can access common files with no restrictionsor limitations. In the future, I'll blog about some other interestingWindows features, including: Our DTrace analytics system which can observe CIFS traffic like never before. Our full support for NT ACLs and ACL inheritance.

Mike Shapiro and Bryan Cantrill have given a great overview of our new storage appliance, the Sun Storage Server 7xxx Series. I'm going to dive right in and give a summary of how our appliance fits...