Secure administration of Oracle VM Server for SPARC (Logical Domains)

A few days ago I was showing new features of Oracle VM Server for SPARC (informally shortened to "OVMSS", or still called LDoms) to a customer, and during the demo he asked "Why are you using root userid for this? Is that necessary?"

I guess the most literal answers would have been "Uh, because I never bothered about it before," and "No, root is not necessary." You can manage OVMSS without root by using Solaris' Role Based Access Control to assign just the needed authorizations to a non-root userid. In real deployments (unlike my little demo lab) that's really the best way to go.

(Irrelevant aside) I'm forcing myself to use the Chicago Manual of Style convention in which punctuation goes inside quoted text. I dislike it, myself. No, it's not an Oracle standard, AFAIK, but publishers seem to insist on it.

Why this is important

A bit of history and editorialization...

The all-powerful root 'super-user' is an artifact of Unix from its earliest days. In the original Unix security model, a userid was either root ("uid 0"), which can run any command, read, write, or remove any file, kill any process, shutdown the system, or a regular user (uid!=0) subject to authorization checks and restricted to its own playpen. It can't do any of those other fun things.

While convenient to concentrate all power in a system administrator userid, it was also risky. It's too easy to do a destructive "oops" while logged in as root, and has horrid security implications. Anyone who obtained the root password or otherwise managed to fool the system into thinking he or she was root could do anything. The root password had to be shared among administrators so they could login to do their administrative tasks - making it easy to compromise the password, and impossible to audit. If a root user accidentally ran a malicious binary (say, by not setting PATH carefully to include only trusted directories) that command would run with root privileges and could in turn do evil things - including setting trap doors that might swing open later.

I always felt that Unix 'root' was a mistake, and that separation of functions should have been considered from the outset. In all fairness, Unix grew up in trusting environments where this wasn't a consideration. For what it's worth, several operating systems have a similar history ("Whee! I can kill login sessions, shut the system down, and store arbitrary values into any location of RAM!" - from another OS), and evolved granular control over administrative privileges over time.

Solaris has, of course, provided rigorous security features for many years - which I leverage in this article.

OVMSS Security

Security is even more important in a virtual machine environment, since compromising a virtual machine monitor compromises the guests running underneath it. Fortunately, Oracle VM Server for SPARC was designed with security in mind. Some security features are provided by the underlying architecture, and others leverage Solaris security capabilities.

Separation of function

OVMSS architecture provides separation of function, using a firmware-based based hypervisor that runs on a processor invisible to guest domains. Other functions (administration, virtual devices) are delegated to a control domain that serves as an administrative control point, and service domains that provide virtual I/O to guest domains that run applications. (To fill out the picture and use the proper definitions: an I/O domain has access to a PCI bus and its devices; a service domain is therefore usually an I/O domain, and since a control domain needs a bus and I/O devices to boot, it is usually used as a service and I/O domain as well.) This is an architectural advantage over designs where all administrative power and access to all system resources resides in a single monolithic hypervisor.

Domain isolation

All domains run on their own CPU threads and RAM, providing a high degree of physical isolation. Each has a separate Solaris instance, so they are separate in terms of security scope.

Since the control domain is the administrative control point for the server, it is further protected by making it inaccessible to network access from guest domains. Guest domains cannot even ping the control domain via the virtual switch! This is by design, and prevents a compromised guest from mounting a network-based attack on the control domain. This default behavior can be changed by plumbing the virtual switch (as documented in the OVMSS administrative guide). After that, the guest domain and control domain can access one another via TCP/IP as usual. Still, the default behavior is to start with strict isolation.

Securing the control domain

How then, do we secure the control domain? The first thing is to apply whatever site-specific Solaris standards are applicable. Next advice: don't permit login by arbitrary users who may otherwise have legitimate access to your servers, since the only purpose for the control domain is to administer the virtualization environment or get access to guest domain consoles. If somebody has no business being on the control domain, they shouldn't even be able to get on.

No clear text password is allowed for authorized users - that's so last-century! Instead, we always login via ssh so passwords and session contents fly across the wire encrypted. Which reminds me: some popular virtual machine products do not encrypt memory contents during virtual machine migration - which exposes their contents (which may include passwords, Social Security ids, credit card numbers) to snooping. Be wary!

Deploying RBAC

Now to the meat of things: using RBAC to authorize selected non-root users to issue commands to the logical domain manager.

Authorization comes at two levels: read access, to view the configuration, and read/write access which lets you read or alter the domain environment. The corresponding Solaris authorizations are and solaris.ldoms.write. These authorizations are defined on the Solaris instance, stored in /etc/security/auth_attr when the LDoms manager software is installed. You can see that there are related authorizations, such as the one to manage the domain service, and authorizations for guest domain consoles ("vntsd" stands for "virtual network terminal server daemon" - quite a mouthful to pronounce.) Note that in the examples below (captured from terminal sessions I've just done), a prompt sequence with "#" indicates I'm logged in as root, and anything else indicates I'm logged in as a "regular" user.

# cat /etc/security/auth_attr |grep LDoms
solaris.ldoms.:::LDoms Administration::
solaris.ldoms.grant:::Delegate LDoms Configuration:: LDoms Configuration::
solaris.ldoms.write:::Manage LDoms Configuration::
solaris.smf.manage.ldoms:::Manage Start/Stop LDoms::
solaris.vntsd.:::LDoms vntsd Administration::
solaris.vntsd.consoles:::Access All LDoms Guest Consoles::
solaris.vntsd.grant:::Delegate LDoms vntsd Administration::

Further, these authorizations are collected into profiles stored in /etc/security/prof_attr

# cat /etc/security/prof_attr |grep ^LDoms
LDoms Management:::Manage LDoms domains:auths=solaris.ldoms.*
LDoms Review:::Review LDoms

We haven't been consistent with upper and lower case, have we? Well, each file is consistent with its own stylebook.

Now, I'll create two plain old userids using the normal commands:

# useradd -d /export/home/ldmuser1 -s /bin/bash ldmuser1
# zfs create rpool/export/home/ldmuser1
# chown -R ldmuser1 /export/home/ldmuser1
# passwd ldmuser1
New Password: 
Re-enter new Password: 
passwd: password successfully changed for ldmuser1

I do the same for user ldmuser2. So far, so boring - this is SA 101. I'll log into one of them and show that by default it cannot execute ldm commands.

-bash-3.00$ export PATH=/usr/sbin:$PATH
-bash-3.00$ ldm list
Authorization failed

Now, back as root, I'll add the read authorization

# usermod -A ldmuser1
UX: usermod: ldmuser1 is currently logged in, some changes may not take effect until next login.
Despite the above warning, it works right away:
-bash-3.00$ ldm list
primary          active     -n-cv-  SP      16    2G       2.6%  13d 5h 52m
rover            active     -n----  5000    8     1G       0.2%  13d 18h 5m
I can read, but can I also modify? Let's try to change the domain I used in my previous few articles.
-bash-3.00$ ldm set-vcpu 16 rover
Authorization failed
No problem - working as desired, and we can change that easily.
# usermod -A solaris.ldoms.write ldmuser1
UX: usermod: ldmuser1 is currently logged in, some changes may not take effect until next login.
And sure enough, on user ldmuser1:
-bash-3.00$ ldm set-vcpu 16 rover
-bash-3.00$ ldm list
primary          active     -n-cv-  SP      16    2G       0.6%  13d 5h 54m
rover            active     -n----  5000    16    1G       0.0%  13d 18h 7m
-bash-3.00$ ldm set-vcpu 8 rover

That was easy. If I want to retract the ability I can do that easily too.

# usermod -A "" ldmuser1
UX: usermod: ldmuser1 is currently logged in, some changes may not take effect until next login.
and again on user ldmuser1
-bash-3.00$ ldm list
Authorization failed

There are several ways of adding magic powers to a userid. In the preceding example I added the specific authorizations, but I can also add a profile to the user, and the profile inherits the authorizations defined for it in /etc/security/prof_attr. Note the change to the user entry in /etc/user_attr

# usermod -P "LDoms Management" ldmuser1
UX: usermod: ldmuser1 is currently logged in, some changes may not take effect until next login.
# cat /etc/user_attr|grep ldmuser
ldmuser1::::type=normal;profiles=LDoms Management
Sure enough, we're back in business:
-bash-3.00$ ldm set-vcpu 16 rover
-bash-3.00$ ldm list
primary          active     -n-cv-  SP      16    2G       1.0%  13d 6h 27m
rover            active     -n----  5000    16    1G       0.1%  13d 18h 40m
-bash-3.00$ ldm set-vcpu 8 rover

Would you like a Role in your RBAC?

Finally, we can do this with roles. Roles are a special type of user account that you don't directly log into. Instead, they are associated with a profile (see above), and users are designated as being able to assume that role.

The benefit is that the user assumes the role only at the specific times when they need to perform the relevant task, instead of running with "extra power" at all times. This enhances both safety (protection against "oops!") and security, since the user has to explicitly assume the role and authenticate with a password.

In this case, I define a role called LDomDemo, assign it the 'LDoms Management' profile, and then set ldmuser2 to be able to switch into that role. Since LDomDemo is a role, not a regular user, you can't log into it - but it gets a password anyway to guard switching into it via su.

# roleadd LDomDemo
# rolemod -P 'LDoms Management' LDomDemo
# usermod -R LDomDemo ldmuser2
# passwd LDomDemo 
New Password: 
Re-enter new Password: 
passwd: password successfully changed for LDomDemo

Now I log into ldmuser2 to try it out. Note that it initially has no additional profiles, and fails to run an ldm command, until I assume the LDomDemo role via su.

-bash-3.00$ profiles
Basic Solaris User
-bash-3.00$ ldm list
Authorization failed
-bash-3.00$ roles
-bash-3.00$ su LDomDemo
$ ldm list
primary          active     -n-cv-  SP      16    2G       0.4%  13d 7h 44m
rover            active     -n----  5000    8     1G       0.1%  13d 19h 56m
$ exit
Note the difference in shell prompt: while logged in as myself I'm running bash, but run the protected shell ("$") when assuming the role.

Again, the advantage of this method is that your userid doesn't have additional powers until you assume the relevant role, which helps protect you from mistakes that accidentally use super-powers when you don't mean to. Using a role also provides additional password protection to complement role assignment.

Let's migrate a domain

Now that this has been done, let's log into ldmuser1 and migrate the domain rover to our neighbor machine (which I've also set up with the equivalent userids)

-bash-3.00$ ldm migrate rover ldmuser1@
Target Password: 
Cannot enable FILE_DAC_READ privilege
Huh? I try every combination of command syntax (use the -p option, omit the target userid, whatever) and it makes no difference. Okay, I can take a hint, and I add the required privilege. This is strong medicine, because it lets a user account read /etc/shadow.
# cat user_attr|grep ldmuser1
ldmuser1::::type=normal;defaultpriv=basic,file_dac_read;profiles=LDoms Management
Let's try again:
-bash-3.00$ ldm migrate rover ldmuser1@
Target Password: 
Cannot enable FILE_DAC_SEARCH privilege
That's progress, I suppose. Let's add the remaining privilege that it is explicitly telling me to add - no investigation is needed!
# cat user_attr|grep ldmuser1
ldmuser1::::type=normal;defaultpriv=basic,file_dac_read,file_dac_search;profiles=LDoms Management
I now go back to my terminal window for ldmuser1 and try again, and it works fine. Note that the nifty ppriv command tells me what privileges my shell enjoys.
-bash-3.00$ ppriv $$
27728:  -bash
flags = 
        E: basic,file_dac_read,file_dac_search
        I: basic,file_dac_read,file_dac_search
        P: basic,file_dac_read,file_dac_search
        L: all
-bash-3.00$ ldm migrate rover ldmuser1@
Target Password: 

It works, but it's really not the right way, as I'll explain next.

Let's migrate a domain using a role, and why roles are better

I didn't understand why the Admin Guides for OVMSS 2.1 and 2.0 do not mention the requirement for file_dac_read and file_dac_search, while the older document for Logical Domains 1.3 (the version before being renamed) does, and tells you how to add them. It was easy to figure out, since the command tells you exactly what it is missing, but puzzling.

Menno Lageman explained this to me (thanks, Menno!). Correct practice is to use a role, so the guide doesn't illustrate directly adding the powerful file_dac_read and file_dac_search privileges to a user account. Domain migration uses these privileges to read root-owned private key and certificate files used to setup the SSL connection between the source and target hosts, a security privilege that should be carefully controlled. Directly adding file_dac_read and file_dac_search to a userid as I did above means that it has those powers all the time, when running any binary! Instead, leveraging a role means that the privileges are only set when running the ldm binary which itself has an execution attribute associated with the "LDoms Management" RBAC profile. This adds a layer of protection: gaining the privilege requires running the binary, and running the binary requires assuming the password-protected RBAC role and running under a profile-aware shell or pfexec.

While using a non-root userid to manage domains is better than using all-powerful root, a userid that can't exercise special powers until you assume the password-protected relevant role and run the correct binary is even better.

So, here's how it looks using ldmuser2, which I previously set up to use the role LDomDemo. I've logged into ldmuser2, and show that it can't issue an ldm command until I assume the LDomDemo role (which requires an additional password. Good). After that I can issue the migrate command without any extra magic incantations.

-bash-3.00$ id
uid=103814(ldmuser2) gid=1(other)
-bash-3.00$ profiles
Basic Solaris User
-bash-3.00$ ldm list
Authorization failed
-bash-3.00$ roles
-bash-3.00$ su LDomDemo
$ ldm list
primary          active     -n-cv-  SP      16    2G       1.2%  15d 6h 59m
atl-sewr-pool-155 active     -n----  5001    8     2G       0.1%  18d 18m
rover            active     -n----  5000    8     1G       0.1%  15d 19h 11m
$ ldm migrate rover
Target Password: 
$ ldm list
primary          active     -n-cv-  SP      16    2G       0.3%  15d 7h 1m
atl-sewr-pool-155 active     -n----  5001    8     2G       0.2%  18d 20m

So, the moral of the story is: use the RBAC roles - that's what the "R" stands for!

Summary, and where to learn more

This article describes how to use RBAC to secure an Oracle VM Server for SPARC system by eliminating use of the root userid and restricting power to specific users and roles when they need them. That, along with restricting what userids can log into a control domain in the first place, should be considered for any domain environment. Other tasks you may wish to consider include using RBAC to control access to guest domains consoles and to enable security auditing.

Reference information for these tasks can be found in Chapter 3 of the Oracle VM Server Administration Guide. The OVM for SPARC document library can be found at


Jeff, thank you for providing the tutorial for integrating RBAC with OVMSS (LDoms).
Implementing this functionality with an application team's operational model for
business continuity would seem to be a great fit. The app team could initiate the
failover and failback testing (Live Migration) without the need to coordinate with
other parties in an organization. Possibilities for software release management or
promoting code into production as well. Thanks again, mike b.

Posted by guest on July 10, 2011 at 09:48 AM MST #

You're quite welcome, Mike - thanks for the kind words. And I have to give a big thanks to Menno for explaining some parts I hadn't fully understood.

To your point - as you observe this permits secure separation of function and privileges, a really important capability for secure deployment and use.

regards, Jeff

Posted by Jeff Savit on July 10, 2011 at 01:05 PM MST #

Post a Comment:
Comments are closed for this entry.



« April 2014