Thursday Sep 13, 2012

To encryption=on or encryption=off a simple ZFS Crypto demo

I've just been asked twice this week how I would demonstrate ZFS encryption really is encrypting the data on disk.  It needs to be really simple and the target isn't forensics or cryptanalysis just a quick demo to show the before and after.

I usually do this small demo using a pool based on files so I can run strings(1) on the "disks" that make up the pool. The demo will work with real disks too but it will take a lot longer (how much longer depends on the size of your disks).  The file hamlet.txt is this one from

# mkfile 64m /tmp/pool1_file
# zpool create clear_pool /tmp/pool1_file
# cp hamlet.txt /clear_pool
# grep -i hamlet /clear_pool/hamlet.txt | wc -l

Note the number of times hamlet appears

# zpool export clear_pool
# strings /tmp/pool1_file | grep -i hamlet | wc -l

Note the number of times hamlet appears on disk - it is 2 more because the file is called hamlet.txt and file names are in the clear as well and we keep at least two copies of metadata.

Now lets encrypt the file systems in the pool.
Note you MUST use a new pool file don't reuse the one from above.

# mkfile 64m /tmp/pool2_file
# zpool create -O encryption=on enc_pool /tmp/pool2_file
Enter passphrase for 'enc_pool': 
Enter again: 
# cp hamlet.txt /enc_pool
# grep -i hamlet /enc_pool/hamlet.txt | wc -l

Note the number of times hamlet appears is the same as before

# zpool export enc_pool
# strings /tmp/pool2_file | grep -i hamlet | wc -l

Note the word hamlet doesn't appear at all!

As a said above this isn't indended as "proof" that ZFS does encryption properly just as a quick to do demo.

Wednesday Jul 04, 2012

Delegation of Solaris Zone Administration

In Solaris 11 'Zone Delegation' is a built in feature. The Zones system now uses fine grained RBAC authorisations to allow delegation of management of distinct zones, rather than all zones which is what the 'Zone Management' RBAC profile did in Solaris 10.

The data for this can be stored with the Zone or you could also create RBAC profiles (that can even be stored in NIS or LDAP) for granting access to specific lists of Zones to administrators.

For example lets say we have zones named zoneA through zoneF and we have three admins alice, bob, carl.  We want to grant a subset of the zone management to each of them.

We could do that either by adding the admin resource to the appropriate zones via zonecfg(1M) or we could do something like this with RBAC data directly:

First lets look at an example of storing the data with the zone.

# zonecfg -z zoneA
zonecfg:zoneA> add admin
zonecfg:zoneA> set user=alice
zonecfg:zoneA> set auths=manage
zonecfg:zoneA> end
zonecfg:zoneA> commit
zonecfg:zoneA> exit

Now lets look at the alternate method of storing this directly in the RBAC database, but we will show all our admins and zones for this example:

# usermod -P +'Zone Management' -A alice

# usermod -A alice

# usermod -P +'Zone Management' -A bob
# usermod -A bob

# usermod -P +'Zone Management' -A carl
# usermod -A carl
# usermod -A carl
# usermod -A carl

In the above alice can only manage zoneA, bob can manage zoneB and zoneC and carl can manage zoneC through zoneF.  The user alice can also login on the console to zoneB but she can't do the operations that require the authorisation on it.

Or if you have a large number of zones and/or admins or you just want to provide a layer of abstraction you can collect the authorisation lists into an RBAC profile and grant that to the admins, for example lets great an RBAC profile for the things that alice and carl can do.

# profiles -p 'Zone Group 1'
profiles:Zone Group 1> set desc="Zone Group 1"
profiles:Zone Group 1> add profile="Zone Management"
profiles:Zone Group 1> add
profiles:Zone Group 1> add
profiles:Zone Group 1> commit
profiles:Zone Group 1> exit
# profiles -p 'Zone Group 3'
profiles:Zone Group 1> set desc="Zone Group 3"
profiles:Zone Group 1> add profile="Zone Management"
profiles:Zone Group 1> add
profiles:Zone Group 1> add
profiles:Zone Group 1> add
profiles:Zone Group 1> commit
profiles:Zone Group 1> exit

Now instead of granting carl  and aliace the 'Zone Management' profile and the authorisations directly we can just give them the appropriate profile.

# usermod -P +'Zone Group 3' carl

# usermod -P +'Zone Group 1' alice

If we wanted to store the profile data and the profiles granted to the users in LDAP just add '-S ldap' to the profiles and usermod commands.

For a documentation overview see the description of the "admin" resource in zonecfg(1M), profiles(1) and usermod(1M)

Tuesday May 01, 2012

Podcast: Immutable Zones in Oracle Solaris 11

In this episode of the "Oracle Solaris: In a Class By Itself" podcast series, the focus is a bit more technical. I was interviewed by host Charlie Boyle, Senior Director of Solaris Product Marketing. We talked about a new feature in Oracle Solaris 11: immutable zones. Those are read-only root zones for highly secure deployment scenarios.

See also my previous blog post on Enctypted Immutable Zones.

Wednesday Feb 29, 2012

Solaris 11 has the security solution Linus wants for Desktop Linux

Recently Linus Torvalds was venting (his words!) about the frustrating requirement to keep giving his root password for common desktop tasks such as connecting to a wifi network or configuring printers.

Well I'm very pleased to say that the Solaris 11 desktop doesn't have this problem thanks to our RBAC system and how it is used including how it is tightly integrated into the desktop.

One of the new RBAC features in Solaris 11 is location context RBAC profiles, by default we grant the user on the system console (ie the one on the laptop or workstation locally at the physical keyboard/screen) the "Console User" profile.  Which on a default install has the necessary authorisations and execution profiles to do things like joining a wireless network, changing CPU power management, and using removal media.   The user created at initial install time also has the much more powerful "System Administrator" profile granted to them so they can do even more without being required to give a password for root (they also have access to the root role and the ability to use sudo).

Authorisations in Solaris RBAC (which dates back in main stream Solaris to Solaris 8 and even further 17+ years in Trusted Solaris) are checked by privileged programs and the whole point is so you don't have to reauthenticate.  SMF is a very heavy user of RBAC authorisations.  In the case of things like joining a wireless network it is privileged daemons that are checking the authorisations of the clients connecting to them (usually over a door)

In addition to that GNOME in Solaris 11 has been explicitly integrated with Solaris RBAC as well, any GNOME menu entry that needs to run with elevated privilege will be exectuted via Solaris RBAC mechanisms.  The panel works out the least intrusive way to get the program running for you.  For example if I select "Wireshark" from the GNOME panel menu it just starts - I don't get prompted for any root password - but it starts with the necessary privileges because GNOME on Solaris 11 knows that I have the "Network Management" RBAC profile which allows running /usr/sbin/wireshark with the net_rawaccess privilege.   If I didn't have "Network Management" directly but I had an RBAC role that had it then GNOME would use gksu to assume the role (which might be root) and in which case I would have been prompted for the role password.  If you are using roleauth=user that password is yours and if you are using pam_tty_tickets you won't keep getting prompted.

GNOME can even go further and not even present menu entries to users who don't have granted to them any RBAC profile that allows running those programs - this is useful in a large multi user system like a Sun Ray deployment.

If you want to do it the "old way" and use the CLI and/or give a root password for every "mundane" little thing, you can still do that too if you really want to.

So maybe Linus could try Solaris 11 desktop ;-)

Monday Feb 20, 2012

Solaris 11 Common Criteria Evaluation

Oracle Solaris 11 is now "In Evaluation" for Common Criteria at EAL4+.  The protection profile is OSPP with the following extended packages: AM - Advanced Management  EIA - Extended Identification and Authentication, LS - Label Security, VIRT - Virtualization.  For information on other Oracle products that are evaluated under Common Criteria or FIPS 140 please see the general Oracle Security Evalutions page.

Please email for all inquiries regarding Oracle security evaluations, I can't answer questions about the content of the evaluation on this blog or directly by email to me.

Friday Feb 03, 2012

What Free/Open Source software is Solaris 11 still missing

Note this is not a commitment from Oracle to deliver anything as a result of your answers, nor is it an official survey of any kind.

Okay first my dirty little secret... my family home desktop machine runs Windows 7.  Earlier this week I had a need to check the MD5 or SHA256 checksum on an iso image I'd downloaded.  On Solaris I'd just run 'digest -a sha256' or sha256sum on Solaris or any Linux distro.  But on Windows 7 the best I could come up with was code it up in Java myself or install the GNU versions via Cygwin.

So that got me thinking, the Solaris 11 repository has a lot more "upstream" Free/Open Source tools and frameworks than any other release of Solaris ever had.  We have Python (which is really a core part of Solaris 11 now), Ruby loads of the GNU runtime and development toolchains and much much more.   However many common Linux distributions still have more than we do but some of that isn't target at server use cases.

So what Free/Open Source software is Solaris 11 still missing that you use to run your business on your Solaris servers?

Even if you don't have Solaris 11 installed you can quickly search for packags at

Please add details in the comments.

Again note this is not a commitment from Oracle to deliver anything as a result of your answers, nor is it an official survey of any kind, just my curiosity.  I will of course log the relevant bugs for viable things if any come up.

Update 1: Thanks for all the submissions so far, some great suggestions in there - keep them coming and don't worry about looking for duplicates in others comments (in fact I'd rather things were listed my multiple people since it shows more interest in a given component).

Update 2: comments are moderated (site requirement), submitting multiple times unfortunately sometimes results in you being told your comment is spam but I still see it and will approve it. Thanks for your patience.

Tuesday Nov 22, 2011

HOWTO Turn off SPARC T4 or Intel AES-NI crypto acceleration.

Since we released hardware crypto acceleration for SPARC T4 and Intel AES-NI support we have had a common question come up: 'How do I test without the hardware crypto acceleration?'.

Initially this came up just for development use so developers can do unit testing on a machine that has hardware offload but still cover the code paths for a machine that doesn't (our integration and release testing would run on all supported types of hardware anyway).  I've also seen it asked in a customer context too so that we can show that there is a performance gain from the hardware crypto acceleration, (not just the fact that SPARC T4 much faster performing processor than T3) and measure what it is for their application.

With SPARC T2/T3 we could easily disable the hardware crypto offload by running 'cryptoadm disable provider=n2cp/0'.  We can't do that with SPARC T4 or with Intel AES-NI because in both of those classes of processor the encryption doesn't require a device driver instead it is unprivileged user land callable instructions.

Turns out there is away to do this by using features of the Solaris runtime loader ( First I need to expose a little bit of implementation detail about how the Solaris Cryptographic Framework is implemented in Solaris 11.  One of the new Solaris 11 features of the linker/loader is the ability to have a single ELF object that has multiple different implementations of the same functions that are selected at runtime based on the capabilities of the machine.  The alternate to this is having the application coded to call getisax() and make the choice itself.  We use this functionality of the linker/loader when we build the userland libraries for the Solaris Cryptographic Framework (specifically, and the unfortunately misnamed due to historical reasons

The Solaris linker/loader allows control of a lot of its functionality via environment variables, we can use that to control the version of the cryptographic functions we run.  To do this we simply export the LD_HWCAP environment variable with values that tell to not select the HWCAP section matching certain features even if isainfo says they are present. 

For SPARC T4 that would be:

export LD_HWCAP="-aes -des -md5 -sha256 -sha512 -mont -mpmul" 

and for Intel systems with AES-NI support:

export LD_HWCAP="-aes"

This will work for consumers of the Solaris Cryptographic Framework that use the Solaris PKCS#11 libraries or use interfaces directly.  It also works for the Oracle DB and Java JCE.  However does not work for the default enabled OpenSSL "t4" or "aes-ni" engines (unfortunately) because they do explicit calls to getisax() themselves rather than using multiple ELF cap sections.

However we can still use OpenSSL to demonstrate this by explicitly selecting "pkcs11" engine  using only a single process and thread. 

$ openssl speed -engine pkcs11 -evp aes-128-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      54170.81k   187416.00k   489725.70k   805445.63k  1018880.00k

$ LD_HWCAP="-aes" openssl speed -engine pkcs11 -evp aes-128-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      29376.37k    58328.13k    79031.55k    86738.26k    89191.77k

We can clearly see the difference this makes in the case where AES offload to the SPARC T4 was disabled. The "t4" engine is faster than the pkcs11 one because there is less overhead (again on a SPARC T4-1 using only a single process/thread - using -multi you will get even bigger numbers).

$ openssl speed -evp aes-128-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      85526.61k    89298.84k    91970.30k    92662.78k    92842.67k

Yet another cool feature of the Solaris linker/loader, thanks Rod and Ali.

Note these above openssl speed output is not intended to show the actual performance of any particular benchmark just that there is a significant improvement from using hardware acceleration on SPARC T4. For cryptographic performance benchmarks see the postings.

Wednesday Nov 09, 2011

Completely disabling root logins on Solaris 11

Since Solaris 8 it has been possible to make the root account a role.  That means you can't login directly as root (except in single user mode) but have to login as an authorised user first and assume (via su) the root role.  This still required the root account to have a valid and known password as it is needed for the su step and for single user access.

With Solaris 11 it is possible to go one step further and completely disable all need for a root password even for access in single user mode.

There are two complementary new features that make this possible.  The first is the ability to change which password is used when authenticating to a role.  A new per role property called roleauth was added, if it isn't present the prior behaviour of using the role account password is retained, if roleauth=user is set instead then the password of the user assuming the role is used.

The second feature was one that existed in the Solaris 11 Express release which changed how the sulogin command worked, prior releases all just asked for the root password.  The sulogin program was changed to authenticate a specific user instead so now asks for a username and the password of that user.  The user must be one authorised to enter single user mode by being granted the 'solaris.system.maintenance' authorisation - and obviously be one that can actually connect to the system console (which I recommend is protected by "other means" eg ILOM level accounts or central "terminal server")

The following sequence of commands takes root from being a normal root account (which depending on how you install Solaris 11 it maybe, or it might already be a role) and granting the user darrrenm the ability to assume the root role and enter single user mode.

# usermod -K type=role root
# usermod -R +root -A +solaris.system.maintenance darrenm
# rolemod -K roleauth=user  root
# passwd -N root

Note that some of the install methods for Solaris 11 will have created an initial user account that is granted the root role and has been given the "System Administrator" profile, in those cases only the last two steps are required as the equivalent of the first two will already have been done at install time for the initial non root user.

Note that we do not lock (-l) the root account but instead ensure it has no valid password (-N) this is because the root account does still have some cron jobs that we ideally want to run and if it was locked then the PAM module would prevent cron from running those jobs.

When root is a role like this you authenticate to the system first as yourself, in this case the user darrenm logs in first.  Once darrenm has logged in we use su(1M) to be come root - just like we would have if root wasn't a role.  The main difference here is that the password given to su(1M) in the above config is darrenm's password.

If you boot the system in single user mode (boot -s) you will be asked for a username and password, we give the username of darrenm and darrenm's password. Once you do that you get a # prompt that is truely root in single user mode.  The distinction here is we have an audit trail and know it was darrenm that authenticated and we have no separate root password to manage.

In some deployment cases there may not be any locally defined accounts, in those cases it is necessary to allow the root to allow direct login on the system console in multiuser mode.  This is achived by adding the following to /etc/pam.conf, and also give the root account a valid password.

login account required

By having that entry we do not have active for console login so the root account will be able to login directly.  The assumption here is that access to the system console is sufficiently secured (and audite) by means external to the Solaris instance.  For example the ILOM of the system is on an access restricted management network that has specific user accounts for SSH access to the ILOM.  You may also want to only give out that root password in emergency cases.  This will allow direct root login only on the console but require that users authenticate to root using their own password when using su.


If you have made root as role and you want to go back to a traditional direct login capability for root you can do so by simply running:

 # rolemod -K type=normal root

Update 1 to answer the first question: Basically exactly the same as if the password was locked, expired or forgotten if you just used root directly.  Failed account locking is not enabled by default.  As for forgetting who was the authorised account that isn't a problem Solaris can fix on its own that is part of your administative procedures.  You can have any number of authorised users and the userattr, roles, profiles commands can be used tell you who they are and manage them.

Update 2 to make it clearer how you use this in multi-user and single user.

Update 3 add information on how to allow root on console.

Password (PAM) caching for Solaris su - "a la sudo"

I talk to a lot of users about Solaris RBAC but many of them prefer to use sudo for various reasons.  One the common usability features that users like is the that they don't have to continually type their password.  This is because sudo uses a "ticket" system for caching the authentication for a defined period (by default 5 minutes).

To bring this usability feature to Solaris 11 I wrote a new PAM module (pam_tty_tickets) that provides a similar style of caching for Solaris roles. 

By default the tickets are stored in /system/volatile/tty_tickets (/var/run is a symlink to /system/volatile now). 

When using su(1M) the user you currently are is set in PAM_USER and PAM_AUSER is the user you are becoming (ie the username argument to su or root if one is not specified).  The PAM module implements the caching using tickets, the internal format of the tickets is the same as what sudo uses. The location can be changed to be compatible with sudo so the same ticket can be used for su and sudo.

To enable pam_tty_tickets for su put the following into /etc/pam.conf (the module is in the pkg:/system/library package so it is always installed but not configured for use by default):

su      auth required 
su      auth sufficient
su      auth requisite
su      auth required 

So what does it now look like:

braveheart:pts/3$ su -
root@braveheart:~# id -a
uid=0(root) gid=0(root) groups=0(root),1(other),2(bin),3(sys),4(adm),5(uucp),6(mail),7(tty),8(lp),9(nuucp),12(daemon)
darrenm@braveheart:~# exit
braveheart:pts/3$ su -

If you want to enable it in the desktop for gksu then you need to add a similar set of changes to /etc/pam.conf with the service name as "embedded_su" with the same modules as is  listed above.  The default timeout matches the sudo default of 5 minutes, the timeout= module option allows specifying a different timeout.

[ NOTE: The man page for pam_tty_tickets was mistakenly placed in section 1 for Solaris 11, it should have been in section 5. ]

Update for Solaris 11.1, now that we have /etc/pam.d/ support it is recommended that instead of updating /etc/pam.conf the following lines be placed into /etc/pam.d/su

auth sufficient
auth definitive
auth requisite
auth required
auth required

Monday Aug 10, 2009

Sending a Break to Solaris hosted in Virtualbox

I've recently starting using VirtualBox instead of physical machines for some of my basic functional testing.  When doing some types of kernel development it is often necessary to force the system into kmdb.

The F1-A keystroke does this on Solaris x86 systems by default, however that isn't going to work with VirtualBox because that keystore will be grabbed by some very low level kernel routines in the host an never reaches the guest.

So we need an alternate way of getting a break to the guest Solaris from the host one.

I was sure someone else must have worked this out before.  I didn't get the full answer from a quick google search but I did find all the parts.

The CLI for VirtualBox can send an NMI (Non Maskable Interupt) to any running guest. Solaris can beconfigured to drop into kmdb or force a panic when receiving an NMI.

In the guest put this into /etc/system and reboot:

set pcplusmp:apic_kmdb_on_nmi=1

Or to set it interactively do:

# echo apic_kmdb_on_nmi/W1 | mdb -kw

# mdb -K

Then with the VirtualBox CLI we can send an NMI to our guest:

$ VBoxManage debugvm ZFS_Crypto_Test injectnmi

Nice easy solution.  Though I do now wonder why we don't have some default action for when an NMI is received - but then not everyone cares about getting a dump or getting into kmdb!

Updated 2013-10-8: at somepoint this changed from controlvm to debugvm




  • General
« April 2014