Friday Dec 03, 2010

File Based Repo

It was always possible to mount an ISO and use it as the package repository, but you also had to configure and run a local package server. With Solaris 11 Express, it is now possible to serve the packages directly from the local repository files, bypassing the server.

Step 1: Download the Repository Image

The repository image comes in two 2 GB chunks and they're available on the Solaris 11 Express Downloads page. After downloading parts A & B, unzip and concatenate them as follows:

bleonard@solaris:~$ cd Download/
bleonard@solaris:~/Download$ unzip 
  inflating: sol-11-exp-201011-repo-full.iso-a  
bleonard@solaris:~/Download$ unzip 
  inflating: sol-11-exp-201011-repo-full.iso-b  
bleonard@solaris:~/Download$ cat sol-11-exp-201011-repo-full.iso-a sol-11-exp-201011-repo-full.iso-b > sol-repo-full.iso

Note, I kept the ISO name generic, as this allows me to upgrade to a newer version of the repository by simply replacing the underlying ISO.

Step 2: Mount the Repository Image

Create a directory at which to mount the ISO:

 bleonard@solaris:~$ sudo mkdir /repo

Mount the ISO to the /repo directory:

bleonard@solaris:~$ sudo mount -F hsfs ~/Download/sol-repo-full.iso /repo

Step 3: Make the Mount Persistent

The mount will not survive a reboot. You have a couple of approaches to make it persist:

The Quick and Dirty Way

Add the following file to /etc/rc3.d:

# cat /etc/rc3.d/S99mountiso
mount -F hsfs /export/home/bleonard/Download/sol-repo-full.iso /repo 

The SMF Way

Save the following repo.xml SMF manifest to /var/svc/manifest/system/filesystem, which defines a new service, repo.

Install the repo service:

bleonard@opensolaris:~$ svccfg import /var/svc/manifest/system/filesystem/repo.xml

Start the repo service:

bleonard@opensolaris:~$ svcadm enable repo 
bleonard@opensolaris:~$ svcs -l repo
fmri         svc:/system/filesystem/repo:default
name         Solaris ISO repository mounter
enabled      true
state        online
next_state   none
state_time   Mon Oct 11 09:40:07 2010
logfile      /var/svc/log/system-filesystem-repo:default.log
restarter    svc:/system/svc/restarter:default
dependency   require_all/none svc:/system/filesystem/local (online)

Step 4: Add the Local Repository as a Publisher

This is the new part in that a publisher can now point to a local file, bypassing the need to set up a local package server.

You can either set up the local file as a mirror or on its own. Use the following syntax to set it up as a mirror:

bleonard@solaris:~$ sudo pkg set-publisher -m file:///repo/repo/ -P solaris
bleonard@solaris:~$ pkg publisher
PUBLISHER                             TYPE     STATUS   URI
solaris                  (preferred)  origin   online
solaris                  (preferred)  mirror   online   file:///repo/repo/

To remove the mirror:

bleonard@solaris:~$ sudo pkg set-publisher -M file:///repo/repo/ solaris

To set it up on its own:

bleonard@solaris:~$ sudo pkg set-publisher -G -g file:///repo/repo/ -P solaris
bleonard@solaris:~$ pkg publisher
PUBLISHER                             TYPE     STATUS   URI
solaris                  (preferred)  origin   online   file:///repo/repo/

In the above we're removing the default remote repository and replacing it with the new local repository. You can see the status of the repository as follows:

bleonard@solaris:~$ pkgrepo info -s /repo/repo
solaris   3941     online           2010-11-11T00:04:39.171739Z

At this point, the repository is ready for use.

However, one minor limitation is that the repository doesn't include a search index (this was done to reduce the initial size of the repository):

bleonard@solaris:~$ pkg search hex
pkg: Some repositories failed to respond appropriately:
file protocol error: code: 11 reason: Search temporarily unavailable.
Repository URL: 'file:///repo/repo'. (happened 4 times)

And since the ISO is mounted directly (and therefore, read-only), one can't be built. If you happen to extract the ISO, a search index could be build with the following command:

sudo pkgrepo -s /repo/repo refresh


I like this approach because when a new repository becomes available, I simply need to replace the ISO file and I'm all set. For further details, including how to set up a local repository server, see the article How To Copy an Oracle Solaris 11 Express Software Package Repository.

Thursday Dec 02, 2010


We here in Solaris land have expended a lot of energy in the past explaining pfexec. In the simplest case it was described as an alias for sudo (and when I first come to Solaris, I'm somewhat embarrassed to admit I did just that, created an alias for sudo to pfexec). But having an alternative to sudo was one of those things that made Solaris "different". When OpenSolaris was first released we tracked unsuccessful searches against our package repository - and sudo was as the top of that list.

As part of the modernization effort for Solaris, sudo eventually found its way into OpenSolaris (beginning with the 2008.11 release). However, by that time I was pretty comfortable with pfexec and never looked back - until now that is.

A big change in the Solaris 11 Express release is that pfexec has been rendered relatively toothless out of the box. The "Primary Administrator" profile is no longer assigned to the user created during installation. If you've upgraded from an earlier release of OpenSolaris, you are unaffected by this change. However, on a fresh installation of Solaris 11 Express, commands that used to work will no longer. For example:

bleonard@solaris:~$ pfexec zfs create rpool/myfs
cannot create 'rpool/myfs': permission denied

However, sudo now works just fine:

bleonard@solaris:~$ sudo zfs create rpool/myfs

One big difference you'll notice is that sudo requires a password - and this your password, not the root password (which I'll address in a moment). The lack of a password prompt was the whole reason for the "Primary Administrator" role being dropped in the first place - although sudo can be configured to behave the same.

If you've upgraded to Solaris 11 Express, you have the opposite problem, pfexec still works as you're accustomed, however, sudo reports you to the sudo police.

bleonard@solaris:~$ sudo zfs create rpool/myfs

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

bleonard is not in the sudoers file.  This incident will be reported. 

The report actually shows up in the /var/adm/messages file:

Dec  2 11:21:57 solaris sudo: [ID 702911 auth.alert] bleonard : user NOT in sudoers ; TTY=pts/2 ; PWD=/export/home/bleonard ; USER=root ; COMMAND=/usr/sbin/zfs create rpool/myfs

I'll address setting up sudo at the end of this entry.

The root Password

In the continued simplification of the Solaris 11 Express installer, it now only asks for one password, which is used as the password for both the root account and the initial user account:

However, the root password is immediately expired, as you'll see if you try to switch to root:

bleonard@solaris:~$ su
su: Password for user 'root' has expired
New Password: 

As you no longer have Primary Administrator privileges, GUI tools requiring administrator privileges will now also prompt you for the root password. For example, if you try to start the Package Manger GUI, you'll first be presented with:

There is one little glitch to be aware of - if you attempt to run a GUI that prompts for the root password, and the root password is expired, the GUI just exits. No warning or prompt for a new password is provided. This issue is being addressed: Gksu does not report expired password. So just make sure you attempt an su from the command, and set a new root password, before trying to use the GUI tools that require the root password.

The root Role

If you look at the installer screen capture above, you'll see that the initial user is assigned administrative privileges. Although this is not in the form of the "Primary Administrator" profile, the user created at installation time does have the root role assigned to them.

bleonard@solaris:~$ roles

Some people mistakingly think that having the root role allows them to use pfexec. pfexec stands for Profile Execute, and executes commands against your assigned profiles - not your assigned roles. The root role simply allows you to su to the root user account.

The etc/sudoers file

Now to the reason why sudo works on a fresh install of Solaris 11 Express, but not on a version upgraded from OpenSolaris.  When a command is prefixed with pfexec, it first checks to see if the user executing the command has a profile which allows the execution of that command. Very similarly, when a command is prefixed with sudo, the /etc/sudoers files is first consulted to see the user is allowed to execute that command.

The /etc/sudoers file is well documented and you can defined very fine grained rules as to what a particular user is allowed to do. In the case of the user created during installation, the user is allowed to do everything (just as if they were root). Here's what the entry for my user, bleonard, looks like:

bleonard ALL=(ALL) ALL

The entry above is stating that user bleonard can run any command on any host as any user. For further details on how to fine tune a user's privileges, see the sudoers man page.

So, to configure an instance of Solaris 11 Express upgraded from OpenSolaris to operate like a freshly installed instance, you need to add a line like the above to the /etc/sudoers file. Note that the file is read-only and should be edited using the visudo editor - I hope you like vi :-).

One last note, if you want sudo to behave like pfexec (sans password), make the following tweak to your entry:


Finally, if you're on a fresh install of Solaris 11 Express and want to continue using pfexec, you can add the "Primary Administrator" profile as follows:

bleonard@solaris:~$ sudo usermod -P "Primary Administrator" bleonard
UX: usermod: bleonard is currently logged in, some changes may not take effect until next login.

Now creating that file system works just fine:

bleonard@solaris:~$ pfexec zfs create rpool/myfs

Happy sudoing or pfexecing, whichever you prefer.

Tuesday Nov 30, 2010

Upgrading ZFS

Upgrading to the latest release of Solaris doesn't automatically upgrade your existing ZFS pools and datasets. So if you're trying to take advantage of cool new features like dedup and crypto on those datasets, you'll have to update ZFS first.

Upgrading Your ZFS Pools

Beware - upgrading your root ZFS pool will essentially invalidate your prior boot environments as you will no longer be able to boot into them.

The version property will tell where you're currently at:

bleonard@opensolaris:~$ zpool get version rpool
rpool  version   14       local

And the -v (lower case) option will tell you what's available:

bleonard@opensolaris:~$ zpool upgrade -v
This system is currently running ZFS pool version 31.

The following versions are supported:

---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
 14  passthrough-x aclinherit
 15  user/group space accounting
 16  stmf property support
 17  Triple-parity RAID-Z
 18  Snapshot user holds
 19  Log device removal
 20  Compression using zle (zero-length encoding)
 21  Deduplication
 22  Received properties
 23  Slim ZIL
 24  System attributes
 25  Improved scrub stats
 26  Improved snapshot deletion performance
 27  Improved snapshot creation performance
 28  Multiple vdev replacements
 29  RAID-Z/mirror hybrid allocator
 30  Encryption
 31  Improved 'zfs list' performance

For more information on a particular version, including supported releases,
see the ZFS Administration Guide.

You can move to any version you'd like using the -V (upper case) option, however, I'm just going to move the the latest and greatest:
bleonard@opensolaris:~$ pfexec zpool upgrade rpool
This system is currently running ZFS pool version 31.

Successfully upgraded 'rpool' from version 14 to version 31

Just to verify the upgrade:

bleonard@opensolaris:~$ zpool get version rpool
rpool  version   31       default

Upgrading Your ZFS Datasets

The process is very similar. First, if you're interested in the version you're current at:

bleonard@opensolaris:~$ zfs get version rpool
rpool  version   3        -

Then to see what's available:

bleonard@opensolaris:~$ zfs upgrade -v
The following filesystem versions are supported:

---  --------------------------------------------------------
 1   Initial ZFS filesystem version
 2   Enhanced directory entries
 3   Case insensitive and File system unique identifier (FUID)
 4   userquota, groupquota properties
 5   System attributes

For more information on a particular version, including supported releases,
see the ZFS Administration Guide.

I'm going to upgrade all the file systems at once. However, there is the option to upgrade just a particular file system, for example:

bleonard@opensolaris:~$ pfexec zfs upgrade rpool/export/home
1 filesystems upgraded 

Or a file system and its descendants:

bleonard@opensolaris:~$ pfexec zfs upgrade -r rpool/export
1 filesystems upgraded
1 filesystems already at this version

But to simply upgrade all:

bleonard@opensolaris:~$ pfexec zfs upgrade -a
25 filesystems upgraded
2 filesystems already at this version

This process may take several minutes to complete, depending of the number file systems that need to be upgraded.

Thursday Nov 18, 2010

Upgrading from OpenSolaris 2009.06 to Solaris 11 Express 2010.11

In this entry I document my experience of upgrading from OpenSolaris 2009.06 to Solaris 11 Express 2010.11. I selected to include lots of output, in case you want to compare your experience to mine.[Read More]

Tuesday Nov 16, 2010

Video Tutorial: Installing Solaris 11 Express in VirtualBox

Jim Laurent has put together a nice 12 minute screencast on installing Solaris 11 Express in VirtualBox, including how to setup shared folders with the host OS.

The video also covers an important tip, which is that the root password is expired at install time and must be immediately changed. Otherwise, GUI tools prompting for the root password will not work. You can read the details in the release notes: Gksu Does Not Report Expired Password.

Wednesday Nov 03, 2010


I was about to write a new blog on Solaris' Fault Management Architecture when I realized my friend Bob Netherton has beaten to it by almost 3 years.

We like to tout the benefits of FMA a lot, but it's often hard to demonstrate because you don't want to go around destroying CPUs and memory modules just to see FMA in action. However, by creating a ZFS pool with files as disks, it's quite easy to demonstrate and that's exactly what Bob does in his blog ZFS and FMA - Two great tastes ......

Note, there's also an FMA Demo Kit that you can use to simulate faults to other hardware components, but I haven't played with that myself yet. Other resources I found helpful:

Friday Oct 15, 2010

Associating a PID with a Service

The question of how to associate a process id with a SMF service recently came across an internal alias and I thought it was worth sharing.

Take this generic looking Java process:

bleonard@opensolaris:/system$ ps -fp 949
     UID   PID  PPID   C    STIME TTY         TIME CMD
    root   949   947   0   Oct 11 ?           3:46 /usr/jdk/jdk1.6.0_13/bin/java -Xms4M -Xmx128M -D

If I'd like to trace back to which SMF service started this process, I start by getting the process' contract ID:

bleonard@opensolaris:/system$ ps -o ctid -p 949

I can then use the ctstat command to cross reference the contract ID to the SMF service:

bleonard@opensolaris:/system$ ctstat -vi 59
59      0       process owned   7       0       -       -       
	cookie:                0x20
	informative event set: none
	critical event set:    core signal hwerr empty
	fatal event set:       none
	parameter set:         inherit regent
	member processes:      947 949
	inherited contracts:   none
	service fmri:          svc:/application/management/common-agent-container-1:default
	service fmri ctid:     59
	creator:               svc.startd
	aux:                   start

And just to go full circle:

bleonard@opensolaris:/system$ svcs -lp  common-agent-container-1
fmri         svc:/application/management/common-agent-container-1:default
name         Cacao, a common Java container for JDMK/JMX based management solution
enabled      true
state        online
next_state   none
state_time   Mon Oct 11 12:54:59 2010
logfile      /var/svc/log/application-management-common-agent-container-1:default.log
restarter    svc:/system/svc/restarter:default
contract_id  59 
dependency   require_all/none svc:/system/filesystem/local (online)
dependency   require_all/none svc:/network/initial (online)
process      947 /usr/lib/cacao/lib/tools/launch -w /var/run/cacao/instances/default/run -L 1638
process      949 /usr/jdk/jdk1.6.0_13/bin/java -Xms4M -Xmx128M -D

For a nice little tutorial on contracts, check out this Contract Subsystem Lab.

Wednesday Oct 06, 2010

IPS Search and Actions

When searching for an IPS package, I usually type something quick and simple like:

bleonard@opensolaris:~$ pkg search netbeans
INDEX      ACTION    VALUE                     PACKAGE
description set       NetBeans                  pkg:/libnb-php@6.5.1-0.111
description set       NetBeans                  pkg:/libnb-groovy@6.5-0.86
description set       NetBeans                  pkg:/netbeans-java@6.5.1-0.111

And scroll through the result list hoping to find what I'm looking for. However, in my example above, that's 437 lines!

bleonard@opensolaris:~$ pkg search netbeans | wc -l

Often times I'll shorten the list by grepping for my build:

bleonard@opensolaris:~$ pkg search netbeans | grep 111
description set       NetBeans                  pkg:/libnb-php@6.5.1-0.111
description set       NetBeans                  pkg:/netbeans-java@6.5.1-0.111
description set       NetBeans                  pkg:/libnb-visualweb@6.5.1-0.111

Which reduces the list down to 134 for this example:

bleonard@opensolaris:~$ pkg search netbeans | grep 111 | wc -l

However, I generally tend to ignore the first two columns, INDEX and ACTION. Actions, in the case of IPS, are actually nouns and not verbs as the name would lead you to expect. Actions are the things that get installed (files, directories, links, drivers, licenses, users, groups, etc). Here's a nice description of the Actions in IPS.

When you perform a search, the search string is checked against all of these actions types, which as seen above, can produce a lot of results. For example, 5 of the 134 results above all refer to the same package, pkg:/sunstudioexpress@0.2009.3.1:

bleonard@opensolaris:~$ pkg search netbeans | grep 111 | grep pkg:/sunstudioexpress@0.2009.3.1
basename   dir       opt/SunStudioExpress/prod/nb-dbxtool/ide10/docs/org/netbeans pkg:/sunstudioexpress@0.2009.3.1-0.111
basename   dir       opt/netbeans-6.5ss/ide10/docs/org/netbeans pkg:/sunstudioexpress@0.2009.3.1-0.111
basename   file      opt/SunStudioExpress/prod/nb-dbxtool/bin/netbeans pkg:/sunstudioexpress@0.2009.3.1-0.111
basename   file      opt/netbeans-6.5ss/bin/netbeans pkg:/sunstudioexpress@0.2009.3.1-0.111
basename   link      opt/SunStudioExpress/netbeans pkg:/sunstudioexpress@0.2009.3.1-0.111

That's because the search results are returning 3 directories, 2 files and 1 link, all in the same package, that contain the search string "netbeans".

If you just want to see a list packages that match the search string, use the -p option.

bleonard@opensolaris:~$ pkg search -p netbeans | grep 111 
pkg:/amp-dev@0.5.11-0.111 (
pkg:/developer/netbeans/plugin/nb-dtrace@1.0-0.111 (
pkg:/developer/sunstudio12u1@12.1.1-0.111 (

This returns a much more reasonable 60 results with just the package names, making the result list much more easier to read through. I wish -p was the default option, rather than the -a (which shows the matching actions).

Wednesday Sep 29, 2010

ps tip

I often use the ps -ef command when looking for a running process. However, sometimes I'm frustrated by the fact that ps truncates the the CMD output. As an example:

bleonard@opensolaris:~$ ps -ef | grep evolution-data
bleonard 15803     1   0   Sep 27 ?           0:00 /usr/lib/evolution-data-server-1.2/evolution-data-server-2.24 --oaf-activate-i

In a recent opensolaris-help forum post, Mike Meyer provided some undocumented BSD compatible options to get more details:

I habitually use "ps auxww" (note no "-" - bsd ps doesn't require it, and that triggers the BSD mode of /usr/bin/ps), which is all processes (ax) in a "user" (u) format. No w option truncates each line at 80 characters, one w option truncates each line at 132 characters, and two w options doesn't truncate it at all.

So, for my example:

bleonard@opensolaris:~$ ps auxww | grep evolution-data
bleonard 15803  0.0  0.22351610840 ?        S   Sep 27  0:00 /usr/lib/evolution-data-server-1.2/evolution-data-server-2.24 --oaf-activate-iid=OAFIID:GNOME_Evolution_DataServer_CalFactory:1.2 --oaf-ior-fd=28

Of course, as it was also pointed out, once you know the process id you can get the same argument information, in a more readable format using, the pargs command:

bleonard@opensolaris:~$ pargs 15803
15803:	/usr/lib/evolution-data-server-1.2/evolution-data-server-2.24 --oaf-activate-ii
argv[0]: /usr/lib/evolution-data-server-1.2/evolution-data-server-2.24
argv[1]: --oaf-activate-iid=OAFIID:GNOME_Evolution_DataServer_CalFactory:1.2
argv[2]: --oaf-ior-fd=28

Or, in single line as pointed out in the comments below:

bleonard@opensolaris:~$ pgrep evolution-data | xargs pargs 
15803:	/usr/lib/evolution-data-server-1.2/evolution-data-server-2.24 --oaf-activate-ii
argv[0]: /usr/lib/evolution-data-server-1.2/evolution-data-server-2.24
argv[1]: --oaf-activate-iid=OAFIID:GNOME_Evolution_DataServer_CalFactory:1.2
argv[2]: --oaf-ior-fd=28

Tuesday Sep 28, 2010

Understanding the Space Used by ZFS

Until recently, I've been confused and frustrated by the zfs list output as I try to clear up space on my hard drive.

[Read More]

Monday Sep 27, 2010

The Second Sighting

The ObservatoryTwo years ago we started The Observatory shortly after the first release of OpenSolaris hit the streets. Since then we've published over 200 entries to which we've received almost 1000 comments and we've consistently been a top 20 blog at (or what we simply call bsc).

A couple of weeks back, John Fowler, EVP of Systems here at Oracle, delivered a strategy webcast where he outlined the future of Solaris. In that webcast, John finally announced what we've all been anticipating as we've watched the evolution of OpenSolaris - Solaris 11. And just last week at Oracle OpenWorld, Solaris 11 Express, which had only been a rumor up to this point, was finally officially announced.

To that extent, we also want to evolve The Observatory as we move forward with the future of Oracle Solaris. Therefore, we are making a slight adjustment to the tagline of this blog, moving from "A Closer Look at Using OpenSolaris" to "A Closer Look at Using Oracle Solaris", which will cover all Oracle Solaris technologies, including OpenSolaris (call me crazy like a fox, but I'm still running OpenSolaris 2009.06).

So, for the most part, the song remains the same, we just have an updated album cover.

Friday Jul 16, 2010

Process Arguments and Environment Variables

If you want to quickly determine the arguments or environment variables used by a process, pargs, one of a handful of very useful proc tools, is for you.

For example, NetBeans is started with a bunch of arguments. Rather than hunting through the netbeans configuration files, I can locate netbeans' process id:

bleonard@opensolaris:~$ ps -ef | grep netbeans
bleonard  5215     1   0 17:35:16 ?           0:00 /bin/bash /usr/netbeans/bin/../platform10/lib/nbexec --userdir /export/home/ble
Then simply run:
bleonard@opensolaris:~$ pargs 5215
5215:	/bin/bash /usr/netbeans/bin/../platform10/lib/nbexec --userdir /export/home/ble
argv[0]: /bin/bash
argv[1]: /usr/netbeans/bin/../platform10/lib/nbexec
argv[2]: --userdir
argv[3]: /export/home/bleonard/.netbeans/6.7
argv[4]: --jdkhome
argv[5]: /usr/java
argv[6]: --branding
argv[7]: nb
argv[8]: --clusters
argv[9]: /usr/netbeans/nb6.7:/usr/netbeans/ergonomics1:/usr/netbeans/ide11:/usr/netbeans/java2:/usr/netbeans/bin/../xml2:/usr/netbeans/apisupport1:/usr/netbeans/webcommon1:/usr/netbeans/websvccommon1:/usr/netbeans/enterprise5:/usr/netbeans/bin/../mobility8:/usr/netbeans/profiler3:/usr/netbeans/ruby2:/usr/netbeans/bin/../python1:/usr/netbeans/php1:/usr/netbeans/bin/../visualweb2:/usr/netbeans/bin/../soa2:/usr/netbeans/identity2:/usr/netbeans/bin/../uml6:/usr/netbeans/harness:/usr/netbeans/cnd2:/usr/netbeans/dlight1:/usr/netbeans/groovy1:/usr/netbeans/bin/../extra:/usr/netbeans/bin/../javafx2:
argv[10]: -J-Dnetbeans.importclass=org.netbeans.upgrade.AutoUpgrade
argv[11]: -J-Dnetbeans.accept_license_class=org.netbeans.license.AcceptLicense
argv[12]: -J-Xmx512m
argv[13]: -J-client
argv[14]: -J-Xss2m
argv[15]: -J-Xms32m
argv[16]: -J-XX:PermSize=32m
argv[17]: -J-XX:MaxPermSize=200m
argv[18]: -J-Xverify:none
argv[19]: -J-Dapple.laf.useScreenMenuBar=true
argv[20]: -J-Dsun.java2d.noddraw=true

To see the environment variables available to a process, use the -e option. For example, here are some of the relevant variables used by sqlplus:

bleonard@opensolaris:~$ pargs -e `pgrep sqlplus`
7114:	sqlplus
envp[0]: ORACLE_HOME=/export/home/bleonard/Download/instantclient_11_2
envp[12]: LD_LIBRARY_PATH=/export/home/bleonard/Download/instantclient_11_2
envp[48]: _=/export/home/bleonard/Download/instantclient_11_2/sqlplus

Thursday Jul 15, 2010


Continuing on the thread of who logged in last, Richard Hamilton has provided a nice little C program to dump the contents of /var/adm/lastlog. Here's what he has to say about lastlog:

/var/adm/lastlog: this file is an array of fixed-sized binary records, containing a single timestamp (time of last login), the tty name, and for remote logins, the host name or IP (in text form, but only 16 characters long). The UID of the user is the record number. That means the file may appear gigantic, but it's actually sparse on disk, not nearly as large as it appears. But most copy/backup/archive utilities do not preserve sparseness, so they would produce a copy that was as large as it appeared.

I've attached the source for a program that will dump out this file in readable form. Remember, there's only one entry per UID, so it will show only the single most recent login time (even if they're logged in more than once at a time), and it does not show logouts. But with a fixed set of users, it doesn't grow, so people tend to leave it alone and not blow it away. In other words, it may not be all the information you want, but it's more likely to be there.

To build the program, you'll need a C compiler. If you don't already have one installed, there are several to choose from, but for this small C program I'm going with The GNU C compiler.

bleonard@os200906:~$ pfexec pkg install SUNWgcc
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                    4/4   2100/2100   30.26/30.26 

PHASE                                        ACTIONS
Install Phase                              2537/2537 

Once SUNWgcc is installed, download lastlog.c and compile it as follows:

bleonard@os200906:~/Downloads$ gcc lastlog.c -o lastlog

Then run it to see the contents of /var/adm/lastlog:

bleonard@os200906:~/Downloads$ ./lastlog 
root       console  Fri Dec  5 18:47:28 2008 
bleonard   console  Wed Jul 14 11:26:48 2010 
karl       pts/5    Thu Jul 15 11:12:57 2010

Wednesday Oct 07, 2009


Netperf is a cool little utility that I discovered while working with Network Virtualization. It's technically a network benchmarking tool, but it's fun to use to hammer your network with load and test out the bandwidth control features provided by network virtualization.

Netperf used to be available in the now decommissioned contrib repository. That's OK, it's easy enough to build.

Building Netperf for Solaris

If you want to save yourself the steps below, just save the following binaries to your /usr/bin directory: netperf and netserver. Here are the instructions from the Netperf manual.

  1. Download Netperf.  At the time of this writing the latest stable build is netperf-2.4.5.

  2. Extract the archive:
    bleonard@solaris:~/Downloads$ tar -xvf netperf-2.4.5.tar.bz2 
  3. Make sure you have gcc-3 and header-math installed:
    bleonard@solaris:~$ pkg list gcc-3 header-math
    NAME (PUBLISHER)                              VERSION         STATE      UFOXI
    developer/gcc-3                               3.4.3- installed  -----
    system/library/math/header-math               0.5.11- installed  -----
  4. Run configure, overriding the default install directory of /usr/local to /usr:
    bleonard@solaris:~/Downloads/netperf-2.4.5$ ./configure --prefix=/usr
    checking build system type... i386-pc-solaris2.11
    config.status: executing depfiles commands

  5. Run make:
    bleonard@solaris:~/Downloads/netperf-2.4.5$ make
    make  all-recursive
    make[1]: Entering directory `/home/bleonard/Downloads/netperf-2.4.5'
    make[1]: Leaving directory `/home/bleonard/Downloads/netperf-2.4.5'
  6. The run make install:
    bleonard@solaris:~/Downloads/netperf-2.4.5$ sudo make install
    Making install in src
    make[1]: Entering directory `/home/bleonard/Downloads/netperf-2.4.5/src'
    make[1]: Leaving directory `/home/bleonard/Downloads/netperf-2.4.5'
  7. Verify the binaries were made:
    bleonard@solaris:~$ which netperf netserver

Using Netperf

To verify the installation, first start the benchmark server:

bleonard@solaris:~$ netserver
Starting netserver at port 12865
Starting netserver at hostname port 12865 and family AF_UNSPEC

Then run netperf over the loopback interface, which will run a TCP_STREAM test of 10 seconds:

bleonard@solaris:~$ netperf -H
TCP STREAM TEST from ::ffff: ( port 0 AF_INET to ::ffff: ( port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

128000  49152  49152    9.99     19036.97   

Now that we know Netperf is set up correctly, we can use it to test bulk data transfer performance between 2 hosts (or 2 zones). TCP stream performance is the default test type and it's simply performed by running the command:

netperf -H <remotehost>

which will run a 10 second test between the local and remote system.

So I'm going to use Netperf to run some tests between the global zone and a local zone, which has been set up to use a virtual NIC exclusively. To create such a configuration on your own, see the blog entry Zones and Network Virtualization.

The local zone, myzone, is currently up and running:

bleonard@solaris:~$ zoneadm list -cv
  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              native   shared
   - myzone           running    /zones/myzone                  ipkg     excl 

The first thing I'm going to do is copy the netserver program over to the local zone:

bleonard@solaris:~$ sudo cp /usr/bin/netserver /zones/myzone/root/usr/bin/.

And then start the netserver:

bleonard@solaris:~$ sudo zlogin myzone /usr/bin/netserver
Starting netserver at port 12865
Starting netserver at hostname port 12865 and family AF_UNSPEC

Now let's test the connection between the global and local zone:

bleonard@solaris:~$ netperf -H
TCP STREAM TEST from ::ffff: ( port 0 AF_INET to ::ffff: ( port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

128000  49152  49152    10.00    1722.31  

Here we can see the throughput between my zones is 1722 Mbit/s. Now let's reduce the max bandwidth of the virtual NIC to 500 Mbit/s and try the test again:

bleonard@solaris:~$ sudo dladm set-linkprop -p maxbw=500 myzone0

bleonard@solaris:~$ dladm show-vnic
LINK         OVER         SPEED  MACADDRESS           MACADDRTYPE         VID
myzone0      e1000g0      500    2:8:20:59:0:b5       random              0

bleonard@solaris:~$ netperf -H
TCP STREAM TEST from ::ffff: ( port 0 AF_INET to ::ffff: ( port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

128000  49152  49152    10.00     482.77   

Or at a ridiculously low 2 Mbit/s:

bleonard@solaris:~$ sudo dladm set-linkprop -p maxbw=2 myzone0

bleonard@solaris:~$ netperf -H
TCP STREAM TEST from ::ffff: ( port 0 AF_INET to ::ffff: ( port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

128000  49152  49152    10.38       1.07  

Good stuff.

I've just scratched the surface of Netperf, but this simple introduction suits my purposes for testing network virtualization. For more fun check out the Netperf manual.

Tuesday Jul 14, 2009

Zones and Network Virtualization

If you're like me and working with zones on your laptop and/or desktop, you probably only have one network interface card to work with. Therefore, the zones I've created share the single network interface with the global zone (ip-type=shared).

Behind the scenes, Solaris creates a logical interface for the zone to use. The logical interface appears in ifconfig as your physical interface with an instance number. For example:

bleonard@solaris:~$ ifconfig -au4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
	inet netmask ff000000 
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
	zone myzone
	inet netmask ff000000 
e1000g0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 2
	inet netmask ffffff00 broadcast
e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
	zone myzone
	inet netmask ffffff00 broadcast

You can see both the loopback (loO) and physical (e1000g0) have an instance (lo0:1 and e1000g0:1) that was created for the zone myzone. These logical interfaces only exist when the zone is running. If you halt the zone, they disappear.

From inside the zone, I only see the logical interfaces:

root@myzone:~# ifconfig -au4
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet netmask ff000000 
e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet netmask ffffff00 broadcast

However, I have no control over them. For example, if I try to bring down e1000g0:1:

root@myzone:~# ifconfig e1000g0:1 inet down
ifconfig: setifflags: SIOCSLIFFLAGS: e1000g0:1: permission denied 

The global zone is responsible for managing the local zone's network configuration.

Network Virtualization

Oracle Solaris 11 introduces network virtualization technology. For example, I can create a virtual network interface card (vnic) that has all the properties of a physical nic.

bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 myzone0
bleonard@solaris:~$ dladm show-link 
e1000g0     phys     1500   up       --
iwh0        phys     1500   down     --
vboxnet0    phys     1500   unknown  --
myzone0     vnic     1500   up       e1000g0

Now it's as if my laptop has 2 physical network interface cards. Using this "new" card, I can create a zone with an exclusive IP stack. My zone config would look something like follows:

bleonard@solaris:~$ cat myzone.config
set zonepath=/zones/myzone
set ip-type=exclusive	
add net
set physical=myzone0

Note there's no longer an IP address associated with the zone configuration. With a dedicated IP stack the zone will be able to manage its own IP.

Create the zone:

bleonard@solaris:~$ sudo zonecfg -z myzone -f myzone.config

Install the zone:

bleonard@solaris:~$ sudo zoneadm -z myzone install
A ZFS file system has been created for this zone.
   Publisher: Using solaris ( ).
       Image: Preparing at /zones/myzone/root.
 Credentials: Propagating Oracle_Solaris_11_Express_Support.key.pem
 Credentials: Propagating Oracle_Solaris_11_Express_Support.certificate.pem
       Cache: Using /var/pkg/download.
Sanity Check: Looking for 'entire' incorporation.
  Installing: Core System (output follows)
               Packages to install:     1
           Create boot environment:    No
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1         1/1      0.0/0.0

PHASE                                        ACTIONS
Install Phase                                  11/11 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 
               Packages to install:    45
           Create boot environment:    No
               Services to restart:     3
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                45/45 12511/12511    89.1/89.1

PHASE                                        ACTIONS
Install Phase                            17958/17958 

PHASE                                          ITEMS
Package State Update Phase                     45/45 
Image State Update Phase                         2/2 
  Installing: Additional Packages (output follows)
               Packages to install:    46
           Create boot environment:    No
               Services to restart:     2
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                46/46   4498/4498    26.5/26.5

PHASE                                        ACTIONS
Install Phase                              6143/6143 

PHASE                                          ITEMS
Package State Update Phase                     46/46 
Image State Update Phase                         2/2 

        Note: Man pages can be obtained by installing SUNWman
 Postinstall: Copying SMF seed repository ... done.
 Postinstall: Applying workarounds.
        Done: Installation completed in 486.420 seconds.

  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.

Create a configuration file for the zone. Note, here we can define the zone's IP configuration (or we could do it later):

bleonard@solaris:~$ cat sysidcfg
network_interface=myzone0 {

Copy the sysidcfg file to the zone:

bleonard@solaris:~$ sudo cp sysidcfg /zones/myzone/root/etc/.

Boot the zone:

bleonard@solaris:~$ sudo zoneadm -z myzone boot

Log into zone. The first login will take some time as the zone completes it's system configuration:

bleonard@solaris:~$ sudo zlogin -C myzone
[Connected to zone 'myzone' console]
Hostname: myzone
Loading smf(5) service descriptions: 3/3
 network_interface=myzone0 {
myzone0 is not a valid network interface  line 3 position 19
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: myzone0.
Note the message about myzone0 being an invalid network interface. This appears to be benign as a few lines down we see myzone0 getting configured. If you used the root_password setting from above, you can log in as root/abc123:
myzone console login: root
Password: abc123
May 31 08:30:02 myzone login: ROOT LOGIN /dev/console
Oracle Corporation      SunOS 5.11      snv_151a        April 2011

As with shared IP, you can see the interface using ifconfig:

root@myzone:~# ifconfig -au4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet netmask ff000000 
myzone0: flags=1000863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet netmask ffffff00 broadcast
        ether 2:8:20:59:0:b5 

However, now you can also manage it. For example:

root@myzone:~# ifconfig myzone0 down
root@myzone:~# ifconfig -au4
lo0: flags=2001000849<⁞UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet netmask ff000000 

And back in the global zone, there's no more logical interfaces cluttering up the ifconfig output:

bleonard@solaris:~$ ifconfig -au4
lo0: flags=2001000849<UP,LOOPBACK,⁞RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
	inet netmask ff000000 
e1000g0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 2
	inet netmask ffffff00 broadcast
In addtion to this, virtual nics provide a whole bunch of control over the data passing through the network interface. For a brief introduction to that see Fun with Crossbow.

The Observatory is a blog for users of Oracle Solaris. Tune in here for tips, tricks and more as we explore the Solaris operating system from Oracle.

Connect with Oracle Solaris:


« April 2014