Tuesday Nov 29, 2011

Package Version Numbers, why are they so important

One of the design goals of IPS has been to allow people to easily move forward to a supported "Surface" of component. That is to say, when you 

# pkg update

your system, you get the latest set of components which all work together, based on the packages you already have installed. During development, this has meant simply you update to the latest "build" of the components. (During development, we build everything and publish everything every two weeks).

Now we've released Solaris 11 using the IPS technologies, things are a bit more complicated. We need to be able to reflect all the types of Solaris release we are doing. For example Solaris Development builds, Solaris Update builds and "Support Repository Updates" (the replacement for patches) in the version scheme. So simply saying "151" as the build number isn't sufficient to articulate what you are running, or indeed what is available to update to

In my previous blog post I talked about creating your own package, and gave an example FMRI of


But it's probably more instructive to look at the FMRI of a Solaris package. The package "core-os" contains all the common utilities and daemons you need to use Solaris.

 $ pkg info core-os
          Name: system/core-os
       Summary: Core Solaris
   Description: Operating system core utilities, daemons, and configuration
      Category: System/Core
         State: Installed
     Publisher: solaris
       Version: 0.5.11
 Build Release: 5.11
Packaging Date: Wed Oct 19 07:04:57 2011
          Size: 25.14 MB
          FMRI: pkg://solaris/system/core-os@0.5.11,5.11-

The FMRI is what we will concentrate on here. In this package "solaris" is the publisher. You can use the pkg publisher command to see where the solaris publisher gets it's bits from

$ pkg publisher
PUBLISHER                             TYPE     STATUS   URI
solaris                               origin   online   http://pkg.oracle.com/solaris/release/

So we can see we get solaris packages from pkg.oracle.com.  The package name is system/core-os. These can be arbitrary length, just to allow you to group similar packages together. Now on the the interesting? bit, the versions, everything after the @ is part of the version. IPS will only upgrade to a "higher" version.


core-os = Package Name

0.5.11 = Component - in this case we're saying it's a SunOS 5.11 package

, = separator

5.11 = Built on version - to indicate what OS version you built the package on

- = another separator = Branch Version

: = yet another separator

20111019T070457Z = Time stamp when the package was published

So from that we can see the Branch Version seems rather complex. It is necessarily so, to allow us to describe the hierarchy of releases we do In this example we see the following

0.175: is known as the trunkid, and is incremented each build of a new release of Solaris. During Solaris 11 this should not change 

0: is the Update release for Solaris. 0 for FCS, 1 for update 1 etc

0: is the SRU for Solaris. 0 for FCS, 1 for SRU 1 etc

0: is reserved for future use

2: Build number of the SRU

1: Nightly ID - only important for Solaris developers

Take a hypothetical example


This would be build 4 of SRU 5 of Update 1 of Solaris 11

This is actually documented in a MOS article 1378134.1 Which you can read if you have a support contract.

Tuesday Oct 25, 2011

Adventures in Regular Expressions

I'm one of those people who will get stuck in and solve problems, even if I don't know everything about an area a problem lies in. As such I often find I'm learning new and unexpected things. Hey, that's why I love coming to work.

So the project I'm working on at the moment relates to how we build the SRUs (the Support Repository Updates), which replace patches in Solaris 11. As such I'm learning a lot about IPS - the Image Packaging System, and in particular how the tools it provides help you deliver consistent and upgradeable packages.

The trick I picked up this week is about Regular Expressions. I have need to change some values in an FMRI of a package. I had been doing it in a set of shell scripts until Mark Nelson of the SMF team pointed out I was rewriting pkgmogrify(1)

So reading the pkgmogrify(1) man page left me feeling less than knowledgeable, I'd sort of gathered it worked with regular expressions. Now these are well known in the industry, just I've never needed to use them.

So after a bit of experimenting I find I can substitute values in a string using the "edit" directive. This is the relevant portion of the man page, which makes sense now I know the answer

     edit      Modifies an attribute of the action.  Three arguments are
               taken.  The first is the name of the attribute, the second
               is a regular expression matching the attribute value, and
               the third is the replacement string substituted for the
               portion of the value matched by the regular expression.
               Unlike the regular expression used to match an action, this
               expression is unanchored.  Normal regular expression
               backreferences, of the form '\1', '\2', etc., are available
               in the replacement string, if there are groups defined in
               the regular expression.

The last sentence is the clincher for what I need to do. I can search a string for a pattern, and load it in to the "groups", and then reference them in the replacement string. So for example if I want to change the package version string from "-" to "-", I can do it using a transform, but where 175 might change and I want that to be reflected in the resulting manifest


$ cat trans/mytransform 
<transform set name=pkg.fmri -> edit value \

'@*-0.([0-9])([0-9])([0-9]).+' '-0.\1\2\'>

The "Groups" are defined in the round brackets eg. ([0-9]) for example. Then I can simply run pkgmogrify to get the  result


$ pkgmogrify -I trans -O <output file> <input manifest> mytransform


This performs the substitution just as I needed it to.

Tuesday Oct 04, 2011

How to create your own IPS packages

It's been ages since I blogged, so apologies for that. I'm at Oracle Openworld 2011 this week, and I was helping out on the Solaris booths in the DemoGrounds. One of the people I talked to asked a question, which I've heard internally at Oracle a few times recently. That is "How can I package up my own app as an IPS package?" I've answered it a few times, and so suddenly it struck me as a good subject for a blog.

 Most of this information is available either in the documentation,  or other blogs, but this is also for me to reference back to (rather than my notes)

With IPS, packages are published to a repository (or repo). This can either be a special pkg web server or simply a directory. For simplicity I've used the latter

$ export PKGREPO=/path/to/repo

$ pkgrepo create $PKGREPO 

This populates the $PKGREPO with the required repository structure, you only need to do this once.

Packages are described by manifests. You need one per package. If you have existing System V packages, you can allow the tools to generate it all for you, but simply passing the SystemV package file in the a tool called pkgsend

$ pkgsend generate sysvpkg > sysvpkg.mf

However, often my own tools are just added manually by hand or archived together with tar. So if I want to turn those in to an IPS package, I need to create my own manifest. Again fortunately pkgsend can help, but you'll need to add some details.

 So let's assume I've put all my tools in /opt/mytools

$ export ROOT=/opt/mytools
$ pkgsend generate $ROOT > mytools.mf

This manifest needs a line adding to it to describe the package

set name=pkg.fmri \

This states the publisher (tools), package name (mytools) and the version (1.3,0.5.11-0.0.0)

So once you have the working manifest - and there's a lot more detail we can add to these,  they can be published. (to the $PKGREPO directory we created earlier)

$ pkgsend publish -s $PKGREPO -d $ROOT mypkg.mf

Note for each successfully published you get a "Published" message, but silent failure (exit code is 1 though so it can be detected in a script)

You can then add the repo you've just populated to your test machine. You'll need to do this a as a privileged user, such as root

# pkg set-publisher -O $PKGREPO tools

and add a package

# pkg install mytools  Packages to install:  1
Create boot environment: No

DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1     716/716      2.4/2.4

PHASE                                        ACTIONS
Install Phase                                746/746

PHASE                                          ITEMS
Package State Update Phase                       1/1
Image State Update Phase                         2/2

PHASE                                          ITEMS
Reading Existing Index                           8/8
Indexing Packages                                1/1

So this is now installed on my system. There are so many interesting things you can add to the manifest, that if you're interested, I'll try to blog about in the future. However one thing I've glossed over here is the package version. It's surprisingly important that you understand what you set that to, so I'll make that the subject of a future blog.

Thursday Nov 01, 2007

Installing Indiana/Opensolaris

For a few days recently I have been looking at the future of packaging, pkg(5) or IPS. IPS looks really powerful and quite simple. It will allow us to generate fixes and deliver them much more simply. What I've been thinking about is how and when will we generate fixes using this mechanism.

Any way as a result I've signed up for pkg-discuss-AT-opensolaris-DOT-org and indiana-discuss-AT-opensolaris.org. Both of these are very active and full of interesting discussions (and arguments) and ideas. Anyway, it's not surprising there has been so much activity recently. Today indiana-discuss announced the launch of the developer preview of the opensolaris binary distribution. So I tried it out on a couple of machines. My laptop first, an Acer Ferrari 4005. Everything just worked. The LiveCD booted up, really quickly actually, well done the team for getting the performance up so well. Even wireless worked, though that's probably because I've already swapped the Broadcom wireless miniPCI card for an Atheros one. Unfortunately I have no spare slices available on the laptop so I move on to my next machine.

This is my home PC, usually running WindowsXP for the kids, it has never successfully Solaris for reasons that will become apparent. I have just upgraded the hard drive so theres 60Gb partition free for me to do some damage.

Booting the livecd failed, or rather Xorg failed to display anything. My machine is an old Athlon XP2600 with an AGP radeon x1600pro graphics card. Great for games, but unfortunately the Solaris/OpenSolaris Radeon driver doesn't support it. Fortunately Stephan Hahn blogged about how to get Xorg to use the vesa driver from the livecd. With that in place I got the gnome gui up and gave the install a go.

The installer uses dwarf-caiman, a cut down slim line installer which is nice and easy to navigate. The install itself was really quick - there's only a CD's worth installed. The rest should be added later over the web from the IPS repository. Unfortunately that is where my old machine creaked too much. The onboard ethernet is an nforce2 gigabit ethernet. It should work with the nge driver but I think it's just too old. I tried adding an alias for it using 

# add_drv '"pci10de,66"' nge

But even though I could plumb it there was no traffic going through it :-( I guess I'll have to find another ethernet card.

The install claimed it failed, but it did come up fine after a reboot, though I had to add a user again at single user because the useradd hadn't worked. Warning here. root is just a role that users can take on - so you can't log in as root as you might expect from a "normal" solaris system.

I'm pretty impressed. Nice installer, lightweight liveCD to get you started. zfs root and pkg(5) to add new stuff (or it will when I get a new ethernet adapter. I wonder if I can get one of my old USB wireless sticks to work :-). Plus it seems to be more responsive under OpenSolaris than windows XP.

 Do give it a go, it is one vision of the future of opensolaris






Chris W Beal-Oracle


« February 2017