Wednesday Oct 24, 2012

Controlling server configurations with IPS

I recently received a customer question regarding how they best could control which packages and which versions were used on their production Solaris 11 servers.  They had considered pointing each server at its own software repository - a common initial approach.  A simpler method leverages one of dependency mechanisms we introduced with Solaris 11, but is not immediately obvious to most people.

Typically, most internal IT departments qualify particular versions for production use.  What this customer wanted to do was insure that their operations staff only installed internally qualified versions of Solaris on their servers.  The easiest way of doing this is to leverage the 'incorporate' type of dependency in a small package defined for each server type.  From the reference " Packaging and Delivering Software With the Image Packaging System in Oracle® Solaris 11.1":

 The incorporate dependency specifies that if the given package is installed, it must be at the given version, to the given version accuracy. For example, if the dependent FMRI has a version of 1.4.3, then no version less than 1.4.3 or greater than or equal to 1.4.4 satisfies the dependency. Version 1.4.3.7 does satisfy this example dependency. The common way to use incorporate dependencies is to put many of them in the same package to define a surface in the package version space that is compatible. Packages that contain such sets of incorporate dependencies are often called incorporations. Incorporations are typically used to define sets of software packages that are built together and are not separately versioned. The incorporate dependency is heavily used in Oracle Solaris to ensure
that compatible versions of software are installed together. An example incorporate dependency is:

depend type=incorporate fmri=pkg:/driver/network/ethernet/e1000g@0.5.11,5.11-0.175.0.0.0.2.1

So, to make sure only qualified versions are installed on a server, create a package that will be installed on the machines to be controlled.  This package will contain an incorporate dependency on the "entire" package, which controls the various components used to be build Solaris.  Every time a new version of Solaris has been qualified for production use, create a new version of this package specifying the new version of "entire" that was qualified.  Once this new control package is available in the repositories configured on the production server, the pkg update command will update that system to the specified version.  Unless a new version of the control package is made available, pkg update will report that no updates are available since no version of the control package can be installed that satisfies the incorporate constraint.

Note that if desired, the same package can be used to specify which packages must be present on the system by adding either "require" or "group" dependencies; the latter permits removal of some of the packages, the former does not.  More details on this can be found in either the section 5 pkg man page or the previously mentioned reference document.

This technique of using package dependencies to constrain system configuration leverages the SAT solver which is at the heart of IPS, and is basic to how we package Solaris itself.


Tuesday Apr 05, 2011

OTN focus on IPS

The marketing (er, product management) folks have been putting together  various pieces of information on IPS.  You can find all the information at this link.

 Thanks guys!


Monday Mar 28, 2011

Publishing your own packages with IPS - getting started.

It's been a while since I blogged on packaging... and we've been busy. I've had several people ask for our developers documentation, but that's still being written, so I thought I'd blog about how to publish a simple package with IPS.

I've been having mysterious networking issues (somewhere, a router doesn't like my system), so I decided I'd package a tool I use often to diagnose flaky networks - mtr. Like many open source programs, it needed some work to get it to compile properly on Solaris. The primary issue is that we've currently hidden libncurses off in /usr/gnu/lib, which is silly - there is no "Solaris" version in /usr/lib, so it should have gone there. Secondly, the configure.in file insists on bringing in -ltermcap - but this causes problems since it exports data that differs in size from the definitions in libncurses. So, after commenting out the offending line:

# AC_CHECK_LIB(termcap, tgetent)

things worked:

export CC=cc
export LDFLAGS="-zguidance -zdirect -zlazyload -zignore -R/usr/gnu/lib -L/usr/gnu/lib -lncurses"
export DESTDIR=/home/barts/publish/proto
./configure --exec-prefix=/usr --prefix=/usr
make install

The -zignore option tells the linker to not include the libraries that aren't actually needed; for some reason configure insists on placing the transitive closure of the library dependencies on the link line rather than  just the ones needed by mtr itself.

So, after all this fiddling, we end up w/ two files installed into our proto area: a man page: usr/share/man/man8/mtr.8 and usr/sbin/mtr, the binary itself.  The installation of the man page into section 8 isn't right for Solaris, but we can fix that in packaging.  We should patch the man page to correct its section number, but this blog entry is supposed to be about packaging, not coercing OSS into the right form for Solaris.

Well, to package the software the first thing we need is a manifest, which contains actions - files, directories, etc.  The pkgsend command will generate a manifest for us from a variety of different inputs, including SVR4 packages, tar files, and directory hierarchies. The latter will be useful here:

: barts@cyber[186]; pkgsend generate /home/barts/publish/proto
dir group=bin mode=0755 owner=root path=usr
dir group=bin mode=0755 owner=root path=usr/share
dir group=bin mode=0755 owner=root path=usr/sbin
dir group=bin mode=0755 owner=root path=usr/share/man
dir group=bin mode=0755 owner=root path=usr/share/man/man8
file usr/share/man/man8/mtr.8 group=bin mode=0644 owner=root path=usr/share/man/man8/mtr.8
file usr/sbin/mtr group=bin mode=04755 owner=root path=usr/sbin/mtr
: barts@cyber[187]; 

Here we can see the directories with default owner & group permissions, and our two files.  Note that the files have a path to the content in the proto area, and a path attribute that indicates where to install them.  They match by default, but we can edit the installation directory in the manifest; this is often easier than getting the configure script to move things around for us.  We also want to delete the directory entries; they're not needed for this package as they already exist, and we would need to make them match exactly.

So, let's dump that in a file:

: barts@cyber[187]; pkgsend generate /home/barts/publish/proto > mtr.p5m

and use pkgmogrify(1) to edit the manifest.  Of course we could do this by hand - but automating the transformations needs to produce a finished manifest from the raw compilation is an important part of making software changes easy, and reducing mistakes.

If you read the man page for pkgmogrify, you'll find several examples; here we simply modify the manifests with the following transforms:

<transform -> edit path man8/mtr.8 man1m/mtr.1m>
<transform dir -> drop>

So, with the above lines in a file named section8, we can invoke pkgmogrify:

: barts@cyber[189]; pkgmogrify mtr.p5m section8 
file usr/share/man/man8/mtr.8 group=bin mode=0644 owner=root path=usr/share/man/man1m/mtr.1m
file usr/sbin/mtr group=bin mode=04755 owner=root path=usr/sbin/mtr
: barts@cyber[190]; 

Ok, we now dump this into another file and run pkgdepend generate on the manifest to discover our dependencies:

: barts@cyber[195]; pkgmogrify mtr.p5m section8 > mtr.p5m.1 
: barts@cyber[196]; pkgdepend generate -md /home/barts/publish/proto mtr.p5m.1 > mtr.p5m.2
: barts@cyber[197]; cat mtr.p5m.2
file usr/share/man/man8/mtr.8 group=bin mode=0644 owner=root path=usr/share/man/man1m/mtr.1m
file usr/sbin/mtr group=bin mode=04755 owner=root path=usr/sbin/mtr
depend fmri=__TBD pkg.debug.depend.file=libresolv.so.2 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libnsl.so.1 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libsocket.so.1 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libgdk_pixbuf-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libgtk-x11-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libm.so.2 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libncurses.so.5 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libgdk-x11-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libc.so.1 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libglib-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libpthread.so.1 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
depend fmri=__TBD pkg.debug.depend.file=libgobject-2.0.so.0 pkg.debug.depend.path=lib pkg.debug.depend.path=usr/gnu/lib pkg.debug.depend.path=usr/lib pkg.debug.depend.reason=usr/sbin/mtr pkg.debug.depend.type=elf type=require
: barts@cyber[198]; 

As you can see, the manifest now contains dependency prototypes specifying which files are needed. We can now run the resolve step to examine the packaging manifests on the system (or in a repository) to determine on which packages we depend. This takes a bit of time to run, since it loads in all the manifests first; this still needs some performance work. It dumps its output in a file with the same name as the input but with ".res" appended; this is so multiple packages can be resolved at once:

barts@cyber[200]; pkgdepend resolve -m mtr.p5m.2          
: barts@cyber[201]; cat mtr.p5m.2.res 
file usr/share/man/man8/mtr.8 group=bin mode=0644 owner=root path=usr/share/man/man1m/mtr.1m
file usr/sbin/mtr group=bin mode=04755 owner=root path=usr/sbin/mtr
depend fmri=pkg:/library/desktop/gtk2@0.5.11-0.161 type=require
depend fmri=pkg:/library/glib2@0.5.11-0.161 type=require
depend fmri=pkg:/library/ncurses@0.5.11-0.161 type=require
depend fmri=pkg:/system/library/math@0.5.11-0.161 type=require
depend fmri=pkg:/system/library@0.5.11-0.163 type=require
: barts@cyber[202]; 

The dependencies have all been resolved... the version numbers match whatever packages I have installed on my system. Now let's create a file based repository, set the publisher name and publish the package:

: barts@cyber[231]; pkgrepo create /home/barts/publish/repo
: barts@cyber[232]; pkgrepo add-publisher -s /home/barts/publish/repo bart
: barts@cyber[233]; pkgsend -s /home/barts/publish/repo publish -d /home/barts/publish/proto mtr@0.80,5.11-0.1 mtr.p5m.2.res
pkg://bart/mtr@0.80,5.11-0.1:20110328T202205Z
PUBLISHED
: barts@cyber[234]


We can now install this package; let's do a dry run with lots of output so we can see what will happen:

: barts@cyber[234]; sudo pkg install -nvvg /home/barts/publish/repo mtr
Password: 
               Packages to install:     1
           Create boot environment:    No
              Rebuild boot archive:    No
Changed fmris:
  None -> pkg://bart/mtr@0.80,5.11-0.1:20110328T202205Z
Services:
  None
Actions
  None -> pkg://bart/mtr@0.80,5.11-0.1:20110328T202205Z
  None -> set name=pkg.fmri value=pkg://bart/mtr@0.80,5.11-0.1:20110328T202205Z
  None -> file e76b633a531d5b1081d13e449efe7534df25105f chash=16844052aeb4716f4b7dec96d04d76e709c1885f group=bin mode=0644 owner=root path=usr/share/man/man1m/mtr.1m pkg.csize=2047 pkg.size=4939
  None -> file f71673bbaa607a5aeb24f73849154b7034ed6026 chash=88d4f4183ab9ff6866c5a699a04254aa27f6b260 elfarch=i386 elfbits=32 elfhash=735cf51f4cbaadb2c1fc9f935b59343354a53b56 group=bin mode=04755 owner=root path=usr/sbin/mtr pkg.csize=48466 pkg.size=131228
  None -> depend fmri=pkg:/library/desktop/gtk2@0.5.11-0.161 type=require
  None -> depend fmri=pkg:/library/glib2@0.5.11-0.161 type=require
  None -> depend fmri=pkg:/library/ncurses@0.5.11-0.161 type=require
  None -> depend fmri=pkg:/system/library/math@0.5.11-0.161 type=require
  None -> depend fmri=pkg:/system/library@0.5.11-0.163 type=require

: barts@cyber[235]; 


Looks plausible; let's install and test it:

: barts@cyber[235]; sudo pkg install -g /home/barts/publish/repo mtr
               Packages to install:     1
           Create boot environment:    No
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1         2/2      0.0/0.0

PHASE                                        ACTIONS
Install Phase                                    8/8 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 

PHASE                                          ITEMS
Reading Existing Index                           8/8 
Indexing Packages                                1/1
: barts@cyber[236]; 

Seems to work; user interface is a little ... clunky... when updating the host to be tracerouted....

Ok, this is a good starting point. Some more things to do before declaring success include:

  • Fix the man page as noted earlier – some sed script seems appropriate.

  • Add a facet to the man page so that it's not installed if the user doesn't want documentation. This is easily done w/ pkgmogrify.

  • Adding a file containing exec-attr info for this binary to match what Solaris does w/ /usr/sbin/traceroute.

  • Add gnome-menu times, icons, etc. and add appropriate restart_fmri tags (see pkg(5)) so that gnome caches are properly refreshed.


A quick recap:


We used pkgsend generate to generate a manifest... and modified the generated manifest with pkgmogrify. We then used pkgdepend to generate and resolve package dependences... and then published with pkgsend again.


There are four steps to publishing your own packages: generate, transmogrify, determine dependencies, publish.

More on this topic later on... hope to get some of the commonly used pkgmogrify transforms into /usr/share/ips or similar; this will make things easier yet.



About

An engineer's viewpoint on Solaris...

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today