Monday Mar 21, 2011

some pkg spring cleaning

With the putback of changeset 2268:1f313c3e7cdf to the pkg(5) gate, publication tools such as pkgsend, pkgrecv, and pkgsign now always require that a destination repository be specified for operations that require it using environment variables or options to the program.  pkgsign also accepts filesystem-based repository locations in path form.

What changed?

For historical reasons that are no longer understood, publication tools such as pkgrecv, pkgsend, and pkgsign previously would automatically attempt to connect to a pkg.depotd(1M) process on port 10000 if a target repository was not specified either through environment or program options.

This is no longer the case.  (This behaviour was never intended for long-term use.)

Operations that publication tools perform that require a destination repository will now always exit with an error if the user has not specified a destination repository.

Additionally, pkgsign now accepts relative and absolute paths to filesystem-based repositories instead of requiring that their location is in URI form.

How could this change impact me?

If a consumer has relied on the publication tool's historical (and sometimes undocumented) behaviour of defaulting to port 10000 for a destination repository, whatever publication operations it was performing will now fail with a usage error.

The consumers in question must be updated to either set the relevant environment variables to indicate the destination repository, or must specify the destination repository using the relevant options to the publication tool in question.

See pkgsend(1), pkgrecv(1), or pkgsign(1) for more information on how to specify a destination repository for each tool.

What are the related bug entries?

The following RFEs/Bugs were resolved by changeset 2268:
  • 17728 publication tools should require a repository to be specified
  • 18041 pkgsign should accept paths for filesystem-based repositories

Wednesday Mar 16, 2011

A spoonful of syntactic sugar

With the putback of changeset 2224 to the pkg(5) gate, pkg(1) and the other parts of the package system now provide convenient ways for a user to specify the latest version of a package, and to specify fully-qualified package names without having to use the formal FMRI scheme.  These changes are also available in build 159 and later.

What changed?

pkg(5) programs such as pkg(1) now provide a way for users to specify that the latest version of a package should be used for an operation (such as install or update) without having to determine what the latest version is first and then specifying it manually.  The reason this matters is because the input to operations such as install or update match all versions of a package if a version is not specified, and then the solver will determine the newest version that can be installed.

This behaviour, although intentional, can sometimes be surprising to users as they may not realise that the dependency relationships between packages already installed may prevent installing the latest version of a package.  In such cases, an older version of a package than the user was expecting may be installed as it is the only version that can satisfy dependencies of packages already installed.

To provide users a way to request that pkg(1) install or update to the latest version of a package, support for specifying the version of a package as '@latest' was added.  This allows the user to direct pkg(1) to require that the solver install or upgrade to the latest version of a package, which in turn will cause the operation to fail with reasons why if it cannot.

As an example, let's say that there was a change in the build 161 version of the shell/zsh package that was desired, but the system is currently running build 160.  When a user execute 'pkg install shell/zsh' on that build 160 system, the solver will evaluate each version of the shell/zsh package that matches what the user specified.  In that case, pkg(1) would match the versions for builds 151-161 and pass those to the solver for evaluation.  The solver would then determine that the build 160 version of shell/zsh should be installed:

# pkg install zsh 
               Packages to install:     1
           Create boot environment:    No
              Rebuild boot archive:    No
Changed fmris:
  None -> pkg://solaris/shell/zsh@4.3.10,5.11-0.160

As can be seen from the output above, the solver (as expected) determined that the build 160 version of zsh should be installed.  Now what if the user really wanted the build 161 version?  Well, they could explicitly request the 161 version by saying 'pkg install shell/zsh@\*-0.161'.  However, what if they didn't know that the build 161 version was the latest?  By using the new '@latest' syntax, they could explicitly request that the latest version be installed:

# pkg install shell/zsh@latest
pkg install: No matching version of shell/zsh can be installed:
  Reject:  pkg://solaris/shell/zsh@4.3.10,5.11-0.161
  Reason:  This version is excluded by installed incorporation pkg://solaris/consolidation/sfw/sfw-incorporation@0.5.11,5.11-0.160

Now in the case above, the solver tells them why the latest version can't be installed -- because the dependency relationship declared by the sfw-incorporation says the 161 version isn't acceptable.  So it should be clear why that isn't the default behaviour of the system -- sometimes the latest version is not the version that should be (or can be) installed.

Likewise, if a user was trying to update all of their packages to the latest version and wanted to determine exactly why that particular combination of packages couldn't be updated to, they could say:

 pkg update '\*@latest'

Another change made was to allow users to specify package names without having to type the 'pkg:' portion of a package FMRI or pattern.  For example, suppose that you have two packages named 'sunstudio' and 'developer/sunstudio'.  Normally, when pkg(1) is asked to list 'sunstudio' it would list packages with names ending in '/sunstudio' or that are 'sunstudio':

 $ pkg list sunstudio
developer/sunstudio  12.1.1-0.111    known      -----
sunstudio            12.1.1-1        known      --r--

If a user only wanted to list or install the 'sunstudio' package, they would have had to specify the package name like this:

 $ pkg list pkg:/sunstudio
sunstudio            12.1.1-1        known      --r--

...or like this:

 $ pkg list pkg://
sunstudio            12.1.1-1        known      --r--

However, users may now specify the same package names or patterns to pkg(5) programs without the 'pkg:' in one of the following ways:

 $ pkg list /sunstudio
sunstudio            12.1.1-1        known      --r--

...or like this:

 $ pkg list //
sunstudio            12.1.1-1        known      --r--

While the changes described above may not seem significant, it's a spoonful of syntactic sugar that makes life with the packaging system a little easier.

pkg(5) archive and temporary source support

With the putback of changeset 2219 to the pkg(5) gate, pkg(1) supports the use of temporary package sources and package archives, and pkgrecv(1) supports the creation of package archives. These changes are also present in build 159 and later.

Support for temporary package sources enables users to perform operations such as install and update without first configuring the source for use with the image.

What changed?

pkgrecv(1) now provides functionality to create package archives from existing package repositories.  Examples:

Create a package archive containing the package "SUNWemacs" and all of its dependencies from the repository located at

  $ pkgrecv -s -d emacs.p5p -a -r SUNWemacs

Copy all of the packages in a package archive to an existing repository located at '/export/repo':

  $ pkgrecv -s archive.p5p -d /export/repo '\*'

pkg(1) now supports the use of temporary package sources for operations such as list, info, and contents.  For example, a user can list the package available for installation in a repository or package archive as follows:

  $ pkg list -g /my/archive.p5p
  $ pkg list -g

Multiple sources can be combined:

  $ pkg list -g /my/archive.p5p -g

pkg(1) now supports the use of temporary package sources for operations such as install, update, change-variant, and change-facet.  For example, a user could install a package from any package repository or package archive as follows (any required dependencies will also be installed automatically):

  $ pkg install -g <pkg_name1> <pkg_name2> ...
  $ pkg install -g /my/archive.p5p <pkg_name1> <pkg_name2> ...

Likewise, they could update any packages found in a package archive or in any package repository as follows:

  $ pkg update -g
  $ pkg update -g /my/newarchive.p5p

Or update specific packages from a package archive as follows:

  $ pkg update -g /my/newarchive.p5p <pkg_name1> <pkg_name2> ...

Multiple sources can also be combined:

  $ pkg install -g -g /my/archive.p5p \\
    <pkg_name1> <pkg_name2> ...

pkg(1) now allows the location of filesystem repositories to be specified as relative or absolute paths.  Previously, they had to be specified as absolute paths using URI syntax (e.g. file:///path/to/repo).  Example:

  $ pkg set-publisher -p myrepo         (relative to $CWD)
  $ pkg set-publisher -p /path/to/repo  (absolute)

pkg(1) now supports the use of package archives as an origin for a publisher.  For example, you could copy all of the packages from an existing repository into an archive and then add a publisher using that archive as a source of package data:

  $ pkgrecv -s \\
     -a -d /my/on-nightly.p5p '\*'
  $ pkg set-publisher -p /my/on-nightly.p5p

What are the potential issues?

Packages installed from a package archive or other temporary source from a publisher that was not previously configured in the image will be added without any origin or mirror information.  If a program is parsing the output of the pkg system and expecting origins or mirrors to always be listed, it will fail.

For now, publishers without origins or mirrors show as follows in the output of 'pkg publisher':

$ pkg publisher
PUBLISHER        TYPE     STATUS   URI  origin   online

The above simply indicates that publisher 'example' doesn't have any origins.

Also be aware that pkgrecv(1) currently only supports the creation of new package archives.  Existing package archives may not be updated. This limitation may be removed at a future date.


Many thanks to the individuals who collaborated on, reviewed, or contributed to this project.



« March 2011