Monday Mar 21, 2011

some pkg spring cleaning

With the putback of changeset 2268:1f313c3e7cdf to the pkg(5) gate, publication tools such as pkgsend, pkgrecv, and pkgsign now always require that a destination repository be specified for operations that require it using environment variables or options to the program.  pkgsign also accepts filesystem-based repository locations in path form.

What changed?

For historical reasons that are no longer understood, publication tools such as pkgrecv, pkgsend, and pkgsign previously would automatically attempt to connect to a pkg.depotd(1M) process on port 10000 if a target repository was not specified either through environment or program options.

This is no longer the case.  (This behaviour was never intended for long-term use.)

Operations that publication tools perform that require a destination repository will now always exit with an error if the user has not specified a destination repository.

Additionally, pkgsign now accepts relative and absolute paths to filesystem-based repositories instead of requiring that their location is in URI form.

How could this change impact me?

If a consumer has relied on the publication tool's historical (and sometimes undocumented) behaviour of defaulting to port 10000 for a destination repository, whatever publication operations it was performing will now fail with a usage error.

The consumers in question must be updated to either set the relevant environment variables to indicate the destination repository, or must specify the destination repository using the relevant options to the publication tool in question.

See pkgsend(1), pkgrecv(1), or pkgsign(1) for more information on how to specify a destination repository for each tool.

What are the related bug entries?

The following RFEs/Bugs were resolved by changeset 2268:
  • 17728 publication tools should require a repository to be specified
  • 18041 pkgsign should accept paths for filesystem-based repositories

Wednesday Mar 16, 2011

A spoonful of syntactic sugar

With the putback of changeset 2224 to the pkg(5) gate, pkg(1) and the other parts of the package system now provide convenient ways for a user to specify the latest version of a package, and to specify fully-qualified package names without having to use the formal FMRI scheme.  These changes are also available in build 159 and later.

What changed?

pkg(5) programs such as pkg(1) now provide a way for users to specify that the latest version of a package should be used for an operation (such as install or update) without having to determine what the latest version is first and then specifying it manually.  The reason this matters is because the input to operations such as install or update match all versions of a package if a version is not specified, and then the solver will determine the newest version that can be installed.

This behaviour, although intentional, can sometimes be surprising to users as they may not realise that the dependency relationships between packages already installed may prevent installing the latest version of a package.  In such cases, an older version of a package than the user was expecting may be installed as it is the only version that can satisfy dependencies of packages already installed.

To provide users a way to request that pkg(1) install or update to the latest version of a package, support for specifying the version of a package as '@latest' was added.  This allows the user to direct pkg(1) to require that the solver install or upgrade to the latest version of a package, which in turn will cause the operation to fail with reasons why if it cannot.

As an example, let's say that there was a change in the build 161 version of the shell/zsh package that was desired, but the system is currently running build 160.  When a user execute 'pkg install shell/zsh' on that build 160 system, the solver will evaluate each version of the shell/zsh package that matches what the user specified.  In that case, pkg(1) would match the versions for builds 151-161 and pass those to the solver for evaluation.  The solver would then determine that the build 160 version of shell/zsh should be installed:

# pkg install zsh 
               Packages to install:     1
           Create boot environment:    No
              Rebuild boot archive:    No
Changed fmris:
  None -> pkg://solaris/shell/zsh@4.3.10,5.11-0.160
Services:
  None

As can be seen from the output above, the solver (as expected) determined that the build 160 version of zsh should be installed.  Now what if the user really wanted the build 161 version?  Well, they could explicitly request the 161 version by saying 'pkg install shell/zsh@\*-0.161'.  However, what if they didn't know that the build 161 version was the latest?  By using the new '@latest' syntax, they could explicitly request that the latest version be installed:

# pkg install shell/zsh@latest
pkg install: No matching version of shell/zsh can be installed:
  Reject:  pkg://solaris/shell/zsh@4.3.10,5.11-0.161
  Reason:  This version is excluded by installed incorporation pkg://solaris/consolidation/sfw/sfw-incorporation@0.5.11,5.11-0.160

Now in the case above, the solver tells them why the latest version can't be installed -- because the dependency relationship declared by the sfw-incorporation says the 161 version isn't acceptable.  So it should be clear why that isn't the default behaviour of the system -- sometimes the latest version is not the version that should be (or can be) installed.

Likewise, if a user was trying to update all of their packages to the latest version and wanted to determine exactly why that particular combination of packages couldn't be updated to, they could say:

 pkg update '\*@latest'

Another change made was to allow users to specify package names without having to type the 'pkg:' portion of a package FMRI or pattern.  For example, suppose that you have two packages named 'sunstudio' and 'developer/sunstudio'.  Normally, when pkg(1) is asked to list 'sunstudio' it would list packages with names ending in '/sunstudio' or that are 'sunstudio':

 $ pkg list sunstudio
NAME (PUBLISHER)     VERSION         STATE      UFOXI
developer/sunstudio  12.1.1-0.111    known      -----
sunstudio            12.1.1-1        known      --r--

If a user only wanted to list or install the 'sunstudio' package, they would have had to specify the package name like this:

 $ pkg list pkg:/sunstudio
NAME (PUBLISHER)     VERSION         STATE      UFOXI
sunstudio            12.1.1-1        known      --r--

...or like this:

 $ pkg list pkg://opensolaris.org/sunstudio
NAME (PUBLISHER)     VERSION         STATE      UFOXI
sunstudio            12.1.1-1        known      --r--
 
  

However, users may now specify the same package names or patterns to pkg(5) programs without the 'pkg:' in one of the following ways:

 $ pkg list /sunstudio
NAME (PUBLISHER)     VERSION         STATE      UFOXI
sunstudio            12.1.1-1        known      --r--

...or like this:

 $ pkg list //opensolaris.org/sunstudio
NAME (PUBLISHER)     VERSION         STATE      UFOXI
sunstudio            12.1.1-1        known      --r--

While the changes described above may not seem significant, it's a spoonful of syntactic sugar that makes life with the packaging system a little easier.

pkg(5) archive and temporary source support

With the putback of changeset 2219 to the pkg(5) gate, pkg(1) supports the use of temporary package sources and package archives, and pkgrecv(1) supports the creation of package archives. These changes are also present in build 159 and later.

Support for temporary package sources enables users to perform operations such as install and update without first configuring the source for use with the image.

What changed?

pkgrecv(1) now provides functionality to create package archives from existing package repositories.  Examples:

Create a package archive containing the package "SUNWemacs" and all of its dependencies from the repository located at http://example.com:10000:

  $ pkgrecv -s http://example.com:10000 -d emacs.p5p -a -r SUNWemacs

Copy all of the packages in a package archive to an existing repository located at '/export/repo':

  $ pkgrecv -s archive.p5p -d /export/repo '\*'

pkg(1) now supports the use of temporary package sources for operations such as list, info, and contents.  For example, a user can list the package available for installation in a repository or package archive as follows:

  $ pkg list -g /my/archive.p5p
  $ pkg list -g http://example.com:10000

Multiple sources can be combined:

  $ pkg list -g /my/archive.p5p -g http://example.com:10000

pkg(1) now supports the use of temporary package sources for operations such as install, update, change-variant, and change-facet.  For example, a user could install a package from any package repository or package archive as follows (any required dependencies will also be installed automatically):

  $ pkg install -g http://example.com:10000 <pkg_name1> <pkg_name2> ...
  $ pkg install -g /my/archive.p5p <pkg_name1> <pkg_name2> ...

Likewise, they could update any packages found in a package archive or in any package repository as follows:

  $ pkg update -g http://example.com:10000
  $ pkg update -g /my/newarchive.p5p

Or update specific packages from a package archive as follows:

  $ pkg update -g /my/newarchive.p5p <pkg_name1> <pkg_name2> ...

Multiple sources can also be combined:

  $ pkg install -g http://example.com:10000 -g /my/archive.p5p \\
    <pkg_name1> <pkg_name2> ...

pkg(1) now allows the location of filesystem repositories to be specified as relative or absolute paths.  Previously, they had to be specified as absolute paths using URI syntax (e.g. file:///path/to/repo).  Example:

  $ pkg set-publisher -p myrepo         (relative to $CWD)
  $ pkg set-publisher -p /path/to/repo  (absolute)

pkg(1) now supports the use of package archives as an origin for a publisher.  For example, you could copy all of the packages from an existing repository into an archive and then add a publisher using that archive as a source of package data:

  $ pkgrecv -s http://example.com:10000/on-nightly \\
     -a -d /my/on-nightly.p5p '\*'
  $ pkg set-publisher -p /my/on-nightly.p5p

What are the potential issues?

Packages installed from a package archive or other temporary source from a publisher that was not previously configured in the image will be added without any origin or mirror information.  If a program is parsing the output of the pkg system and expecting origins or mirrors to always be listed, it will fail.

For now, publishers without origins or mirrors show as follows in the output of 'pkg publisher':

$ pkg publisher
PUBLISHER        TYPE     STATUS   URI
opensolaris.org  origin   online   http://pkg.opensolaris.org/release/
example

The above simply indicates that publisher 'example' doesn't have any origins.

Also be aware that pkgrecv(1) currently only supports the creation of new package archives.  Existing package archives may not be updated. This limitation may be removed at a future date.

Thanks

Many thanks to the individuals who collaborated on, reviewed, or contributed to this project.

Wednesday May 12, 2010

filesystem-based repository access

With the putback of changeset 1895 to the pkg(5) gate, pkg(1) now supports filesystem-based repository access.  Please note that this functionality is not expected to be available in an OS build until after b139.

What changed?

There are no user-visible changes to command syntax or client operation.  The only noticeable change is that any command that accepted a repository URI such as 'http://localhost:10000' now also accepts URIs for filesystem locations such as 'file:/path/to/my/repository'.

For example, if you were creating a new image and had a repository located at '/export/myrepo', you could do this:

  pkg image-create -p file:/export/myrepo /path/to/image

See the pkg(1) man page for more examples.

What does this mean?

This change means that a pkg.depotd server is no longer required for serving package data if a client can access the package repository via the filesystem (e.g. NFS mounts, etc.).

What are the related bug entries?

The following RFEs/Bugs were resolved by changeset 1895:

15762 client support for filesystem-based repository access
10244 caching dictionaries as a class variable prevents multi-image and repo search
14802 ability to have separate read / write download caches
15763 test suite should not start depot server unless necessary
15764 file_manager insert needs to stat() less and return more

What are the potential issues?

Currently, the client will not cache data from a repository that is accessed via the filesystem in the image's download cache (e.g. /var/pkg/download), with the assumption that access to the repository via filesystem is reliable and/or reasonably fast.

In some use case scenarios (such as NFS, SMB, etc.) filesystem-based access may be slow, unreliable, or otherwise unsuitable for client transport.  In those cases, it is advised that http-based access to the repository via pkg.depotd(1m) is used instead.  It is hoped that the client can account for the performance of available filesystem-based resources automatically in the future when retrieving package content.

Search operations performed using a file-based repository can use substantially more memory when a repository contains a large number of packages (such as /dev).  This is because the client is essentially acting as the search server in addition to the client.

To avoid this overhead, the related publisher can be configured with both an http(s) origin and the file-based origin.  When performing search operations, the client will use the http(s) based origin(s) first to perform searches, but check the file-based origin first for all package file retrieval operations.  For example:

  pkg set-publisher -g file:/export/myrepo \\
    -g http://localhost:10000 example.com

Thanks

Many thanks to the individuals who collaborated on, reviewed, or contributed to this project.

Monday Feb 08, 2010

Simplifying Publisher Configuration

Changeset 1736 in the pkg(5) gate made the following changes:

  11522 pkg should require publisher prefix to match repository information
  7156 client image api needs image creation interface
  12744 update_publisher over-zealously testing publisher validity
  14203 image-create usage doesn't mention mirror / origin options

What changed?

The pkg(1) client and the pkg.client.api will now attempt to retrieve publisher configuration information from package repositories.  This information is used to auto-configure new publishers, validate configuration requests, and to update the configuration of existing publishers.

These changes were made with a focus on simplifying publisher addition and updates of existing publisher configuration.

Further improvements and additions to this functionality are planned for future releases.

Are there any pkg.depotd(1m) changes?

While the pkg.depotd(1m) program did not change, clients are now more reliant on its correct configuration.

As already mentioned in the past, please be certain that you set the publisher.prefix property found in the repository's cfg_cache file correctly.  The pkg.depotd man page contains instructions on how to set this property.

Please also note that the repository.origins and/or repository.mirrors properties should be set as clients will use these to automatically configure new publishers and to update existing publishers.  You might also want to consider providing a description so that users have an idea of what sort of packages the repository contains.

Part of an example cfg_cache might look like this:

[publisher]
prefix = example.com
alias = None
...
[repository]
origins = http://pkg-us.example.com/,http://pkg-ca.example.com/
mirrors = http://pkg-us-mirror1.example.com/,http://pkg-ca-mirror1.example.com/
...

How has pkg(1) image-create changed?

The client remains completely compatible syntax-wise, so no changes to existing scripts are required, and behaviour when using the older syntax (other than validating publisher configuration) remains unchanged.

However, image-create now also accepts -p <uri> where <uri> is the URI of a package repository and example usage is as follows:

pkg image-create -p <uri> /target

When using this syntax, image-create will retrieve all of the publisher's configuration information from the target package repository and add all of the publishers it finds to the image.

How has pkg(1) set-publisher changed?

The syntax of the set-publisher command remains backwards compatible, so no changes to existing scripts, etc. are required.  However, set-publisher now accepts the -p option which accepts the URI of a package repository from which all publisher configuration information will be retrieved.

Any new publishers found in the retrieved configuration information will be added, while existing ones will be updated if the provided URI is already in the list of configured origins for the publisher being updated.

If a publisher name is provided, then -p will only use publisher configuration information that matches the provided name.

As an example, the old syntax to add the contrib repository might have been:

pkg set-publisher -g http://pkg.opensolaris.org/contrib \\
  contrib.opensolaris.org

The new syntax is:

pkg set-publisher -p http://pkg.opensolaris.org/contrib

How has pkg.client.api changed?

Incompatible with clients using API versions 0-31.

The ImageInterface class has changed as follows:

  • The add_publisher and update_publisher methods now validate the image's publisher configuration against the origins of the publisher.  If any of the origins are found to not match, an UnknownRepositoryPublishers exception will be raised. If one of the new repository origins does not provide publisher configuration information or it is incomplete, a RepoPubConfigUnavailable exception will be raised.

The pkg.client.api module has changed as follows:

  • A new method named image_create has been added.  See 'pydoc pkg.client.api' for details.


The pkg.client.api_errors modules has changed as follows:

  • UnknownRepositoryPublishers, RepoPubConfigUnavailable, and UnknownErrors expections have been added for use by the pkg.client.api.  API consumers are reminded that they should catch all ApiException class exceptions, although catching specific exception subclasses for case-by-case handling in addition to that is acceptable.


Feedback is welcomed on the pkg-discuss mailing list on opensolaris.org.

Thursday Jan 07, 2010

New pkg(5) license functionality

For those of you that require license acceptance or display for your packages, you should be aware that starting with OpenSolaris development build b131, new functionality will be delivered to support license acceptance and display.

The following fixes and enhancements were integrated:

  • 5943 add 'must-accept' attribute to license actions
  • 5586 licenseinfo api needs to expose license action attributes
  • 13155 add 'must-display' attribute to license actions
  • 13158 change pkg.client.api to understand and require license acceptance
  • 13160 pkg(1) needs update for client api license acceptance changes 

These changes were made to support packages that require display or acceptance of license related data during package install and update operations.  As a result of these changes, the pkg(1) client and client API require explicit acceptance and/or indication of license display during install and update operations.

Please note that (currently) no packages exist that use this functionality, but they are expected in the near future.

Client Considerations

If any of the license actions contained in a package being installed or updated have must-accept=true, the following pkg(1) subcommands require that the new --accept option be provided before the operation will proceed:

  • install
  • fix
  • image-update
  • change-variant
  • change-facet

In addition, all of the above subcommands also now have a --licenses option to display the payload of all the licenses for packages part of the operation.  For example:

pkg install -n --licenses foo

The above command would display all of the licenses for the packages that would be installed or updated if the package 'foo' were installed or updated.  If the --accept option is not provided, and a license requires acceptance, the pkg(1) client will now exit with exit code 6, indicating license acceptance failure.  If a license requires display, the pkg(1) client will display it during install/update operations; this cannot be suppressed.

Client API Changes

Version 29:
Incompatible with clients using versions 0-28:
    The ImageInterface class has changed as follows:

  • set_plan_license_status() was added.  This is used to indicate
              whether licenses for the packages being operated on have been
              accepted or displayed.  Clients must do this if the related
              license requires acceptance or display.

    The LicenseInfo class has changed as follows:

  • get_text() may now trigger a remote retrieval of the license
              payload if needed to return the text.
  • The related package FMRI and license attributes are now
              properties: fmri, license, must_accept, and must_display.

    The PlanDescription class has changed as follows:

  • get_changes() is now a generator function.
  • get_licenses() was added to allow clients to retrieve the
              list of licenses related to the plan's operations as well
              as the current accepted and displayed status of each.  Please
              note that this function returns _all_ licenses related to the
              operation; not just those that require acceptance or display.

Publication Considerations

To use this new license acceptance functionality, simply add must-accept=true or must-display=true (as appropriate) to license actions in your package manifest.  An example pkgsend sequence might look like this:

open licensed@1.3,5.11-0
add depend type=require fmri=baz@1.0
add file /tmp/libc.so.1 mode=0555 owner=root group=bin path=/lib/libc.so.1
add license /tmp/libc.copyright license=libc.copyright must-display=True
add license /tmp/libc.license license=libc.license must-accept=True
close

Please note that this functionality is not supported before build 131, and that you should use this functionality sparingly.  must-accept=true should not be placed on the majority of open source licenses (BSD, GPL, etc.) and must-display should only be set if absolutely necessary. 

Comments or concerns should be sent to the pkg-discuss mailing list on opensolaris.org.

Wednesday Nov 18, 2009

Enabling pkg(5) Availability

With the putback for the following issues in revision 1504:

9969 client support for multiple origins desired
11715 ImageConfig does not handle None for publisher correctly
11793 image-create example partially disagrees with the usage

...the client now supports multiple origins for publisher repositories.  The purpose of this change was to enable greater pkg (5) server availability and redundancy.

What is an origin?

An origin is simply a location of a package repository that contains package file, manifest, and catalog data; such as: http://pkg.opensolaris.org/dev.  The default mode for the pkg.depotd process is to run as an 'origin'.

A mirror, in contrast, only contains package file data and is currently provided by a pkg.depotd server running in --mirror mode.

What does this change mean?

This change means that redundancy can now more easily be built into package server infrastructure.  For example, you could have http://pkg1.example.com/, http://pkg2.example.com/, and http://pkg3.example.com/.

A client, once configured, would have pkg publisher output similar to the following:

PUBLISHER                TYPE   STATUS URI
example.com  (preferred) origin online http://pkg1.example.com/
example.com  (preferred) origin online http://pkg2.example.com/
example.com  (preferred) origin online http://pkg3.example.com/

During package operations, the client will automatically attempt to select and use the best repository origin if any data needs to be retrieved for that publisher.

In addition, ipkg branded zone creation and attach has been updated to use this additional origin and mirror information when configuring zones.

How do I use this?

For consistency, and to support this new client functionality, the image-create and set-publisher subcommands for pkg(1) have changed as follows:

image-create
 New Options:
  [-g|--origin ...] [-m|--mirror ...]

set-publisher
 New Options:
  [-g origin_to_add | --add-origin=origin_to_add ...]
  [-G origin_to_remove | --remove-origin=origin_to_remove ...]

 Deprecated Options:
  [-O uri]

So, as an example, to create a new, full image, with publisher example.com, that also has an additional mirror, two additional origins and that is stored at /aux0/example_root:

$ pkg image-create -F -p example.com=http://pkg.example.com:10000 \\
  -g http://alternate1.example.com:10000/ \\
  -g http://alternate2.example.com:10000/ \\
  -m http://mirror.example.com:10000/ \\
  /aux0/example_root

Alternatively, to add a new publisher or update an existing publisher with the above information:

$ pkg set-publisher \\
  -g http://pkg.example.com:10000 \\
  -g http://alternate1.example.com:10000/ \\
  -g http://alternate2.example.com:10000/ \\
  -m http://mirror.example.com:10000/ \\
  example.com

Are there are any client compatibility concerns?

No.  Clients upgrading from older versions of the pkg client software to this version of the client software will not experience any issues.  In addition, some care has been taken to ensure older clients that understand the current image format are compatible with this new configuration, although they will be limited to using the first origin for a repository.

Do I need to update my scripts or client usage?

While the '-O' option has been deprecated for the set-publisher command and is no longer documented, it will continue to work exactly as it has in the past so no change is strictly necessary.  The command-line option compatibility for -O is currently planned to be maintained through the next major OpenSolaris release (2010.x), but will be dropped after that.

However, if you are using any scripts or programs that parse the output of the 'pkg publisher' command, and only expect a single origin to be listed for each publisher, they will need to be updated to account for this.

Comments or concerns should be sent to the pkg-discuss mailing list on opensolaris.org.

Friday Oct 23, 2009

Do you want to go faster?

Please read this if you publish packages or run a pkg.depotd server!

Overview

The short version?  These changes make the client take a lot less time to perform common operations (info, list, etc.).  Although, a greater benefit is only realized when the client is configured to get package data from a repository that is being served by the new pkg.depotd server in this putback.

The Nitty Gritty

Revision 1431:62b6033670e4 in the pkg(5) gate integrated the following fixes and enhancements:

  • 10416 server catalog v1 support desired

  • 243 need localized descriptions, other metadata at catalog level

  • 2424 Need to use UTC consistently everywhere

  • 3092 messaging api/framework needed for pkg clients (cli, gui, etc.)

  • 7063 "pkg list -a -s" needs performance improvement

  • 7163 manifests are modified by client before being written to disk

  • 8217 package fmri should be added to manifest during publishing

  • 9061 importer should not refresh indexes

  • 9446 traceback for cfg_cache operations if read-only filesystem

  • 10415 client catalog v1 support desired

  • 11094 Client transport for catalog v1

  • 11523 only permit FMRIs from same publisher for network repositories

  • 11831 server api version incompatible templates can cause traceback

  • 11832 depot needs ability to seed / alter repository configuration

  • 11954 importer shows zero packages processed unless debug enabled

  • 12006 merge utility should have a test suite

Client Software Considerations

The pkg(1) client now requires that the name (prefix) that you have configured your publisher with matches that reported by its repository (origin) when the package repository provides a version 1 catalog.


For example, if you have your client configured this way:


$ pkg publisher
PUBLISHER       TYPE     STATUS URI
dev (preferred) origin   online http://pkg.opensolaris.org/dev

Then the next time a pkg refresh is performed, you will see the following warning:


$ pkg refresh
Refreshing catalog 1/1 test
The catalog retrieved for publisher 'dev' only contains package data for these publisher(s): opensolaris.org.  To resolve this issue, update this publisher to use the correct repository origin, or add one of the listed publishers using this publisher's repository origin.

To correct the repository origin, execute the following command as a privileged user:

pkg set-publisher -O <url> dev

To add a new publisher using this publisher's repository origin, execute the following command as a privileged user:

pkg set-publisher -O http://pkg.opensolaris.org/dev <publisher>

After the new publisher has been added, this one should be removed by executing the following command as a privileged user:

⁞pkg unset-publisher test

Please note that this means that available packages from that publisher (excluding already installed packages) will not be shown in pkg list, pkg info, etc.

This change is the first step in a set of changes to not require that a user specify the publisher's name (prefix) when using image-create, set-publisher, etc. and to further simplify configuration.

Disk space usage will also increase for the client when it is configured to use a publisher repository that contains catalog v1 data.  As an example, when pkg.opensolaris.org/dev's depot software is upgraded, it now requires roughly 36MB of disk space (uncompressed) for local metadata storage (51K+ packages).  This information does compress extremely well (around 7MB gzip'd).

Disk space usage will continue to slightly increase for each new build until historical and obsolete packages are implemented, at which point it will significantly decrease again.

Depot Software Considerations

As a result of these changes, if you use the new depot server with an existing repository without using the --readonly option it will be automatically upgraded to a new repository format.


If you use an older repository with the newer depot software, but want to use it with the newer depot software, you must specify --readonly and --writable-root.  If you do that, a new catalog will be written to the --writable-root directory based on the contents of the repository.


In addition, there is now a hard requirement that you specify a default publisher prefix for the repository.  Carefully consider what you specify here as it will be used during publication if the FMRI specified at publication time does not include a publisher prefix.  The publisher prefix of any published packages will be placed into package manifests, the catalog, etc. and so changing it once set will not update the existing package data inside the repository as it is only a default.

Once a repository has been upgraded to the new format, you will be unable to use older pkg(5) depot server software, pksend, or pkgrecv with the new repository.  If you attempt to use an older client with the upgraded repository, it will fail.

As a reminder, parallel publication is not supported at this time via file://, only with http://.  Various import tests have been performed to verify that publication performance is roughly the same as it was before.

Finally, please note that this upgrade behaviour also applies if you are using 'file://' locations for destination repositories with pkgrecv or pkgsend.

Thanks

Enough thanks cannot be given to team members that helped complete this phase of the catalog v1 project over the past month.  This was truly a team effort.  Transport work provided by our resident guru: johansen.


Tuesday Mar 10, 2009

What's in a name? That which we call an Authority...

The pkg(5) team is always looking for ways to improve the user experience for those using the pkg(5) system and to promote a better understanding of it.  One of the ways to do that is to ensure that clear, consistent terminology is used to describe ideas and concepts.  That is why it was recently decided that a name change was in order for one of the core concepts in the pkg(5) system: authorities.

An authority is a forward or reverse domain name that identifies an organization (or group of individuals) responsible for providing one or more packages.  However, the name that was chosen for that concept caused some confusion among our users.  This was also further exacerbated by the interchangeable usage of the words "Authority" and "Repository" which are decidedly different in meaning.  As a result, starting with revision 926 of the pkg gate code,  an "Authority" is now called a "Publisher".

Before I move on to the details of the changes involved, I felt it was important to summarise what the current overall vision is for the structure of publishers and repositories and how the client is intended to use them:

  • Publisher - a forward or reverse domain name (e.g. "opensolaris.org" or "org.opensolaris") that can be used to identify an organization that is responsible for one or more packaging repositories.  While a publisher may have (publish to) one or more repositories, pkg(5) clients are only intended to access one of them at a time (such as the "development" or "release" repositories for OpenSolaris).  Please note that for compatibility reasons, the client is currently restricted to a single repository per publisher; this will change after the 2009.06 release of OpenSolaris.
  • Repository - a location where clients can publish and retrieve package content (files contained within the package such as programs, documents, etc.) and / or metadata (information about the package such as its name, description, etc.).  It has the following characteristics:
    • may have one or more origins (URIs) used for publication and retrieval of package metadata and content (the client currently only supports one of these)
    • may have zero or more mirrors (URIs) for retrieval of package content
    • can only contain packages from a single publisher

Now that you know what a publisher and a repository is, here is a high-level summary of the changes that were needed to make this happen:

  • All references to "authority" were replaced with "publisher".
  • A publisher management and client history logging interface was added to pkg.client.api so that all pkg(5) clients can consistently and easily manage publisher and repository information.
  • The output of the "pkg publisher" subcommand (aka "pkg authority") was changed to include mirrors in the default listing.
  • The cli and gui clients included with the pkg(5) system were updated to use the new publisher management interfaces.
  • Support for generating and parsing .p5i data (pkg(5) publisher and package information files) was added to the api and to the gui client.  This functionality will eventually allow users to trigger installation of new software just by clicking a link in their web browser.
  • Fixed SSL Certificate handling to be more robust so that the client won't fail with a Python traceback when performing network operations (such as downloading package data from a repository) and so that the user will know why the failure occurred (expired certificate, etc.).
  • Fixed "pkg publisher" command to return a partial failure code of 3 if the information for one of the publishers or repositories couldn't be read (such as the SSL Certificate).


As a result of these changes the following commands have been renamed and altered as follows:

  • pkg authority -> pkg publisher
    • The output of this command also changed.  Before, executing "pkg authority" would have given you a list of authorities and the origin URIs for each repository.  Now it shows repository mirrors as well and indicates their status and type and ensures that the preferred publisher is always listed first and the rest in alphabetical order by name.  Here is a before and after comparison from my own system:
      $ pkg authority
      AUTHORITY                           URL
      opensolaris.org                     http://pkg.opensolaris.org/dev/
      ipkg (preferred)                    http://ipkg.sfbay/dev/
      extra                               https://pkg.sun.com/opensolaris/extra/
      
      $ pkg publisher
      PUBLISHER                             TYPE     STATUS   URI
      ipkg                     (preferred)  origin   online   http://ipkg.sfbay/dev/
      extra                                 origin   online   https://pkg.sun.com/opensolaris/extra/
      opensolaris.org                       origin   online   http://pkg.opensolaris.org/dev/
      
  • pkg set-authority -> pkg set-publisher
  • pkg unset-authority -> pkg unset-publisher


All of the man pages and other technical documentation provided with the packaging system has been updated, and the IPS User Guide is in the process of being updated.  Please note that the old commands are still recognised for the purposes of backwards compatibility, but they will be removed after the 2009.06 release of OpenSolaris.

For those that are interested, the technical details are listed below:

Revision 926 / 6ee411c9026a

  • 5871 publisher apis desired
  • 3682 pkg publisher should show mirrors in default listing
  • 5071 provide direct way to change publisher to not use certificate
  • 5373 pkg commands shouldn't traceback when client can't read a key/cert file
  • 6843 problems with empty string for ssl_cert, ssl_key
  • 6897 "authority" should be "publisher"
  • 7046 client intent is not sent if same api object used for multiple operations / reasons

Friday Feb 13, 2009

"Direct" package publishing

Who says good things don't happen on Friday the 13th?

For several weeks now, I've been working on improvements to our publishing process.  The primary one, among those, is the ability to publish using pkgsend or solaris.py directly to disk without running the pkg.depotd server.  Here's a summary of the major changes:

Revision 873 / b95d76c53b5e

This revision integrated enhancements and fixes for the following issues:

  • 2691 ability to publish packages to local disk repository
  • 850 pkgsend traceback if invalid or unavailable repository url specified
  • 894 pkgsend should do more action verification
  • 1236 pkgsend open should not allow both -e and -n
  • 2430 pkgsend add of non-existent file causes traceback
  • 5088 server can still create a repo when in —readonly mode
  • 5294 pkgsend doesn't provide server response when operations fail
  • 5944 pkgsend traceback if invalid or no action type specified
  • 6566 catalog is always rebuilt when server config is not readonly

As a result of these changes, the following changes in behaviour should be noted:

  • pkg.depotd will no longer automatically create or populate the inst_root specified using pkg/inst_root or the -d option if started with —readonly. Instead, it will exit with an error message. Starting the server in a mode other than readonly will create the specified directory and any necessary subdirectories or files.
  • pkg.depotd will no longer automatically rebuild the repository catalog if not started with —readonly. However, search indices will continue to be refreshed automatically if needed. If, for some reason, the package catalog does need to be rebuilt on pkg.depotd startup or the old behaviour is desired, pkg.depotd can be started with the —rebuild option. This change should result in significantly faster startup (and less disk I/O) for large repositories served by pkg.depotd in publishing mode.
  • pkgsend and the importer now support file:// URLs for publishing. In the case of pkgsend:

    PKG_REPO=file:///path/to/repo pkgsend open foo@1.0
    pkgsend -s file:///path/to/repo open foo@1.0

    In the case of solaris.py (the importer), you will need to set the REPO variable in the makefile or override it another way. Be aware that if you specify a directory that is not a valid pkg(5) repository, publishing will fail with an error indicating that. You must create a repository first by starting the pkg.depotd server with the desired path and then exiting. A subcommand to create repositories will be added to pkgsend in the near future (bug 6574).
  • It is important to note that only one user at a time should publish directly to a file:// repository URL; the result of more than one user publishing to a file:// repository URL at a time is completely untested and is likely to fail. Bug 2696 will help enforce this by ensuring that a repository lock is obtained during publishing mode. If you need the ability to allow multiple users to simultaneously publish to a repository, you must use pkg.depotd and publish to a http:// repository URL. The tradeoff is that file:// repository URL publishing can be significantly faster and will result in significantly reduced memory usage and disk I/O.
  • Significantly more verification is now done during the publishing process. Expect things that may have appeared to work before to no longer be accepted by the server or by the pkgsend client. Additionally, pkgsend once more produces useful error information when operations fail.
I'd be interested in any feedback others might have about these changes or ways we can improve the process.  These changes should appear in build 108 of OpenSolaris 2009.  Enjoy!

Wednesday Dec 10, 2008

A New Face on Things

As part of the drive for the OpenSolaris 2008.11 release, the pkg(5) team has been working hard on improving the user experience, and fixing as many outstanding issues as we possibly can in time for the release.  Among those recent improvements is a face-lift of the depot server's BUI (browser user interface).  The short summary of these changes is that users can now search package metadata and browse packages by release and branch.  You can see the result of these changes at the primary OpenSolaris packaging repository website.

This face-lift was more than just "skin-deep" though.  As part of the changes implemented, all of the depot web-facing pages are now Mako templates and there is a new python API specially designed for these pages that can be found in pkg.server.api  (see pydoc pkg.server.api for more information).

What does this mean?  Projects that want to integrate their own web content or customise the presentation of their depot server can now do so with almost no restrictions.  It also means that future ehancements to the depot pages will be far easier.  Please note that these changes are not yet included in an OpenSolaris release; if you'd like to run your own depot server and benefit from these changes, you'll need to build and install the latest version from the pkg gate.

The good news is that there is also a great number of improvements still planned.  The following enhancements are set to be implemented in the future (if you don't see what you need or want in the list below, please open an RFE):

  • Advanced Depot Statistics Page
  • Advanced Search Capabilities (ability to filter searches by release and branch, only show latest versions, etc.)
  • Category-based RSS/Atom Feed (so users can easily get updates from a feed reader about changes to specific packages, or for security updates for any packages, etc.)
  • Comprehensive Package Browsing Capability (so that you can easily navigate through the files contained within a package, it's dependencies, etc.)
  • Depot Page Customisation Documentation
  • Improved Package Information page (instead of the place-holder text-only display we currently have) that ties into the comprehensive browsing mentioned above.
  • Package Count by Release and Branch
  • Package Popularity / Ratings Information
  • Recent Catalog Additions
  • Useful Links section

 Finally, today also marks the official release of OpenSolaris 2008.11.  This release is the culmination of the hard work of many great people in the OpenSolaris community inside and outside of Sun (including independent community members; and folks from Intel, AMD, and others) and other communities as well.  My greatest hope is that if you're reading this, and you haven't tried OpenSolaris yet, you will now.

Congratulations and thanks to all who worked so hard to make this release a reality.

Thursday Jul 10, 2008

A New Beginning

Monday, July 7th, 2008 was my first day as a Sun Employee. I never dreamed I would be here. The road here has certainly be an interesting one.

I've spent many years involved in various open source projects and communities. However, the OpenSolaris community was the first one to really grab my heart and attention for any significant length of time.

That's why when I was finally given the opportunity to work for Sun, I could think of nothing else I'd rather do. I believe in what Sun is doing; they're not just changing their business model, opening source code, or doing their best to keep shareholders happy -- they're trying to bridge the digital divide.

The digital divide is what separates our society as a whole from truly communicating or achieving their goals. When people are of one heart and one mind, they can achieve great things.

It is my belief that Sun, through open standards, open technology, and open processes is the company that can empower others to bridge that divide and ultimately bring people together.

Lots of companies claim to be innovators, but ultimately are just lining their own pockets.

I believe that Sun has realized what few others have: True innovation is when you empower people in such a way that they can improve their own lives and that of others.

I can't think of anything greater that I'd want to be a part of, can you?

Update: I've had a few people ask what I will be doing here at Sun and in what area. In short, I'm part of Solaris Kernel Engineering. I'll be working with the great folks on the ips team to bring the new packaging system up to speed. Specifically, I will initially be doing a lot of work on the web interface / repository and assisting wherever else is needed. I'm currently working from Overland Park, KS.

About

user12609878

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today