Do you want to go faster?
By swalker on Oct 23, 2009
Please read this if you publish packages or run a pkg.depotd server!
The short version? These changes make the client take a lot less time to perform common operations (info, list, etc.). Although, a greater benefit is only realized when the client is configured to get package data from a repository that is being served by the new pkg.depotd server in this putback.
The Nitty Gritty
Revision 1431:62b6033670e4 in the pkg(5) gate integrated the following fixes and enhancements:
10416 server catalog v1 support desired
243 need localized descriptions, other metadata at catalog level
2424 Need to use UTC consistently everywhere
3092 messaging api/framework needed for pkg clients (cli, gui, etc.)
7063 "pkg list -a -s" needs performance improvement
7163 manifests are modified by client before being written to disk
8217 package fmri should be added to manifest during publishing
9061 importer should not refresh indexes
9446 traceback for cfg_cache operations if read-only filesystem
10415 client catalog v1 support desired
11094 Client transport for catalog v1
11523 only permit FMRIs from same publisher for network repositories
11831 server api version incompatible templates can cause traceback
11832 depot needs ability to seed / alter repository configuration
11954 importer shows zero packages processed unless debug enabled
12006 merge utility should have a test suite
Client Software Considerations
The pkg(1) client now requires that the name (prefix) that you have configured your publisher with matches that reported by its repository (origin) when the package repository provides a version 1 catalog.
For example, if you have your client configured this way:
$ pkg publisher PUBLISHER TYPE STATUS URI dev (preferred) origin online http://pkg.opensolaris.org/dev
Then the next time a pkg refresh is performed, you will see the following warning:
$ pkg refresh Refreshing catalog 1/1 test The catalog retrieved for publisher 'dev' only contains package data for these publisher(s): opensolaris.org. To resolve this issue, update this publisher to use the correct repository origin, or add one of the listed publishers using this publisher's repository origin. To correct the repository origin, execute the following command as a privileged user: pkg set-publisher -O <url> dev To add a new publisher using this publisher's repository origin, execute the following command as a privileged user: pkg set-publisher -O http://pkg.opensolaris.org/dev <publisher> After the new publisher has been added, this one should be removed by executing the following command as a privileged user: ⁞pkg unset-publisher test
Please note that this means that available packages from that publisher (excluding already installed packages) will not be shown in pkg list, pkg info, etc.
This change is the first step in a set of changes to not require that a user specify the publisher's name (prefix) when using image-create, set-publisher, etc. and to further simplify configuration.
Disk space usage will also increase for the client when it is configured to use a publisher repository that contains catalog v1 data. As an example, when pkg.opensolaris.org/dev's depot software is upgraded, it now requires roughly 36MB of disk space (uncompressed) for local metadata storage (51K+ packages). This information does compress extremely well (around 7MB gzip'd).
Disk space usage will continue to slightly increase for each new build until historical and obsolete packages are implemented, at which point it will significantly decrease again.
Depot Software Considerations
As a result of these changes, if you use the new depot server with an existing repository without using the --readonly option it will be automatically upgraded to a new repository format.
If you use an older repository with the newer depot software, but want to use it with the newer depot software, you must specify --readonly and --writable-root. If you do that, a new catalog will be written to the --writable-root directory based on the contents of the repository.
In addition, there is now a hard requirement that you specify a default publisher prefix for the repository. Carefully consider what you specify here as it will be used during publication if the FMRI specified at publication time does not include a publisher prefix. The publisher prefix of any published packages will be placed into package manifests, the catalog, etc. and so changing it once set will not update the existing package data inside the repository as it is only a default.
Once a repository has been upgraded to the new format, you will be unable to use older pkg(5) depot server software, pksend, or pkgrecv with the new repository. If you attempt to use an older client with the upgraded repository, it will fail.
As a reminder, parallel publication is not supported at this time via file://, only with http://. Various import tests have been performed to verify that publication performance is roughly the same as it was before.
Finally, please note that this upgrade behaviour also applies if you are using 'file://' locations for destination repositories with pkgrecv or pkgsend.
Enough thanks cannot be given to team members that helped complete this phase of the catalog v1 project over the past month. This was truly a team effort. Transport work provided by our resident guru: johansen.