Wednesday Dec 16, 2009

Updating Production Systems Using pkg(5)

As people get exposed to the notion of using the Image Packaging System to deliver their layered applications, they often ask us about the extent to which the pkg(5) system can be used to update production systems.  The following FAQ entry doesn't delve into a great amount of detail, but it attempts to explain the overall role of pkg(5) in this context and how the orchestration of inventorying and applying updates to large numbers of production systems is typically addressed by enterprise system management (ESM) tools - many of which integrate with underlying packaging systems.

FAQ: How can I use pkg(5) to update production systems?

Wednesday Nov 18, 2009

Update Center Toolkit 2.3 Release

The Update Center team is pleased to announce the availability of release 2.3 of the Update Center Toolkit. The focus for this release has been to improve the overall quality of the toolkit by reducing the bug backlog. This release also includes the following new features:

  • Support for the AIX operating system (command line only) and the Windows 7 operating system (user folder only)
  • A new version of pkg(5) that is aligned with OpenSolaris build 122. This version includes a new transport library based on libcURL.
  • Support for package icons in updatetool (see pkg.icon.24px)
  • Ability for updatetool to show only a few recent versions rather than all or one.
  • Support for an "excludedirs" capability in the makepkgs tool
  • Support for specifying a default ownership for files when using makepkgs
  • The copypkgs tool can copy packages to/from a pkg.depotd server
  • The HTML templates for pkg.depotd support showing only the most recent version of packages
  • Improved on-line help for the software update interface
  • Improved startup time for the software update interface when started from the notifier
  • Support for promoting new software applications via the software update interface

Downloads and release notes for the 2.3 release are now available as are updated screenshots of the graphical user interfaces.

This release will make its debut in the GlassFish v3 release that is scheduled for later this year.

Wednesday Sep 16, 2009

Using pkg(5) GlassFish v2.1 Packages

We don't offer a standalone distribution (download bundles + dedicated repos) for GlassFish v2.1, but we do produce and maintain v2.1 packages in support of GlassFish Web Space Server deployments (see my earlier post for background).   Although Web Space Server is currently the only distribution that embeds these packages, you can easily experiment with the packages by installing them from our layered collection development repository.  The beauty of the packaging system and our repositories is that it's quite simple to try out these and other packages:

  1. Download a copy of the Update Center Toolkit. Alternatively, if you're already on OpenSolaris or you already have the toolkit available with the pkg(1) CLI, move on to the next step.
  2. pkg image-create --user -p   ~/gfv2.1
  3. cd ~/gfv2.1
  4. pkg install glassfish2
  5. cd glassfish2
  6. ../ant/bin/ant -f setup.xml 
  7. bin/asadmin start-domain
  8. Access http://localhost:4848 and log in with admin / adminadmin

Voila!  That's it.

If you'd like to run the Update Tool GUI to further explore the other packages, execute "pkg install updatetool" and run "bin/updatetool".  Otherwise, you can execute "pkg list -a" to see the list of all packages that are available for installation on your platform. 

Tuesday Sep 15, 2009

Repository Analytics: Helping to make informed decisions

Over the past several months our Update Center project has been rounding out our overall pkg(5)-based content delivery solution by providing usage reports to adopting project teams: GlassFish Application Server v3, Web Space Server, Web Stack and soon, Message Queue.

You may be familiar with the term web analytics, defined as the "analysis and reporting of internet data for purposes of understanding and optimizing web usage" by Wikipedia. In the context of delivering content via network package repositories, the Update Center team has deployed the beginnings of a reporting solution that helps product teams improve their offerings by providing greater insight into how add-ons and updates are being adopted by end users.

Over the past several months, our toolkit project has:

  • Defined a database schema and deployed a MySQL database to manage the usage data
  • Developed and deployed scripts to tranform raw repository access log data and to load it into the database
  • Developed an overall dashboard and an initial series of product-specific reports
  • Started tweaking reports based on feedback from users

Project teams can currently obtain the following information via the reports:

  • Active unique installations (i.e. "active users")
  • Add-on (new) packages installed
  • Updated packages
  • Operating system usage
  • Client tool usage (Update Tool GUI, Software Update GUI, pkg(1) CLI, pkg(5) Java API)
  • Add-on activity by registered installations

Additionally, we report on the versions of the toolkit that are in use across all installations.  This information is a real boon to our toolkit development project in that it lets us make informed decisions about how much we invest in older versions.  It also helps us monitor the roll out of new releases of the toolkit.

Over the next several weeks we'll publish new blog entries that explain the reports and how projects are using them to improve their products and ultimately the experience of their users.

Wednesday Jul 29, 2009

Open MQ 4.4 Adopts pkg(5) and the Update Center Toolkit

The Open MQ team has been busy preparing the initial promoted builds of their upcoming new minor feature release, Open MQ 4.4.  It's impressive that without any real prodding from our Update Center Toolkit team, the MQ team has pretty much of their own accord begun adopting pkg(5) and the Update Center Toolkit as the main delivery format for this next release.  The MQ team has produced a series of installer wrapped download bundles containing pre-installed pkg(5) images for their supported platforms and they are in the process of delivering a companion set of raw zip downloads using virtually the same underlying pre-installed pkg(5) images.  You can obtain their preliminary promoted builds of the installer-wrapped bundles from their MQ 4.4 release site.

Shape of the pkg(5)-based Open MQ Distribution

The Open MQ 4.4 builds currently consists of at least 9 pkg(5) packages containing content ranging from portable Java components of the core product to platform-specific components such as the optional C language client API. Platforms addressed by these preliminary builds include: Windows, Linux, Mac OS X, Solaris and OpenSolaris.  Once AIX is supported by the Update Center project's port of the pkg(5) system in v2.3 of the toolkit, the Open MQ AIX bundle will likely migrate to using pkg(5) packages of their AIX install bundles.

Benefits of Sustainable, Modular Binary Packages

In addition to forming the basis of the MQ 4.4 distribution, these same pkg(5) packages are integrated into builds of the GlassFish v3 distribution. Thereby demonstrating the value of reusing the same modular binary packages across multiple product distributions.  Historically, the lack of formal packaging of raw zip and tar.gz bundles has made it difficult for development and sustaining teams to deliver and share the same updated binaries.  With the adoption of the pkg(5) system and the Update Center Toolkit, development and sustaining teams only need to deliver one set of updated packages that can be shared across multiple product distributions. 

Tuesday Jul 28, 2009

GlassFish v2.1 Packages: The Untold Story

GlassFish v3 garners a lot of attention in our Update Center 2 development project because of its relatively long history of adopting the Image Packaging System and the Update Center Toolkit for its layered, multi-platform distributions.  But there's also a story around the GlassFish Enterprise Server v2.1 release, Sun's premier Java Enterprise Edition 5  platform.

Rationale for Retrofitting Only Packages

Since the GlassFish v2 product offering already had a series of download and install bundles in unpackaged zip and self-extracting formats, it didn't make sense to retrofit those existing deliverables in pkg(5) format for multi-platform use and re-release the entire product distribution.  However, the GlassFish team and several product teams dependent on GlassFish saw an opportunity to make the GlassFish v2.1 product and follow-on support updates available as a series of pkg(5) packages that other products could incorporate into their own pkg(5)-based distributions.

For example, the Web Space Server product's dependency on GlassFish v2.1 was satisfied by including the GlassFish v2.1 pkg(5) packages in the Web Space Server package repositories.  Doing so yielded a soup-to-nut pkg(5)-based distribution of the Web Space Server + GlassFish v2.1 stack.  As part of their set of initial download and install bundles, the Web Space Server team produced a bundle containing GlassFish v2.1, its dependencies and Web Space Server pre-installed using the pkg(5) packages of these components.  Evaluators and customers of the Web Space Server product have benefited from this integrated initial install and ongoing update and add-on experience.

Packaging Kickstart

A special thanks to Christophe Ravel of the shared components team in the Sun middleware group for his instrumental work in getting the initial GlassFish v2.1 packages and their dependencies (including Message Queue, JavaDB and Ant) built for a variety of OS platforms and working with the GlassFish Sustaining team to integrate the package creation process into their existing build system.  Over much of this year, the GlassFish Sustaining team has been refreshing the pkg(5) packages as they've developed update releases for the GlassFish v2.1 product.  These updated packages of GlassFish v2.1 have been published to the Web Space Server support package repository for delivery to customers with support subscriptions. 

Based on his experience, Christophe published a presentation, Multi-platform IPS Packaging for Software Engineers, that is part of the Image Packaging System Best Practices the Update Center project offers to projects adopting pkg(5) for layered, multi-platform use.

Web Space Server Continues to Roll Out Updates and Add-ons

Back in Februrary the GlassFish Web Space Server team rolled out their 10.0 release of their new web portal product.  Following the GlassFish application server project, the Web Space Server team was the second major project team to adopt the Image Packaging System and the Update Center Toolkit for layered, multi-platform use. 

Deliveries Thus far

Since their initial release, the Web Space Server team has been busy rolling out:

  • 5 update releases consisting of updated packages published to support repositories that are accessible to customers with support subscriptions
  • GlassFish Enterprise Server 2.1 packages containing latest fixes for customers
  • More than 30 new add-on packages including many portlets and other extensions to the base platform

Additionally, the Web Space Server team has been aggressively staying on top of the Update Center Toolkit releases by publishing updated toolkit packages to both their release and support repositories.  Doing so ensures that their users get the latest toolkit fixes and features.  They are on v2.2 of the toolkit and will soon publish v2.2 Update 1.

A Satisfied Adopter

Based on more than 6 months of experience supporting their product in the field, James Falkner, a lead on the Web Space Server project, had this to say about the value of adopting the pkg(5) system and the Update Center Toolkit:

"The integration of the pkg(5) system and the Update Center Toolkit as part of the Web Space Server product offering has empowered our customers with out-of-the-box, easy to use tools for managing cross-platform Web Space Server deployments, which allows our product team to focus on delivering high quality releases and updates through an efficient pipeline. Specific benefits include:

    • Ease of rolling in external dependencies (e.g. GFv2.1+dependencies) simply by including the packages in the appropriate repository
    • Desktop notifier, which keeps our customers developing on desktops up-to-date
    • Cross-platform support for our cross-platform product
    • GUI + CLI for addressing many real world use cases
    • Usage metrics"

Closed Network Support: New Territory

Another area where the Web Space Server team is breaking new ground is by enabling customers with deployments that don't have access to the internet-based package repositories to obtain copies of the Web Space Server package repositories and run them locally to feed clients within the customers' closed networks.  More on this front in a future post.

Thursday Jul 23, 2009

OpenDS: pkg(5) Ant Integration Updated

Ludovic Poitou has graciously integrated some of my recent build enhancements into the OpenDS source tree.  These changes were mainly oriented toward enabling developers to build the pkg(5)-enabled OpenDS download bundles while disconnected from the network.  Use of the custom Ant tasks for pkg(5) developed by the Update Center 2 project is a key aspect of the overall OpenDS and pkg(5) integration.  

The OpenDS build enhancements include:

  • modifications to enable building of OpenDS pkg(5)-based image when your disconnected from the network
  • updates to the package metadata to include OpenDS project links and up-to-date package descriptions
  • streamlined developer set up steps

The updated developer experience is documented in Building OpenDS in pkg(5) Format Using the Update Center 2 Toolkit on the OpenDS project site.

This integration paves the way to build and publish both pkg(5)-enabled OpenDS download bundles and the associated packages such that OpenDS developers and users can start getting their feet wet with the benefits of this combination.

Thursday May 28, 2009

Update Center Toolkit 2.2 Release

The Update Center team is pleased to announce the availability of release 2.2 of the Update Center Toolkit. This release includes the following major new features:

  • Software Update interface for simplifying the application of updates to multiple images and for finding out about new applications and add-ons to the applications that you already have,
  • Featured Software section in the Update Tool interface for informing the user about new applications and add-ons to existing applications,
  • An updated version of the pkg(5) software that provides the same capabilities as the version in OpenSolaris 2009.06,
  • Enhanced Update Tool interface that:
    • makes use of background threads to provide a more interactive experience, 
    • provides a package search interface,
    • provides URLs in the package overview with more information about each package, and
    • enhances the interface with right click menus, additional icons, and other ease of use features
  • Improved development tools for publishing packages and creating images with pre-installed packages, and
  • Support for SUSE Linux with KDE and OpenSolaris 2009.06.

A look at the new Software Update Interface:

Update Center Software Update Interface

Downloads and release notes for the 2.2 release are now available as are updated screenshots of the graphical user interfaces.

Monday May 11, 2009

Update Center Toolkit 2.2 Build 28 Available

The UC2 development team is pleased to announce the availability of build 28 of the Update Center Toolkit.  Coming only a week later, this build resolves 47 issues beyond what was available in the previous build.  The remaining builds will fix stopper bugs only as the team works towards delivery of the 2.2 release later this month. 

Tuesday May 05, 2009

Update Center Toolkit 2.2 Build 27 Available

The UC2 development team is pleased to announce the availability of build 27 of the Update Center Toolkit.  This build resolves 44 issues beyond what was available in the previous build. This was the last open build for the 2.2 release.  The remaining builds will fix stopper bugs only as the team works towards delivery of the 2.2 release later this month. 

Monday May 04, 2009

Rollback with pkg(5) User Images

Undo Icon Today, a colleague asked about how the Update Center 2 toolkit and the Image Packaging System handles rollback to a previous version of a user image.  He noticed that the Update Tool GUI doesn't seem to have an option to do this.  

Right now, one has to either:

  • Uninstall the package(s) that you don't want and reinstall the older version.  The older version is available in the GUI by using the View|Show All Versions menu choice.  From the command line, this can be done by specifying  some-package@previous.version on the install command after first uninstalling the current version. While not atomic, this has the same effect as a "patchrm" in SVR4.
  • If using a ZFS file system, run "zfs snapshot" on your image directory before doing the update.  If you don't like the update, you can rollback to the snapshot.
  • Create a backup of the image directory before doing the update.  Restore from the backup if you don't like the update.

A few notes:

  1. Currently there is no notion of an "update" defined in Update Center, i.e., all we have are installs of new versions of packages.  So if you install new versions of 3 packages yesterday and 2 packages today, there isn't any way to identify that as 2 "updates".  Rather, it is merely 5 installs of new package versions.  The operational history does record the operations that are done, so one would be able to figure out from the history that two "update" operations were done.  But there isn't anything in the GUI or CLI that takes advantage of this.
  2. From the CLI, installing an older version of a package doesn't downgrade the package.  So if you have version 2 of a package installed and you do "pkg install some-package@1", it will output a message saying that "No updates are available for the image". Technically, there isn't any reason why IPS can't calculate what it takes to go backward directly, although there could be some trickiness relative to dependency calculations, but so far there hasn't been emphasis on that.  This is primarily because the OpenSolaris team depends on ZFS for rollback of operating system updates.  So rollbacks of user image updates has not received as much attention. Eventually, I would like to see the CLI support something like a "pkg rollback" operation that would take a list of older package versions, and the result would be that the image software would be restored to that previous version of the packages.
  3. Uninstalling a package also requires uninstalling the packages that depend on it. So accomplishing the first option can be difficult when dependencies are involved. If there were a "pkg rollback" command, it could take care of this by possibly rolling back updates to the packages that depend on the one being rolled back.
  4. The second option requires root access, which generally is not required for user images. 
  5. When a rollback is done with the second and third option, all other history in that image not related to the update is also lost.  For example, with a database in the image, any updates that were made between the update and the rollback would be lost. Generally, when people think of patch backout with SVR4 packages, they assume that data changes made while the patch was there are not lost.  This actually presents problems sometimes if the patch allows new data structures to be introduced or causes data reformatting, and then the old code has to deal with the new data (and sometimes it can't).  These rollbacks are fool proof because they literally get you back to where you were, but that might not be where you want to be.

Sunday Apr 26, 2009

Update Center Toolkit 2.2 Build 26 Available

The UC2 development team is pleased to announce the availability of build 26 of the Update Center Toolkit.  New features in this build include:

  •  The Software Update UI is feature complete.  It is started by clicking on the notifier update notification balloon.   It can also be started from the command line by entering "updatetool -u".   See the screen shot below.
  • "Featured Software" is now feature complete. This capability enables the promotion and easy installation of software applications and add-ons from the Update Tool GUI and Software Update tool.
  • The search feature is available within the updatetool interface and is now also feature complete.

This build resolves 38 issues beyond what was available in the previous build.  Additionally build 111 of pkg(5) has been incorporated in this build.  This build fixes an array of pkg(5) issues.

The Software Update user interface provides a simply and streamlined method  to apply updates to multiple application images: 

Software Update UI

Monday Apr 06, 2009

Update Center Toolkit 2.2 Build 25 Available

The UC2 development team is pleased to announce the availability of build 25 of the pkg(5) toolkit.  New features in this build include:

  • the Software Update UI is available in preliminary form by entering "updatetool -u".  This UI is also launched when clicking on the notifier balloon. 
  • the feature software feature is available in preliminary form.  Look at the top of the left hand panel in updatetool. 
  • the search feature is available within the updatetool interface.  There is still some work to do on the layout, but the basic search functionality is there.

This build resolves 14 issues beyond what was available in the previous build.

Monday Mar 30, 2009

Repository Feeds: Making Progress!

Although the feed feature of the pkg(5) repository server pkg.depotd has been available for a few months, we're just starting to get it configured properly for some of the repositories that our project team helps to manage.  For example, recently Teresa of our Release Engineering team modified several repository instances to use a 60 day window rather than the default of 24 hours.  This extended window will help new users see a broader set of package additions and updates.  The setting of interest was:



Where "1440" is the number of hours.

The URL for repository feeds is simply:

Here's a view from Thunderbird of the feed from,  a repo that acts as a clearing house of development quality packages of Sun middleware:

Several comments on the user experience:

1) Date/time stamps: Are a bit misleading based on how we typically manage publication of packages to our repositories.  Since we use a "copypkgs" utility to copy a submitted package to the destination repository and this utility preserves the publication date/time stamp, the date/time displayed in the feed can be much earlier than the date/time at which the package appeared in the repository.

2) Since this is a feed for an external repository, we need to configure pkg.depotd to use a more appropriate "From" value than "SWI Release Engineering".  Perhaps, simply "Sun Microsystems" in this example. The "From" field in Thunderbird's display comes from the <author><name> element in the feed:

<author><name>SWI Release Engineering</name><email></email>

3) It's very nice to see the distinction between "Added" and "Updated" for new and updated packages.

4) Perhaps dropping the publication date/time stamp from the "Subject" would help reduce the complexity of the title and let the users focus on the package name and non-date/time stamp portion of the version string.  The "Subject" field in Thunderbird's display comes from the <title> element in the feed:

<title>Updated pkg:/pkg-toolkit@2.2.0,0-22.1765:20090221T002421Z</title>

5) Once we enhance our repository web front end, there will be a much better user experience when the user clicks on the website link to see the package details.  For example, displaying change summary information would be a very useful feature.

Clearly, there's room for a more user friendly display of data.  The UC2 project is keen on seeing a series of repo web UI enhancements along these lines.

News for projects adopting the multi-platform port of the Image Packaging System - aka pkg(5)


« October 2016