Tuesday Sep 15, 2009

Repository Analytics: Helping to make informed decisions

Over the past several months our Update Center project has been rounding out our overall pkg(5)-based content delivery solution by providing usage reports to adopting project teams: GlassFish Application Server v3, Web Space Server, Web Stack and soon, Message Queue.

You may be familiar with the term web analytics, defined as the "analysis and reporting of internet data for purposes of understanding and optimizing web usage" by Wikipedia. In the context of delivering content via network package repositories, the Update Center team has deployed the beginnings of a reporting solution that helps product teams improve their offerings by providing greater insight into how add-ons and updates are being adopted by end users.

Over the past several months, our toolkit project has:

  • Defined a database schema and deployed a MySQL database to manage the usage data
  • Developed and deployed scripts to tranform raw repository access log data and to load it into the database
  • Developed an overall dashboard and an initial series of product-specific reports
  • Started tweaking reports based on feedback from users

Project teams can currently obtain the following information via the reports:

  • Active unique installations (i.e. "active users")
  • Add-on (new) packages installed
  • Updated packages
  • Operating system usage
  • Client tool usage (Update Tool GUI, Software Update GUI, pkg(1) CLI, pkg(5) Java API)
  • Add-on activity by registered installations

Additionally, we report on the versions of the toolkit that are in use across all installations.  This information is a real boon to our toolkit development project in that it lets us make informed decisions about how much we invest in older versions.  It also helps us monitor the roll out of new releases of the toolkit.

Over the next several weeks we'll publish new blog entries that explain the reports and how projects are using them to improve their products and ultimately the experience of their users.

Wednesday Aug 12, 2009

Road to AIX Support: First Build Milestone for pkg(5)

Yesterday, August 11, 2009, the Image Packaging System, pkg(5), was compiled and run for the first time on IBM's AIX® 6.1 operating system. All of the API unit tests ran successfully, and 69 of the CLI tests passed. Since I don't have root access on the AIX system being used, the other CLI tests were not expected to pass. AIX support for the command line interfaces is one of the features in the Update Center 2.3 toolkit.

The hardest part about porting pkg(5) to AIX has been to assemble the build environment that is needed to compile the software, especially without root access.  Although there are some RPMs available with some of the tools for AIX, these require root access to install.  That's one of the nice things about pkg(5) packaging is that it supports user images that do not require root access. Compilation of pkg(5) requires Python, Mercurial, a C compiler, setuputils, pyXML, simplejson, patch, cURL,  OpenSSL, and tidy. This list and the instructions for performing the build have been recorded on a project wiki page.

Once the build environment was assembled, modifications to only four files (nine lines) were needed to accomplish the initial port. These changes consisted of modifying the build and test scripts and the utility for getting a canonical operating system name to recognize 'aix' as a valid platform. Work still remains to complete more thorough testing and debugging, packaging of pkg(5) into the toolkit for AIX, and making sure other parts of the toolkit such as the Java API are supported on AIX, but this is a big first step.

With the addition of AIX in the 2.3 release, this expands the multi-platform support for the Image Package System to 5 major platforms (Solaris, Linux, Mac/OS, Windows, and AIX).

® AIX is a registered trademark of IBM. 


Monday Jun 08, 2009

New Software Update GUI

The recent release of Update Center 2.2 included a new Software Update GUI:

Update Center Software Update Interface

For details about this new feature visit Chris Kasso's blog.

Monday Mar 30, 2009

Repository Feeds: Making Progress!

Although the feed feature of the pkg(5) repository server pkg.depotd has been available for a few months, we're just starting to get it configured properly for some of the repositories that our project team helps to manage.  For example, recently Teresa of our Release Engineering team modified several repository instances to use a 60 day window rather than the default of 24 hours.  This extended window will help new users see a broader set of package additions and updates.  The setting of interest was:

cfg_cache:

[feed]
window=1440

Where "1440" is the number of hours.

The URL for repository feeds is simply:

http://pkg.sun.com/layered-collection/dev/feed

Here's a view from Thunderbird of the feed from http://pkg.sun.com/layered-collection/dev,  a repo that acts as a clearing house of development quality packages of Sun middleware:


Several comments on the user experience:

1) Date/time stamps: Are a bit misleading based on how we typically manage publication of packages to our repositories.  Since we use a "copypkgs" utility to copy a submitted package to the destination repository and this utility preserves the publication date/time stamp, the date/time displayed in the feed can be much earlier than the date/time at which the package appeared in the repository.

2) Since this is a feed for an external repository, we need to configure pkg.depotd to use a more appropriate "From" value than "SWI Release Engineering".  Perhaps, simply "Sun Microsystems" in this example. The "From" field in Thunderbird's display comes from the <author><name> element in the feed:

<author><name>SWI Release Engineering</name><email>swi-re@sun.com</email>

3) It's very nice to see the distinction between "Added" and "Updated" for new and updated packages.

4) Perhaps dropping the publication date/time stamp from the "Subject" would help reduce the complexity of the title and let the users focus on the package name and non-date/time stamp portion of the version string.  The "Subject" field in Thunderbird's display comes from the <title> element in the feed:

<title>Updated pkg:/pkg-toolkit@2.2.0,0-22.1765:20090221T002421Z</title>

5) Once we enhance our repository web front end, there will be a much better user experience when the user clicks on the website link to see the package details.  For example, displaying change summary information would be a very useful feature.

Clearly, there's room for a more user friendly display of data.  The UC2 project is keen on seeing a series of repo web UI enhancements along these lines.

Thursday Nov 13, 2008

Update Tool UI Features: New Toolbar and Files Tab

Manpreet Singh, the lead developer on the Update Tool GUI component of the multi-platform Image Packaging System toolkit, has just delivered a couple of cool and useful features.  These features are part of Build 18 of the toolkit that will be published to the developer quality repositories in a couple of weeks and they will be part of the 2.1 release scheduled to release in product quality form in January.

Files Details Tab:

Now you can easily list the files and directories that each package delivers by simply clicking on this tab.  It was a bit of a challenge to load the file list for package such as JRuby that contain over 13,000 files (!), but Manpreet quickly figured out how to do it efficiently and reliably.

The first thing I thought of when using this feature was: For an installed package, let me click on the file of interest and view it.  That would be cool.  

The second thought that occurred to me was that it would be great to have a Search component on the toolbar that would enable me to search all of the available and/or installed packages for a specific file of interest.

I'm sure we'll get around to both of these fancy features, but now that Manpreet has charted course for adding tabs to the GUI, in the near future we're planning to add several more tabs to address listing of dependencies and license agreements.

New Toolbar:

On the eye candy front, we've moved the Refresh, Install and Remove buttons from the bottom of the component lists up to the toolbar.  Doing so brings the GUI more in line with its cousin the OpenSolaris Package Manager and adds more visual pizazz and weight to the toolbar.   One could also argue that it's more conventional to place these actions up in the toolbar. 

Next up for the toolbar will be the search component mentioned above that will not only search for files, but also packages of interest.  Since the latest development version of the OpenSolaris Package Manager already exposes this capability of pkg(5), we might be able to crib some code from it.



Update Tool UI Features: New Toolbar and Files Tab

Manpreet Singh, the lead developer on the Update Tool GUI component of the multi-platform Image Packaging System toolkit, has just delivered a couple of cool and useful features.  These features are part of Build 18 of the toolkit that will be published to the developer quality repositories in a couple of weeks and they will be part of the 2.1 release scheduled to release in product quality form in January.

Files Details Tab:

Now you can easily list the files and directories that each package delivers by simply clicking on this tab.  It was a bit of a challenge to load the file list for package such as JRuby that contain over 13,000 files (!), but Manpreet quickly figured out how to do it efficiently and reliably.

The first thing I thought of when using this feature was: For an installed package, let me click on the file of interest and view it.  That would be cool.  

The second thought that occurred to me was that it would be great to have a Search component on the toolbar that would enable me to search all of the available and/or installed packages for a specific file of interest.

I'm sure we'll get around to both of these fancy features, but now that Manpreet has charted course for adding tabs to the GUI, in the near future we're planning to add several more tabs to address listing of dependencies and license agreements.

New Toolbar:

On the eye candy front, we've moved the Refresh, Install and Remove buttons from the bottom of the component lists up to the toolbar.  Doing so brings the GUI more in line with its cousin the OpenSolaris Package Manager and adds more visual pizazz and weight to the toolbar.   One could also argue that it's more conventional to place these actions up in the toolbar. 

Next up for the toolbar will be the search component mentioned above that will not only search for files, but also packages of interest.  Since the latest development version of the OpenSolaris Package Manager already exposes this capability of pkg(5), we might be able to crib some code from it.

Sunday Oct 26, 2008

pkg(5) Custom Ant Tasks in Action with OpenDS

The Update Center 2 project is please to announce an updated version of our experiment to package and deliver OpenDS using the pkg(5) system.  Since the end of May we've produced several iterations of this experiment, but this latest upgrade demonstrates a nice new developer feature of the toolkit:  custom Ant tasks for pkg(5). 

Using these Ant tasks, we've greatly streamlined the manner in which producing pkg(5) packages, package repositories and installation images can be performed in Ant-based build environments.  There's now a clear separation between the pkg(5) tooling and application-specific Ant build xml code.

Rajkumar Srinivasan of Sun's India Engineering Center and a member of the Update Center 2 project developed the custom Ant tasks in response to an ever growing demand for pkg(5) integration tooling.

Check out the upgraded example.

Saturday Oct 25, 2008

New pkg(5) version for the Toolkit

The multiplatform pkg(5) Toolkit 2.0 was based on an Image Packaging System drop from July 16, 2008. The UC2 project stopped taking new features from the pkg(5) team at that time so as to stabilize the build for the 2.0 release. In preparation for delivering the 2.1 Toolkit, we are bringing in the latest pkg(5) software. This article highlights some of the new pkg(5) features that will be part of the toolkit and explains what changes were needed in the toolkit updatetool software to accommodate the new release.

Summary of New Features

 Feature  Description
Operational History

The pkg(5) software now maintains a history of operations that have been performed on the image. The history can be accessed with the "pkg history" command and can be cleared out with the "pkg purge-history" command.

Improved Search The search feature has been improved to use more complete indexing and better data structures to improve performance.  This is for both the local image and for the server.
Repository HTML Interface The HTML interface for the repository has been improvided to provide links to the meta data about a package and the manifest for the package.  An RSS feed capability has been added so that an RSS reader can be used to track additions to a repository. The RSS link is available on the HTML page for the repository.
Data Collection The pkg(5) client now forwards information about the operations that are being performed to the the pkg.depotd server using a new HTTP header, X-IPkg-Intent.
New Programming API The pkg(5) software now has a new client programming API that is intended to be more stable.
Actuators The actuator feature allows package actions to be associated with a behavior that is activated either before or after the action is installed or removed.  The only implementation of actuators at this time is for an SMF refresh.  In the future, the plan is to implement some sort of user image actuator that would provide support for implementing configuration logic in a user image.
Automatic Refresh Commands that change the repository URL for an authority now automatically update the catalog for that authority. An option is provided to not do this automatic refresh.
Updated pkg.depotd Server Infrastructure The pkg.depotd server is now based on the CherryPy 3.1 framework.
New pkg(1) command line options
  • fix - fix errors reported by verify
  • refresh can now operate on a single authority
  • history, purge-history - displays the history of commands that have been performed on the image
  • rebuild-index - rebuilds the index used for searching the local image

Bringing this new version of pkg(5) into the Update Center 2 project requires several changes to the toolkit software. The pkg(5) internal changes and the toolkit changes that result include:

  •  The operational history feature requires that the name of the client be set when an Image object is created.  This results in an img.history.client_name  = 'updatetool' statement following the creation of each Image object.
  • The ImagePlan constructor and Image.make_install_plan now take an additional argument, a function that indicates whether the command has been canceled. Throughout the toolkit, a function that always returns false is passed in for now. The toolkit may be able to leverage this later.
  • The Image.load_optional_dependencies method takes an additional argument, a progress tracker. Since updatetool wasn't tracking this before, a quiet tracker is used for now.
  • An ImagePlan now deletes the list of PkgPlan objects (pkg_plans) after the the plan is executed to save memory. Updatetool has one situation where it uses the list of PkgPlans after the plan is executed, so the list has to be saved away now before it gets deleted by the ImagePlan.
  • The Image.split_authority method now returns the list of mirrors too. All calls this this method have been modified to expect the new return value.
  • There is a new module, global_settings, for setting global settings. The misc.MAX_TIMEOUT_COUNT value was moved to that new module.
  • The Image.find_root method now throws an ImageNotFoundException rather than a ValueError when the directory that is passed in is not within an image.
  • The noexecute argument for the Image.make_install_plan method is now required rather than being a keyword argument.
  • The ProgressTracker class has 3 new methods that must be defined by any derived class.
  • The pkg.depotd process now runs a subprocess of itself to do indexing of the repository.  This required some changes in the makepkgs and makeimage tools.
  • The new version of CherryPy changes the way that the customized DepotResponse is passed into the server.  This requires changes to the way the toolkit starts the server.

As development of 2.1 proceeds, we expect that the pkg(5) gate will be brought into the multi-platform toolkit builds more frequently up until it becomes necessary to stabilize the build in preparation for the release of 2.1.


Wednesday Oct 01, 2008

Windows Vista Support

Windows Vista is one of the supported platforms for the Update Toolkit 2 toolkit, but recently we ran into some problems with using the toolkit on an image that was installed under the Program Files folder. In Windows Vista, the Program Files folder is protected so that standard users are not able to write to that part of the file system. If a user does attempt to write there, something called a virtual store is created to receive the written data. This gives programs that are using the virtual store the illusion that they are able to write to Program Files when they really can't.  This worked fine for some operations, but there are some file operations that fail and so updatetool, running as a standard user, eventually got a permission failure. 

To fix this problem, the UC2 team will be introducing appropriate application manifests into the Windows executables that are part of the toolkit, so that the programs run as administrator when they need to on Windows Vista. As a work-around, the tools can be run as an administrator manually using the following:

  • If running updatetool from Windows Explorer or the start menu, right click on the icon and select "Run as administrator".
  • If running command line tools such as the pkg(1) CLI or the UC2 bootstrap tools, start the command prompt window that will be used to run the tools using the same "Run as administrator" option. This requires finding the Command Prompt icon in the start menu rather than using the Run "cmd" option.
Issue 704 contains more details about the root causes and the solution for this problem.
About

News for projects adopting the multi-platform port of the Image Packaging System - aka pkg(5)

Search

Categories
Archives
« April 2015
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
  
       
Today