OpenDS in pkg(5) Format Using the UC2 Toolkit: An Experiment

Updated Jan 14, 2010

The Update Center 2 Toolkit project is demonstrating how the Image Packaging System, also known as the pkg(5) System, can deliver the following features for multi-platform layered applications:

  • Multi-OS platform packaging of packages and updates
  • Pre-installed installation images
  • Ongoing installation image management
  • Network repository-based delivery of add-ons and updates 
  • Desktop notification of updates and new add-ons

This article describes the results of the first of several experiments intended to demonstrate how projects' build environments can be enhanced to deliver pkg(5) packages and pkg(5)-based install bundles. In this article we address the first stage of an experiment based on the popular OpenDS project.

Update:  As of late 2009, building the pkg(5) form of delivering OpenDS is available in the official OpenDS source tree and is documented on the OpenDS wiki.   After satisfying a couple of dependencies, you can build the packages and download bundle image straight from the official OpenDS source.  See the document Build and Use OpenDS in pkg(5) Format for details.

Try the End Result

Note: You will need access to Sun's internal network to try the following image. This limitation will be addressed once we publish an experimental repository outside the firewall. Even if you don't have access to Sun internal network, you can try the approach of building the image and your own local repository as explained later in this document.

If you'd like to experience the end result of this experiment, you can download the following file and try out a multi-OS platform, pkg(5)-based install bundle of OpenDS.

OpenDS Multi-platform pkg(5) Image (11.6 MB, multi-OS platform)

This single download applies to the following platforms:

  • Windows x86
  • Mac OS X x86 and PPC
  • Linux x86
  • Solaris x86 and SPARC
  • OpenSolaris x86

Once you've expanded the zip file, execute the following command to complete installation of the Update Center 2 toolkit runtime:

bin/updatetool

Since some of the toolkit dependencies include native binaries such as a minimized Python runtime, there is some magic happening behind the scenes to dynamically complete the UC2 toolkit installation based on the platform you are using. 

The dynamic completion process requires network access to the associated package repository.

Once the UC2 toolkit components have been installed, you can execute the same bin/updatetool command again to start the GUI-based Update Tool to browse and install OpenDS-related add-on components and updates as they become available.

You can execute the opends/setup command to launch the OpenDS QuickSetup tool to configure and start your directory server instance.

See the opends/install.txt file for more information about how to set up and explore OpenDS.

What's Different Between the OpenDS Zip Images?

First off, the size difference between the current unpackaged OpenDS zip download and the pkg(5)-based zip download is negligible: both are about 13 MB. 

Use of the pkg(5) system in this experiment involved zero changes to the OpenDS code base other than extensions that were added to the build scripts.  Modest changes were made to the filesystem layout of the current unpackaged OpenDS binary zip distribution in that the core OpenDS component is placed in its own subdirectory of the installation.  This change makes it easier to add other software components without having to install them within the core OpenDS directory structure.

  • OpenDS in the pkg(5) image is installed one level deeper than in the unpackaged zip file.
    • opends/ is the exact same layout as in the unpackaged zip file
  • bin/, pkg/, and updatetool/ directories are adjacent to the opends/ installation area
    • The new top level bin/ area enables components to represent symbolic links or wrapper scripts for commonly used commands.  For example, the OpenDS project could represent very frequently used commands in this top level bin/ location.
  • pkg(5) control files including the local package database are housed under a hidden directory .org.opensolaris,pkg/

Objectives of this Experiment

The initial stage of this experiment focused on extending the OpenDS build environment to generate:

  • OpenDS pkg(5) packages,
  • an initial pkg(5) package repository and
  • a pre-installed pkg(5) image derived from the repository

The UC2 project team expects the results of this experiment to be helpful to both the OpenDS project team in their effort to better understand integration with pkg(5).

Packaged Content

The content packaged in this experiment includes:

  • OpenDS core runtime
  • OpenDS Directory Services Markup Language (DSML) Gateway
Other packages can be easily added to this experiment.  For example, the SNMP support for OpenDS could either be added to the core OpenDS package or delivered as a separate package.  Additionally, to help facilitate use of the Java EE-based DSML Gateway, we could include GlassFish v2 and/or v3 pkg(5) packages in the same repository.

The OpenDS core runtime is included in the downloadable zip file of the pre-installed pkg(5) image while the other components are available as add-on packages in the associated pkg(5) package repository.

Existing Build Features Left Intact

Since we did not want to adversely impact the OpenDS project's ability to continue generating unpackaged zip-based install bundles of OpenDS, the experiment did not modify the existing zip-based build features.  It only added a set of build features to enable the generation of pkg(5)-based packages, an associated repository and pre-installed zip image.

The OpenDS project may decide in the future to supplant the existing unpackaged zip creation function of the build environment with the generation of the pkg(5)-based zip image, but that decision is obviously one owned by the OpenDS project team.

Additions Not Yet Checked In

The additions to the OpenDS build environment have not yet been checked into the OpenDS source repository.  As you'll see below, in order to try out these enhancements, you'll need to download several files and apply their contents on top of an existing OpenDS build environment.  Once the UC2 and OpenDS project teams have reviewed these enhancements and made suitable modifications, some of these enhancements may eventually be incorporated into the OpenDS source repository.

Try Building it Yourself

If you are either a NetBeans user or are comfortable with the command line Subversion and Ant tools and you would like to try out the extensions produced by this experiment, follow these instructions to publish the pkg(5) packages and assemble a zip-based pkg(5) image of OpenDS.

Once a form of this pkg(5) integration is bought back by the OpenDS project, you won't have to do anything other than execute an Ant build.xml target or two.  In the meantime, you'll need to download several files and overlay their content on top of an existing OpenDS source workspace.


Assemble the pkg(5) Image and Repository

1) Install NetBeans (optional) and use Subversion to check out the OpendDS source tree:

svn checkout --username guest https://opends.dev.java.net/svn/opends/trunk/opends

2) Download pkg(5) Toolkit

Download the pkg(5) toolkit for your client platform and expand it under the following directory of your newly checked out OpenDS source:

opends/ext/pkg-toolkit/

3) Add Supplemental Ant Build File

3.1) Download the following file into the top level of the OpenDS source tree next to the existing build.xml file.  This file contains new targets to carry out pkg(5)-based packaging and image assembly.

build-pkg.xml

into:

opends/

Although the pkg(5)-specific Ant code is currently contained in this separate Ant file, the Ant code could be easily integrated into the existing OpenDS build.xml file. 

3.2) Modify the existing OpenDS build.xml file to include the pkg(5) Ant build file:

Add the following <import> line in bold near the top of the existing build.xml file:

...

<project name="Directory Server" basedir="." default="package">
  <description>
    This is the build script for the OpenDS Directory Server.  See the BUILDING
    file in this directory for build instructions.
  </description>

<import file="build-pkg.xml"/>

  <!-- Build JVM properties -->
  <property name="MEM" value="256M"/>

...

4) Add Several Packaging Prototype Files

Expand an archive containing several OpenDS-specific package definition files:

opends-pkg-defs.zip

into:

opends/src/build-tools/

These package prototype files are used to convey the package metadata and the list of files and directories to include in the pkg(5) packages.

5) Execute the following Ant build targets to build the existing, non-pkg(5) OpenDS zip download:

ant clean
ant dsml

6) Execute the following new target to create the pkg(5)-based packages and publish them to a local package repository and to assemble the pkg(5)-based install bundle:

ant package-pkg-make-image

What Happened?

Two key Ant build targets were executed:

1) package-pkg-make-packages

3) package-pkg-make-image

The package-pkg-make-packages target creates a reference installation of the OpenDS content and invokes the custom Ant task makepkgs to create a repository of the OpenDS packages.  Several package prototype files are used to define the scope of the packages and to provide metadata for the packages.  The result of this target is the OpenDS-1.1.0-repo.zip file.

Next, the package-pkg-make-image is executed to create the pre-installed image, install several packages, set some identitfying information, set the source repository and archive it in the OpenDS-<version>-image.zip file.

For now, only the following components are included in the pre-installed image:

  • opends package installed under opends/
  • pkg-java package installed under pkg/. This package contains the pkg(5) Java Bootstrap facility under pkg/lib/pkg-bootstrap.jar that will install the Python-based pkg(5) client and its native dependencies.

  • under bin/, several wrapper stub scripts for the pkg and updatetool executables. The wrapper scripts were copied from pkg/lib/pkg-bootstub.sh and pkg-bootstub.bat.

It's important to point out that the pre-installed pkg(5) image in its zip archive format is very similar in layout and size when compared to the un-packaged OpenDS zip install bundle that already exists.  However, the pkg(5) form includes a local package database and the special pkg(5) bootstrap wrapper stub scripts and the Java-based pkg-bootstrap.jar file.  An existing OpenDS user could happily use OpenDS in the form of the new pkg(5)-based image without really caring that we've incorporated the pkg(5) capabilities within it.

Note that the pkg(5)-based zip distro in this experiment is what we refer to as a "universal" image in that it doesn't contain any OS platform-specific pkg(5) toolkit binaries and the OpenDS content is multi-platform (although it contains files to accommodate various OS platforms). 

If such universal images are not needed or the dynamic completion process is unwanted, a project can generate OS platform-specific pre-installed pkg(5) images that already contain both pkg(5) toolkit OS platform-specific packages and their dependencies as well as project-specific binaries and packages that are OS platform-specific.  In these case, the wrapper stub scripts and the pkg-java package do not need to be included in the pre-installed image because the pkg(5) and Update Tool portions of the toolkit along with their dependencies would already be included in the pre-installed image.

Try Out the Image

1) Start the local repository

<workspace>/ext/pkg-toolkit/pkg/bin/pkg.depotd -d <workspace>/build/package/pkg/repo-complete-<platform> -p 10001

For example, on a Mac:

$ /Users/ckamps/NetBeansProjects/opends/ext/pkg-toolkit/pkg/bin/pkg.depotd -d /Users/ckamps/NetBeansProjects/opends/build/package/pkg/repo-complete-darwin-universal -p 10001

You can access the repository via your web browser:



2) Run the Update Tool GUI from the pkg(5) Image

<workspace>/build/package/pkg/image/OpenDS/bin/updatetool

Answer "Y" to dynamically complete the installation of the pkg(5) toolkit components and execute the command again to launch the GUI.

Browse the "Available Add-ons" and install them.

Build Environment Enhancements

The following parts of the OpenDS build environment were modified to support the pkg(5) integration:

  • build-pkg.xml - added
  • build.xml - enhanced to import build-pkg.xml
  • src/build-tools/pkg/ - added
  • ext/pkg-toolkit/ - added

build-pkg.xml

This supplemental Ant build file was introduced to generate the pkg(5) package repositories and a universal OpenDS pkg(5) image.

build.xml

A single line was added to import the build-pkg.xml build file.  See the instructions above for the enhancement.  The OpenDS project may choose to integrate the content of this supplemental build file or some derivative of it into the main build.xml file.

src/build-tools/pkg/

Contains pkg(5) package prototype files and other information used during package publishing and image assembly.

ext/pkg-toolkit/

Contains pkg(5) toolkit.

build/package/pkg/

This directory and its content are created dynamically while building the pkg(5) image and generating the pkg(5) repositories.  The following deliverables result from this process and are housed in this directory:

  • OpenDS-<version>-repo.zip - OpenDS packages
  • OpenDS-<version>-image.zip - Install bunde containing the universal pre-installed image

Improvements and Next Steps

Working with the OpenDS project, the UC2 Toolkit project expects to address the following limitations in further stages of this experiment. Some of these enhancements are absolutely necessary before the OpenDS team would consider integrating generation of pkg(5) packages, repositories and pre-installed images into their build environment.

Package Versions

We should probably enhance the pkg(5) integration to use the OpenDS build number followed by the SVN revision number in the revision portion of the package versions.  Currently, only a placeholder value of "0" followed by the SVN revision number is being represented in the second half of the package version string.

Design of OpenDS Packages

We leave it to the OpenDS project to determine how many packages should represent the OpenDS binaries.  Our experiment only broke OpenDS into two packages, but given the domain expertise of the OpenDS team, they may opt to define more packages.

OpenDS QuickSetup Java WebStart Integration

The UC2 project would like to demonstrate an experimental integration of a pre-installed OpenDS pkg(5) image being delivered by a slightly enhanced version of the OpenDS Java WebStart QuickSetup feature.  Doing so would likely be quite easy given that the pkg(5) pre-installed image can be delivered in zip form just as the unpackaged OpenDS zip is delivered today.  Beyond transporting the pkg(5) pre-installed image of OpenDS, we should consider triggering the dynamic completion of the toolkit installation during the Java WebStart-based initial installation.

OpenDS Status Panel Integration

It makes sense to at least represent whether updates are available within the OpenDS status panel and ideally enable the user to review and apply those updates. Since there is now a basic Java client API for pkg(5), demonstrating an "Updates Available" button in the OpenDS status panel should be relatively easy.

OpenDS Packages for OpenSolaris Distro

This experiment does not address how to enhance the layout of the OpenDS installation and to define pkg(5) package(s) that would be published to the OpenSolaris pkg(5) repository.  This experiment is focused on the assembly of an OpenDS image suitable for cross-platform, layered installation situations.  There are critical differences in requirements for these two environments that were not reconciled in this experiment. For example, default install locations and possibly specification of dependencies on other packages in the OpenSolaris distribution are key considerations.

Comments:

Great post!

Would it be possible to provide the xml files you have created so that somebody outside Sun could build the image on his own? The links to the ant build script do not work outside Sun.

Thanks, Johannes

Posted by Johannes Nicolai on January 14, 2010 at 03:40 AM CST #

Hi Johannes,

The XML files are now available in the publicly available OpenDS source. Meaning that after satisfying a couple of dependencies, you can build the packages and download bundle image straight from the official OpenDS source.

Here's a link to the instructions:

https://www.opends.org/wiki/page/BuildAndUseOpenDSInPkg5FormatUsingTheUC2Toolkit

Chris

Posted by Christopher Kampmeier on January 14, 2010 at 03:49 AM CST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

ckamps

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Feeds