On the Road to Nightly Updates: Split Workspace
By mkupfer on Oct 22, 2005
One of the milestones we're working towards for OpenSolaris is nightly updates. Currently we deliver updates every two weeks in the form of tarballs. Nightly updates would have some advantages for external developers, particularly if we do the deliveries using a source code management system instead of tarballs. First, it's usually easier to merge multiple sets of smaller changes than one massive set of changes. So it would be easier to synchronize project workspaces with the master sources. Second, if something breaks, it's easier to track down a problem if you know it was introduced on a particular date. That's because there are a lot fewer deltas to look through, compared to examining all the putbacks for a two-week period.
One of the requirements for nightly updates is making the delivery totally automated. The current biweekly deliveries to opensolaris.org require several staff hours each. Part of the problem is that the master workspace, /ws/onnv-gate, is organized as a single source tree that contains both open and closed source. The deliveries are done from a "split" workspace, with separate open and closed trees. But each delivery requires pulling over changes from /ws/onnv-gate to the split workspace and reviewing it for changes. New files are reviewed to see if they should go into the closed tree. Makefile changes have to be merged to ensure that the open tree still builds standalone. We can't afford to do this every day.
So one of the requirements for nightly updates is splitting /ws/onnv-gate into open and closed trees. This pushes the makefile changes and the review of new files closer to the people who are doing the bug fixing and project work. That scales better.
This begs the question, why hasn't /ws/onnv-gate been split already? Why not just put back the workspace that we're doing the biweekly deliveries from?
The short answer is that /ws/onnv-gate has requirements that our delivery workspace does not. For example, /ws/onnv-gate has to be able to build both the open and closed trees. The delivery workspace only has to build the open tree. /ws/onnv-gate has to produce both debug and non-debug builds. The delivery workspace only produces debug builds.
So one of the things I've been working on since before the OpenSolaris launch is a set of changes that splits /ws/onnv-gate and meets all the requirements like being able to build both the open and closed trees. The changes require edits to around 600 files, and approximately 3300 files will get moved.
I've been meaning to blog about this for some time, but it's only recently that I made this a priority. I had originally figured that since the work mostly had to do with support for the closed tree, external developers wouldn't care much about it. But a recent thread on opensolaris-discuss got me thinking that since the changes are visible to external developers, it's better to share my plans, and to do it now, rather than when I put back.
With two parallel trees, one for open code (usr/src) and one
for closed code (usr/closed), it seems natural to build the open
tree and then build the closed tree. Unfortunately, a portion
libc must remain closed for now, and it must be
libc can be built as a whole.
Another approach might be to build the closed tree first. That won't work, because there is code in the closed tree that depends on having the headers installed from the open tree.
It might work to build both trees in stages, doing both trees for each stage before proceeding to the next stage. That is, install all the headers, build the open kernel code and then the closed kernel code, build the closed libraries and then the open libraries, etc. But this approach is fragile. As soon as one closed library requires that an open library be built first, this approach will break.
So what I've done is drive the entire build from the open tree, and let it jump into closed subdirectories as needed. For example, usr/src/lib/Makefile has
SUBDIRS= \\ [...] ../cmd/sgs/libdl .WAIT \\ $(CLOSED)/lib/libc_i18n \\ libc .WAIT \\ [...]
and the rule to build
$(SUBDIRS) abi: FRC @if [ -f $@/Makefile ]; then \\ cd $@; pwd; $(MAKE) $(TARGET); \\ else \\ true; \\ fi
In this scheme, the build dependency between
and the closed i18n code is expressed as
libc_i18n sources are present, the
[ -f $@/Makefile ]
test will succeed, and the code will get built.
If they aren't present, the test fails, /usr/bin/true is executed, and the
libc_i18n target is a
For OpenSolaris builds, the closed binaries tarball will get copied into the proto area at the start of the build. Thus the fact that the closed tree is missing is harmless.
Status and the Future
The status of this work is that it's mostly implemented. We still have some cleanup to do. I expect our testing will find a few things that will need fixing prior to putback, which is currently targeted for snv_28.
Besides splitting the tree, there are a few other things that need to be done before we can have automated nightly deliveries. One item is providing better automation for generating the closed binaries tarball. Currently this is handled by building a full ON workspace, building an OpenSolaris workspace, and generating the delta between the two workspaces. This is awkward to automate because it depends on coordinating the successful builds of two workspaces, and our build scripts (nightly.sh in particular) aren't really set up for that. Of course, we could modify our scripts. But a better approach would be to modify the closed tree makefiles, so that a "make install" populates a "closed binaries" tree as well as the proto area. This would also make it easier to deliver non-debug kernel modules.
Another task is to finish automating the generation of other deliverables, such as the BFU and SUNWonbld tarballs, and tie everything together so it can be run via nightly.sh.
A third task is to move the OpenSolaris downloads from the Sun Download Center (SDLC). The Studio compiler downloads need to stay on the SDLC, but the OpenSolaris sources, BFU archives, SUNWonbld binaries, and closed binaries will be moved to opensolaris.org. The reason for this move is that a certain amount of manual work is required any time we update files on the SDLC. Hosting the files on opensolaris.org gives us the freedom to implement an approach that can be 100% script-driven.
At some point we will start delivering the sources using a read-only Subversion repository. I'm not sure whether we will have that in place before we move to nightly updates.
Besides nightly updates, another OpenSolaris milestone that splitting the tree facilitates is using opensolaris.org to host project work. Right now it's difficult for internal projects to move to opensolaris.org because most internal engineers don't have a clear picture of what files can be exposed externally and which ones must be kept confidential. After the tree has been split, the picture becomes clear: files in usr/src are okay to post on opensolaris.org, files in usr/closed must stay internal.
 actually, we bringover from the snapshot workspace for the particular build that we're delivering.
 we'll probably need to fix this before we move the master workspace to be external.
 this isn't the final source code management setup. The point is to deploy something that's easier to work with than tarballs.
Technorati tags: OpenSolaris