Endless Nights

How long is your night?

Or, how long does your nightly build take?

It's been about six months since my first article on Unconsolidating (as opposed to 'Consolidations', the very unique practice of the OpenSolaris organization of hosting the source code of all known applications in a single source tree instead of the more commonly accepted engineering practice of having individual applications in their own source repositories) in which I looked at the then-available packages in SFW and how long it took to build it (see the article for more).

Seems like a good time to do a refresh of the numbers and a sanity check on the predictions. Back in December I just assumed a linear increase in build time as package counts went up. Clearly there is a huge variation on build times of individual packages but I figured it would largely average out over a large number of packages.

Today (actually, last week when I looked), SFW is producing 205 packages and it took 3 hours 37 minutes to build (on the same dual Opteron w/2GB RAM which I used in December). That's not exactly on the line predicted in December, but it's actually closer than I thought it would be, so the linear approximation is not far off at all, so far.

So, once we have 20,000 packages we can expect the nightly build to take over two weeks. I guess we'll need to change the SFW release cycle since currently it pushes bits out every two weeks!

Even if we only get to 5000 packages in the foreseeable future, you'll only get to do one build per work week, so better make sure nobody broke anything ;-(

At least back in the days of batch processing and punchcards one could expect the results back the next day!

(Of course, from what I'm seeing, most would-be contributor to OpenSolaris are running within VirtualBox, so your builds will take much longer than this, I'm afraid.. I've been meaning to do a current SFW build within VirtualBox on my laptop, but I know it'll be painful so haven't found the time yet...)

Future problems aside, this is certainly a problem already. As the build time closes in on 4 hours, we can already only get a single build in during a work day, which tends to make even the simplest changes take multiple days instead of a few hours as they would in more normal circumstances.

Here's a graph showing current and predicted build times. The two lines are the predictions using the numbers from December'07 and from now, June'08. As you can see, they're identical for all practical purposes.

If we still haven't managed to solve this problem by December'08, I'll revisit the numbers and graph once again at the end of this year...


I've been thinking about this for a while. Your post has inspired me to write up my thoughts. See http://mgerdts.blogspot.com/2008/06/opensolaris-build-service-proposal.html for details

Posted by Mike Gerdts on June 11, 2008 at 12:20 PM PDT #

Thanks for the thoughts, that's an interesting angle that should be explored...

The other part of the problem which I didn't graph but is also significant is space.

Today (June'08) it takes about 10GB on disk to build SFW. That doesn't seem horribly bad given disk sizes, but even that is already hurting us since we have several developers on the servers and each one has multiple workspaces for various bugs/features being concurrently worked on.... 10GB here 10GB there and soon enough we run out of disk periodically.

(In December'07 it took about 7.5GB)

Given the same linear increase assumptions, we might be at 240GB per workspace at 5000 packages and about 1TB per workspace at 20,000 packages. Not looking forward to that!

Posted by Jyri on June 12, 2008 at 10:44 AM PDT #

It's not 10 GB per workspace. It is 10 GB per (initial) full build. If one full build is done per week (twice as often as ON creates a new build tag), that is only half a terabyte per year. Arguably most builds from 6+ months ago are no longer interesting and can be whacked.

Each developer has a private workspace that is not in the build area - most likely on his or her workstation or laptop. When the developer wishes to build, the diffs are pulled in by the build system. In the case of typical bug fixes (as opposed to upgrading to the next gnome release) those diffs are a few kilobytes.

The build system creates a clone of the pre-built workspace. The cost of that clone is a few kilobytes. The diff is applied to the cloned workspace, using several kilobytes more. An incremental build (nightly -i) is performed - typically resulting in several megabytes. Unless a core library or header file is modified it will be a lot less than even one gigabyte - much less ten.

Once the build is done the resulting objects that changed are transferred into an IPS repository. This is likely several more megabytes - in all likelihood significantly smaller than the amount of (new) space consumed by the cloned workspace. Once the transfer to the IPS repository is complete, the (temporary) build workspace can be destroyed.

The net increase for the typical small change is typically several megabytes. Presumably the packages also expire (and are deleted) from such development IPS repositories after some fixed time (likely less than a month) so the rate of storage growth should be quite manageable.

Those that really need a full clobber build or are bumping the version of GNOME are rather more likely to have adequate personal build resources.

Those that are needing multiple concurrent repositories likely just need the source - not a full build. They should be able to use the build farm for for that.

Posted by Mike Gerdts on June 12, 2008 at 11:26 AM PDT #

Post a Comment:
Comments are closed for this entry.



Top Tags
« July 2016