Endless Night (Take Three)

With a mixture of sadness (because it hurts OpenSolaris adoption) and great amusement (because, really, how can they still be doing this!) it is now once again time for my biannual SFW build statistics update.

You may recall my original article on unconsolidating back in December 2007 where I pointed out all the problems with this peculiar practice and the inevitability of it collapsing under the weight of its own build time and size.

Later in June 2008 I posted an update on SFW build times. To summarize, at the time SFW was up to 205 packages and a build took about three and a half hours and 10GB of disk space. (Refer to the previous articles for more info on the simplifying assumptions behind the numbers.)

Fast forward to February 2009, where are we? I ran an SFW build overnight on the current bits (build 108 closed last night) and it took seven and a half hours and 12.3GB of disk space (on the same machine as I've run the previous two build tests).

A few observations...

  • The build time has increased faster than the previous linear prediction. Using the current time/pkgs ratio, we'd be looking at 124 hours build time at 5000 packages (and 496 hours for 20,000 packages).
  • The build disk size usage has increased slower than the previous linear prediction. Using current size/pkgs ratio, we'd be looking at 203GB to build 5000 packages (and 815GB to build 20,000 packages).
  • The build is now up to a full working day. So any developer working on integrating (or fixing or updating) open source applications into OpenSolaris gets one shot per day of getting a clean nightly build (a requirement for integration).

Here is a graph of the data points so far. The dotted boxes at 5000 and 20000 packages show the range of predicted future build times.

And here is the data for the build size. While 12GB doesn't seem too bad given modern disk sizes, in practice it is also a big problem.

In my Web Stack engineering group we have several shared lab servers for doing development work and we are chronically running out of disk space (to the point that often builds fail due to lack of space, which isn't too amusing given the build took all day). With a handful of developers all of whom have a handful of workspaces (for different integration projects) going at the same time, 12GB at a time adds up surprisingly fast.

On the positive side, there has been some good news since my previous update. With the contrib repo up and running, there is an alternative to SFW (and of course, Web Stack project has its own Web Stack project repository but this is only for web tier components and not for general purpose components).

Sadly, this doesn't really help as much as it could because it is only being leveraged for packages which are considered unimportant and/or unsupported. So we continue to stuff most packages into SFW.

As before, it continues to be inevitable that SFW will collapse, it is only a matter of when it becomes so painful that it will no longer be possible to ignore the problem.

Any bets on the timeline?


Is there a rule that says that packages in the contrib repo \*may not\* be supported? I thought the plan was just that support was not \*required\*. With a small change in our assumptions it should be possible to use the contrib process for both unsupported and supported open source packages. I'm talking about unsupported meaning "throw over the wall", and supported meaning "pull updates from external source, and add porting tweaks". This reasoning doesn't apply to the fully supported and Sun-owned stuff.

Posted by Chris Quenelle on March 05, 2009 at 01:16 AM PST #

A significant limitation is that OpenSolaris is not built on OpenSolaris. Instead, they build Nevada on Nevada and later transform the bits into OpenSolaris. What this means is that at build time there is no access to any IPS content. So... anything in /contrib cannot be depended on by anything else.

Thus, /contrib is ok for top of the stack applications as long as there is no need for any other package to depend on it. It is useless for libraries or anything like that.

Posted by Jyri on March 05, 2009 at 05:34 AM PST #

So the end users who put stuff into pending are not allowed to build on OpenSolaris? Who checks? If the bits work, they work. Who cares how it was built? This sounds like another rule that can easily be relaxed for software going into the pending repo.

Posted by Chris Quenelle on March 05, 2009 at 06:10 AM PST #

End users can do as they wish, of course. The limitations are on packages built by Sun.

Posted by Jyri on March 05, 2009 at 08:40 AM PST #

Post a Comment:
Comments are closed for this entry.



Top Tags
« April 2014