Thursday Jul 30, 2009

The World of Web Stack (1.5)

As you may have seen, Web Stack 1.5 is out. Go ahead and give it a try!

One of the slightly confusing parts of the Web Stack distribution is that it varies by platform so there are several ways of "giving it a try". So it might be worth summarizing how and where to get it:

If you are on OpenSolaris 2009.06

Web Stack 1.5 is integrated into OpenSolaris 2009.06 out of the box. There is nothing to download. Simply install your favorite Web Stack components via the pkg(5) command. One shortcut is to install the 'amp' metapackage which will bring in a number of AMP-related Web Stack components (check the amp manifest to see precisely which ones it includes).

% pfexec pkg install amp

Note the 'amp' metapackage doesn't install all Web Stack components and you can just as well install only the ones you need, individually.

Yes, I realize OpenSolaris 2009.06 shipped last month! That means Web Stack 1.5 has been available in OpenSolaris for over a month now. So what is this week's announcement all about?

If you are on Solaris 10 or RedHat Linux

Well, unfortunately our shipping dates are a bit out of sync on different platforms (this is something I want to get aligned for the next time around) so what we're announcing this week is the availability of Web Stack 1.5 components on Solaris 10 and RedHat Linux.

Download link for Web Stack 1.5 for Solaris 10 and RedHat Linux

There are two different packaging formats you may download for these platforms: native packages and update center images.

The native packages (svr4 for Solaris 10 and rpm for RedHat Linux) are what you'd expect, similar to the Web Stack 1.4 packages (and yes, you can upgrade your previous install if you wish).

The update center image is new in this release and it is quite interesting. Instead of downloading actual Web Stack components you only download an IPS user image which you may unzip anywhere you like, such as in your home directory. From within this user image you will then invoke the pkg(5) CLI to download and install those components you wish to use (there is also a GUI, updatetool, if you're into that).

Yes, this means you can install Web Stack components into any location you like while running as your regular nonprivileged (non-root) user. This is quite nice for experimenting and development work. Please refer to the README and documentation for details. Give it a try and let us know how you like it.

The other brand new cool thing in 1.5 is the Enterprise Manager GUI, which provides a really nice monitoring interface to component statistics.

Others from my team have written in more detail about various features so I won't repeat that here, just check these out:

Finally, please note that while the full marketing name this quarter is Glassfish Web Stack, the Web Stack product is separate and completely unrelated to the Glassfish Application Server! I realize this has been the source of much confusion lately, particularly at OSCON last week.

Hopefully this helps clarify a bit how and where to obtain Web Stack 1.5! With that out of the way, now go give it a try!

Friday Jul 24, 2009

Web Stack at OSCON

It's been so hectic here lately I actually forgot to blog about this before it happened but thanks to everyone who attended our ( CVR and myself) Web Stack session at OSCON 2009 this week.

As we mentioned, we should be announcing the availability of Web Stack 1.5 'real soon now', watch this space for the details...

Monday Jun 29, 2009

Web Server 7 Meets Slowloris

Lately there's been some noise about slowloris, a perl script which sends HTTP requests slowly. While there's nothing new about this technique, I've been asked about it a few times so I wanted to show how easy it is to protect against it if you're lucky enough to be using Sun's Web Server 7.

In a nutshell, the script opens a connection to the target web server and sends valid request headers and then continues to send more headers, slowly. Specifically, it first sends:

GET / HTTP/1.1
Host: $hostname
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.503l3; .NET CLR .0.4506.2152; .NET CLR 3.5.30729; MSOffice 12)  
Content-Length: 42
X-a: b

Then it continues to send:

X-a: b

after every $timeout delay. It has a default $timeout of 100 seconds but you can change this with -timeout switch.

Let's look at the more general cases here instead of just slowloris specifically.

The most rudimentary form of this attack is to open a connection to the web server and either don't send anything or send a partial request and nothing else after that (as described above, this is not what slowloris does).

You'll want your web server to eventually time out and close the connection if this happens. In Web Server 7 this is controlled by the io-timeout element in server.xml. The default value is 30 (seconds). Let's try it:

% date;telnet localhost 80;date
Mon Jun 29 19:05:43 PDT 2009
Connected to localhost.
Escape character is '\^]'.
Connection to localhost closed by foreign host.
Mon Jun 29 19:06:14 PDT 2009

As you can see, 31 seconds went by before the connection was closed. You can change io-timeout to be shorter if you wish:


% date;telnet localhost 8080;date
Mon Jun 29 19:15:12 PDT 2009
Connected to localhost.
Escape character is '\^]'.
Connection to localhost closed by foreign host.
Mon Jun 29 19:15:27 PDT 2009

Above I changed the io-timeout to 15 and indeed it took 15 seconds before closing the mute connection. Let's try the same thing but send a partial request:

% date;telnet localhost 8080;date
Mon Jun 29 19:14:38 PDT 2009
Connected to localhost.
Escape character is '\^]'.
GET / HTTP/1.1
Host: localhost
HTTP/1.1 408 Request Timeout
Server: Sun-Java-System-Web-Server/7.0
Date: Tue, 30 Jun 2009 02:14:54 GMT
Content-length: 148
Content-type: text/html
Connection: close

<HTML><HEAD><TITLE>Request Timeout</TITLE></HEAD>
<BODY><H1>Request Timeout</H1>
The server timed out waiting for the client request.
</BODY></HTML>Connection to localhost closed by foreign host.
Mon Jun 29 19:14:54 PDT 2009

Ok, let's try to make the attack more interesting. Instead of just going silent, the client can continue sending more request data, just slowly. This is what slowloris does. As long as the client sends a little bit of valid request data often enough to not get disconnected by the timeout it can hold on to the connection.

Fortunately Web Server 7 also monitors the time it takes to receive all the request headers. This can be configured using the request-header-timeout element in server.xml. This can be used to defeat a slowloris-type attack. Even thought the slowloris request never actually completes (since it just keeps sending more headers forever), Web Server 7 will stop waiting and close it off after request-header-timeout seconds go by.


Of course, if you set request-header-timeout to 5s you could then run slowloris with a -timeout of less than 5 seconds. However, this quickly starts to defeat the premise of this style of attack. The idea behind a slowloris-style attack is to attempt to tie up the web server quietly without the client having to generate hundreds or more connections per second. For fun, I set my request-header-timeout to 1s and ran slowloris with a -timeout of 1s. The result is the client machine uses up all its CPU generating new connections while Web Server 7 continues to be happily responsive.

A variant of this attack is to send a POST request, send all the request headers and then start to send the body data, slowly. Note that slowloris does not implement this (the -httpready flag sends a POST instead of a GET, but it continues to send X-a:b request headers, not request body data). However it is easy enough to write a tool to do this instead.

If you encounter that scenario you're in luck because Web Server 7 also monitors the time for the request body to arrive and you can set a timeout on that as well, using the request-body-timeout element:


That's all there is to it to protect against slowlaris and similar slow-client attacks if you're using Sun Web Server 7! Enjoy!

Monday Jun 15, 2009

Email Backlog

It's no secret that email overload is a problem these days, here's just a few of many articles on the topic:

A quote from the second article above is particularly interesting (or scary):

   In this study Dr Jackson found that it takes an average of 
   64 seconds to recover your train of thought after interruption 
   by email. So people who check their email every five minutes 
   waste 8 and 1/2 hours a week figuring out what they were 
   doing moments before.

In nearly 20 years (wow!) of reading lots of email daily this has never been much of a problem for me though. I always managed to keep my inbox almost empty from day to day (I long considered 100 emails to be the maximum threshhold to ever have pending in the inbox).

Thinking back, I'd say historically the bulk of my incoming email has been either

  • Administrivia (meeting announcements and such): quickly dispatched without thought or mental interruptions
  • Engineering content, directly related to whatever I'm working on that day: these take time to read and process but since the emails are relevant to the current project they don't cause a mental context switch and may even help further the project at hand so there is a net win

As resources get tighter and I find myself filling more and more roles simultaneously the dynamic has changed in the last 6-9 months or so. From a perpetually clean inbox I've now gone to a significant backlog. Even more annoying is that I find there are many days where all I get to do is read email!

After some months it is clear this is not a temporary crunch, so I need to change strategies from what has worked in the past. I spent some time monitoring my email activity to figure out what is different. It's not really quantity, I've always received lots of email but it hasn't been a problem. The key difference appears to be that now I'm involved in many projects each one with many unrelated trains of discussion.

As emails arrive, each one is more often than not unrelated to the previous ones and also unrelated to what I'm actually trying to get done at the moment. And thus, I find myself facing the case made in the Dr. Jackson study quoted above.

As each email arrives I read it and start thinking of that particular project/problem for a few moments (a few seconds to a minute or two). It is not enough time to solve or address the issue, just enough to get distracted. Hoping to get back to the real work I was doing instead of spending more time on this new train of though, I don't actually process the email, so it remains in the queue.

By then, several other emails have arrived so I repeat the cycle with each one. By the time I finally get back to what I was actually working on, that project is so many mental context switches behind I no longer have any idea what I was doing and need to spend several minutes getting back into it. By which time, of course, ten more emails have arrived... and the cycle repeats all day.

So I need to address the interruption and context switch problem. A few weeks ago I started to allocate limited time to email. Specifically:

  • Only read email in specific blocks of time preallocated for email on that day.
  • If I can answer or resolve the issue in less than 5 minutes, do so right then, within the time allocated for email handling.
  • If it's going to take any longer than that, add a task to the bottom of my to-do list and move on.
  • The rest of the time, quit mutt and resist all urges to go look at email.

I started by allocating two hours a day to email, one in the morning and one in the afternoon. Quickly it became apparent this is not enough to keep up, so I increased it to three hours. I'll gather more data before settling on the final timing but looks like it'll have to be a bit over three hours a day for email processing.

Here's a graph, showing only a few days from last month. I'll post another one with much more data once some more time goes by so I have more numbers. The yellow area is my current email backlog and the blue line is the number of minutes a day spent processing email.

Friday May 15, 2009

Web Stack Deep Dive at CommunityOne

In addition to the unconference, Web Stack will also have an afternoon-long deep dive session at CommunityOne. It will be on Wednesday (June 3rd). Here is a link to the deep dive sessions page.

As of today, the agenda shown on the above page is fairly preliminary and we'll be tailoring it depending on the schedule of the speakers as well as feedback we get in the next few days on topic interest (so feel free to let us know if there is anything in particular you want explored in depth -

It'll be informative and a good chance to meet several of the Web Stack component developers and maintainers, so I hope to see you all there!

Thursday May 14, 2009

Web Stack Unconference (CommunityOne)

At the Web Stack webinar this week I mentioned the free unconference on the Sunday before CommunityOne (Sunday May 31st). Here is the signup sheet:

As noted in the above page, you don't need to be a participant in CommunityOne nor JavaOne in order to attend the unconference. Just sign up on the above page (add your name to the table) and show up! If you have an interest on any particular Web Stack topic areas please add them to the bottom of the page.

See you there!

Wednesday Apr 22, 2009

Web Stack at the MySQL Conference

Kind of a last minute announcement, but for those at the MySQL Conference today, CVR and myself will be hosting a Web Stack BoF tonight. We'll cover Web Stack in general but of course with an eye on MySQL on OpenSolaris in particular. We should have a few demos from Sriram and some slides, but mostly just open to any Web Stack discussion. Join us at 7:30pm!

Friday Mar 06, 2009

Web Stack Road Trip (2)

A couple new opportunities to hear about Web Stack are coming up this month..

Next week (March 10th) as part of the 'Glassfish Boot Camp' here in Santa Clara we'll have a hands-on lab session on Web Stack. We'll do some quick demos and I expect there to be open time for questions, discussion and trying anything you might be curious about. Unfortunately (only if you are not signed up ;-) looks like the event is sold out by now.

The week after (March 18th) I'll be at CommunityOne East in New York City where we'll have a session titled 'OpenSolaris and the Web Stack: Apache, MySQL, PHP, NetBeans PHP IDE and Dtrace Integration'. I'll give a quick intro/background to Web Stack and then we'll have a number of demos. Check it out!

Tuesday Feb 10, 2009

Updated varnish and nginx packages

Tonight I updated the nginx and varnish packages in the Web Stack project repository to their respective latest stable releases (nginx-0.6.35 and varnish-2.0.2).

There are no enhancements to the packages this time other than the version so both are still rough (refer to my previous entries on each for their current state). These continue to be experimental components, so if you'd like to play with them please do and let know how it works for you and any suggestions for improvement. If there is interest in these (or other) components, we can look into making the OpenSolaris integration stronger!

Support for Web Stack now available...

If you've been following at all today you've surely seen all the entries about the Glassfish portfolio announcements so I won't repeat all that here.

For us in Web Stack, the important part of this is that Sun is now selling support for the Web Stack components (and all other Glassfish portfolio components). So for those of you who've asked if or when can you get production support for Web Stack, now you can. Head over to and click on the "Contact Sales" link for more.

That's my ad for the week! Meanwhile, I'm pushing out some interesting updates to OpenSolaris Web Stack, I'll blog about those shortly....

Tuesday Feb 03, 2009

Endless Night (Take Three)

With a mixture of sadness (because it hurts OpenSolaris adoption) and great amusement (because, really, how can they still be doing this!) it is now once again time for my biannual SFW build statistics update.

You may recall my original article on unconsolidating back in December 2007 where I pointed out all the problems with this peculiar practice and the inevitability of it collapsing under the weight of its own build time and size.

Later in June 2008 I posted an update on SFW build times. To summarize, at the time SFW was up to 205 packages and a build took about three and a half hours and 10GB of disk space. (Refer to the previous articles for more info on the simplifying assumptions behind the numbers.)

Fast forward to February 2009, where are we? I ran an SFW build overnight on the current bits (build 108 closed last night) and it took seven and a half hours and 12.3GB of disk space (on the same machine as I've run the previous two build tests).

A few observations...

  • The build time has increased faster than the previous linear prediction. Using the current time/pkgs ratio, we'd be looking at 124 hours build time at 5000 packages (and 496 hours for 20,000 packages).
  • The build disk size usage has increased slower than the previous linear prediction. Using current size/pkgs ratio, we'd be looking at 203GB to build 5000 packages (and 815GB to build 20,000 packages).
  • The build is now up to a full working day. So any developer working on integrating (or fixing or updating) open source applications into OpenSolaris gets one shot per day of getting a clean nightly build (a requirement for integration).

Here is a graph of the data points so far. The dotted boxes at 5000 and 20000 packages show the range of predicted future build times.

And here is the data for the build size. While 12GB doesn't seem too bad given modern disk sizes, in practice it is also a big problem.

In my Web Stack engineering group we have several shared lab servers for doing development work and we are chronically running out of disk space (to the point that often builds fail due to lack of space, which isn't too amusing given the build took all day). With a handful of developers all of whom have a handful of workspaces (for different integration projects) going at the same time, 12GB at a time adds up surprisingly fast.

On the positive side, there has been some good news since my previous update. With the contrib repo up and running, there is an alternative to SFW (and of course, Web Stack project has its own Web Stack project repository but this is only for web tier components and not for general purpose components).

Sadly, this doesn't really help as much as it could because it is only being leveraged for packages which are considered unimportant and/or unsupported. So we continue to stuff most packages into SFW.

As before, it continues to be inevitable that SFW will collapse, it is only a matter of when it becomes so painful that it will no longer be possible to ignore the problem.

Any bets on the timeline?

Friday Jan 16, 2009

Notes on Web Server Open Sourcing

Brian Aker wrote about the open sourcing of our web server and it got picked up on slashdot today.

I was reading through the comments and figured I'd throw in a few notes about what this code is and is not...

(I worked directly on the Web Server product for some years and while it is not my day job today, I'm still very closely affiliated with the group who works on the commercial version of this product inside Sun.)

  • First, the released code is not a snapshot of the Netscape Enterprise Server from the 90's!
  • What it is, is a snapshot of the very latest source code for JES Web Server 7.0 (with some non-core parts removed, such as the administration infrastructure - see full list of differences here.
  • The commercial version of this product is actively maintained and sold by Sun (note it is free to download and use, however - so feel free to download both the source and the commercial binaries and try/compare both, if you wish).
  • That said, the code is indeed a direct descendant of the Netscape Enterprise Server. The marketing name changes over the years have not marked rewrites of the core code, it's been the same code all along.
  • While the revision history is not part of the open sourced snapshot (sorry), I can mention that in the internal repository of this code I see cvs comments dating back to 1995.
  • With over ten years of development and bug fixing a lot has changed, naturally. On the other hand, if you were involved with the original product way back then, you'll definitely find some familiar bits and pieces here and there. As with any mature software product, there are always some parts which have not changed in ages.
  • So, while not a mummified snapshot, the code is indeed interesting as a piece of Internet history. Furthermore, it is also interesting as a modern living product.
  • Extreme scalability in multi-CPU (or multi-core) hardware is perhaps the most interesting angle from which to look at the code. (Funnily enough, with the rise of parallelism in modern hardware, maybe the code is becoming more interesting these days instead of less!)
  • As to who or why be interested, that doesn't really have any one answer. If you find it interesting or useful for either reason (or some other of your own), enjoy! Being under BSD license, there are many ways to take advantage of it.

Tuesday Jan 13, 2009

Announcing Open Source Web Server

I'm happy to announce that our Web Server product (about which I've been writing here for a few years now) is now open sourced and available as part of the OpenSolaris Web Stack community!

Well, technically it is not exactly the Web Server product, since the open sourced code does not include some of the value-add components such as the administration framework. But it is the real deal, the massively scalable web server core which is used in the JES Web Server 7.0 product is now all open source!

This marks another milestone in the very long history of this web server. Back in the 90's this was the Netscape Enterprise Server, which later morphed into the iPlanet Web Server during the Sun|Netscape Alliance. After some years it was renamed the SunONE Web Server and most recently renamed again to the JES Web Server (Sun just like to keep you confused, thus the constant renaming of the product!)

The code is placed under BSD license, this should allow for good cross pollination with other web tier projects.


Source code is available via:

% hg clone ssh://

Build instructions are here:

(The code itself is highly portable as you can see based on the supported platforms of the commercial product. Building on other platforms is a bit more involved due to dependencies so the build instructions only cover the more flexible platforms.)

(edit: adding link to top level info page)

More info here:

Monday Dec 22, 2008

Sun Web Stack 1.4

I'm happy to announce Sun Web Stack 1.4 is now available for download!

Despite the confusing version number this is the first release of this new product. If you've been following my articles on OpenSolaris Web Stack then this product will seem quite familiar. In fact, Sun Web Stack 1.4 is a port of the primary OpenSolaris Web Stack components from 2008.11 to Solaris 10 (and RedHat Enterprise Linux).

So now you can take advantage of the work delivered by the Web Stack project on your existing Solaris 10 production systems and not only on OpenSolaris... Cool!

Speaking of cool, the strange version number (for a first release) is due to the fact that Sun Web Stack takes over where CoolStack left off. Since the final CoolStack release was 1.3, we decided to continue the version numbering as a nod to that continuity. Unlike CoolStack though, Sun Web Stack is a full-fledged Sun product, which means you'll have the option of purchasing support for it.

The packages are freely available of course. Here's some useful links:

I want to stress that Sun Web Stack is a port of the OpenSolaris Web Stack work. That means that future component upgrades and feature additions all go to OpenSolaris first and then will get promptly backported to Solaris 10 and RedHat under the Sun Web Stack banner. This also means that if you want to discuss future Web Stack directions, the best place is still The support forum I listed above is specifically for support questions related to using Sun Web Stack binaries on Solaris 10 or RedHat Linux. As the architect for both OpenSolaris Web Stack and Sun Web Stack, I'll continue covering both in this blog so stay tuned for more!

The initial design of this project started in September and it was mostly completed towards the end of last month. In conjunction with the OpenSolaris 2008.11 work, that made for very busy three months! So the timing of this release is perfect, just in time for the holiday break!

In short, if you've been following Web Stack work but are still on Solaris 10, go check out Sun Web Stack, give it a try and let us know what else you'd like to see included in the future.

Early in 2009 we will be planning on what to enhance and add to Web Stack for the next release of OpenSolaris (and thus, the next release of Sun Web Stack as well) so this is a good time to get your Web Stack wishes in!

Monday Dec 15, 2008

Web Server in OpenSolaris

Web Server 7.0u4 has been released and of particular interest is that it is now formally supported in OpenSolaris... In conjunction with the OpenSolaris Web Stack project, another piece of the puzzle falls in place for scalable web tier solutions on OpenSolaris!



Top Tags
« June 2016