Thursday Jul 09, 2009

Rack and content_type for multipart requests

I wrote this entry a while back but never posted it. Seems a shame to waste it.

 While updating the Apache Olio app to do caching we found that on the new test rig, adding events and people (it's a Social networking app) would fail. We had moved to the new rig because our time on the lab systems had run out, but fortunately we diligently left an overlap and were able to go back and look at the Rails stack to see what had changed. The problem that we were seeing gave us a clue as all of the parameters in the  HTTP POST request that contained the person or the event details looked like this:

Parameters: {"address"=>{"city"=>#<File:/tmp/RackMultipart4952-35>, "zip"=>#<File:/tmp/RackMultipart4952-37>, "country"=>#<File:/tmp/RackMultipart4952-38>, "street1"=>#<File:/tmp/RackMultipart4952-33>, "street2"=>#<File:/tmp/RackMultipart4952-34>, "state"=>#<File:/tmp/RackMultipart4952-36>}, "commit"=>#<File:/tmp/RackMultipart4952-20>, "event_image"=>#<File:/tmp/RackMultipart4952-30>, "event_document"=>#<File:/tmp/RackMultipart4952-31>, "authenticity_token"=>#<File:/tmp/RackMultipart4952-39>, "event"=>{"title"=>#<File:/tmp/RackMultipart4952-21>, "event_timestamp(1i)"=>#<File:/tmp/RackMultipart4952-25>, "event_timestamp(2i)"=>#<File:/tmp/RackMultipart4952-26>, "event_timestamp(3i)"=>#<File:/tmp/RackMultipart4952-27>, "telephone"=>#<File:/tmp/RackMultipart4952-24>, "description"=>#<File:/tmp/RackMultipart4952-23>, "summary"=>#<File:/tmp/RackMultipart4952-22>, "event_timestamp(4i)"=>#<File:/tmp/RackMultipart4952-28>, "event_timestamp(5i)"=>#<File:/tmp/RackMultipart4952-29>}, "tag_list"=>#<File:/tmp/RackMultipart4952-32>}

instead of like this:

Parameters: {"address"=>{"city"=>"aaynaiuotrtgs", "zip"=>"81602", "country"=>"USA", "street1"=>"49857 Pk Ln", "street2"=>"", "state"=>"BC"}, "commit"=>"Create", "event_image"=>#<File:/tmp/RackMultipart4833-2>, "event_document"=>#<File:/tmp/RackMultipart4833-3>, "authenticity_token"=>"VdNDLR/dCCJe96Ua3zEC9ZOwPg2DxujQ5D6pxI9E0ws=", "event"=>{"title"=>"aa rygrtokldq t ", "event_timestamp(1i)"=>"2008", "event_timestamp(2i)"=>"10", "event_timestamp(3i)"=>"20", "telephone"=>"0014879641640", "description"=>"kw sjnieb vui fslzpn jokjw xjijsm jzeweyio dthti vckudre osoempc jurldvyi adusy twghtlzwluh cowiczskxg wql ctulke km yxtuost enixrl qv to ltszeriord lpxrlp cokjtrehwc mbrnchxh fdnxwie x nuuzpvvv pqlwqghg thwtgc svuzbnzdokgv iqwsrvokviuw l z gnr trkmc aspwbgckozcg so jq dcjxl vluosk dypk rkhg iseurrximrvk qnepyyzxu iugxbgmvcui mahnpibcoa wbhvplqym ogompcsikpz engr ugipr uvj w duk dqefcurj zoztkh ", "summary"=>"x c ztsg ncccoca e dspe azhzwvcz blfdtdllh zpbothd gctqotpln eunpoudzboef fcbzcstxh ", "event_timestamp(4i)"=>"20", "event_timestamp(5i)"=>"10"}, "tag_list"=>"tag1"}

So all of the parameters in the request were being treated as file uploads and it very much looked like Rack might be the cause. We had been using Thin as our Rails runtime and a check on the two systems showed that the old system had Thin 1.0.0 and Rack 0.9.1 and the new system had Thin 1.2.2 and Rack 1.0.0. Going back to the older versions fixed the issue.

Rack processes multipart form data with a couple of passes through the class method parse_multipart(env) (rack/utils.rb) where env is a wrapper around the request (as StringIO). The first pass processes the StringIO and extracts the form data and it's parts. To determine if a part is a file upload it used to run the following check:

filename = head[/Content-Disposition:.\* filename="?([\^\\";]\*)"?/ni, 1]
if filename
  body ="RackMultipart")
  body.binmode  if body.respond_to?(:binmode)

Which basically locates lines like the following in the form data:

    Encapsulated multipart part:  (image/jpeg)
        Content-Disposition: form-data; name="event_image"; filename="event.jpg"\\r\\n

In Rack 1.0.0 the conditional changed to:

if content_type || filename
  body ="RackMultipart")
  body.binmode  if body.respond_to?(:binmode)

The check for content_type had always been there and I won't list the code, but needless to say the conditional was now: if either content_type or filename (or both) are set then treat this part as a file upload.

In the Apache Olio Rails Driver (the code that drives load to the App) we have to assemble POST requests by hand and the code is all based on Apache HttpClient 2 (we use 3 now but the same code using deprecated methods in 3). What we had been doing to add text params to the POST request up to this point was:

MultipartPostMethod post = new MultipartPostMethod(addEventResultURL)
post.addParameter("event[title]", <randomly generated String data>);

This had the unfortunate effect of adding a content_type to the form-data with the result that the request looked like this:

     Encapsulated multipart part:  (text/plain)
        Content-Disposition: form-data; name="event[title]"\\r\\n
        Content-Type: text/plain; charset=US-ASCII\\r\\n
        Content-Transfer-Encoding: 8bit\\r\\n\\r\\n
        Line-based text data: text/plain

We modified the code to use a StringPart and addPart() instead of addParameter():

StringPart tmpPart = new StringPart("event[title]", <randomly generated String data>);


and we also had to explicitly set content_type to NULL on the new part:


and the form data in the request now looks like this:

Encapsulated multipart part: 
        Content-Disposition: form-data; name="event[title]"\\r\\n
        Content-Transfer-Encoding: 8bit\\r\\n\\r\\n
        Data (19 bytes)

Friday Jun 19, 2009

EventMachine and the elusive emptys

Having built Ruby 1.9 packages for Solaris Nevada we had to try installing the commonly used gems, particularly those for the Rails stack and specifically the ones we use with the Apache Olio Rails app. One of these was eventmachine which is used by Thin and which is a great piece of Software. It installed fine on our Solaris Nevada b116 x64 box but failed to install on our Nevada b116 SPARC box. The error was pretty bizarre too:

usr/ccs/bin/as: error: no input filename given
usage: /usr/ccs/bin/as [-V] [-Q{y,n}] [-q] [-s]
        [-S] [-K {pic,PIC}] [-o objfile] [-L] [-T]
        [-P [[-Yc,path] [-Ipath] [-Dname] [-Dname=def] [-Uname]]...]
        [-m [-Ym,path]] [-n] [-ul] [-xF]
        [-m32] [-m64]
        [-xarch={v7,v8,v8a,v8plus,v8plusa,v8plusb,v9,v9a,v9b,sparc,sparcvis, sparcvis2,sparcfmaf,sparcima}]
        [-xcode={pic13,pic32}] file.s...
\*\*\* Error code 1
make: Fatal error: Command failed for target `binder.o'

What looked suspicious here, was the build line, which looked like this:

g++ -I. -I/usr/ruby/1.9/include/ruby-1.9.1/sparc-solaris2.11 -I/usr/ruby/1.9/include/ruby-1.9.1/ruby/backward -I/usr/ruby/1.9/include/ruby-1.9.1 -I/var/ruby/1.9/gem_home/gems/eventmachine-0.12.8/ext -DBUILD_FOR_RUBY -DHAVE_RB_THREAD_BLOCKING_REGION -DHAVE_TBR -DHAVE_WRITEV -DHAVE_WRITEV -DOS_UNIX -DOS_SOLARIS8 -DWITH_SSL -I/usr/sfw/include -fPIC -g -O3 -fPIC  -g -obinder.o -c binder.cpp

There's no space in '-obinder.o' which you'd normally expect to read '-o binder.o'. It looked odd, though it's not unusual for spaces to be ommited when passing names to options, I'd just never seen it on a build line before so it stood out. Looking at the output with -v set as one of the flags passed to g++, gave more data but not anything useful. We looked at it on the x64 system (again with -v as a flag to g++) and it went through the same steps but didn't fail.  But it did use a different assembler! gcc/g++ on x64 uses /usr/sfw/bin/gas but on sparc uses /usr/ccs/bin/as. It also became apparent by running the assembler manually that it was choking on -obinder.o, it seems that gas is more tolerant than as :o(

The Makefile that gets generated by extconf.rb when you install the gem has lines in it's .SUFFIXES section of the form:

        $(CXX) $(INCFLAGS) $(CPPFLAGS) $(CXXFLAGS) $(COUTFLAG)$@ -c $<

and earlier in the file COUTFLAG is defined as:


Which ultimately results in -obindir.o (same for all of the other objects it builds).

Matz seems to have hit this problem before, because the Ruby 1.9.1 Makefile contains a fix for a similar problem in Ruby (although it still happens when building the ext/\*.c sources)

empty =
OUTFLAG = -o $(empty)
COUTFLAG = -o $(empty)

Which results in a space being inserted after the '-o' for $(OUTFLAG) and for $(COUTFLAG)


The Makefile is of course generated by extconf.rb which gets it's instructions on how to create the Makefile from mkmf.rb which Ruby provides. So the fix would be to make the changes outlined above in lib/ruby/1.9.1/mkmf.rb or to have it create the SUFFIXES lines with a space between the COUTFLAG variable and the output file name. We can do that for any packages that we build, but we also need to raise a bug with Ruby.

Building Ruby 1.9.1 on Solaris SPARC

We've been working on building Ruby 1.9.1 (p129) packages on Solaris Nevada (the platform that OpenSolaris is mostly built from). We hit a couple of problems on the way, one was easy to fix the other not so.

The first issue was Sun Studio borking when it found a function declared with a return type of void but which actually contained a return statement. gcc actually thinks this ok which seems odd to me, maybe it's just our version of gcc though... When gcc hits this it says:

pty.c:425: warning: `return' with a value, in a function returning void

But with Sun Studio cc you get:

pty.c:425: void function cannot return a value
cc: acomp failed for pty.c

The file causing the error is ext/pty/pty.c and line 425 has the offending line in function getDevice() (declared as 'static void'):

return Qnil;

If you comment out this line it will build ok. There is a bug for this issue......

The other problem only occured on SPARC initially, we were using one of the public build systems that we have for the SFW consolidation (the part that delivers most of the F/OSS into Solaris Nevada and OpenSolaris) and that was running Solaris Nevada build 114, for x64 we were building on Solaris Nevada build 116 which was the latest version available at time of writing. x64 built fine.

The error seen on SPARC was:

Undefined                       first referenced
symbol                             in file
rb_cFalseClass                      enc/emacs_mule.o  (symbol scope specifies local binding)
rb_cTrueClass                       enc/emacs_mule.o  (symbol scope specifies local binding)
rb_cFixnum                          enc/emacs_mule.o  (symbol scope specifies local binding)
rb_cSymbol                          enc/emacs_mule.o  (symbol scope specifies local binding)
rb_cNilClass                        enc/emacs_mule.o  (symbol scope specifies local binding)
ld: fatal: symbol referencing errors. No output written to .ext/sparc-solaris2.11/enc/  

when linking enc/ It's not a well documented error, but the implication was obvious, the required symbols had been found but they had been declared as local and so couldn't be used to build this shared object. Using nm on emac_mule.o and on seemed to indicate that the symbols were needed by emacs_mule.o (UNDEF) and were available in We asked the compiler experts and they thought that perhaps the symbols were declared as HIDDEN. We tried elfdump on both emacs_mule.o and and guess what, when analyzing, elfdump threw up loads of errors of the type:

"bad symbol entry: <address> lies outside of containing section"

This suggested that the shared library was broken in some way.

We isolated the linker lines from the build for and ran them individually (after touching ruby.c). There were two lines, the first was the linker line which actually built the shared library. That ran ok and when we ran elfdump on the resultant library there were no errors. The second line was:

/usr/sfw/bin/gobjcopy -w -I "Init_\*"

After running this manually we saw the same error when using elfdump on the resultant library.

At the same time as this we were running a build on a SPARC system that we'd had upgraded to Nevada build 116 and that completed OK. A check on the version of gobjcopy on the two systems showed that we had gobjcopy 2.15 on the build 114 system and 2.19 on the build 116 system. Further checking showed that gobjcopy was delivered into Solaris Nevada in SUNWbinutils and that had been updated in Nevada build 116. So the problem wasn't the fact we were building on SPARC but that we were building on different OS revs, the problem also exists on x64.

At the moment we haven't looked into what was going wrong when gobjcopy tried to make the Init_\* symbols local, but it was apparently corrupting the library.

At the moment this makes it tough to build Ruby 1.9 on OpenSolaris which is based on Nevada build 111b and we are looking at how best to get around this. Maybe make the packages available from the /webstack repository. In the meantime we'll file a bug against OpenSolaris and come up with a workaround.

Thursday May 14, 2009

Apache Olio: Web 2.0 toolkit for Java EE

This week saw the Apache Olio project release the code for it's Java EE version adding to the versions already available for PHP and Rails. 

If you know Apache Olio and want to know more about the specifics of the Java EE version then I'll cover them first, if you want to know more about Apache Olio in general, read on.

To run the Java EE version of Olio, you'll need:

  • Java SE 5 or 6
  • Java EE 5 compliant application server (tested so far with GlassFish v2)
  • A Java Persistence API (JPA) provider (Eclipselink is the JPA provider packaged with GlassFish v2)
  • MySQL Database (any DB could be used but we have scripts and instructions for MySQL)

Some of the technologies that the Java EE version features:

  • JPA for the Object-Relational persistence solution
  • AJAX 
  • Rest based Services (partially implemented)
  • JMaki widgets wrappers for Yahoo and dojo widgets

In planning are the following changes/features:

  • Re-implementation of the memcached layer for caching (this was stripped out for this release but needs to be put back)
  • Rest based services with JSR-311, JAX-RS.  I've started this already using the Jersey implementation.
  • Replacement of the jMaki widgets with appropriate alternative
  • Minor features to 'catch up' with the PHP and JRuby version.
  • Investigation of file distribution system eg. Hadoop (current implementation only uses local filesystem)

If you want to get involved then visit our page at

You can contibute patches, submit bugs or RFEs or just generally tell us what components you have successfully used the app with.

What is Apache Olio?

Apache Olio is a Web 2.0 toolkit, basically it's a Web 2.0 application and a load generator. You deploy the application to a configuration that you want to test, fire up the load generator, drive load to the application and then analyze the results. The application isn't that fussy about what it runs on, for the Java EE app you need a Java EE Web container (Glassfish or Tomcat for example), you also need a Database and a schema is provided that can be used to set that up. You need a filestore and you need a Web Server to act as a remote Web Service (for looking up geolocations). 

Apache Olio uses Faban to drive load along with a custom Faban driver. Faban is a benchmark driver framework and harness that is designed to allow you to model the usage of your application and drive load for 1000s of simulated users. It also can be used to manage the runtime environment and it gathers the results from test runs.

Once you've deployed the application you can load it up with dummy users and events (it's a Social Networking app) and use the driver to simulate load. At the end of a test run, you get all of the data from the run presented to you in graphical form (depending on the platform). I spend a vast amount of time using Olio and Faban and can't recommend them enough.

Kim is the lead developer of the Java EE version of Apache Olio and he has a blog entry that goes into lots of detail on how Apache Olio Java EE works and what it looks like.

Wednesday Mar 25, 2009

Ruby 1.8.7 and RubyGems 1.3.1 on OpenSolaris

Last week, all of the work that Chris Zhu, Prashant Srinivasan and I did on integrating Ruby 1.8.7 p72 and RubyGems 1.3.1 into OpenSolaris bore fruit as the packages became available in the /dev repository of OpenSolaris.

You can still use Ruby 1.8.6 as that's the version that was in the 2008.11 release of OpenSolaris, these packages won't get promoted to the main repository until the next release of OpenSolaris in May/June. 

In order to upgrade to Ruby 1.8.7 in OpenSolaris in the meantime you need to upgrade all of the packages so that they are at the same level. Once you have upgraded you'll no longer have Ruby 1.8.6 available to you, so make sure that's really what you want to do. Prashant wrote a brief guide on how to update Ruby in OpenSolaris a while back and you can find that here.

We'd be happy to receive any feedback on your experiences in upgrading and your thoughts on what you really would like to see for Ruby and RubyGems in OpenSolaris. As Ruby/RubyGems is part of Sun WebStack you can ask questions and post comments on the  alias (you need to subscribe first at or you can post a comment on either of our blogs.

Tuesday Mar 24, 2009

Community One East

I had the good fortune to present, along with my colleague Prashant, at the Community One event in New York last week. We talked about all of the testing work that we have been doing with Ruby on Rails on OpenSolaris these last 2 months and gave some pointers on what stack components to use and how to scale both vertically and horizontally. Got to see some of the sessions, although I spent a lot of the time in the speaker room on the first day. It was interesting to hear Dave Douglas talking about Kutiman, well actually it was more weird than interesting :o) I've listened to Kutiman quite a bit over the last couple of years, but had missed the Youtube Mashup thing that he's done. If you are interested check out

In the same Keynote, Dave Douglas re-purposed Sun's 25 year old vision: "The network is the computer" for the Cloud and announced the Sun Open Cloud Platform. Dave believes (and I tend to agree) that there will be many clouds, some public, some private, some internal, some internal, geographic based clouds, market based clouds and so on. The Sun Open Cloud Platform will provide the infrastructure for powering these clouds, starting with the Sun Cloud. The Sun Cloud is currently manifested as a RESTful API which you can sign up for at (it comes out of an internal project called Kenai and from QLayer), the API is published under Creative Commons which is cool. Other bits will follow later this year, Sun Cloud Compute Service and Sun Cloud Storage Service, which all I know about is limited to the press release. Now all we need to do is get the word out about what we are doing and then listen to the people who use it, who want to use it and who used it and didn't like it, and make it the best Cloud possible.

I didn't get a chance to see Simon Phipps' talk on The 3rd wave of Open Source live as I was talking at the time. I have watched the recording since and it's pretty interesting. I like the mention of Open Source as the Firmware for the Systems group at Sun (where I work), for example the Sun Storage 7000 series has OpenSolaris and ZFS as it's 'firmware', that these software components are fundamental parts of these systems and that no additional software and software licenses are required to make them work. Indeed Open Source really is now at the core of Sun Microsystems. Something else he said half jokingly, he used the word "victims" to describe the users of closed source, proprietary software, not sure why that gels so well with me. "More and more procurement driven projects are ending in failure" and the alternative is an adoption led process, where there's a lose specification of requirements, prototyping and several iterations that lead into a pilot that will either succeed or 'fail early' and that's fine, better to fail early than to fail after spending lots of money on licenses, professional services and support contracts.

Anyway, rather than give a commentary on the talks I saw you can see video of the main track and pick up the slides for the others at:

I had the pleasure to meet Dave Pascoe and Mark Niedzielski from OurStage who attended our talk. I think I learned more from them than they did from me :o)   If you are at all interested in new music you should check them out. Anyway, they use Rails big time and they use OpenSolaris for storage. The kind of scaling that they do is something that given the limited opportunities for access to equipment, I could only ever dream about. It was thought provoking to say the least. Mark showed me a software monitoring tool called Rails Performance Monitor from a company called New Relic. It's the kind of app that anyone involved in managing and monitoring performance long for and tells you pretty much everything about how your Rails app is performing and where it's spending it's time. I've been using the Lite version for the last couple of days.

The second day of Community One was reserved for Deep Dives and I decided to attend the MySQL sessions (dev in the morning, dba in the afternoon) and see if I could pick up some useful tips. We spent a little too long looking at the online docs, but apart from that they were good sessions. If nothing else, finding out that I could log queries that didn't use an index made the whole thing worth while.

NYC was it's usual self:  expensive, noisy, bustling and full of character. I had a 1.5 hour tour of Manhattan on my way from the airport on the Shuttle bus and took the E line subway in the rush hour on the return trip.

Oh and some people were talking about IBM, something about an outrageous plot to take over the world...

Monday Mar 23, 2009

Building eventmachine on OpenSolaris and Solaris Nevada

If you are installing the eventmachine Ruby Gem on OpenSolaris (which you do whenever you install Thin). Make sure that you use GNU 'make' and not Solaris 'make' otherwise the installation will fail. You can do this by making sure that you have the SUNWgmake package installed and that /usr/gnu/bin is in your PATH and is before /usr/bin and/or /usr/ccs/bin

GNU 'make' implicitly sets the CXX variable used in the eventmachine Makefile to 'g++' and as g++ is in /usr/bin (assuming you have the SUNWgcc package installed)  you don't have to set it yourself. eventmachine will build with the Solaris version of 'make' but you would need to set CXX manually before installing the gem, i.e.:

CXX=/usr/sfw/bin/g++ gem install eventmachine

On Solaris Nevada Distributions (SXCE) GNU make is only available as 'gmake' and so by default eventmachine will use /usr/bin/make which is the Solaris make. In that case you need to set CXX as described above.

Remember: four is the number of Make commands available on OpenSolaris and the number of Make commands available on OpenSolaris is four, no more no less.

Monday Feb 16, 2009

Thin on OpenSolaris

Since the start of the year we've been testing with the Apache Olio Rails application on OpenSolaris. Early on we found that when using Thin as our Rails runtime, every now and then a Thin instance would abort and core dump after an assertion failure in EventMachine. This really only happened under high loads and was hard to recreate. The problem is detailed here:

We talked to the EventMachine folks on the eventmachine-talk alias and after a couple of days they were able to come up with a fix that we then tested at length. The fix is now available in EventMachine 0.12.4 . The support that they gave us was really fantastic and helped us resolve a problem that really threatened to hold up our project. Thanks particularly to Aman Gupta.

So if you are using or planning on using Thin and EventMachine on OpenSolaris, EventMachine 0.12.4 is the way to go.

Tuesday Feb 10, 2009

OpenSolaris at Fosdem '09

I didn't spend as much time as I'd have liked on the OpenSolaris booth at Fosdem '09 and so I missed this celebrity interview.

Monday Feb 09, 2009

Weekend away with 5000 other F/OSS fans

Just spent a long weekend in Brussels attending Fosdem '09. It was my first time there and I really loved the down to earth nature of the whole thing. It came at just the right time as I really needed a break from the rubbish weather we've been having in the UK, although early on Saturday morning on the ULB campus in Brussels it seemed like the weather had decided to follow me over the channel.

I'm not a big fan of keynotes and so was pleased to find that Fosdem kept them at the minimum, somehow our very own Simon Phipps managed to get 10 minutes to talk about fixing a 29 year old bug in Sun's RPC code, seemed that in order to get the issue fixed the code had to be re-licensed and Simon has just managed to make that happen in the last week. It was a strange segue in some ways and led into Mark Surman of the Mozilla Foundation talking about what Free means today and how it will influence the future. It was an interesting talk but one I can't remember much about so I'm looking forward to the slides/video becoming available. There was also the Fosdem Dance which is funny and disturbing at the same time.

I spent most of the rest of the day wandering around the booths and getting to see what everyone was doing. I did catch Mark Rheinhold and Joe Darcy's updates on OpenJDK7, OpenJDK6 and the formative feature set of Java SE 7. Last time I'd seen a Java SE 7 presentation we were still talking about JAMs and  JSR 277, seems that the landscape has changed and now it's JSR 294 that's being adopted. I never really got excited by the idea of SuperPackages and JAMs so hopefully this is a good move, although listening to the Project Felix Lightning Talk later in the day is seems that this may be a set back for OSGI in Java,  but I'm sure there's loads of other stuff going on in that area. Lenz Grimmer also talked about using Bazaar for source code management and it seems worth checking out (i'll add it to the list of SCM tools that I need to look at). While at the lightning talks I finally got to meet Nick Kew, Nick is a leading contributor to the Apache HTTP server and now works for the Sun WebStack team, he's based in the UK but a long way from where I am in Surrey so it was great to meet him at last.

On Day 2 I spent all of my time in the Ruby, MySQL and Java Dev rooms. Of most interest was getting to listen to Alex Buckley talk about invokedynamic and other features that support dynamic languages that are being integrated into the (J)VM. The J is paranthesized as a nod to the fact that in future the Java VM will support many different languages. One of the main reasons for Java's success over the last decade is the Hotspot Compiler and all of the work done to make the VM performant. As Virtual Machines go it seems to have few peers and taking all that and making it available to other languages can only be a good thing. I say this as someone who's spent a lot of time working with various flavors of Ruby VMs in the last couple of years. It'll be interesting to see where all of this goes.

I really liked Kris Buytaert's talk on monitoring MySQL. My group in Sun have been talking a lot recently about integrating Hyperic HQ and Nagios into OpenSolaris and this talk just showed how much this is needed. It seems we should also be looking at Zabbix which Kris had a lot of good things to say about. I also really enjoyed The Data Charmer's talk on MySQL partitions, if you aren't familiar with him, The Data Charmer is Guiseppe Maxia - The community team lead for MySQL here at Sun. He explained logical vs physical partitions using maps of Brussels and a pair of scissors, something that earned a deserved round of applause (ok he didn't use scissors, it sounds better when describing it though). Peter Vanbroekhaven gave an interesting talk on some of the more interesting idioms of Ruby classes, objects and modules but I missed most of the other Ruby sessions due to timing.

The Fosdem organisers laid on a bus to take people to Brussels central station, I decided to try to get on the 16:20 bus and queued at what I thought was the right place. It wasn't and I wasn't alone in getting this wrong as when the Bus arrived and parked round the corner, everyone waiting where I was ran down the road like an angry football crowd after a game (we weren't angry though). I made it on to the bus but only just, I'm a nervous traveller at the best of times and I think I may have been pretty miserable if I hadn't made it on, particularly having been one of the first to queue up for it.

Whatever happens over the next year, I'll be planning on attending Fosdem in 2010. It's a must attend event for anyone involved in Free and Open Software in Europe. I think next time though I'll travel light and avoid airplanes and taxi's as they seem so out of keeping with the whole ethos of the event.

Tuesday Jan 20, 2009

Lighttpd and Olio Rails

We were trying to use Lighttpd to run the Apache Olio Rails application on OpenSolaris recently and we found that because the Lighttpd workers run as a non-root user (in this case as webservd), the image_science gem was unable to access the shared library built for it by RubyInline. The error that we saw was:

ActionView::TemplateError (Permission denied - /root/.ruby_inline) on line #10 of events/_filtered_events.html.erb (although the exact error varies depending on whether you are looking at the error page returned to the browser or the logfile). We knew from some of the problems that we had with getting image_science up and running on OpenSolaris that RubyInline defaulted to building libraries in the root users home directory, but up until then we had been using Mongrel and Thin and running them as root (which is food for thought). 

The fix is simple, RubyInline defaults to building libraries in $HOME/.ruby_inline unless the environment variable $INLINEDIR is set in which case it builds them in $INLINEDIR/.ruby_inline. You can pass environment variables on to the FastCGI processes that Lighttpd spawns by setting them in the fastcgi.server directive in the Lighttpd config file. An example of this is the one from our rig:

fastcgi.server =  ( ".fcgi" =>
                    ( "localhost" =>
                      ( "min-procs" => 1,
                        "max-procs" => 5,
                        "socket" => "/tmp/ruby-olioapp.fastcgi",
                        "bin-path" => "/export/faban/olio_rails/olioapp/public/dispatch.fcgi",
                        "bin-environment" => (
                           "RAILS_ENV" => "production",
                           "INLINEDIR" => "/export/faban/olio_rails/olioapp/tmp"

I've included the whole thing as it's sometimes tough to see the nesting of the options. Basically, if you don't have a 'bin-environment' section add one after 'bin-path' (watch for the commas).

With this config file, RubyInline will build (rebuild in this case) the libraries of the gems that make use of it in /export/faban/olio_rails/olioapp/tmp/.ruby_inline so as long as the user that Lighttpd is running it's worker processes as has access to that directory you should be good to go.

BTW: In case you are wondering, image_science is a native Ruby Gem that can resize images and create thumbnails, but instead of being built on install, it's built and managed by the RubyInline gem when you first go to use it.

Tuesday Oct 14, 2008

Lighttpd on CMT running Web 2.0 workloads

Earlier in the year I spent many a night burning candles at both ends testing with a Web 2.0 workload on a Lighttpd/PHP/Memcached/MySQL stack. The results have been made into a Sun Blueprint entitled An Open Source Web Solution - Lighttpd Web Server and Chip Multithreading Technology The bottom line is that we were able to get 416 ops/sec with 2250 users before we ran out of network bandwidth. That was with a 5 second cycle time between operations, so 450 ops/sec would have been the best we could theoretically get. This was all done on a Sun SPARC Enterprise T5120 running with 8 cores (64 threads) at 1.2GHz. In reflection I wish I'd spent more time on Memcached and MySQL analysis, but we wasted a lot of cycles trying to team a couple of network cards using a switch that was not up to the job and we just ran out of time.

The keys to all of the testing were the Faban test harness which not only ran all of the tests for us but collected and presented most of the data used in the Blueprint Document, and the Web 2.0 kit which is now in incubation on as Project Olio (

Friday Oct 10, 2008

Alfresco on Glassfish (short version)

I posted a complete walkthrough on how to get Alfresco up and running on Glassfish earlier . Here's a version for those who don't need all the detail.

Glassfish has an undocumented property that when used in conjunction with the Classloader Delegate feature will allow MyFaces libraries to be loaded before the JSF libraries. You set both of these in the server specific deployment descriptor which is named sun-web.xml and which lives in the WEB-INF directory of the War file. The complete file looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE sun-web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Application Server 8.1 Servlet 2.4//EN" "">
 <class-loader delegate="false"/>
 <property name="useMyFaces" value="true"/>

So create the file, place it in a directory named WEB-INF and then add it to the alfresco.war file:

$ jar uvf alfresco.war WEB-INF/sun-web.xml
adding: WEB-INF/sun-web.xml(in = 316) (out= 236)(deflated 25%)

Then deploy the alfresco.war file. If you've done everything else that's needed, you should be able to fire up a browser and connect to Alfresco at http://localhost:8080/afresco

Alfresco on Glassfish

I recently posted an updated version of this walkthrough that covers Alfresco 3.2r2.

This walkthrough is for OpenSolaris, it was made on the b98 development build but I'd expect it to work on the release version of 2008.05. It probably works on all other platforms as well with some changes to the pathnames and some of the commands. I'll work on something similar for Solaris, Ubuntu and for Windows.

I've posted a shorter version for those who have previously tried to get it working but were just missing the vital ingredient.

Initial Setup

I'm going to assume you are doing this as a non-root user. If you do this as root, just drop the pfexec statement from any commands you issue.

First install Glassfish and MySQL:

$ pfexec pkg refresh 
$ pfexec pkg install glassfishv2 
$ pfexec pkg install SUNWmysql5

Unlike with the best tutorials, I won't first remind you to turn your system on or not to install the packages if they are already installed.

OpenSolaris has this nifty feature called SMF but we haven't yet managed to hook the installation of packages up such that Services are automatically registered with it yet although that should be there in the November release. For now run the following to register it:

$ pfexec svccfg import /var/svc/manifest/application/database/mysql.xml

and then start the MySQL service:

$ pfexec svcadm enable mysql

This will do all of the work required to get MySQL up and running for the first time. Now create a DB and a user to access it with.

$ /usr/mysql/50/bin/mysqladmin -u root create alfresco
$ /usr/mysql/50/bin/mysql -u root
mysql> grant all privileges on alfresco.\* to 'alfresco'@'%' identified by 'alfresco';
Query OK, 0 rows modified (0.00sec) 
mysql> grant all privileges on alfresco.\* to 'alfresco'@'localhost' identified by 'alfresco';
Query OK, 0 rows modified (0.00sec) 

Now setup a Glassfish domain:

$ pfexec /usr/sbin/asadmin create-domain --adminport 4848 domain1

You need a JDBC MySQL driver:

$ wget

The URL used for the MySQL JDBC driver is just one of many mirrors. You might be better off going to and choosing where to get it from.

Unpack this somewhere and copy mysql-connector-java-5.1.6-bin.jar to the Glassfish domain's lib directory:

$ pfexec cp mysql-connector-java-5.1.6-bin.jar /var/appserver/domains/domain1/lib

While you are doing this you may as well create an alfresco directory in the same place, you'll need it later.

$ pfexec mkdir /var/appserver/domains/domain1/lib/classes/alfresco

Alfresco War file and configuration

This testing was done with Alfresco labs 3b, it should work with other versions, but if it doesn't post a comment on this blog and I'll look into it.

Next we'll need the Alfresco War file, which you can probably get like this:

$ wget

But as that looks like a URI than might have been created by SkyNet a machine, you may need to fallback to going here:

Once you have it, extract the bundle somewhere, you won't need it after the setup is finished.

You now need to setup the Alfresco bits that aren't contained in the WAR file. From the directory in which you unpacked the Alfresco bundle, copy the entire contents of the extensions directory to the directory you created earlier:
$ pfexec cp -r extensions/\* /var/appserver/domains/domain1/lib/classes/alfresco

You don't want to copy over the extensions directory itself, just it's contents (extension and messages directories).

Alfresco needs somewhere to keep all of it's documents and indexes. This is usually named alf_data, if you don't specify a path to alf_data or the path doesn't exist it will create one for you... somewhere. Let's not leave things to chance and create one and then configure Alfresco to find it:

$ pfexec mkdir /var/alfresco/alf_data

Now cd to /var/appserver/domains/domain1/lib/classes/alfresco/extension and modify the config files so that Alfresco can find alf_data and can find the database. The files that need changing are:

In do the following:

Uncomment the dir.root line and change it's path to /var/alfresco/alf_data




Uncomment the db.username and db.password lines




The last part of the file has sections for specific databases, by default it's setup to use Derby. Comment out the lines for derby and uncomment the lines for MySQL




In, comment out the line for Derby and uncomment the line for MySQL

The Glassfish specific deployment descriptor

Now for the fun bit. Go to the directory in which you unpacked the Alfresco bundle, create a directory called WEB-INF and in that create a file called sun-web.xml with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE sun-web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Application Server 8.1 Servlet 2.4//EN" "">
 <class-loader delegate="false"/>
 <property name="useMyFaces" value="true"/>

and then add that file to the alfresco.war file:

$ jar uvf alfresco.war WEB-INF/sun-web.xml
adding: WEB-INF/sun-web.xml(in = 316) (out= 236)(deflated 25%)

You're now ready to start Glassfish and deploy alfresco.war to the Glassfish domain:

$ pfexec /usr/bin/asadmin start-domain domain1
$ /usr/sbin/asadmin deploy alfresco.war
Please enter the admin user name>admin
Please enter the admin password> 
Command deploy executed successfully

Now you should be able to fire up a browser and connect to Alfresco at http://localhost:8080/afresco


This really just covers the basics and uses a clean OpenSolaris install just to show what actually needs to be done to get it working. Hopefully it can be extrapolated from to help install Alfresco on Glassfish in more complicated scenarios. I'll post this on the Alfresco forums once I'm happy that nothing silly creeped in. Hopefully we can extend this to cover different setups and configurations.

At some point it would be nice to show how to get the PDF to Flash functionality working as currently that's missing,


Check that it's using the MySQL database that you created by connecting to MySQL through the mysql command and looking to see if the Alfresco database has a bunch of tables. Also check that /var/alfresco/alf_data contains directories if it doesn't then it probably hasn't picked up the configuration in /var/appserver/domains/domain1/lib/classes/alfresco/extensions. Check that you've used the correct path and that the changes that you made to and are correct.

Remember to stop and start MySQL (if you need to) using svcadm.

The sun-web.xml file has along line in it, if you cut and paste make sure that it doesn't get split across multiple lines.

If you see any issues or mistakes then post a comment on this blog.

Friday Oct 03, 2008

Lighttpd in OpenSolaris

Lighttpd went into the SFW consolidation 6 weeks ago and made it into build 97 of Solaris Nevada (which sees the light of day as Solaris Express Community Edition). It appeared in the OpenSolaris package repository in the first week of September and you can install Lighttpd using the following command:

pfexec pkg install SUNWlighttpd14

if it doesn't find it try running: pfexec pkg refresh and then try it again.

BTW: You don't need to use the pfexec part of the commands if you are logged in as root

I've not blogged about this before because I tried to be clever with dependencies and built the Lighttpd MySQL vhost module against the MySQL 5.0 version that's intergrated into SFW but I deliberately didn't call out the dependency in the packages. This should have meant that when you installed Lighttpd, it wouldn't pull down MySQL, but once you had installed the MySQL packages (if you chose to) then MySQL vhost support would just work. The missing library dependency was spotted by the Gatekeeper and he added the MySQL dependency to the packages by hand. So now when you install Lighttpd, you get MySQL too. Sorry about that.

We have a CR open for this and in the next couple of weeks we'll integrate a fix that packages MySQL support separately. If in the meantime I come up with a way of bypassing the dependency check I'll blog it.

In the meantime, give it a try and if you find anything that doesn't work the way it should or if there's some change you'd like to suggest, reply to this blog entry or join the webstack-discuss alias at and send us an email. 

BTW: We don't have lua support at the moment so the likes of mod_magnet won't work, we are hoping to fix this in the near future


Bloggity, blog


« April 2014