Thursday Jul 09, 2009

Rack and content_type for multipart requests

I wrote this entry a while back but never posted it. Seems a shame to waste it.

 While updating the Apache Olio app to do caching we found that on the new test rig, adding events and people (it's a Social networking app) would fail. We had moved to the new rig because our time on the lab systems had run out, but fortunately we diligently left an overlap and were able to go back and look at the Rails stack to see what had changed. The problem that we were seeing gave us a clue as all of the parameters in the  HTTP POST request that contained the person or the event details looked like this:

Parameters: {"address"=>{"city"=>#<File:/tmp/RackMultipart4952-35>, "zip"=>#<File:/tmp/RackMultipart4952-37>, "country"=>#<File:/tmp/RackMultipart4952-38>, "street1"=>#<File:/tmp/RackMultipart4952-33>, "street2"=>#<File:/tmp/RackMultipart4952-34>, "state"=>#<File:/tmp/RackMultipart4952-36>}, "commit"=>#<File:/tmp/RackMultipart4952-20>, "event_image"=>#<File:/tmp/RackMultipart4952-30>, "event_document"=>#<File:/tmp/RackMultipart4952-31>, "authenticity_token"=>#<File:/tmp/RackMultipart4952-39>, "event"=>{"title"=>#<File:/tmp/RackMultipart4952-21>, "event_timestamp(1i)"=>#<File:/tmp/RackMultipart4952-25>, "event_timestamp(2i)"=>#<File:/tmp/RackMultipart4952-26>, "event_timestamp(3i)"=>#<File:/tmp/RackMultipart4952-27>, "telephone"=>#<File:/tmp/RackMultipart4952-24>, "description"=>#<File:/tmp/RackMultipart4952-23>, "summary"=>#<File:/tmp/RackMultipart4952-22>, "event_timestamp(4i)"=>#<File:/tmp/RackMultipart4952-28>, "event_timestamp(5i)"=>#<File:/tmp/RackMultipart4952-29>}, "tag_list"=>#<File:/tmp/RackMultipart4952-32>}

instead of like this:

Parameters: {"address"=>{"city"=>"aaynaiuotrtgs", "zip"=>"81602", "country"=>"USA", "street1"=>"49857 Pk Ln", "street2"=>"", "state"=>"BC"}, "commit"=>"Create", "event_image"=>#<File:/tmp/RackMultipart4833-2>, "event_document"=>#<File:/tmp/RackMultipart4833-3>, "authenticity_token"=>"VdNDLR/dCCJe96Ua3zEC9ZOwPg2DxujQ5D6pxI9E0ws=", "event"=>{"title"=>"aa rygrtokldq t ", "event_timestamp(1i)"=>"2008", "event_timestamp(2i)"=>"10", "event_timestamp(3i)"=>"20", "telephone"=>"0014879641640", "description"=>"kw sjnieb vui fslzpn jokjw xjijsm jzeweyio dthti vckudre osoempc jurldvyi adusy twghtlzwluh cowiczskxg wql ctulke km yxtuost enixrl qv to ltszeriord lpxrlp cokjtrehwc mbrnchxh fdnxwie x nuuzpvvv pqlwqghg thwtgc svuzbnzdokgv iqwsrvokviuw l z gnr trkmc aspwbgckozcg so jq dcjxl vluosk dypk rkhg iseurrximrvk qnepyyzxu iugxbgmvcui mahnpibcoa wbhvplqym ogompcsikpz engr ugipr uvj w duk dqefcurj zoztkh ", "summary"=>"x c ztsg ncccoca e dspe azhzwvcz blfdtdllh zpbothd gctqotpln eunpoudzboef fcbzcstxh ", "event_timestamp(4i)"=>"20", "event_timestamp(5i)"=>"10"}, "tag_list"=>"tag1"}

So all of the parameters in the request were being treated as file uploads and it very much looked like Rack might be the cause. We had been using Thin as our Rails runtime and a check on the two systems showed that the old system had Thin 1.0.0 and Rack 0.9.1 and the new system had Thin 1.2.2 and Rack 1.0.0. Going back to the older versions fixed the issue.

Rack processes multipart form data with a couple of passes through the class method parse_multipart(env) (rack/utils.rb) where env is a wrapper around the request (as StringIO). The first pass processes the StringIO and extracts the form data and it's parts. To determine if a part is a file upload it used to run the following check:

filename = head[/Content-Disposition:.\* filename="?([\^\\";]\*)"?/ni, 1]
if filename
  body ="RackMultipart")
  body.binmode  if body.respond_to?(:binmode)

Which basically locates lines like the following in the form data:

    Encapsulated multipart part:  (image/jpeg)
        Content-Disposition: form-data; name="event_image"; filename="event.jpg"\\r\\n

In Rack 1.0.0 the conditional changed to:

if content_type || filename
  body ="RackMultipart")
  body.binmode  if body.respond_to?(:binmode)

The check for content_type had always been there and I won't list the code, but needless to say the conditional was now: if either content_type or filename (or both) are set then treat this part as a file upload.

In the Apache Olio Rails Driver (the code that drives load to the App) we have to assemble POST requests by hand and the code is all based on Apache HttpClient 2 (we use 3 now but the same code using deprecated methods in 3). What we had been doing to add text params to the POST request up to this point was:

MultipartPostMethod post = new MultipartPostMethod(addEventResultURL)
post.addParameter("event[title]", <randomly generated String data>);

This had the unfortunate effect of adding a content_type to the form-data with the result that the request looked like this:

     Encapsulated multipart part:  (text/plain)
        Content-Disposition: form-data; name="event[title]"\\r\\n
        Content-Type: text/plain; charset=US-ASCII\\r\\n
        Content-Transfer-Encoding: 8bit\\r\\n\\r\\n
        Line-based text data: text/plain

We modified the code to use a StringPart and addPart() instead of addParameter():

StringPart tmpPart = new StringPart("event[title]", <randomly generated String data>);


and we also had to explicitly set content_type to NULL on the new part:


and the form data in the request now looks like this:

Encapsulated multipart part: 
        Content-Disposition: form-data; name="event[title]"\\r\\n
        Content-Transfer-Encoding: 8bit\\r\\n\\r\\n
        Data (19 bytes)

Monday Feb 16, 2009

Thin on OpenSolaris

Since the start of the year we've been testing with the Apache Olio Rails application on OpenSolaris. Early on we found that when using Thin as our Rails runtime, every now and then a Thin instance would abort and core dump after an assertion failure in EventMachine. This really only happened under high loads and was hard to recreate. The problem is detailed here:

We talked to the EventMachine folks on the eventmachine-talk alias and after a couple of days they were able to come up with a fix that we then tested at length. The fix is now available in EventMachine 0.12.4 . The support that they gave us was really fantastic and helped us resolve a problem that really threatened to hold up our project. Thanks particularly to Aman Gupta.

So if you are using or planning on using Thin and EventMachine on OpenSolaris, EventMachine 0.12.4 is the way to go.


Bloggity, blog


« February 2016