Thursday Jul 09, 2009

Rack and content_type for multipart requests

I wrote this entry a while back but never posted it. Seems a shame to waste it.

 While updating the Apache Olio app to do caching we found that on the new test rig, adding events and people (it's a Social networking app) would fail. We had moved to the new rig because our time on the lab systems had run out, but fortunately we diligently left an overlap and were able to go back and look at the Rails stack to see what had changed. The problem that we were seeing gave us a clue as all of the parameters in the  HTTP POST request that contained the person or the event details looked like this:

Parameters: {"address"=>{"city"=>#<File:/tmp/RackMultipart4952-35>, "zip"=>#<File:/tmp/RackMultipart4952-37>, "country"=>#<File:/tmp/RackMultipart4952-38>, "street1"=>#<File:/tmp/RackMultipart4952-33>, "street2"=>#<File:/tmp/RackMultipart4952-34>, "state"=>#<File:/tmp/RackMultipart4952-36>}, "commit"=>#<File:/tmp/RackMultipart4952-20>, "event_image"=>#<File:/tmp/RackMultipart4952-30>, "event_document"=>#<File:/tmp/RackMultipart4952-31>, "authenticity_token"=>#<File:/tmp/RackMultipart4952-39>, "event"=>{"title"=>#<File:/tmp/RackMultipart4952-21>, "event_timestamp(1i)"=>#<File:/tmp/RackMultipart4952-25>, "event_timestamp(2i)"=>#<File:/tmp/RackMultipart4952-26>, "event_timestamp(3i)"=>#<File:/tmp/RackMultipart4952-27>, "telephone"=>#<File:/tmp/RackMultipart4952-24>, "description"=>#<File:/tmp/RackMultipart4952-23>, "summary"=>#<File:/tmp/RackMultipart4952-22>, "event_timestamp(4i)"=>#<File:/tmp/RackMultipart4952-28>, "event_timestamp(5i)"=>#<File:/tmp/RackMultipart4952-29>}, "tag_list"=>#<File:/tmp/RackMultipart4952-32>}

instead of like this:

Parameters: {"address"=>{"city"=>"aaynaiuotrtgs", "zip"=>"81602", "country"=>"USA", "street1"=>"49857 Pk Ln", "street2"=>"", "state"=>"BC"}, "commit"=>"Create", "event_image"=>#<File:/tmp/RackMultipart4833-2>, "event_document"=>#<File:/tmp/RackMultipart4833-3>, "authenticity_token"=>"VdNDLR/dCCJe96Ua3zEC9ZOwPg2DxujQ5D6pxI9E0ws=", "event"=>{"title"=>"aa rygrtokldq t ", "event_timestamp(1i)"=>"2008", "event_timestamp(2i)"=>"10", "event_timestamp(3i)"=>"20", "telephone"=>"0014879641640", "description"=>"kw sjnieb vui fslzpn jokjw xjijsm jzeweyio dthti vckudre osoempc jurldvyi adusy twghtlzwluh cowiczskxg wql ctulke km yxtuost enixrl qv to ltszeriord lpxrlp cokjtrehwc mbrnchxh fdnxwie x nuuzpvvv pqlwqghg thwtgc svuzbnzdokgv iqwsrvokviuw l z gnr trkmc aspwbgckozcg so jq dcjxl vluosk dypk rkhg iseurrximrvk qnepyyzxu iugxbgmvcui mahnpibcoa wbhvplqym ogompcsikpz engr ugipr uvj w duk dqefcurj zoztkh ", "summary"=>"x c ztsg ncccoca e dspe azhzwvcz blfdtdllh zpbothd gctqotpln eunpoudzboef fcbzcstxh ", "event_timestamp(4i)"=>"20", "event_timestamp(5i)"=>"10"}, "tag_list"=>"tag1"}

So all of the parameters in the request were being treated as file uploads and it very much looked like Rack might be the cause. We had been using Thin as our Rails runtime and a check on the two systems showed that the old system had Thin 1.0.0 and Rack 0.9.1 and the new system had Thin 1.2.2 and Rack 1.0.0. Going back to the older versions fixed the issue.

Rack processes multipart form data with a couple of passes through the class method parse_multipart(env) (rack/utils.rb) where env is a wrapper around the request (as StringIO). The first pass processes the StringIO and extracts the form data and it's parts. To determine if a part is a file upload it used to run the following check:

filename = head[/Content-Disposition:.\* filename="?([\^\\";]\*)"?/ni, 1]
if filename
  body = Tempfile.new("RackMultipart")
  body.binmode  if body.respond_to?(:binmode)
end

Which basically locates lines like the following in the form data:

    Encapsulated multipart part:  (image/jpeg)
        Content-Disposition: form-data; name="event_image"; filename="event.jpg"\\r\\n

In Rack 1.0.0 the conditional changed to:

if content_type || filename
  body = Tempfile.new("RackMultipart")
  body.binmode  if body.respond_to?(:binmode)
end

The check for content_type had always been there and I won't list the code, but needless to say the conditional was now: if either content_type or filename (or both) are set then treat this part as a file upload.

In the Apache Olio Rails Driver (the code that drives load to the App) we have to assemble POST requests by hand and the code is all based on Apache HttpClient 2 (we use 3 now but the same code using deprecated methods in 3). What we had been doing to add text params to the POST request up to this point was:

MultipartPostMethod post = new MultipartPostMethod(addEventResultURL)
post.addParameter("event[title]", <randomly generated String data>);

This had the unfortunate effect of adding a content_type to the form-data with the result that the request looked like this:

     Encapsulated multipart part:  (text/plain)
        Content-Disposition: form-data; name="event[title]"\\r\\n
        Content-Type: text/plain; charset=US-ASCII\\r\\n
        Content-Transfer-Encoding: 8bit\\r\\n\\r\\n
        Line-based text data: text/plain

We modified the code to use a StringPart and addPart() instead of addParameter():

StringPart tmpPart = new StringPart("event[title]", <randomly generated String data>);

post.addPart(tmpPart);

and we also had to explicitly set content_type to NULL on the new part:

tmpPart.setContentType(null);

and the form data in the request now looks like this:

Encapsulated multipart part: 
        Content-Disposition: form-data; name="event[title]"\\r\\n
        Content-Transfer-Encoding: 8bit\\r\\n\\r\\n
        Data (19 bytes)
 
  

About

Bloggity, blog

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today