Thursday Oct 29, 2009

Ruby MySQL nearly in OpenSolaris

One of the big pain points when installing Native Ruby Gems is the need to have various build packages installed. Packages that deliver the likes of gcc, gmake and ginstall. You also need to know where the libraries and C header files that you want to build against are located. On OpenSolaris the last part should be a no-brainer at least with packages installed from the repository, but some packages such as MySQL don't install to /usr/lib and /usr/include and the mysql_config that the MySQL package ships is not on the default $PATH and even if it was, it emits Compiler and Linker information for Sun Studio, not for gcc.

What this means is that you have to install all of the tools above and then install the MySQL gem with options telling the build where to find the MySQL libs and the MySQL headers.

Making the MySQL gem available as an OpenSolaris package, means you don't have to worry about any of that. You just run:

pfexec pkg install ruby-mysql

and voila!, you have MySQL support in Ruby... But it doesn't work :o(

The ruby-mysql package was promoted to the /contrib repository this week. Unfortunately it was promoted before I had tested it fully (which shows we have some major holes in the processes used to get packages into /contrib). The version that's there currently is unusable as it causes a segmentation violation when running with Rails. If you want to use this package today then you can get it from the /pending repository. Details of how to make use of OpenSolaris repositories can be found on a separate blog entry here.

Friday Sep 25, 2009

Rails: Setting selected options in a ListBox


I really struggled with this yesterday. I'm writing a Rails CRUD app and when creating a new entry one of the fields presented is a ListBox that allows multiple selections. That was straightforward, but i had problems when editing an existing entry. What didn't want to work for me are the recognised methods for having the current items for the entry being edited be selected in the ListBox.

The way that I understand that it should work is like this:

collection_select(:user, :group_ids, Group.find(:all), :id, :name,{},{:multiple=>,:name=>'user[group_ids][]'})

But needless to say it didn't. In the end I found this from Wes Gamble (weyus on Ruby Forum). It's a bit more DIY but it works a treat. I'm reproducing it here in case it's useful to anyone else (thanks Wes!)

<select id="user_roles" name="roles[]" multiple="multiple">
<% Role.find(:all).each do |r| %>
     <option value="<%= %>" <%= @user.roles.include?(r) ? 
"selected=\\"selected\\"" : '' %>"><%= %></option>
<% end %>

Rails 2.3.3 in OpenSolaris /contrib repo

A while back an OpenSolaris repository called /contrib was introduced to basically allow anyone with an account on to submit Open Source applications/libraries to an OpenSolaris repo. The details of how to go about doing this are best saved for another blog entry (add to ToDo list). This new repository gave us a real opportunity to package up Ruby on Rails and make it available to OpenSolaris users in the package format that they are familiar with. We'd avoided the main /release and /dev repositories because Rails rev'd too frequently for us to be able to keep up with the changes given that the processes for getting into those repositories are fairly lengthy. Besides there was always the 'gem install' command.

So we've now added Ruby on Rails 2.3.3 and it's dependent Gems to the /contrib repository. Since then Rails 2.3.4 has been released, more on that later.

To install the rails package on a system with OpenSolaris 2009.06 or later you need to run the following commands:

% pfexec pkg set-publisher -O  contrib

% pfexec pkg refresh

% pfexec pkg install ruby-rails

% pfexec gem install rack

The first command gives you access to the /contrib repository and all of the packages that it contains and is worth doing anyway as there's lots of great packages there besides Ruby on Rails. It's a one off command, once you've added /contrib it will remain in your list of package publishers. The second command isn't strictly necessary the first time, but it's useful to remember as /contrib updates on a weekly basis and your local cache of package names and versions doesn't refresh automatically.

The next command installs the 'ruby-rails' package and if it's not already installed, installs the SUNWruby18 package which contains Ruby 1.8.7 and RubyGems 1.3.1 (versions correct at the time of writing). The last command installs rack, which is now required by Rails, something we didn't find out until the last moment we moved to Rails 2.3.3. We are looking to package rack and add it as a dependency to the ruby-rails package. The ruby-rails package installs to what is effectively the vendor gem location in /usr and the rails and rake executables install to /usr/bin (as symbolic links). You can still use the gem command to install Ruby on Rails versions later than 2.3.3 if you need to and the /usr/bin/rails command will pick up the later Rails version.

At the moment the benefits of using 'pkg' to install Rails over using 'gem' are mainly just the convenience of having dependencies pulled in automatically and having the 'rails' command on the PATH. In the future, we'll add native gems that are used to provide infrastructure to Ruby on Rails, gems that usually require a compiler to build, along with knowledge of the location of any dependent libraries. Ultimately we'll be able to provide a single package that installs a complete, optimised, Ruby on Rails infrastructure.

Thursday Jul 09, 2009

Rack and content_type for multipart requests

I wrote this entry a while back but never posted it. Seems a shame to waste it.

 While updating the Apache Olio app to do caching we found that on the new test rig, adding events and people (it's a Social networking app) would fail. We had moved to the new rig because our time on the lab systems had run out, but fortunately we diligently left an overlap and were able to go back and look at the Rails stack to see what had changed. The problem that we were seeing gave us a clue as all of the parameters in the  HTTP POST request that contained the person or the event details looked like this:

Parameters: {"address"=>{"city"=>#<File:/tmp/RackMultipart4952-35>, "zip"=>#<File:/tmp/RackMultipart4952-37>, "country"=>#<File:/tmp/RackMultipart4952-38>, "street1"=>#<File:/tmp/RackMultipart4952-33>, "street2"=>#<File:/tmp/RackMultipart4952-34>, "state"=>#<File:/tmp/RackMultipart4952-36>}, "commit"=>#<File:/tmp/RackMultipart4952-20>, "event_image"=>#<File:/tmp/RackMultipart4952-30>, "event_document"=>#<File:/tmp/RackMultipart4952-31>, "authenticity_token"=>#<File:/tmp/RackMultipart4952-39>, "event"=>{"title"=>#<File:/tmp/RackMultipart4952-21>, "event_timestamp(1i)"=>#<File:/tmp/RackMultipart4952-25>, "event_timestamp(2i)"=>#<File:/tmp/RackMultipart4952-26>, "event_timestamp(3i)"=>#<File:/tmp/RackMultipart4952-27>, "telephone"=>#<File:/tmp/RackMultipart4952-24>, "description"=>#<File:/tmp/RackMultipart4952-23>, "summary"=>#<File:/tmp/RackMultipart4952-22>, "event_timestamp(4i)"=>#<File:/tmp/RackMultipart4952-28>, "event_timestamp(5i)"=>#<File:/tmp/RackMultipart4952-29>}, "tag_list"=>#<File:/tmp/RackMultipart4952-32>}

instead of like this:

Parameters: {"address"=>{"city"=>"aaynaiuotrtgs", "zip"=>"81602", "country"=>"USA", "street1"=>"49857 Pk Ln", "street2"=>"", "state"=>"BC"}, "commit"=>"Create", "event_image"=>#<File:/tmp/RackMultipart4833-2>, "event_document"=>#<File:/tmp/RackMultipart4833-3>, "authenticity_token"=>"VdNDLR/dCCJe96Ua3zEC9ZOwPg2DxujQ5D6pxI9E0ws=", "event"=>{"title"=>"aa rygrtokldq t ", "event_timestamp(1i)"=>"2008", "event_timestamp(2i)"=>"10", "event_timestamp(3i)"=>"20", "telephone"=>"0014879641640", "description"=>"kw sjnieb vui fslzpn jokjw xjijsm jzeweyio dthti vckudre osoempc jurldvyi adusy twghtlzwluh cowiczskxg wql ctulke km yxtuost enixrl qv to ltszeriord lpxrlp cokjtrehwc mbrnchxh fdnxwie x nuuzpvvv pqlwqghg thwtgc svuzbnzdokgv iqwsrvokviuw l z gnr trkmc aspwbgckozcg so jq dcjxl vluosk dypk rkhg iseurrximrvk qnepyyzxu iugxbgmvcui mahnpibcoa wbhvplqym ogompcsikpz engr ugipr uvj w duk dqefcurj zoztkh ", "summary"=>"x c ztsg ncccoca e dspe azhzwvcz blfdtdllh zpbothd gctqotpln eunpoudzboef fcbzcstxh ", "event_timestamp(4i)"=>"20", "event_timestamp(5i)"=>"10"}, "tag_list"=>"tag1"}

So all of the parameters in the request were being treated as file uploads and it very much looked like Rack might be the cause. We had been using Thin as our Rails runtime and a check on the two systems showed that the old system had Thin 1.0.0 and Rack 0.9.1 and the new system had Thin 1.2.2 and Rack 1.0.0. Going back to the older versions fixed the issue.

Rack processes multipart form data with a couple of passes through the class method parse_multipart(env) (rack/utils.rb) where env is a wrapper around the request (as StringIO). The first pass processes the StringIO and extracts the form data and it's parts. To determine if a part is a file upload it used to run the following check:

filename = head[/Content-Disposition:.\* filename="?([\^\\";]\*)"?/ni, 1]
if filename
  body ="RackMultipart")
  body.binmode  if body.respond_to?(:binmode)

Which basically locates lines like the following in the form data:

    Encapsulated multipart part:  (image/jpeg)
        Content-Disposition: form-data; name="event_image"; filename="event.jpg"\\r\\n

In Rack 1.0.0 the conditional changed to:

if content_type || filename
  body ="RackMultipart")
  body.binmode  if body.respond_to?(:binmode)

The check for content_type had always been there and I won't list the code, but needless to say the conditional was now: if either content_type or filename (or both) are set then treat this part as a file upload.

In the Apache Olio Rails Driver (the code that drives load to the App) we have to assemble POST requests by hand and the code is all based on Apache HttpClient 2 (we use 3 now but the same code using deprecated methods in 3). What we had been doing to add text params to the POST request up to this point was:

MultipartPostMethod post = new MultipartPostMethod(addEventResultURL)
post.addParameter("event[title]", <randomly generated String data>);

This had the unfortunate effect of adding a content_type to the form-data with the result that the request looked like this:

     Encapsulated multipart part:  (text/plain)
        Content-Disposition: form-data; name="event[title]"\\r\\n
        Content-Type: text/plain; charset=US-ASCII\\r\\n
        Content-Transfer-Encoding: 8bit\\r\\n\\r\\n
        Line-based text data: text/plain

We modified the code to use a StringPart and addPart() instead of addParameter():

StringPart tmpPart = new StringPart("event[title]", <randomly generated String data>);


and we also had to explicitly set content_type to NULL on the new part:


and the form data in the request now looks like this:

Encapsulated multipart part: 
        Content-Disposition: form-data; name="event[title]"\\r\\n
        Content-Transfer-Encoding: 8bit\\r\\n\\r\\n
        Data (19 bytes)

Monday Mar 23, 2009

Building eventmachine on OpenSolaris and Solaris Nevada

If you are installing the eventmachine Ruby Gem on OpenSolaris (which you do whenever you install Thin). Make sure that you use GNU 'make' and not Solaris 'make' otherwise the installation will fail. You can do this by making sure that you have the SUNWgmake package installed and that /usr/gnu/bin is in your PATH and is before /usr/bin and/or /usr/ccs/bin

GNU 'make' implicitly sets the CXX variable used in the eventmachine Makefile to 'g++' and as g++ is in /usr/bin (assuming you have the SUNWgcc package installed)  you don't have to set it yourself. eventmachine will build with the Solaris version of 'make' but you would need to set CXX manually before installing the gem, i.e.:

CXX=/usr/sfw/bin/g++ gem install eventmachine

On Solaris Nevada Distributions (SXCE) GNU make is only available as 'gmake' and so by default eventmachine will use /usr/bin/make which is the Solaris make. In that case you need to set CXX as described above.

Remember: four is the number of Make commands available on OpenSolaris and the number of Make commands available on OpenSolaris is four, no more no less.


Bloggity, blog


« February 2017