Tuesday Dec 22, 2009

Ruby and Lighttpd updates in OpenSolaris

As of build 129 of OpenSolaris Ruby, Lighttpd and RubyGems have been updated to the following versions:

  • Ruby 1.8.7 patch level 174
  • RubyGems 1.3.5 
  • Lighttpd 1.4.23

Lighttpd is a couple of revs behind, when we started the update 1.4.24 had just been released and had a couple of issues which we would have needed to patch. At that time darix and stbeuhler weren't sure if they would release a 1.4.25 to fix theses issues so we took an executive decision to go with 1.4.23 and update to a later version via the OpenSolaris /dev repository after the release of OpenSolaris 2010.03. 1.4.25 made it out before we integrated but given the processes we have to ensure component and build quality it's not a good idea to change versions at the 11th hour. We'll probably push Lighttpd 1.4.25 out via the /webstack repo.

These same version updates are going into WebStack 1.6 which will be available around the same time as OpenSolaris 2010.03

Thursday Oct 29, 2009

Ruby MySQL nearly in OpenSolaris

One of the big pain points when installing Native Ruby Gems is the need to have various build packages installed. Packages that deliver the likes of gcc, gmake and ginstall. You also need to know where the libraries and C header files that you want to build against are located. On OpenSolaris the last part should be a no-brainer at least with packages installed from the repository, but some packages such as MySQL don't install to /usr/lib and /usr/include and the mysql_config that the MySQL package ships is not on the default $PATH and even if it was, it emits Compiler and Linker information for Sun Studio, not for gcc.

What this means is that you have to install all of the tools above and then install the MySQL gem with options telling the build where to find the MySQL libs and the MySQL headers.

Making the MySQL gem available as an OpenSolaris package, means you don't have to worry about any of that. You just run:

pfexec pkg install ruby-mysql

and voila!, you have MySQL support in Ruby... But it doesn't work :o(

The ruby-mysql package was promoted to the /contrib repository this week. Unfortunately it was promoted before I had tested it fully (which shows we have some major holes in the processes used to get packages into /contrib). The version that's there currently is unusable as it causes a segmentation violation when running with Rails. If you want to use this package today then you can get it from the /pending repository. Details of how to make use of OpenSolaris repositories can be found on a separate blog entry here.

Wednesday Oct 14, 2009

Rake should be on the default PATH...

I can convince myself of anything. When we updated from RubyGems 0.9.4 to 1.3.5 I decided that it was ok for the 'rake' command to be in the $GEM_HOME/bin which then translated to /var/ruby/1.8/gem_home/bin. I justified it because users could run 'gem env', look at the "EXECUTABLE DIRECTORY" and see that they had to add /var/ruby/1.8/gem_home/bin to their $PATH env var. Having had to install and then run rake several times on different systems over the last few days I can see that this is wrong. I want to 'gem install rake' and then run 'rake <some rake task>' without the need to update my PATH. I'm certain that others will agree that this is how it should be.

I could easily change the way that we build RubyGems such that the "EXECUTABLE DIRECTORY" defaults to /usr/bin but I have no way of being certain that when the end user runs 'gem install <some gem with an executable>' that /usr/bin will actually be writable even by root (or the root role as used by pfexec). We have the Rails package in the /contrib repository that also delivers the Rake gem. This package installs the 'rake' executable to /usr/bin, but it's a bit much to ask you to install Rails just to get Rake. I seem to have the following options:

  1. With the SUNWruby18 package on OpenSolaris, provide wrapper scripts in /usr/bin for the popular gems that deliver executables. These would emit a message saying to install the require Gem should it not already be installed and run as normal if it is. This has a few corner cases that have to be considered
  2. Provide a /contrib package that provides wrapper scripts for popular gems which would work as described in (1)
  3. Change "EXECUTABLE DIRECTORY" to /usr/bin and modify RubyGems to detect the writability of /usr/bin. It does this by default but would end up installing any affected gems in root's ~/.gem directory.

I favour option 3, but only if it can be done without significant changes to RubyGems.

Friday Jun 19, 2009

Building Ruby 1.9.1 on Solaris SPARC

We've been working on building Ruby 1.9.1 (p129) packages on Solaris Nevada (the platform that OpenSolaris is mostly built from). We hit a couple of problems on the way, one was easy to fix the other not so.

The first issue was Sun Studio borking when it found a function declared with a return type of void but which actually contained a return statement. gcc actually thinks this ok which seems odd to me, maybe it's just our version of gcc though... When gcc hits this it says:

pty.c:425: warning: `return' with a value, in a function returning void

But with Sun Studio cc you get:

pty.c:425: void function cannot return a value
cc: acomp failed for pty.c

The file causing the error is ext/pty/pty.c and line 425 has the offending line in function getDevice() (declared as 'static void'):

return Qnil;

If you comment out this line it will build ok. There is a bug for this issue......

The other problem only occured on SPARC initially, we were using one of the public build systems that we have for the SFW consolidation (the part that delivers most of the F/OSS into Solaris Nevada and OpenSolaris) and that was running Solaris Nevada build 114, for x64 we were building on Solaris Nevada build 116 which was the latest version available at time of writing. x64 built fine.

The error seen on SPARC was:

Undefined                       first referenced
symbol                             in file
rb_cFalseClass                      enc/emacs_mule.o  (symbol scope specifies local binding)
rb_cTrueClass                       enc/emacs_mule.o  (symbol scope specifies local binding)
rb_cFixnum                          enc/emacs_mule.o  (symbol scope specifies local binding)
rb_cSymbol                          enc/emacs_mule.o  (symbol scope specifies local binding)
rb_cNilClass                        enc/emacs_mule.o  (symbol scope specifies local binding)
ld: fatal: symbol referencing errors. No output written to .ext/sparc-solaris2.11/enc/emacs_mule.so  

when linking enc/emacs_mule.so. It's not a well documented error, but the implication was obvious, the required symbols had been found but they had been declared as local and so couldn't be used to build this shared object. Using nm on emac_mule.o and on libruby.so.1 seemed to indicate that the symbols were needed by emacs_mule.o (UNDEF) and were available in libruby.so.1. We asked the compiler experts and they thought that perhaps the symbols were declared as HIDDEN. We tried elfdump on both emacs_mule.o and libruby.so.1 and guess what, when analyzing libruby.so.1, elfdump threw up loads of errors of the type:

"bad symbol entry: <address> lies outside of containing section"

This suggested that the shared library was broken in some way.

We isolated the linker lines from the build for libruby.so.1 and ran them individually (after touching ruby.c). There were two lines, the first was the linker line which actually built the shared library. That ran ok and when we ran elfdump on the resultant library there were no errors. The second line was:

/usr/sfw/bin/gobjcopy -w -I "Init_\*" libruby.so.1

After running this manually we saw the same error when using elfdump on the resultant library.

At the same time as this we were running a build on a SPARC system that we'd had upgraded to Nevada build 116 and that completed OK. A check on the version of gobjcopy on the two systems showed that we had gobjcopy 2.15 on the build 114 system and 2.19 on the build 116 system. Further checking showed that gobjcopy was delivered into Solaris Nevada in SUNWbinutils and that had been updated in Nevada build 116. So the problem wasn't the fact we were building on SPARC but that we were building on different OS revs, the problem also exists on x64.

At the moment we haven't looked into what was going wrong when gobjcopy tried to make the Init_\* symbols local, but it was apparently corrupting the library.

At the moment this makes it tough to build Ruby 1.9 on OpenSolaris which is based on Nevada build 111b and we are looking at how best to get around this. Maybe make the packages available from the /webstack repository. In the meantime we'll file a bug against OpenSolaris and come up with a workaround.

Wednesday Mar 25, 2009

Ruby 1.8.7 and RubyGems 1.3.1 on OpenSolaris

Last week, all of the work that Chris Zhu, Prashant Srinivasan and I did on integrating Ruby 1.8.7 p72 and RubyGems 1.3.1 into OpenSolaris bore fruit as the packages became available in the /dev repository of OpenSolaris.

You can still use Ruby 1.8.6 as that's the version that was in the 2008.11 release of OpenSolaris, these packages won't get promoted to the main repository until the next release of OpenSolaris in May/June. 

In order to upgrade to Ruby 1.8.7 in OpenSolaris in the meantime you need to upgrade all of the packages so that they are at the same level. Once you have upgraded you'll no longer have Ruby 1.8.6 available to you, so make sure that's really what you want to do. Prashant wrote a brief guide on how to update Ruby in OpenSolaris a while back and you can find that here.

We'd be happy to receive any feedback on your experiences in upgrading and your thoughts on what you really would like to see for Ruby and RubyGems in OpenSolaris. As Ruby/RubyGems is part of Sun WebStack you can ask questions and post comments on the webstack-discuss@opensolaris.org  alias (you need to subscribe first at http://opensolaris.org) or you can post a comment on either of our blogs.

Friday Oct 03, 2008

Segmentation Fault when running Rails with MySQL on Solaris Nevada

We found in recent testing while integrating Ruby 1.8.6 Patch Level 287 into Solaris Nevada that it's possible to create a situation where Ruby on Rails applications will sometimes crash with a Segmentation Fault. We found that the MySQL native gem when installed, had been built with the mysql.h C header file from MySQL 5.0 but had been linked with an older version of the MySQL client library. This comes about if you install the gem as follows:

gem install mysql -- --with-mysql-dir=/usr/mysql/5.0

In this case the compiler is able to find the C headers in /usr/mysql/5.0/include but because the client library is actually located in /usr/mysql/5.0/lib/mysql/lib, the linker isn't able to find it. As we include /usr/sfw/lib in the build and runtime linker paths for native Ruby Gems, it is able to find the MySQL 4.0 client library and at runtime it links with that. Not surprisingly the MySQL 5.0 C header won't work with the MySQL 4.0 client library, what is surprising is that it works at all and doesn't crash as soon as you try to access MySQL from Ruby.

The solution is to install the gem without specifying any flags, in which case it will compile and link with the MySQL4.0  client, or preferably to specify the path to the include and lib directories of MySQL 5.0. So use either of the following:

gem install mysql -- --with-mysql-lib=/usr/mysql/5.0/lib/mysql -- with-mysql-include=/usr/mysql/5.0/include

gem install mysql

This is not a problem on OpenSolaris as the MySQL 4.0 client library is not there by default, just remember that just using --with-mysql-dir= may not always be enough information for the compiler and linker to do the right thing.

Sunday Aug 31, 2008

RubyGems /etc/gemrc

RubyGems now allows you to set the path to where your (and the systems) gems are installed using a System Wide Config File: /etc/gemrc. This is really useful to those packaging RubyGems for different operating systems as it avoids nasty hacks or coercing the GEM_PATH to a specific value every time RubyGems are run. The format of the file is like this:

gem: --no-rdoc --no-ri
gemhome: /var/ruby/1.8/gem_home
 - /usr/ruby/1.8/lib/ruby/gems/1.8

Here it's specified that by default rdoc and ri should not be built when installing gems and the gemhome and gempath configuration variables are set. With the /etc/gemrc set as above `gem environment` gives this output:

RubyGems Environment:
  - RUBY VERSION: 1.8.6 (2007-09-23 patchlevel 110) [i386-solaris2.11]
  - INSTALLATION DIRECTORY: /var/ruby/1.8/gem_home
  - RUBY EXECUTABLE: /usr/ruby/1.8/bin/ruby
  - EXECUTABLE DIRECTORY: /var/ruby/1.8/gem_home/bin
    - ruby
    - x86-solaris-2.11
     - /var/ruby/1.8/gem_home
     - /usr/ruby/1.8/ruby/lib/gems/1.8
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
     - "gemhome" => "/var/ruby/1.8/gem_home"
     - "gempath" => ["/usr/ruby/1.8/ruby/lib/gems/1.8"]
     - http://gems.rubyforge.org/

The actual values from /etc/gemrc are printed out in the "GEM CONFIGURATION" section, but they have actually had an affect on the "INSTALLATION DIRECTORY" (where gems are installed to), "EXECUTABLE DIRECTORY" (where gem binaries such as rails are installed to) and "GEM PATHS" which is the paths that the gem command will use to locate gems. This comes into effect when you run commands like `gem environment`, `gem query`, 'gem list --local` and most importantly when checking for dependencies when installing new gems.

Now the downside, the values for gempath and gemhome really are only used when running the gem command. They do not come into play when actually using a ruby gem. An example: As part of as system setup, the Ruby on Rails 2.1.0 gem is installed to /usr/ruby/1.8/lib/ruby/gems/1.8 using the --install-dir switch with `gem install`. Then the Ruby on Rails 2.0.2 gem is installed to the default Gem Home (/var/ruby/1.8/gem_home). The `rails` binaries installed by each are just wrapper scripts that call out to the rails gem, but the rails command does allow you to specify the version of Ruby on Rails that you want to use with the following syntax:

rails _2.0.2_ my_test_app

With "_2.0.2_" being the version that you want to use. If you run this with the setup as described above, where rails 2.1.0 is installed in one directory and rails 2.0.2 is installed in another and where you have only set the Gem Path through /etc/gemrc the following happens:

grond% rails _2.0.2_ my_test_app
 /usr/ruby/1.8/lib/ruby/site_ruby/1.8/rubygems.rb:580:in `report_activate_error': RubyGem version error: rails(2.1.0 not = 2.0.2) (Gem::LoadError)
        from /usr/ruby/1.8/lib/ruby/site_ruby/1.8/rubygems.rb:134:in `activate'
        from /usr/ruby/1.8/lib/ruby/site_ruby/1.8/rubygems.rb:49:in `gem'
        from /var/ruby/1.8/gem_home/bin/rails:18

This shows that it could not find the Ruby on Rails 2.0. gem anywhere on the path it was using to locate gems. The path it used was the default path constructed from the various paths in the Ruby config file: rbconfig.rb and as Ruby is installed to /usr/ruby/1.8 this is /usr/ruby/1.8/lib/ruby/gems/1.8 on OpenSolaris.

The fix is to use the GEM_PATH environment variable, if this is set, then the paths that it specifies will be included for any usage of Ruby Gems, be it the using the `gem` command or actually running the gems themselves.

When running the gem command a Gem::ConfigFile instance is create and it is this that actually reads and parses the /etc/gemrc file. The gemhome and gempath values are extracted from the ConfigFile instance by Gem::GemRunner which is the base class for running 'gem' commands, these values are then pushed into the Gem.path class variable which is used throughout RubyGems. However, when running or loading a gem, the only setting of the Gem.path is done by checking to see if GEM_PATH is set, if it is then it's values are used, if it isn't then the default path is used.

I can only suppose that this was done deliberately but I can't at the moment see why that would be. It seems to make sense that GEM_PATH and gempath (in /etc/gemrc) would do the same thing.


Bloggity, blog


« July 2016