Tuesday Mar 18, 2008

Doing The Right Thing, or Why Rails is not in OpenSolaris

If you haven't been watching the discussion on webstack-discuss@opensolaris.org, the news is that Rails will not be integrated with OpenSolaris or Nevada.
ie., an installation of SXCE/SXDE/Nevada/Indiana will have Ruby installed, but Rails will not be available by default.

The recommended way to install Rails and other gems is by using the gem command - using the famous utterance "gem install rails --include-dependencies".
And that's it.

I think this is the right position for us to take. I think it's quite a brave one, and one that shows that we are truly working as a community now, at least in my corner of the universe.
Think about it . . . with other OSes drawing users away from Ruby's native packaging, in order to "do it their way" - what would the natural response from an operating system vendor be? To say "Let it be, a good solution already exists in Rubygems", or "hey, we want XYZ library delivered through our mechanism, irrespective of what the problems are"?

Rubygems is the chosen package manager for Ruby, and abiding by that decision helps Ruby users on Solaris - and people get that. We got a a lot of comments from our user community, as well as from the good folk at Twitter about Solaris packaging for Ruby gems in general - and nobody at Sun, after having heard the story, ever said, "you know, Debian has it, BSD has it, so we should do it too".

And that's a brave step to take.

Thanks to all the folk that provided comments, please keep them coming, and I'm proud of the folk at Sun, for working sincerely with the community.

Friday Feb 15, 2008

SXDE 1/08: First Solars Release that includes Ruby

SXDE 1/8(or the more wordy Solaris Express Developer Edition 1/8) was released two weeks ago, and this is the first Solaris release to have the Ruby programming language available as an operating system feature.

Expect to find it as /usr/ruby/1.8/bin, and linked from /usr/bin/ruby. We used the /usr/ruby/1.8 location to be friendly to different Ruby versions. ie., when version 1.9 comes out, expect to find it in /usr/ruby/1.9/bin.

Rubygems is also included in this release. So that should make it quite simple to be up-and-running with Ruby development . . . did I hear you ask about an IDE?
Well, the famous Netbeans, with built-in Ruby support, is available in Solaris express. Netbeans has been getting so many good reviews these days for it's Ruby support. This release allows you to choose between the native Ruby and jruby for your projects . . . cool.

What's on the roadmap?

Now that we have the Ruby platform, extensions, and Rubygems - the next step is to integrate the Rails stack.
This should also be available as IPS packages for OpenSolaris Indiana.

DTrace support should be available earlier than that, and Gems for MySQL and Postgres should be available along with Rails.

Expect to see these components go into the Solaris Express Community Edition(or Nevada) builds in the near future.

Oh, if you need Ruby/Rails for Solaris 10 - then use Coolstack. This doesn't come with the OS, but needs you to execute a simple pkgadd command to install it. It installs Ruby into /opt/coolstack.

Monday Nov 12, 2007

What's cool in Opensolaris build 78?

I can think of one thing ;-)

We putback Ruby into OpenSolaris build 78! Ruby 1.8.6 patchlevel 110, Rubygems 0.9.4, and Openssl, Curses, Readline, and Tcl/Tk extensions are enabled.

The directory layout for Ruby is per the ARC case. Ruby is in /usr/ruby/, it's linked from /usr/bin so as to be easy to locate - the man page is also appropriately linked.

Same for Rubygems, it's available though a link in /usr/bin - and the default gem repository happens to be /var/ruby/1.8/gem_home (ie., $GEM_HOME).

In the mean time, the wish list has is getting bigger n bigger . . .

\*\*\*\*\*\*\*\*\*  This mail is automatically generated  \*\*\*\*\*\*\*

Your putback for the following fix(es) is complete:

    6622810 Bring Ruby, Rubygems, and Extensions into OpenSolaris.

These fixes will be in release:


The gatekeeper will mark these bugids "fixed" and "integrated" in bugtraq
as soon as the gate has been delivered to the WOS.  However, you can mark
them "fixed" (but not "integrated") now if you wish.

        Your Friendly Gatekeepers


Friday Oct 26, 2007


The Ruby PSARC case got approved quite speedily. It could have been all the prior research into compatibility, file layouts, directory structure done(in response to a lot of good questions from Jyri).

Or it was probably because you hear that the ARCs are a source of delays, and other such scary rumors, so that when you really go through it, and realize that it's not true, you're confused.

MySQL was the other ARC case that seemed to sail through.
So, I'm going to dismiss perceptions of ARC tardiness as fiction(at least in my limited experiences).

My advise to anyone who wants to write an ARC case for OpenSolaris:

-> Read prior ARC cases to get a feel for how they're written(ie., the tone).

-> Think Interfaces, Interfaces, and Interfaces. ie., what interfaces does your software expose, what does it expect? What's the stability of each of these interfaces?

-> Look for precedence. ie., If you're proposing the addition of a new scripting language(like Ruby), look for older cases where scripting languages were approved(like Perl).

-> Think about compatibility/stability across versions. Across what releases is compatibility guaranteed? Across which releases is backward compatibility broken? What constitutes a stable release? What constitutes a development release?

Solaris is all about backward compatibility and stability.

-> And give this research a lot of time - otherwise, you're tempted to cut corners, and somehow wing your case. This impulse is detrimental to Solaris - which is the software you're trying to improve by making an ARC proposal in the first place.

Well, that's enough rambling about ARC cases, I've got to ease into my week end Martinis :-)


Saturday Oct 13, 2007

rubygems for opensolaris

Rubygems seems quite easy to put into OpenSolaris. It is a good candidate for /usr/ruby(with appropriate symbolic links from /usr/bin, of course). The gem repository needs to be outside the /usr hierarchy to allow the repository to be administered by users of sparse root zones. /var/ruby/1.8 looks like a good place for this.

The question of versioning model, compatibility, and how to distinguish a stable version from not, all had extemely pleasing answers for a ardent-compatibility-supporter like yours truly :-) thanks to Eric Hodel.(this is data that I was able to incorporate into my ARC case for Ruby). It also seems that Rubygems will have their own release cycles, independent of being included in the Ruby train.

When Rubygems installs a gem with an executable, for example Mongrel, it ties the runtime to the currently installed Ruby version. ie., It uses "#!/usr/ruby/1.8/bin/ruby". and thus, Mongrel will still invoke Ruby 1.8 even when the end user upgrades to ruby 1.9.(it would be cool if Rubygems used "#!/usr/bin/env ruby" instead of hardcoding . . .)
This makes it impossible to reuse the gem repository across different versions of Ruby installed on the same machine.
So a /var/ruby/x.y/gems directory will ensure that each Ruby version has it's own repository.

Should be easy to write a program to migrate gems between these, given the gem cache directory in Rubygems.

Rubygems tends to complain when when invoked after being configured with the non default gem repository directory. This seems like an RFE to me, since you tell the Rubygems installation program where your repository is, why doesn't it remember?

I know, I know, setting the GEM_HOME variable will solve that. But that takes time. Maybe just a few seconds more, but time that doesn't need to be wasted for this.

Thursday Oct 04, 2007

Whats a production Ruby version? what's a compatible version? And how do you know?

I've recently been working on getting Ruby checked into OpenSolaris. I was trying to find out how a stable Ruby release could be identified - to understand which versions to package, when updates can overwrite previous versions, and when they should co-exist etc.,

The PickAxe and Matz's OReilly book said that odd numbered minor versions of Ruby were considered development versions. ie., <ersion>.<OddNumber-Minor-version>.<Teeny-Version> releases are not production versions.

But since I didn't see any recent mention of this, I decided to ask the source, ruby-core. The response came from none other than Matz, and a few other ruby-corers. Odd numbered minor versions no longer indicate a development release.
What does, is the absence of a point release(or the "TEENY" version field), or a point release of "0". For example, 1.9(==1.9.0) is a dev. release, and development updates to this svn branch are not numbered. Once 1.9 is deemed production ready, it will be released as 1.9.l, which will be a production release.

So, version x.y.z is a development release iff z == 0, or z is undefined; The oddness/evenness of y not withstanding.
The corollary is that version x.y.z is a production release iff z > 0.

On compatibility, release x.y.z is compatible for any value of z > 0.
Corollary: x.y.z is not compatible for x.p.z if y != p. And it _is_ compatible only if y == p.

Saturday Sep 22, 2007

Kinda hard to argue with our good deeds?

Very cool.
From RailsConf Europe, by none other then Mr. Hansson - http://www.loudthinking.com/posts/11-sun-surprises-at-railsconf-europe-2007

Friday Sep 14, 2007

Have ideas for what goes with Ruby/Rails in OpenSolaris?

If you are not interested in Ruby on Solaris, don't read further.

Else . . . help decide! You opinions and rationale are crucial to make OpenSolaris better!

What's your favorite Ruby extension or gem?
Write to the "Open Source applications on OpenSolaris" community at http://www.opensolaris.org/jive/thread.jspa?threadID=39394&tstart=0

Wednesday Sep 05, 2007

Uncooperative irb on Solaris?

 Did you just download Ruby on Solaris and realize that irb is uncooperative with history, tab completion, and line editing?

 The problem is that it needs the GNU readline extension.  Unlike ssl and curses support, the GNU Readline library is not available on Solaris/OpenSolaris yet.
 To enable this support, install GNU Readline, and then build Ruby with the appropriate library/include paths.  I've included a list of steps below to save the interested user some time . . .

(1) Download and Install Sun Studio 12.  It's pretty easy, pick it up from http://developers.sun.com/sunstudio/downloads/index.jsp.  It installs into /opt/SUNWspro
(2) Download the latest stable release of Ruby 1.8.6 from ftp://ftp.ruby-lang.org/pub/ruby/stable-snapshot.tar.gz.
(3) Download GNU Readline from ftp://ftp.cwru.edu/pub/bash/readline-5.2.tar.gz
(4) prefix your path with /opt/SUNWspro/bin.  Ensure that /usr/sfw/bin is not in your path for the next step.
(5) run ./configure in, make, and make install

root test4-sp>export PATH="/opt/SUNWspro/bin:/opt/coolstack/bin:/opt/csw/bin:/usr/sbin:/usr/bin"
root test4-sp>./configure --prefix=/export/readline/binaries
[ . . . snip . . .]
root test4-sp>make
[. . .snip . . .]
root test4-sp>make install

Don't get put off by the last message "install: you may need to run ldconfig".  ldconfig doesn't exist on Solaris.  Use good old crle to add the readline libraries into your path, if you'd like.  But it's not necessary for adding readline support to Ruby/irb.

(6) Next, note the path you installed readline into, in my case, it is /export/readline/binaries.  We'll have to ensure that the Ruby build knows about this library.  So, set your $CFLAGS variable appropriately:

root test4-sp>export CFLAGS="-xO4 -g -L/export/readline/binaries/lib/lib -I/export/readline/binaries/include/readline"

(7) Now, add /usr/local/bin(and /usr/sfw/bin too) to your path for autoconf, and proceed to build Ruby with readline enabled.

export PATH=/opt/SUNWspro/bin:/usr/local/bin:/usr/sfw/bin:/opt/coolstack/bin:/opt/csw/bin:$PATH
export cc=/opt/SUNWspro/bin
./configure --without-gcc --prefix=/export/ruby/builds/ --enable-pthread --with-readline-dir=/export/readline/binaries

(8) then run make, and make install (also make install-doc for ri to work properly).

(9) To turn on tab completion, create a .irbrc file in your home directory which contains the following(courtesy Nick Sieger's blog):

root test4-sp>more ~/.irbrc
require 'irb/completion'
ARGV.concat [ "--readline", "--prompt-mode", "simple" ]

(10) Now you're set.  Emacs style command editing in irb, history, and TAB completion of classes should work like a charm!

Friday Aug 24, 2007

Control-C Control-V? Dont! - aka "Thinking in Ruby . . ."

Code reuse doesn't mean copy-paste, of course. But there are times when you do it because you have always done things that way – and thus don't use better tools when they are available. Habit is a dangerous thing.

I had to write a program to concurrently load data into a file store(MogileFS, if you were curious). The program had to load user data, and event calendar data.

The number of processes that my loader could spawn, to upload this data into this network file system, was the constraint.

The algorithm for the loader is straight forward:

  1. -> start with M users, N events. C is the number of processes that you can optimally create to load data(hardware constraints).

  2. -> iterate C times, select M/C users with each iteration, and fork/exec a process to load the M/C users into the store.

  3. -> wait for these processes to exit(since your concurrency should not exceed C)

  4. -> iterate C times, select N/C events with each iteration, and fork/exec a process to load the N/C events into the store.

That sounds trivial enough - I want to use Ruby, and here is how the program might look(I'm leaving out irrelevant details of loading into my data store for brevity):

bash-3.00# more loader.rb
concurrency = 4
num_users = 40
num_events = 80

puts "Adding Users. . ."
count = 0
concurrency.times do
  count_new = count + num_users/concurrency - 1
  exec "/usr/bin/echo Users #{count} #{count_new} > /dev/null" if fork.nil?
  puts "range is #{count} to #{count_new}"
  count = count_new + 1

(1..concurrency).each do |i|
  puts "Adding users - Process id: #{Process.wait} finished. #{i}/#{concurrency} complete."

puts "Adding Events. . ."
count = 0
concurrency.times do
  count_new = count + num_events/concurrency - 1
  exec "/usr/bin/echo Events #{count} #{count_new} > /dev/null" if fork.nil?
  puts "range is #{count} to #{count_new}"
  count = count_new + 1

(1..concurrency).each do |i|
  puts "Adding events - Process id: #{Process.wait} finished. #{i}/#{concurrency} complete."


That's pretty cool, and we're happy with the way we use the iterators that Ruby provides. This saves us some boiler plate code. And we like not having to use the $ prefix.

But wait, there is another Rubyism that could have worked it's way into this code - the code in the 'yellow' block and the code the 'blue' block look very similar, except for a “puts Adding Users . . .” statement that becomes “puts Adding Events. . .” in the second block, and the reference to num_users in the first block which becomes num_events in the second block.

Indeed, this is because of the rather instinctive ControlC-ControlV that went with the creation of the second block. But luckily, we spotted that instinct to stick to old ways. Ruby helps minimise copy-paste operations, it's code block feature comes to the rescue! The way to remove the copy-paste operation is to recognise that the copied block does exactly the same operation, but on events, rather than users. Ruby Hashes and code blocks come to the rescue, and here is what a more Ruby-friendly implementation looks like:

bash-3.00# more loader.rb
concurrency = 4
num_users = 40
num_events =

sets_to_load_into_mogile =
 "Users" => num_users,
 "Events" => num_events

sets_to_load_into_mogile.keys.each { |set|
 #puts "key is #{set} and value is #{sets_to_load_into_mogile[set]}"

puts "Adding #{set}. . ."
count = 0
concurrency.times do
  count_new = count + sets_to_load_into_mogile[set]/concurrency - 1
  exec "/usr/bin/echo  #{set} #{count} #{count_new} > /dev/null" if fork.nil?
  puts "range is #{count} to #{count_new}"
  count = count_new + 1

(1..concurrency).each do |i|
  puts "Adding #{set} - Process id: #{Process.wait} finished. #{i}/#{concurrency} complete."

The code in yellow is the hash that defines what kinds of sets need to be loaded(ie., Users, and Events). The values define the cardinality of such sets.

The sets_to_load_into_mogile.keys variable is an array of keys in the hash. We iterate over these, and run the code block as many times as needed with the right customizations.

Quite simple. Once we think in Ruby. And in this case that means looking out for the tendency to use Control-C/Control-V where Ruby offers such a strong alternative.

Friday Aug 03, 2007

Compiling Ruby 1.8.6 on Solaris Nevada (or OpenSolaris, or Solaris Express ) using Sun Studio 12

One doesn't need to compile their Ruby from source on Solaris Nevada / OpenSolaris / Solaris Express (SX). It can be downloaded from Blastwave, Sunfreeware, or Cooltools.
sunfreeware.com happens to have the latest stable version(1.8.6) available.

However, if you want an optimized build for your setup, or just like to compile your programs from source, there's a problem to be aware of and avoid while building.

Assuming Sun Studio 12 is installed in /opt/SUNWspro (the dafault location), here's what happens when a compile is attempted:

bash-3.00# pwd

bash-3.00# echo $PATH

bash-3.00# ./configure --without-gcc
checking build system type... i386-pc-solaris2.11
checking host system type... i386-pc-solaris2.11
checking target system type... i386-pc-solaris2.11

[ . . . lots more output snipped . . .]

configure: creating ./config.status
config.status: creating Makefile

bash-3.00# make
make: Warning: Illegal dependency list for target `.DEFAULT'
cc -g  -DRUBY_EXPORT -I. -I. -c array.c

[ . . . lots more output snipped . . .]

cc -g  -DRUBY_EXPORT -I. -I. -c ./missing/isinf.c
"./missing/isinf.c", line 32: syntax error before or at: __builtin_isinf
"./missing/isinf.c", line 33: warning: old-style declaration or incorrect type for: n
"./missing/isinf.c", line 33: syntax error before or at: double
"./missing/isinf.c", line 33: identifier redeclared: n
        current : double
        previous: int : "./missing/isinf.c", line 32
"./missing/isinf.c", line 35: syntax error before or at: return
cc: acomp failed for ./missing/isinf.c
\*\*\* Error code 2
make: Fatal error: Command failed for target `isinf.o'

The Makefile trys to compile ./missing/isinf.c and fails, thus aborting with a fatal build error. The good news is that this file is not needed, on Solaris, since isinf already exists as a part of libm - isinf(3M).
Why, then is this being built with the Ruby distribution? That's because Ruby's configure.in does not account for Solaris "5.11" . It only knows Solaris 10. Hence SXDE/SXCE/Nevada/OpenSolaris will not be recognized, and running make will cause isinf.c to be built(or attempted).

It's straightforward to avoid this problem. Here's how:

(1) replace the configure.in script in ruby-1.8.6/ by this modified version. The modified version adds the below lines, for Solaris Nevada in the appropriate place.

solaris\*2.11)  if test -z "$GCC"; then
               LIBS="-lm $LIBS"

(2) set your PATH,
export PATH=/usr/ccs/bin:/opt/SUNWspro/bin:/usr/sfw/bin:$PATH
/usr/ccs/bin is needed for make, /opt/SUNWspro/bin is the location of the studio 12 compiler, and /usr/sfw/bin is the free software directory, and has GNU Autoconf.

(3) Next, run the configure script.
./configure --without-gcc

(4) Next, run make.


(make will run Autoconf again, and will throw an error and ask itself to be run again, this is normal). The error is:
config.status: creating Makefile
Makefile updated, restart.
\*\*\* Error code 1
The following command caused the error:
{ \\
            echo "all:; -@rm -f conftest.mk"; \\
            echo "conftest.mk: .force; @echo AUTO_REMAKE"; \\
            echo ".force:"; \\
        } > conftest.mk || exit 1; \\
        make -f conftest.mk | grep '\^AUTO_REMAKE$' >/dev/null 2>&1 || \\
        { echo "Makefile updated, restart."; exit 1; }
make: Fatal error: Command failed for target `Makefile'

(5) Run make again.
(/usr/ccs/bin/make is okay - GNU Make is not necessary). The warning "make: Warning: Illegal dependency list for target `.DEFAULT'" warning can be ignored.

(6) Finally, run "make install" - and ruby, rdoc, and irb will be installed into /usr/local/bin (or whatever prefix you specified to ./configure). Also run "make install-doc" to compile the documentation for viewing through ri.




« July 2016