Tuesday Mar 24, 2009

installing ImageScience an art?

ImageScience is a pretty cool gem to use in your Rails application for Thumbnail generation.  It doesn't install correctly out of the box since ImageScience uses an incorrect default path for locating FreeImage. The Makefile shipped with FreeImage also does not build correctly on OpenSolaris.  So here's a way to make sure it works correctly first time if you are on OpenSolaris.

Step1:  Update your Ruby package.  We now package Ruby 1.8.7 p72 with OpenSolaris.  Here is a blog entry I wrote sometime ago on how to upgrade to 1.8.7p72 -> http://blogs.sun.com/prashant/entry/how_does_one_update_ruby

Step2: Ensure that /usr/gnu/bin is prepended to your PATH environment variable. (or, just utter export PATH=/usr/gnu/bin:$PATH in bash).

Step3: Install the requisite OpenSolaris packages.

pfexec pkg install SUNWgmake

pfexec pkg install SUNWgnu-coreutils

The "pfexec" is there to give you enough privileges to install the packages.  If you run as root(which I do - and which is not recommended), then you don't need to use pfexec.

Step4: Build FreeImage. Download it from http://downloads.sourceforge.net/freeimage/FreeImage3110.zip and unzip it.  The Makefile that ships with FreeImage does not build correctly on OpenSolaris, so here is a Makefile that does -> http://blogs.sun.com/prashant/resource/files/Makefile.opensolaris  .

Change directory into the exploded FreeImage source, and utter the following.

gmake -f Makefile.opensolaris

gmake -f Makefile.opensolaris install

Where Makefile.opensolaris is the OpenSolaris Makefile that you just downloaded.

Step5:  The rbconfig.rb file in your Ruby installation needs to be changed.  Edit this file(it's to be found in /usr/ruby/1.8/lib/ruby/1.8/i386-solaris2.11 )

Look for a line that goes thus:


 and replace it with


i.e., change the linker binary such that the GNU Linker is used by RubyInline(which is an ImageScience dependency).

This is a bug that will be fixed in build111.  So if you have a newer version, this bug might already have been fixed.

Step6: Now install ImageScience the usual way. 

gem install image_science

Step7: You should now be set.  But, just to make sure, run the ImageScience tests and make sure they pass.  Here is how to do that.

Edit $GEM_HOME/gems/image_science-1.1.3 test/test_image_science.rb (if you use the default gem home on OpenSolaris the file is to /var/ruby/1.8/gem_home/gems/image_science-1.1.3 test/test_image_science.rb ).

Add the following line to the beginning of the file.

 require 'rubygems'

Now test it by executing:

cd /var/ruby/1.8/gem_home/gems/image_science-1.1.3

(or cd $GEM_HOME/gems/image_science-1.1.3 if your gem home is different).

ruby test/test_image_science.rb

 Loaded suite test/test_image_science
Finished in 0.01314 seconds.

7 tests, 28 assertions, 0 failures, 0 errors

And thats all it takes to have a working ImageScience library.

Wednesday Sep 03, 2008

"no kex alg" on recent Nevada builds

I've begun to see "no kex alg" messages on trying to ssh into machines that were jumpstarted to the later Nevada/OpenSolaris builds.  After some googling, it seems that the OpenSolaris folk are working on how to get this work better. In the mean time, here is how I generate ssh keys on a machine where sshd doesn't accept my connection because it's ssh keys haven't been generated.

bash-3.2# ssh dn02
no kex alg
bash-3.2# rlogin dn02  #or get in through the service processor.
Last login: Wed Sep  3 12:39:03 from dn01
Sun Microsystems Inc.   SunOS 5.11      snv_96  November 2008
# bash
bash-3.2# /lib/svc/method/sshd -c
bash-3.2# svcadm refresh ssh
bash-3.2# exit
# Connection to dn02 closed.
bash-3.2# ssh dn02
The authenticity of host 'dn02 (' can't be established.
RSA key fingerprint is 0b:ec:fe:85:51:82:5e:df:c0:44:10:d3:79:67:49:ea.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'dn02,' (RSA) to the list of known hosts.
Last login: Wed Sep  3 12:42:00 2008 from dn01
Sun Microsystems Inc.   SunOS 5.11      snv_96  November 2008

see http://opensolaris.org/jive/thread.jspa?messageID=176002 and http://defect.opensolaris.org/bz/show_bug.cgi?id=219

It's apparently fixed in OpenSolaris LiveCD - but the fix probably hasn't made it into Nevada, even though the diffs posted in the bug entry are available on my machine.

Saturday Jun 07, 2008

On allocas

The alloca call is used when one would like to dynamically allocate memory within function scope. Such memory is reclaimed after the function call returns, hence obviating the necessity of explicitly freeing the memory.
On SPARC, alloca is a macro, defined in alloca.h.

     55 #if defined(__BUILTIN_VA_ARG_INCR) || \\
     56 	defined(__sparc) || defined(__i386) || defined(__amd64)
     57 #define	alloca(x)	__builtin_alloca(x)

The compiler, when it pre-processes a source file containing an alloca call, replaces it with a call to __builtin_alloca(invoking a "cc -P" generates pre-processed source code into a filename.i, in Sun Studio).

After the file is compiled, however, the allocas may not show up when DTrace is used to profile the application. This is because the compiler generates inline assembly for the __builtin_alloca call. This happens even if inlining is disabled using the "-xinline=" compiler option. All that the alloca implementation needs is to decrement the stack pointer by the number of bytes allocated by the alloca call, and the code to do this is generated by the compiler in place of the alloca call.(there is an exception to this simple algorithm, which is documented by Darryl)
For example,

     #include <stdio.h>
     #include <alloca.h>

     void main(void)
       void \* ptr = alloca(262144);

is compiled into an object, which on disassembly(which can be done using the er_src command with Sun Studio) looks like
Annotated disassembly
Source file: ./allocate.c
Object file: ./allocate
Load Object: ./allocate

     1. #include <stdio.h>
     2. #include <alloca.h>
     4. void main(void)
     5. {
        [5]    10b70:  save        %sp, -104, %sp
     6.   void \* ptr = alloca(262144);
        [6]    10b74:  sethi       %hi(0x40000), %o0
        [6]    10b78:  sub         %sp, %o0, %sp
        [6]    10b7c:  ret         
        [6]    10b80:  restore     %g0, 0, %g0
     7. }

The save and restore calls are made on entry and exit from main, they obtain a fresh set of registers for the main routine, using a SPARC hardware feature called register windows. The alloca call is broken into the sethi and sub instructions. sethi sets the most significant 22 bits of register o0 with the 22 most significant bits of the hex value 0x40000(which is what the %hi achieves). the sub call then subtracts 0x40000(or 262144 in decimal), which is stored in register o0, from the stack pointer(-xO4 optimization was used to compile this code).

The alloca code is inlined in the object, hence a call to alloca(or __builtin_alloca) will not show up through DTrace or nm(unless the code falls into the exception category mentioned above).

A good way to trace this is to use the collector(and analyzer) or SPOT.
It helps if the binary is compiled using "-g -xO4 -xbinopt=prepare". Using -g does not reduce performance if -xO4 or higher is used, and -xbinopt=prepare does not affect performance.

Friday Oct 26, 2007


The Ruby PSARC case got approved quite speedily. It could have been all the prior research into compatibility, file layouts, directory structure done(in response to a lot of good questions from Jyri).

Or it was probably because you hear that the ARCs are a source of delays, and other such scary rumors, so that when you really go through it, and realize that it's not true, you're confused.

MySQL was the other ARC case that seemed to sail through.
So, I'm going to dismiss perceptions of ARC tardiness as fiction(at least in my limited experiences).

My advise to anyone who wants to write an ARC case for OpenSolaris:

-> Read prior ARC cases to get a feel for how they're written(ie., the tone).

-> Think Interfaces, Interfaces, and Interfaces. ie., what interfaces does your software expose, what does it expect? What's the stability of each of these interfaces?

-> Look for precedence. ie., If you're proposing the addition of a new scripting language(like Ruby), look for older cases where scripting languages were approved(like Perl).

-> Think about compatibility/stability across versions. Across what releases is compatibility guaranteed? Across which releases is backward compatibility broken? What constitutes a stable release? What constitutes a development release?

Solaris is all about backward compatibility and stability.

-> And give this research a lot of time - otherwise, you're tempted to cut corners, and somehow wing your case. This impulse is detrimental to Solaris - which is the software you're trying to improve by making an ARC proposal in the first place.

Well, that's enough rambling about ARC cases, I've got to ease into my week end Martinis :-)


Wednesday Aug 22, 2007

MogileFS, and Solaris 10

MogileFS is a pretty cool non-hierarchical distributed file system for files that don't need to be stored in an RDBMS. I recently played with MogileFS as a part of some performance investigations on Solaris. While it was pretty straight forward, there are some gotchas to avoid.

Here is how I did it on Solaris/SPARC, using the Studio 11 compilers. It should work on Solaris x86, if you use Sun Studio 12.( I ran into some problems with compiling some perl dependencies on Solaris x86, with Studio 11. They went away with Studio 12, but I haven't gotten around to doing a complete build of MogileFS on Solaris x86 yet.)

I was helped quite a bit by this article by Brett G. Durrett. and I've added (modified)content from it into my howto to make it easy to get all the content in one page.

How do you install Mogile FS on Solaris SPARC?

  1. Start with either a Solaris 10 or a Nevada/OpenSolaris machine. I used Nevada build 46. This is an ancient build of OpenSolaris – build 70 being the latest.

  2. Install Sun Studio 11 into /opt/SUNWspro. This is where it installs, by default. If you have it in another place(such as an NFS mount), create a link to /opt/SUNWspro

  3. Cooltools, are really cool. Get the cooltools AMP stack(from coolstack 1.1). Select the link titled “Apache 2.2.3, MySQL 5.0.33, PHP 5.2.0, English” under the like of downloads for Solaris 10 SPARC. Install this stack, and make sure that /opt/coolstack/bin is prepended to your $PATH variable.

  4. Also install coolstack perl. This should give you Perl 5.8.8. This is available from http://cooltools.sunsource.net too.

  5. Prepend /usr/sfw/bin to your path. This is where the gnu packages are found, in Solaris 10 onward. In case you were wondering, this is different from /opt/sfw/bin, where the contents of the Solaris companion CD are installed(optionally).

  6. Install GNU Coreutils 6.4. - here is a direct link - ftp://ftp.sunfreeware.com/pub/freeware/sparc/10/coreutils-6.4-sol10-sparc-local.gz These are easily obtained from either http://www.sunfreeware.com or http://www.blastwave.org or mirrors in case the above link does not work.

  7. Prepend /usr/local/bin to your $PATH variable. This is crucial because mogstored depends on the GNU df. You'll see disk capacity/usage/free space show up as zero(and also other mysterious failures) when looking at device stats using mogadm if GNU df is not in your path.

  8. Configure cpan with /opt/coolstack/bin/perl. Use http mirrors if you're behind a corporate firewall since they work better. CPAN does not handle ftp:// urls properly(at least for me), when I use Sun's proxies. This leads to corrupt packages. So if you're behind a corporate firewall, use only http CPAN mirrors for better results.

  9. Download Perlbal first - http://search.cpan.org/CPAN/authors/id/B/BR/BRADFITZ/Perlbal-1.52.tar.gzY and go through the perl Makefile.pl; make; make test; make install process. You will see a lot of dependencies that are unsatisfied in the second step. The list looks like this(including some packages that it doesn't mention explicitly):
    BSD::Resource 0 (you can install the latest, 1.28 as well)
    Danga::Socket 1.44
    HTTP::Date 0
    HTTP::Response 0
    prerequisite Sys::Syscall 0

  10. First try to install them through CPAN, since each of these packages has other dependencies. If CPAN doesn't install some of the above mentioned packages(probably because of failures in “make test&rdquo;), then install them manually(by going into /.cpan/build/<Pkg_name> and run perl Makefile.PL, make, make install directly). You wont need to install the dependencies for these, since CPAN would have finished this already(hence making your installation experience easier).

    Finally run “make” and “make install” on Perlbal. It shouldn't complain about unsatisfied dependencies now.

  11. Obtain dbi and dbd. install these. -- http://cooltools.sunsource.net/coolstack/faq.html (install dbi, use cpan for the dependencies, and then compile dbi directly). use studio 11(make sure you're running this on sparc, not x86)

  12. By now you're past the difficult section, and can afford to breathe a little. Setup the mogile user:
    useradd mogile
    mkdir -p /export/home/mogile
    chown mogile /export/home/mogile
    edit /etc/passwd to change home dir to /export/home/mogile

  13. Setup the mysql user. I installed this on the same machine as the MogileFS tracker. But this may not be most performant.
    groupadd mysql
    useradd mysql
    bash-3.00# mkdir -p /export/home/mysql
    bash-3.00# chown -R mysql:mysql /export/home/mysql

  14. Below is how you quickly set up MySQL for MogileFS.

    cd /opt/coolstack/mysql_32bit/bin
    chown -R mysql:mysql /opt/coolstack/mysql_32bit/
    cd /opt/coolstack/mysql_32bit ; /opt/coolstack/mysql_32bit/bin/mysqld_safe &

    bash-3.00$ /opt/coolstack/mysql_32bit/bin/mysqladmin -u root password 'new-password'
    /opt/coolstack/mysql_32bit/bin/mysqladmin -u root -h MyHostname password 'new-password'
    bash-3.00# pwd
    bash-3.00$ ./mysql -u root -p
    Enter password:
    Welcome to the MySQL monitor. Commands end with ; or \\g.
    Your MySQL connection id is 5
    Server version: 5.0.33-standard Source distribution

    Type 'help;' or '\\h' for help. Type '\\c' to clear the buffer.

    mysql> CREATE DATABASE mogilefs;
    Query OK, 1 row affected (0.03 sec)

    mysql> GRANT ALL PRIVILEGES ON \*.\* TO 'mogile'@'localhost' IDENTIFIED BY 'some_pass' WITH GRANT OPTION;
    Query OK, 0 rows affected (0.00 sec)

    mysql> GRANT ALL PRIVILEGES ON \*.\* TO 'mogile'@'%' IDENTIFIED BY 'some_pass' WITH GRANT OPTION;
    Query OK, 0 rows affected (0.00 sec)

    mysql> flush privileges
    -> ;
    Query OK, 0 rows affected (0.00 sec)

    mysql> exit

  15. Install curl support for PHP(if you plan to use MogileFS through PHP). Do this by editing php.ini and add
    extension=curl.so . php.ini is in /opt/coolstack/php5/lib/php.ini

  16. Now install mogilefs, using the below steps, which is a modified version of http://mogilefs.schtuff.com/howto for linux. The following steps document the solaris specific stuff, as well as presents an updated version of the Linux howto, in one place.

  17. The snapshot of MogileFS that I used is here. You can use this or pick up the latest from svn. MogileFS seems to be evolving at such a fast rate that I decided to put up the snapshot I used, rather leaving it open ended. You can install Mogile with the following commands:

    # cd trunk/server/
    # perl Makefile.PL
    # make
    # make test
    # make install

    If you get any errors during this process it will probably be errors telling you that a dependent module is missing. If during the 'make test' step you get the error, "t/00-startup....DBI connect('mysql','root',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at t/lib/mogtestlib.pl line 16" it can probably be ignored – it should not be necessary for MySQL to be running on any host other than mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/>.

    the following "make test" errors are OK

    bash-3.00# make test
    PERL_DL_NONLAZY=1 /opt/coolstack/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/\*.t
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    All tests successful, 5 tests skipped.
    Files=5, Tests=0, 3 wallclock secs ( 2.75 cusr + 0.30 csys = 3.05 CPU)

  18. You probably want to install some helpful utilities on each tracker or storage server as well (these will be needed for later configuration). These are located in the trunk/utils directory and can be installed with the following commands (starting in the top of the SVN directory you pulled):

    # cd trunk/utils/
    # perl Makefile.PL
    # make
    # make test
    # make install

    You also want the API – the utilities will require this. These are located in the trunk/api/perl directory and can be installed with the following commands (starting in the top of the SVN directory you pulled):

    # cd trunk/api/perl
    # perl Makefile.PL
    # make
    # make test
    # make install

  19. Database configuration: The database is empty and will need a schema applied. The ' trunk/server' directory has a utility named 'mogdbsetup' to make this process simple. By default it assumes the database is located on localhost so if you are running it from a different host you will need to provide the host name on the command line.

    # ./mogdbsetup --dbhost=mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/> --dbname=mogilefs --dbuser=mogile --dbpass=some_pass
    Again, make sure you replace the host and password so that they match you database configuration from above.

    The mogdbsetup utility does not specify a table type by default so your tables will match the defaults for your database. In many cases this will mean that you end up with MyISAM tables. If you prefer InnoDB tables you will either need to make sure your database defaults to InnoDB or you can manually convert the tables (both of these are outside of the scope of this document but there are plenty of examples out there).

  20. Tracker Configuration: On each tracker server (mogiletracker.yourdomain.com <http://mogiletracker.yourdomain.com/>), create a configuration file at /etc/mogilefs/mogilefsd.conf with the following:

    db_dsn DBI:mysql:mogilefs:mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/>
    db_user mogile
    db_pass some_pass
    conf_port 6001
    listener_jobs 5

    db_dsn points to your database instance. If you are running the database on the same machine as the storage server you can omit ":mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/>: and it will use the local machine. db_user and db_pass should match the user and password you configured when setting up your database.

    The program 'mogilefsd' will not run as root so you will need to run this as a non-root user “mogile” that you created earlier.

  21. Starting the trackers: Trackers will not run as root so you will need to run them as another user. If you created the "mogile" user when seetingup the trackers, the following commands will work (assumes you start logged in to mogiletracker.yourdomain.com <http://mogiletracker.yourdomain.com/> as root):

    # su mogile
    $ mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon
    $ exit

    You can confirm that the trackers are running with the following command:

    bash-3.00# ps -ef | grep mogilefsd
     mogile 11612 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11614 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11611 11608   0 15:33:11 pts/2       0:01 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
       root 11635  9604   0 15:34:22 pts/1       0:00 grep mogilefsd
     mogile 11613 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11618 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11621 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11617 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11615 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11616 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11619 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11620 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd

    If you don't get a list of running processes the trackers are not running.

  22. Storage Server Configuration: On each storage server, create the storage directory (make sure it has access permissions for the user you will use to run mogstored):

    # mkdir /var/mogdata

    Configure it:

    On each storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:


  23. Adding storage server information to the trackers:

    bash-3.00#mogadm --trackers=MyTrackerServerHostname:6001 host add MyStorageServerHostname --ip= --port=7500 --status=alive

    You can confirm that your host(s) were added with the following command;

    bash-3.00# mogadm --trackers=MyTrackerServerHostname:6001 host list

  24. Add a device to your storage server:

    # mkdir -p /var/mogdata/dev1

    and let the tracker know . . .

    bash-3.00# mogadm --trackers=dn15:6001 device add dn15 1

  25. List your devices:

    bash-3.00# mogadm --trackers=MyTrackerHostname:6001 device list
    MyTrackerHostname [1]: alive
                      used(G) free(G) total(G)
     dev1: alive      4.745   58.722  63.467


  26. In order to add/remove/read files, you will need to create a domain and a class. Domains are like buckets to store your files in an otherwise non-hierarchical distributed file system. Classes, seem like no more than specifiers for the replication count.

    Create a domain: bash-3.00# mogadm --trackers=MyTrackerHostname:6001 domain add mydomain.sun.com

    Add a class to the domain: bash-3.00# mogadm --trackers=MyTrackerHostname:6001 class add mydomain.sun.com Addresses

  27. Now you're all set! But you'll probably want to play around with some files. If you want to use PHP, heres a client that I use - http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/MogileClient/MogileFS.php?revision=7483&view=markup(It's pretty straight forward to use.)

Tuesday Jul 31, 2007

Project Indiana

In case you stopped tracking project Indiana after the initial buzz - here is the OpenSolaris project page for Indiana: http://www.opensolaris.org/os/project/indiana/

The conversations seem to have started in earnest.

An OpenSolaris binary distro that has the corresponding source will make it way easier for an interested user to try OpenSolaris, and I happen to know a few interested hackers in the waiting list. I've heard Ian Murdock speak on Indiana(at least twice now?), and find myself agreeing enthusiastically with his motivations for Indiana.




« April 2014