Friday Jun 19, 2009

Typo in OpenSolaris RoR AMI on Amazon EC2

We released a new OpenSolaris Ruby on Rails image on Amazon EC2 last week.  This version is different from the previous one on OpenSolaris because not only does this AMI include the usual goodies, updates etc., this is also a Typo appliance. 

Want some marketing key words?

  • Optimized
  • Secure
  • Pre-configured, preinstalled, and very "appliancy"
  • Extremely monitorable

Since I'm not really a marketing guy, I just can't put words out there without feeling a primordial urge to explain myself.

Optimized

The stack (Ruby, extensions, gems, web servers) is compiled with the latest compilers, optimal compiler flags based on performance studies, and Nginx is configured to have its temporary file store on a tmpfs mount.

Secure

Nothing runs as root, even for an instruction.  If you're wondering how Nginx can listen at port 80 without being started as root, the answer is easy - Solaris privileges:  root is not a user, rather it's a role.  So you can confer privileges (like listening on port 80) which are usually associated with root to other users (like the nginx user).  So this obviates the necessity for applications(like Nginx or Apache) to run as root even for the brief set of instructions needed to bind to a port.  The less instructions you run as root, the less your exposure.

Pre-configured, pre-installed, and very "appliancy".

The image contains a production ready configuration of Typo, which is the popular Rails blog engine(usage details are here[1]).  Its architecture: there are two Nginx workers.  Nginx listens at port 80 and is a fast web server which reduces the possibility of application bottlenecks.  In all likelihood, one cannot fully stress two Nginx workers in an EC2 instance (or even one, in my opinion) In addition, Nginx has a low memory footprint, so the benign effect of having two instances outweighs the overhead). 

The Rails servers are powered by Thin.  There are two instances that Nginx communicates with.  Rails is a CPU-intensive/low concurrency application that benefits from multiple instances.  The Thins are sized in anticipation of a single CPU so it may be wise to increase the number of Thin instances depending on the number of CPUs in the image.

MySQL is the database.  The three tiers are pre-configured.  Nginx communicates with Thin, and Thin communicates with the database.  Typo has its schema populated into the MySQL database, Rails and its dependencies are already installed.  The native gems, like MySQL and Thin have been compiled and configured correctly . . . and it "just works".  To use it start the instance, and log on to http://instance to begin setting up your web log.

SMF services exist for the Nginx, Thin, and MySQL components, and the dependencies between these services are also expressed in the manifests.

The catalog entry has information on where the service manifests are stored, and where the components are installed for one who wants further control.

Extremely monitorable

The AMI has DTrace support.  DTrace provides for a low overhead profiling with fine-grained control over which code is being profiled.  Please refer to Reference 2 below[2].

What's not in the AMI:  one needs a backup mechanism for their weblog entries, this is something that the AMI does not provide at this time. 

And that's it, please check out the AMI, and hope it is useful either in itself, as a starting point for an tiered architecture that can benefit from an SMF integration, and/or as a base Ruby on Rails AMI that can be used for building a private stack.

References:

[1] http://blogs.sun.com/ec2/entry/ruby_on_rails_2_ami

[2] http://blogs.sun.com/prashant/entry/dtrace_support2


Saturday Jun 07, 2008

On allocas


The alloca call is used when one would like to dynamically allocate memory within function scope. Such memory is reclaimed after the function call returns, hence obviating the necessity of explicitly freeing the memory.
On SPARC, alloca is a macro, defined in alloca.h.

     55 #if defined(__BUILTIN_VA_ARG_INCR) || \\
     56 	defined(__sparc) || defined(__i386) || defined(__amd64)
     57 #define	alloca(x)	__builtin_alloca(x)
     58  

The compiler, when it pre-processes a source file containing an alloca call, replaces it with a call to __builtin_alloca(invoking a "cc -P" generates pre-processed source code into a filename.i, in Sun Studio).

After the file is compiled, however, the allocas may not show up when DTrace is used to profile the application. This is because the compiler generates inline assembly for the __builtin_alloca call. This happens even if inlining is disabled using the "-xinline=" compiler option. All that the alloca implementation needs is to decrement the stack pointer by the number of bytes allocated by the alloca call, and the code to do this is generated by the compiler in place of the alloca call.(there is an exception to this simple algorithm, which is documented by Darryl)
For example,

     #include <stdio.h>
     #include <alloca.h>

     void main(void)
     {
       void \* ptr = alloca(262144);
     }

is compiled into an object, which on disassembly(which can be done using the er_src command with Sun Studio) looks like
Annotated disassembly
---------------------------------------
Source file: ./allocate.c
Object file: ./allocate
Load Object: ./allocate

   
   
   
     1. #include <stdio.h>
     2. #include <alloca.h>
     3. 
     4. void main(void)
     5. {
        
        [5]    10b70:  save        %sp, -104, %sp
     6.   void \* ptr = alloca(262144);
        [6]    10b74:  sethi       %hi(0x40000), %o0
        [6]    10b78:  sub         %sp, %o0, %sp
        [6]    10b7c:  ret         
        [6]    10b80:  restore     %g0, 0, %g0
     7. }


The save and restore calls are made on entry and exit from main, they obtain a fresh set of registers for the main routine, using a SPARC hardware feature called register windows. The alloca call is broken into the sethi and sub instructions. sethi sets the most significant 22 bits of register o0 with the 22 most significant bits of the hex value 0x40000(which is what the %hi achieves). the sub call then subtracts 0x40000(or 262144 in decimal), which is stored in register o0, from the stack pointer(-xO4 optimization was used to compile this code).

The alloca code is inlined in the object, hence a call to alloca(or __builtin_alloca) will not show up through DTrace or nm(unless the code falls into the exception category mentioned above).

A good way to trace this is to use the collector(and analyzer) or SPOT.
It helps if the binary is compiled using "-g -xO4 -xbinopt=prepare". Using -g does not reduce performance if -xO4 or higher is used, and -xbinopt=prepare does not affect performance.

Monday Nov 12, 2007

What's cool in Opensolaris build 78?

I can think of one thing ;-)

We putback Ruby into OpenSolaris build 78! Ruby 1.8.6 patchlevel 110, Rubygems 0.9.4, and Openssl, Curses, Readline, and Tcl/Tk extensions are enabled.

The directory layout for Ruby is per the ARC case. Ruby is in /usr/ruby/, it's linked from /usr/bin so as to be easy to locate - the man page is also appropriately linked.

Same for Rubygems, it's available though a link in /usr/bin - and the default gem repository happens to be /var/ruby/1.8/gem_home (ie., $GEM_HOME).

In the mean time, the wish list has is getting bigger n bigger . . .


\*\*\*\*\*\*\*\*\*  This mail is automatically generated  \*\*\*\*\*\*\*

Your putback for the following fix(es) is complete:

    6622810 Bring Ruby, Rubygems, and Extensions into OpenSolaris.
    

These fixes will be in release:

        snv_78

The gatekeeper will mark these bugids "fixed" and "integrated" in bugtraq
as soon as the gate has been delivered to the WOS.  However, you can mark
them "fixed" (but not "integrated") now if you wish.

        Your Friendly Gatekeepers

-------------------------------------------------------------

Friday Sep 14, 2007

Have ideas for what goes with Ruby/Rails in OpenSolaris?

If you are not interested in Ruby on Solaris, don't read further.

Else . . . help decide! You opinions and rationale are crucial to make OpenSolaris better!

What's your favorite Ruby extension or gem?
Write to the "Open Source applications on OpenSolaris" community at http://www.opensolaris.org/jive/thread.jspa?threadID=39394&tstart=0

Wednesday Sep 05, 2007

Uncooperative irb on Solaris?


 Did you just download Ruby on Solaris and realize that irb is uncooperative with history, tab completion, and line editing?

 The problem is that it needs the GNU readline extension.  Unlike ssl and curses support, the GNU Readline library is not available on Solaris/OpenSolaris yet.
 To enable this support, install GNU Readline, and then build Ruby with the appropriate library/include paths.  I've included a list of steps below to save the interested user some time . . .

(1) Download and Install Sun Studio 12.  It's pretty easy, pick it up from http://developers.sun.com/sunstudio/downloads/index.jsp.  It installs into /opt/SUNWspro
(2) Download the latest stable release of Ruby 1.8.6 from ftp://ftp.ruby-lang.org/pub/ruby/stable-snapshot.tar.gz.
(3) Download GNU Readline from ftp://ftp.cwru.edu/pub/bash/readline-5.2.tar.gz
(4) prefix your path with /opt/SUNWspro/bin.  Ensure that /usr/sfw/bin is not in your path for the next step.
(5) run ./configure in, make, and make install

root test4-sp>export PATH="/opt/SUNWspro/bin:/opt/coolstack/bin:/opt/csw/bin:/usr/sbin:/usr/bin"
root test4-sp>./configure --prefix=/export/readline/binaries
[ . . . snip . . .]
root test4-sp>make
[. . .snip . . .]
root test4-sp>make install


Don't get put off by the last message "install: you may need to run ldconfig".  ldconfig doesn't exist on Solaris.  Use good old crle to add the readline libraries into your path, if you'd like.  But it's not necessary for adding readline support to Ruby/irb.

(6) Next, note the path you installed readline into, in my case, it is /export/readline/binaries.  We'll have to ensure that the Ruby build knows about this library.  So, set your $CFLAGS variable appropriately:

root test4-sp>export CFLAGS="-xO4 -g -L/export/readline/binaries/lib/lib -I/export/readline/binaries/include/readline"

(7) Now, add /usr/local/bin(and /usr/sfw/bin too) to your path for autoconf, and proceed to build Ruby with readline enabled.

export PATH=/opt/SUNWspro/bin:/usr/local/bin:/usr/sfw/bin:/opt/coolstack/bin:/opt/csw/bin:$PATH
export cc=/opt/SUNWspro/bin
./configure --without-gcc --prefix=/export/ruby/builds/ --enable-pthread --with-readline-dir=/export/readline/binaries

(8) then run make, and make install (also make install-doc for ri to work properly).

(9) To turn on tab completion, create a .irbrc file in your home directory which contains the following(courtesy Nick Sieger's blog):

root test4-sp>more ~/.irbrc
require 'irb/completion'
ARGV.concat [ "--readline", "--prompt-mode", "simple" ]

(10) Now you're set.  Emacs style command editing in irb, history, and TAB completion of classes should work like a charm!


Wednesday Aug 22, 2007

MogileFS, and Solaris 10

MogileFS is a pretty cool non-hierarchical distributed file system for files that don't need to be stored in an RDBMS. I recently played with MogileFS as a part of some performance investigations on Solaris. While it was pretty straight forward, there are some gotchas to avoid.

Here is how I did it on Solaris/SPARC, using the Studio 11 compilers. It should work on Solaris x86, if you use Sun Studio 12.( I ran into some problems with compiling some perl dependencies on Solaris x86, with Studio 11. They went away with Studio 12, but I haven't gotten around to doing a complete build of MogileFS on Solaris x86 yet.)

I was helped quite a bit by this article by Brett G. Durrett. and I've added (modified)content from it into my howto to make it easy to get all the content in one page.

How do you install Mogile FS on Solaris SPARC?

  1. Start with either a Solaris 10 or a Nevada/OpenSolaris machine. I used Nevada build 46. This is an ancient build of OpenSolaris – build 70 being the latest.

  2. Install Sun Studio 11 into /opt/SUNWspro. This is where it installs, by default. If you have it in another place(such as an NFS mount), create a link to /opt/SUNWspro

  3. Cooltools, are really cool. Get the cooltools AMP stack(from coolstack 1.1). Select the link titled “Apache 2.2.3, MySQL 5.0.33, PHP 5.2.0, English” under the like of downloads for Solaris 10 SPARC. Install this stack, and make sure that /opt/coolstack/bin is prepended to your $PATH variable.

  4. Also install coolstack perl. This should give you Perl 5.8.8. This is available from http://cooltools.sunsource.net too.

  5. Prepend /usr/sfw/bin to your path. This is where the gnu packages are found, in Solaris 10 onward. In case you were wondering, this is different from /opt/sfw/bin, where the contents of the Solaris companion CD are installed(optionally).

  6. Install GNU Coreutils 6.4. - here is a direct link - ftp://ftp.sunfreeware.com/pub/freeware/sparc/10/coreutils-6.4-sol10-sparc-local.gz These are easily obtained from either http://www.sunfreeware.com or http://www.blastwave.org or mirrors in case the above link does not work.

  7. Prepend /usr/local/bin to your $PATH variable. This is crucial because mogstored depends on the GNU df. You'll see disk capacity/usage/free space show up as zero(and also other mysterious failures) when looking at device stats using mogadm if GNU df is not in your path.

  8. Configure cpan with /opt/coolstack/bin/perl. Use http mirrors if you're behind a corporate firewall since they work better. CPAN does not handle ftp:// urls properly(at least for me), when I use Sun's proxies. This leads to corrupt packages. So if you're behind a corporate firewall, use only http CPAN mirrors for better results.

  9. Download Perlbal first - http://search.cpan.org/CPAN/authors/id/B/BR/BRADFITZ/Perlbal-1.52.tar.gzY and go through the perl Makefile.pl; make; make test; make install process. You will see a lot of dependencies that are unsatisfied in the second step. The list looks like this(including some packages that it doesn't mention explicitly):
    BSD::Resource 0 (you can install the latest, 1.28 as well)
    Danga::Socket 1.44
    HTTP::Date 0
    HTTP::Response 0
    prerequisite Sys::Syscall 0
    IO::WrapTie
    Compress::Zlib
    IO::AIO
    Net::Netmask

  10. First try to install them through CPAN, since each of these packages has other dependencies. If CPAN doesn't install some of the above mentioned packages(probably because of failures in “make test&rdquo;), then install them manually(by going into /.cpan/build/<Pkg_name> and run perl Makefile.PL, make, make install directly). You wont need to install the dependencies for these, since CPAN would have finished this already(hence making your installation experience easier).

    Finally run “make” and “make install” on Perlbal. It shouldn't complain about unsatisfied dependencies now.

  11. Obtain dbi and dbd. install these. -- http://cooltools.sunsource.net/coolstack/faq.html (install dbi, use cpan for the dependencies, and then compile dbi directly). use studio 11(make sure you're running this on sparc, not x86)

  12. By now you're past the difficult section, and can afford to breathe a little. Setup the mogile user:
    useradd mogile
    mkdir -p /export/home/mogile
    chown mogile /export/home/mogile
    edit /etc/passwd to change home dir to /export/home/mogile

  13. Setup the mysql user. I installed this on the same machine as the MogileFS tracker. But this may not be most performant.
    groupadd mysql
    useradd mysql
    bash-3.00# mkdir -p /export/home/mysql
    bash-3.00# chown -R mysql:mysql /export/home/mysql


  14. Below is how you quickly set up MySQL for MogileFS.

    cd /opt/coolstack/mysql_32bit/bin
    ./mysql_install_db
    chown -R mysql:mysql /opt/coolstack/mysql_32bit/
    cd /opt/coolstack/mysql_32bit ; /opt/coolstack/mysql_32bit/bin/mysqld_safe &

    bash-3.00$ /opt/coolstack/mysql_32bit/bin/mysqladmin -u root password 'new-password'
    /opt/coolstack/mysql_32bit/bin/mysqladmin -u root -h MyHostname password 'new-password'
    bash-3.00# pwd
    /opt/coolstack/mysql_32bit/bin
    bash-3.00$ ./mysql -u root -p
    Enter password:
    Welcome to the MySQL monitor. Commands end with ; or \\g.
    Your MySQL connection id is 5
    Server version: 5.0.33-standard Source distribution

    Type 'help;' or '\\h' for help. Type '\\c' to clear the buffer.

    mysql> CREATE DATABASE mogilefs;
    Query OK, 1 row affected (0.03 sec)

    mysql> GRANT ALL PRIVILEGES ON \*.\* TO 'mogile'@'localhost' IDENTIFIED BY 'some_pass' WITH GRANT OPTION;
    Query OK, 0 rows affected (0.00 sec)

    mysql> GRANT ALL PRIVILEGES ON \*.\* TO 'mogile'@'%' IDENTIFIED BY 'some_pass' WITH GRANT OPTION;
    Query OK, 0 rows affected (0.00 sec)

    mysql> flush privileges
    -> ;
    Query OK, 0 rows affected (0.00 sec)

    mysql> exit
    Bye
    bash-3.00$


  15. Install curl support for PHP(if you plan to use MogileFS through PHP). Do this by editing php.ini and add
    extension=curl.so . php.ini is in /opt/coolstack/php5/lib/php.ini

  16. Now install mogilefs, using the below steps, which is a modified version of http://mogilefs.schtuff.com/howto for linux. The following steps document the solaris specific stuff, as well as presents an updated version of the Linux howto, in one place.

  17. The snapshot of MogileFS that I used is here. You can use this or pick up the latest from svn. MogileFS seems to be evolving at such a fast rate that I decided to put up the snapshot I used, rather leaving it open ended. You can install Mogile with the following commands:

    # cd trunk/server/
    # perl Makefile.PL
    # make
    # make test
    # make install

    If you get any errors during this process it will probably be errors telling you that a dependent module is missing. If during the 'make test' step you get the error, "t/00-startup....DBI connect('mysql','root',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at t/lib/mogtestlib.pl line 16" it can probably be ignored – it should not be necessary for MySQL to be running on any host other than mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/>.

    the following "make test" errors are OK

    bash-3.00# make test
    PERL_DL_NONLAZY=1 /opt/coolstack/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/\*.t
    t/00-startup.........skipped
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    t/10-weighting.......skipped
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    t/domains-classes....skipped
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    t/hosts-devices......skipped
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    t/store..............skipped
    all skipped: Can't create temporary test database: DBI connect('mysql','root',...) failed: Access denied for user 'root'@'localhost' (using password: NO) at /export/pkgs/mogile_new/trunk/server/blib/lib/MogileFS/Store/MySQL.pm line 180
    All tests successful, 5 tests skipped.
    Files=5, Tests=0, 3 wallclock secs ( 2.75 cusr + 0.30 csys = 3.05 CPU)
    bash-3.00#




  18. You probably want to install some helpful utilities on each tracker or storage server as well (these will be needed for later configuration). These are located in the trunk/utils directory and can be installed with the following commands (starting in the top of the SVN directory you pulled):

    # cd trunk/utils/
    # perl Makefile.PL
    # make
    # make test
    # make install


    You also want the API – the utilities will require this. These are located in the trunk/api/perl directory and can be installed with the following commands (starting in the top of the SVN directory you pulled):

    # cd trunk/api/perl
    # perl Makefile.PL
    # make
    # make test
    # make install


  19. Database configuration: The database is empty and will need a schema applied. The ' trunk/server' directory has a utility named 'mogdbsetup' to make this process simple. By default it assumes the database is located on localhost so if you are running it from a different host you will need to provide the host name on the command line.

    # ./mogdbsetup --dbhost=mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/> --dbname=mogilefs --dbuser=mogile --dbpass=some_pass
    Again, make sure you replace the host and password so that they match you database configuration from above.

    The mogdbsetup utility does not specify a table type by default so your tables will match the defaults for your database. In many cases this will mean that you end up with MyISAM tables. If you prefer InnoDB tables you will either need to make sure your database defaults to InnoDB or you can manually convert the tables (both of these are outside of the scope of this document but there are plenty of examples out there).

  20. Tracker Configuration: On each tracker server (mogiletracker.yourdomain.com <http://mogiletracker.yourdomain.com/>), create a configuration file at /etc/mogilefs/mogilefsd.conf with the following:

    db_dsn DBI:mysql:mogilefs:mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/>
    db_user mogile
    db_pass some_pass
    conf_port 6001
    listener_jobs 5

    db_dsn points to your database instance. If you are running the database on the same machine as the storage server you can omit ":mogiledb.yourdomain.com <http://mogiledb.yourdomain.com/>: and it will use the local machine. db_user and db_pass should match the user and password you configured when setting up your database.

    The program 'mogilefsd' will not run as root so you will need to run this as a non-root user “mogile” that you created earlier.

  21. Starting the trackers: Trackers will not run as root so you will need to run them as another user. If you created the "mogile" user when seetingup the trackers, the following commands will work (assumes you start logged in to mogiletracker.yourdomain.com <http://mogiletracker.yourdomain.com/> as root):

    # su mogile
    $ mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon
    $ exit


    You can confirm that the trackers are running with the following command:

    bash-3.00# ps -ef | grep mogilefsd
     mogile 11612 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11614 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11611 11608   0 15:33:11 pts/2       0:01 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
       root 11635  9604   0 15:34:22 pts/1       0:00 grep mogilefsd
     mogile 11613 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11618 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11621 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11617 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11615 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11616 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11619 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
     mogile 11620 11611   0 15:33:12 pts/2       0:00 /opt/coolstack/bin/perl /opt/coolstack/bin/mogilefsd -c /etc/mogilefs/mogilefsd
    bash-3.00#


    If you don't get a list of running processes the trackers are not running.

  22. Storage Server Configuration: On each storage server, create the storage directory (make sure it has access permissions for the user you will use to run mogstored):

    # mkdir /var/mogdata

    Configure it:

    On each storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:

    httplisten=0.0.0.0:7500
    mgmtlisten=0.0.0.0:7501
    docroot=/var/mogdata

  23. Adding storage server information to the trackers:

    bash-3.00#mogadm --trackers=MyTrackerServerHostname:6001 host add MyStorageServerHostname --ip=10.6.141.125 --port=7500 --status=alive


    You can confirm that your host(s) were added with the following command;

    bash-3.00# mogadm --trackers=MyTrackerServerHostname:6001 host list

  24. Add a device to your storage server:

    # mkdir -p /var/mogdata/dev1

    and let the tracker know . . .

    bash-3.00# mogadm --trackers=dn15:6001 device add dn15 1

  25. List your devices:

    bash-3.00# mogadm --trackers=MyTrackerHostname:6001 device list
    MyTrackerHostname [1]: alive
                      used(G) free(G) total(G)
     dev1: alive      4.745   58.722  63.467

    bash-3.00#

  26. In order to add/remove/read files, you will need to create a domain and a class. Domains are like buckets to store your files in an otherwise non-hierarchical distributed file system. Classes, seem like no more than specifiers for the replication count.

    Create a domain: bash-3.00# mogadm --trackers=MyTrackerHostname:6001 domain add mydomain.sun.com

    Add a class to the domain: bash-3.00# mogadm --trackers=MyTrackerHostname:6001 class add mydomain.sun.com Addresses




  27. Now you're all set! But you'll probably want to play around with some files. If you want to use PHP, heres a client that I use - http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/MogileClient/MogileFS.php?revision=7483&view=markup(It's pretty straight forward to use.)




Friday Aug 03, 2007

Compiling Ruby 1.8.6 on Solaris Nevada (or OpenSolaris, or Solaris Express ) using Sun Studio 12

One doesn't need to compile their Ruby from source on Solaris Nevada / OpenSolaris / Solaris Express (SX). It can be downloaded from Blastwave, Sunfreeware, or Cooltools.
sunfreeware.com happens to have the latest stable version(1.8.6) available.

However, if you want an optimized build for your setup, or just like to compile your programs from source, there's a problem to be aware of and avoid while building.

Assuming Sun Studio 12 is installed in /opt/SUNWspro (the dafault location), here's what happens when a compile is attempted:

bash-3.00# pwd
/export/ruby/ruby-1.8.6

bash-3.00# echo $PATH
/usr/ccs/bin:/opt/SUNWspro/bin:/usr/sbin:/usr/bin

bash-3.00# ./configure --without-gcc
checking build system type... i386-pc-solaris2.11
checking host system type... i386-pc-solaris2.11
checking target system type... i386-pc-solaris2.11

[ . . . lots more output snipped . . .]

configure: creating ./config.status
config.status: creating Makefile

bash-3.00# make
make: Warning: Illegal dependency list for target `.DEFAULT'
cc -g  -DRUBY_EXPORT -I. -I. -c array.c

[ . . . lots more output snipped . . .]

cc -g  -DRUBY_EXPORT -I. -I. -c ./missing/isinf.c
"./missing/isinf.c", line 32: syntax error before or at: __builtin_isinf
"./missing/isinf.c", line 33: warning: old-style declaration or incorrect type for: n
"./missing/isinf.c", line 33: syntax error before or at: double
"./missing/isinf.c", line 33: identifier redeclared: n
        current : double
        previous: int : "./missing/isinf.c", line 32
"./missing/isinf.c", line 35: syntax error before or at: return
cc: acomp failed for ./missing/isinf.c
\*\*\* Error code 2
make: Fatal error: Command failed for target `isinf.o'
bash-3.00#

The Makefile trys to compile ./missing/isinf.c and fails, thus aborting with a fatal build error. The good news is that this file is not needed, on Solaris, since isinf already exists as a part of libm - isinf(3M).
Why, then is this being built with the Ruby distribution? That's because Ruby's configure.in does not account for Solaris "5.11" . It only knows Solaris 10. Hence SXDE/SXCE/Nevada/OpenSolaris will not be recognized, and running make will cause isinf.c to be built(or attempted).

It's straightforward to avoid this problem. Here's how:

(1) replace the configure.in script in ruby-1.8.6/ by this modified version. The modified version adds the below lines, for Solaris Nevada in the appropriate place.

solaris\*2.11)  if test -z "$GCC"; then
                      ac_cv_func_isinf=yes
              fi
               LIBS="-lm $LIBS"
              ;;

(2) set your PATH,
export PATH=/usr/ccs/bin:/opt/SUNWspro/bin:/usr/sfw/bin:$PATH
/usr/ccs/bin is needed for make, /opt/SUNWspro/bin is the location of the studio 12 compiler, and /usr/sfw/bin is the free software directory, and has GNU Autoconf.

(3) Next, run the configure script.
./configure --without-gcc

(4) Next, run make.

 make

(make will run Autoconf again, and will throw an error and ask itself to be run again, this is normal). The error is:
config.status: creating Makefile
Makefile updated, restart.
\*\*\* Error code 1
The following command caused the error:
{ \\
            echo "all:; -@rm -f conftest.mk"; \\
            echo "conftest.mk: .force; @echo AUTO_REMAKE"; \\
            echo ".force:"; \\
        } > conftest.mk || exit 1; \\
        make -f conftest.mk | grep '\^AUTO_REMAKE$' >/dev/null 2>&1 || \\
        { echo "Makefile updated, restart."; exit 1; }
make: Fatal error: Command failed for target `Makefile'

(5) Run make again.
make
(/usr/ccs/bin/make is okay - GNU Make is not necessary). The warning "make: Warning: Illegal dependency list for target `.DEFAULT'" warning can be ignored.

(6) Finally, run "make install" - and ruby, rdoc, and irb will be installed into /usr/local/bin (or whatever prefix you specified to ./configure). Also run "make install-doc" to compile the documentation for viewing through ri.

About

prashant

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today