Friday Apr 03, 2015

Cloudera Hive client on Solaris

Goal: From an Oracle Solaris server, access Hive data that is stored in a Cloudera Hadoop cluster, for example, access Hive data in an Oracle Big Data Appliance from a Solaris SPARC RDBMS server.


This blog assumes that the Cloudera Hadoop cluster is already up and running. To test, I started with Oracle Big Data Lite Virtual Machine that was installed using the process documented at Big Data Lite with a static IP address on Oracle Linux 6 / EL6


Step 1: On the Solaris server, create an "oracle" user:

The user doesn't have to be "oracle", but that fits well with my planned usage.

# /usr/sbin/groupadd oinstall
# /usr/sbin/groupadd dba
# /usr/sbin/useradd -d localhost:/export/home/oracle -m -g oinstall -G dba oracle
# passwd oracle
# echo "oracle ALL=(ALL) ALL" > /etc/sudoers.d/oracle
# su - oracle

Set up a VNC server using the process at Installing VNC server on Solaris 11.

Step 2: Visit the Cloudera Manager to determine the version of Cloudera that is running:

  • Click on the "Support" pull down and the "About":


  • In my case, I was using Cloudera Express 5.1.2

Step 3: On the Oracle Solaris server, install the Hive and Hadoop tarballs that match your cluster

  • Choose your version. In my case, CDH 5.1.2.



  • Unpack the tarballs on your Solaris server:
$ tar xzf Downloads/hadoop-2.3.0-cdh5.1.2.tar.gz
$ tar xzf Downloads/hive-0.12.0-cdh5.1.2.tar.gz

  • And verify:
$ ls hadoop-2.3.0-cdh5.1.2
bin                  examples             libexec
bin-mapreduce1       examples-mapreduce1  sbin
cloudera             include              share
etc                  lib                  src

$ ls hive-0.12.0-cdh5.1.2/

bin                examples           LICENSE            RELEASE_NOTES.txt
conf               hcatalog           NOTICE             scripts
docs               lib                README.txt

Step 4: Download the HDFS configuration files to the Solaris server:
  • In the Cloudera Manager, go the the hdfs page:

* From the hdfs page, download the client configuration. Place the client configuration onto the Solaris server


  • Unpack the HDFS client configuration on your Solaris server:

$ unzip Downloads/
Archive:  Downloads/
  inflating: hadoop-conf/hdfs-site.xml 
  inflating: hadoop-conf/ 
  inflating: hadoop-conf/ 
  inflating: hadoop-conf/ 
  inflating: hadoop-conf/ 
  inflating: hadoop-conf/core-site.xml 

Step 6: Configure the HDFS client software on the Solaris server

  • Edit hadoop-conf/ set JAVA_HOME correctly:

export JAVA_HOME=/usr/jdk/instances/jdk1.7.0

  • Move the configuration files into place:

$ cp hadoop-conf/* hadoop-2.3.0-cdh5.1.2/etc/hadoop/

  • Add the Hadoop bin directory to your PATH. You may want to do this in your local shell or .bashrc

$ export PATH=~/hadoop-2.3.0-cdh5.1.2/bin:$PATH

Step 7: Test the HDFS client software on Solaris

  • Verify that the remote hdfs filesystem is visible from your Solaris server

$ hdfs dfs -ls
15/04/02 14:40:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 6 items
drwx------   - oracle oracle          0 2014-08-25 01:55 .Trash
drwx------   - oracle oracle          0 2014-09-23 09:25 .staging
drwxr-xr-x   - oracle oracle          0 2014-01-12 15:15 moviedemo
drwxr-xr-x   - oracle oracle          0 2014-09-24 05:38 moviework
drwxr-xr-x   - oracle oracle          0 2014-09-08 11:50 oggdemo
drwxr-xr-x   - oracle oracle          0 2014-09-20 09:59 oozie-oozi

Step 8: Download the Hive configuration files to the Solaris server:

  • In the Cloudera Manager, go the the hive page:

* From the hive page, download the client configuration. Place the client configuration on the Solaris server


  • Unpack the hive client configuration on your Solaris server:

$ unzip Downloads/
Archive:  Downloads/hive-clientconfig(1).zip
  inflating: hive-conf/hive-site.xml 
  inflating: hive-conf/  
  inflating: hive-conf/ 
  inflating: hive-conf/ 
  inflating: hive-conf/core-site.xml 
  inflating: hive-conf/mapred-site.xml 
  inflating: hive-conf/  
  inflating: hive-conf/yarn-site.xml 
  inflating: hive-conf/hdfs-site.xml 
  inflating: hive-conf/  

Step 9: Configure the Hive client software on the Solaris server

  • Move the configuration files into place:

$ cp hive-conf/* hive-0.12.0-cdh5.1.2/conf/

Step 10: YARN configuration option for Hadoop:

  • The HFDS configuration files don't include yarn-site.xml. Copy the YARN configuration file from the hive tree to the hadoop tree:

$ cp hive-0.12.0-cdh5.1.2/conf/yarn-site.xml hadoop-2.3.0-cdh5.1.2/etc/hadoop/

Step 11: Hide

$ mv hive-0.12.0-cdh5.1.2/conf/ hive-0.12.0-cdh5.1.2/conf/

  • Add the Hive bin directory to your PATH. You may want to do this in your local shell or .bashrc

$ export PATH=~/hive-0.12.0-cdh5.1.2/bin:$PATH

Step 12: Test the Hive client software on Solaris

  • Verify that the remote Hive tables are visible from your Solaris server

$ hive
15/04/03 12:20:37 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
15/04/03 12:20:37 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
15/04/03 12:20:37 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
15/04/03 12:20:37 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
15/04/03 12:20:37 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
15/04/03 12:20:37 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
15/04/03 12:20:37 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
15/04/03 12:20:37 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.

Logging initialized using configuration in jar:file:/home/oracle/hive-0.12.0-cdh5.1.2/lib/hive-common-0.12.0-cdh5.1.2.jar!/
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/oracle/hadoop-2.3.0-cdh5.1.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/oracle/hive-0.12.0-cdh5.1.2/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
hive> show tables;
Time taken: 1.799 seconds, Fetched: 9 row(s)

Conclusion: Successfully accomplished task of configuring an Oracle Solaris server to access Hive data that is stored in a Cloudera Hadoop cluster.

Friday Oct 24, 2014

Onion Security

This blog is part of the SPARC T5-4 RAC and WebLogic Cluster series: including:

I was asked why I used so many subnets in the system which is described in my blog entry SPARC T5-4 LDoms for RAC and WebLogic Clusters:
  • Management Subnet
  • HTTP Public Network
  • JDBC subnet
  • Storage subnet
The short answer: I didn't need to. I could have used one public network and one RAC private network.

Longer answer:
  • Better observability
  • Better isolation
  • It enables a better security model

Onion View

LDom View

If Joe Blackhat is able to compromise our HTTP server, that is a bad thing, but hopefully he will only be able access a subset of the data.  To get any additional data, he will need to request the data from the WebLogic server. The HTTP to WebLogic network layer can be monitored, firewalled, and logged. Again, if Joe Blackhat is able to penetrate one layer deeper, into the WebLogic layer, he will only be able to access additional data via JDBC calls to Oracle RAC. Again, the WebLogic to RAC network layer can be monitored, firewalled, and logged. And so forth...

In case it isn't obvious, the management network is intended to be used only infrequently by DBA's and System Administrators. This network should be tightly controlled and only enabled when system administration is required.

Monday Aug 25, 2014

Installing VNC server on Solaris 11

I tend to forget the exact command line for installing VNC on Solaris 11.

VNC isn't installed by default:

# vncserver
-bash: vncserver: command not found

But it is in the repository. (You need to use the --accept option because java/jre-7  requires BCL License acceptance )

# pkg install --accept solaris-desktop
           Packages to install: 463
           Mediators to change:   1
            Services to change:  16
       Create boot environment:  No
Create backup boot environment:  No
DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            463/463   68277/68277  729.0/729.0  6.2M/s

PHASE                                          ITEMS
Installing new actions                   104012/104012
Updating package state database                 Done
Updating package cache                           0/0
Updating image state                            Done
Creating fast lookup database                   Done
Updating package cache                           2/2
You have new mail in /var/mail/root

Now the VNC server is installed:

$ vncserver

You will require a password to access your desktops.

xauth:  file /home/jeff/.Xauthority does not exist

New 'my-host:1 (root)' desktop is my-host:1

Creating default startup script /home/jeff/.vnc/xstartup
Starting applications specified in /home/jeff/.vnc/xstartup
Log file is /home/jeff/.vnc/my-host:1.log

By default, you get the tiled window manager (twm), but I prefer Gnome.

# vi /home/jeff/.vnc/xstartup

At the bottom of xstartup, change "twm &" -to- "gnome-session &":

Kill the session running twm and restart with Gnome:

$ vncserver -kill :1
Killing Xvnc process ID 2392

$ vncserver

New 'my-host:1 (root)' desktop is my-host:1

Starting applications specified in /home/jeff/.vnc/xstartup
Log file is /home/jeff/.vnc/my-host:1.log

Now you're good to go. Visit the vncserver with a vncclient:


You may also want to review:

Hope this helps.

Wednesday Aug 20, 2014

Quadratic Programming with Oracle R Enterprise

     I wanted to use quadprog with ORE on a server running Oracle Solaris 11.2 on a Oracle SPARC T-4 server

For background, see:

Problem: path to Solaris Studio doesn't match my installation:

$ ORE CMD INSTALL quadprog_1.5-5.tar.gz
* installing to library \u2018/u01/app/oracle/product/12.1.0/dbhome_1/R/library\u2019
* installing *source* package \u2018quadprog\u2019 ...
** package \u2018quadprog\u2019 successfully unpacked and MD5 sums checked
** libs
/opt/SunProd/studio12u3/solarisstudio12.3/bin/f95 -m64   -PIC  -g  -c aind.f -o aind.o
bash: /opt/SunProd/studio12u3/solarisstudio12.3/bin/f95: No such file or directory
*** Error code 1
make: Fatal error: Command failed for target `aind.o'
ERROR: compilation failed for package \u2018quadprog\u2019
* removing \u2018/u01/app/oracle/product/12.1.0/dbhome_1/R/library/quadprog\u2019

$ ls -l /opt/solarisstudio12.3/bin/f95
lrwxrwxrwx   1 root     root          15 Aug 19 17:36 /opt/solarisstudio12.3/bin/f95 -> ../prod/bin/f90

Solution: a symbolic link:

$ sudo mkdir -p /opt/SunProd/studio12u3

$ sudo ln -s /opt/solarisstudio12.3 /opt/SunProd/studio12u3/

Now, it is all good:

$ ORE CMD INSTALL quadprog_1.5-5.tar.gz
* installing to library \u2018/u01/app/oracle/product/12.1.0/dbhome_1/R/library\u2019
* installing *source* package \u2018quadprog\u2019 ...
** package \u2018quadprog\u2019 successfully unpacked and MD5 sums checked
** libs
/opt/SunProd/studio12u3/solarisstudio12.3/bin/f95 -m64   -PIC  -g  -c aind.f -o aind.o
/opt/SunProd/studio12u3/solarisstudio12.3/bin/ cc -xc99 -m64 -I/usr/lib/64/R/include -DNDEBUG -KPIC  -xlibmieee  -c init.c -o init.o
/opt/SunProd/studio12u3/solarisstudio12.3/bin/f95 -m64  -PIC -g  -c -o solve.QP.compact.o solve.QP.compact.f
/opt/SunProd/studio12u3/solarisstudio12.3/bin/f95 -m64  -PIC -g  -c -o solve.QP.o solve.QP.f
/opt/SunProd/studio12u3/solarisstudio12.3/bin/f95 -m64   -PIC  -g  -c util.f -o util.o
/opt/SunProd/studio12u3/solarisstudio12.3/bin/ cc -xc99 -m64 -G -o aind.o init.o solve.QP.compact.o solve.QP.o util.o -xlic_lib=sunperf -lsunmath -lifai -lsunimath -lfai -lfai2 -lfsumai -lfprodai -lfminlai -lfmaxlai -lfminvai -lfmaxvai -lfui -lfsu -lsunmath -lmtsk -lm -lifai -lsunimath -lfai -lfai2 -lfsumai -lfprodai -lfminlai -lfmaxlai -lfminvai -lfmaxvai -lfui -lfsu -lsunmath -lmtsk -lm -L/usr/lib/64/R/lib -lR
installing to /u01/app/oracle/product/12.1.0/dbhome_1/R/library/quadprog/libs
** R
** preparing package for lazy loading
** help
*** installing help indices
  converting help for package \u2018quadprog\u2019
    finding HTML links ... done
    solve.QP                                html 
    solve.QP.compact                        html 
** building package indices
** testing if installed package can be loaded
* DONE (quadprog)


Here is an example from

> require(quadprog)
> Dmat <- matrix(0,3,3)
> diag(Dmat) <- 1
> dvec <- c(0,5,0)
> Amat <- matrix(c(-4,-3,0,2,1,0,0,-2,1),3,3)
> bvec <- c(-8,2,0)
> solve.QP(Dmat,dvec,Amat,bvec=bvec)
[1] 0.4761905 1.0476190 2.0952381

[1] -2.380952

[1] 0 5 0

[1] 3 0

[1] 0.0000000 0.2380952 2.0952381

[1] 3 2

Here, the standard example is modified to work with Oracle R Enterprise

ore.connect("my-name", "my-sid", "my-host", "my-pass", 1521)

  function () {

  function () {
    Dmat <- matrix(0,3,3)
    diag(Dmat) <- 1
    dvec <- c(0,5,0)
    Amat <- matrix(c(-4,-3,0,2,1,0,0,-2,1),3,3)
    bvec <- c(-8,2,0)

[1] 0.4761905 1.0476190 2.0952381

[1] -2.380952

[1] 0 5 0

[1] 3 0

[1] 0.0000000 0.2380952 2.0952381

[1] 3 2

Now I can combine the quadprog compute algorithms with the Oracle R Enterprise Database engine functionality:
  • Scale to large datasets
  • Access to tables, views, and external tables in the database, as well as those accessible through database links
  • Use SQL query parallel execution
  • Use in-database statistical and data mining functionality

Tuesday Jul 22, 2014

VNC Cut & Paste on Solaris 10

I love being able to cut & paste between my laptop (e-mail, web browser, etc.) and my VNC session. This functionality is controlled by vncconfig. From the man page:

vncconfig is used to configure and control a running
instance of Xvnc, or any other X server with the VNC exten-
sion. Note that it cannot be used to control VNC servers
prior to version 4.

I hate this message:

$ vncconfig
No VNC extension on display :1.0

When I hit the problem again, I found my way back to one of my old blog entries: Solaris/x64 VNC with Cut & Paste, but this time, the situation was slightly different:
  • On SPARC
  • SFWvnc was not installed
When I re-read my old blog, it didn't seem to apply.

But I found:

$ which Xvnc
/ usr/local/bin/Xvnc

$ pkginfo -L | grep vnc

$ /usr/sbin/pkgchk -l SUNWxvnc | grep Xvnc
NOTE: Couldn't lock the package database.
Pathname: /usr/X11/bin/Xvnc
Pathname: /usr/X11/share/man/man1/Xvnc.1

Someone had put Xvnc version 3.x in /usr/local. Because /usr/local was near the start of my PATH, I wasn't using the version of Xvnc that is supplied with Solaris 10 Update 10.

Again, the solution is simple.

1) Kill the running Xvnc process

2) Start a new Xvnc process using the Solaris supplied executable

$ export PATH=/usr/X11/bin:$PATH
$ vncviewer

3) Use a VNC viewer to visit the Xvnc server and run vncconfig:

$ vncconfig &


4) Now, cut & paste works between my laptop (e-mail, web browser, etc.) and my VNC session.

5) You will want to make sure that vncconfig is started automatically in your .vnc/xstartup file.

Hope this helps!

Tuesday Mar 18, 2014

What compiler options were used to create an executable?

Not stripped:

$ cc -O hello.c

$ file a.out

a.out: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, not stripped

$ strings -a a.out | grep cc

/opt/solarisstudio12.3/prod/bin/cc -O hello.c

$ dwarfdump -i a.out | grep compile_o

DW_AT_SUN_compile_options Xa;O;R=Sun C 5.12 SunOS_sparc Patch 148917-07 2013/10/18;backend;raw;cd;


$ strip a.out

$ file a.out

a.out: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, stripped

$ strings -a a.out | grep cc


Tuesday May 28, 2013

Adding users in Solaris 11 with power like the initial account

During Solaris 11.1 installation, the system administrator is prompted for a user name and password which will be used to create an unprivileged account. For security reasons, by default, root is a role, not a user, therefore the initial login can't be to a root account. The first login must use the initial unprivileged account. Later, the initial unprivileged user can acquire privileges through either "su" or "sudo".

For enterprise class deployments, the system administrator should be familiar with RBAC and create users with least privileges.

In contrast, I'm working in a lab environment and want to be able to simply and quickly create new users with power like the initial user. With Solaris 10, this was straight forward, but Solaris 11 adds a couple of twists.

Create a new user jenny in Solaris 11:

# useradd -d localhost:/export/home/jenny -m jenny
# passwd jenny

Jenny can't su to root:

jenny@app61:~$ su -
Roles can only be assumed by authorized users
su: Sorry

Because Jenny doesn't have that role:

jenny@app61:~$ roles
No roles

Give her the role:

root@app61:~# usermod -R root jenny

And then Jenny can su to root:

jenny@app61:~$ roles

jenny@app61:~$ su -
Oracle Corporation      SunOS 5.11      11.1    September 2012
You have new mail.

But even when jenny has the root role, she can't use sudo:

jenny@app61:~$ sudo -l
Sorry, user jenny may not run sudo on app61.

jenny@app61:~$ sudo touch /jenny
jenny is not in the sudoers file.  This incident will be reported.

Oh no, she is in big trouble, now.

User jeff was created as the initial account, and he can use sudo:

jeff@app61:~$ sudo -l
User jeff may run the following commands on this host:
    (ALL) ALL

But jeff isn't in the sudoers file:

root@app61:~# grep jeff /etc/sudoers

So how do you make jenny as powerful as jeff with respect to sudo?

Turns out that jeff, created during the Solaris installation, is in here:

root@app61:~# cat /etc/sudoers.d/svc-system-config-user
jeff ALL=(ALL) ALL

My coworker, Andrew, offers the following advice: "The last line of /etc/sudoers is a directive to read "drop-in" files from the /etc/sudoers.d directory. You can still edit /etc/sudoers. It may better to leave svc-system-config-user alone and create another drop-in file for local edits. If you want to edit sudoers as part of an application install then you should create a drop-in for the application - this makes the edits easy to undo if you remove the application. If you have multiple drop-ins in /etc/sudoers.d they are processed in alphabetical (sorted) order.  There are restrictions on file names and permissions for drop-ins. The permissions must be 440 (read only for owner and group) and the file name can't have a dot or ~ in it. These are in the very long man page."

Wednesday May 22, 2013

Debugging Hadoop using Solaris Studio in a Solaris 11 Zone

I've found Orgad Kimchi's How to Set Up a Hadoop Cluster Using Oracle Solaris Zones to be very useful, however, for a development environment, it is too complex. When map/reduce tasks are running in a clustered environment, it is challenging to isolate bugs. Debugging is easier when working within a standalone Hadoop installation. I've put the following instructions together for installation of a standalone Hadoop configuration in a Solaris Zone with Solaris Studio for application development.

A lovely feature of Solaris is that your global zone may host both a Hadoop cluster set up in a manner similar to Orgad's instructions and simultaneously host a zone for development that is running a Hadoop standalone configuration.

Create the Zone

These instructions assume that Solaris 11.1 is already running in the Global Zone.

Add the Hadoop Studio Zone

# dladm create-vnic -l net0 hadoop_studio

# zonecfg -z 
Use 'create' to begin configuring a new zone.
hadoop-studio> create
create: Using system default template 'SYSdefault'
hadoop-studio> set zonepath=/ZONES/hadoop-studio
hadoop-studio> add net
hadoop-studio:net> set physical=hadoop_studio
hadoop-studio:net> end
hadoop-studio> verify
hadoop-studio> commit
hadoop-studio> exit

Install and boot the zone

# zoneadm -z hadoop-studio install
# zoneadm -z 
hadoop-studio boot

Login to the zone console to set the network, time, root password, and unprivileged user.

# zlogin -C hadoop-studio

After the zone's initial configuration steps, nothing else needs to be done from within the global zone. You should be able to log into the Hadoop Studio zone with ssh as the unprivileged user and gain privileges with "su" and "sudo".

All of the remaining instructions are from inside the Hadoop Studio Zone.

Install extra Solaris software and set up the development environment

I like to start with both JDK's installed and not rely on the "/usr/java" symbolic link:

# pkg install  jdk-6
# pkg install --accept jdk-7

Verify the JDKs:

# /usr/jdk/instances/jdk1.6.0/bin/java -version
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) Server VM (build 20.10-b01, mixed mode)

# /usr/jdk/instances/jdk1.7.0/bin/java -version
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) Server VM (build 23.3-b01, mixed mode)

Add VNC Remote Desktop software

# pkg install --accept solaris-desktop

Create a Hadoop user:

# groupadd hadoop
# useradd -d localhost:/export/home/hadoop -m -g hadoop hadoop
# passwd hadoop
# usermod -R root hadoop

Edit /home/hadoop/.bashrc:

export PATH=/usr/bin:/usr/sbin
export PAGER="/usr/bin/less -ins"
typeset +x PS1="\u@\h:\w\\$ "

# Hadoop
export HADOOP_PREFIX=/home/hadoop/hadoop

# Java
export JAVA_HOME=/usr/jdk/instances/jdk1.6.0
export PATH=$JAVA_HOME/bin:$PATH

# Studio
export PATH=$PATH:/opt/solarisstudio12.3/bin
alias solstudio='solstudio --jdkhome /usr/jdk/instances/jdk1.6.0'

Edit /home/hadoop/.bash_profile:

. ~/.bashrc

And make sure that the ownership and permission make sense:

# ls -l /home/hadoop/.bash*      
-rw-r--r--   1 hadoop   hadoop        12 May 22 05:24 /home/hadoop/.bash_profile
-rw-r--r--   1 hadoop   hadoop       372 May 22 05:24 /home/hadoop/.bashrc

Now is a good time to a start remote VNC desktop for this zone:

# su - hadoop

$ vncserver

You will require a password to access your desktops.

xauth:  file /home/hadoop/.Xauthority does not exist

New '
hadoop-studio:1 ()' desktop is hadoop-studio:1

Creating default startup script /home/hadoop/.vnc/xstartup
Starting applications specified in /home/hadoop/.vnc/xstartup
Log file is /home/hadoop/.vnc/

Access the remote desktop with your favorite VNC client

The default 10 minute time out on the VNC desktop is too fast for my preferences:

System -> Preferences -> Screensaver
  Display Modes:
  Blank after: 100
  Close the window (I always look for a "save" button, but no, just close the window without explicitly saving.)

Download and Install Hadoop

For this article, I used the "12 October, 2012 Release 1.0.4" release. Download the Hadoop tarball and copy it into the home directory of hadoop:

$ ls -l hadoop-1.0.4.tar.gz
-rw-r--r--   1 hadoop   hadoop   62793050 May 21 12:03 hadoop-1.0.4.tar.gz

Unpack the tarball into the home directory of the hadoop user:

$ gzip -dc hadoop-1.0.4.tar.gz  | tar -xvf -
$ mv hadoop-1.0.4 hadoop

Hadoop comes pre-configured in Standalone Mode

Edit /home/hadoop/hadoop/conf/, and set JAVA_HOME:

export JAVA_HOME=/usr/jdk/instances/jdk1.6.0

That is all. Now, you can run a Hadoop example:

$ hadoop jar hadoop/hadoop-examples-1.0.4.jar pi 2 10
Number of Maps  = 2
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Starting Job
Job Finished in 10.359 seconds
Estimated value of Pi is 3.80000000000000000000

Install Solaris Studio:

Visit to obtain Oracle_Solaris_Studio_Support.key.pem, Oracle_Solaris_Studio_Support.certificate.pem and follow the instructions for "pkg set-publisher" and "pkg update" or "pkg install"

# sudo pkg set-publisher \
          -k /var/pkg/ssl/Oracle_Solaris_Studio_Support.key.pem \
          -c /var/pkg/ssl/Oracle_Solaris_Studio_Support.certificate.pem \
          -G '*' -g solarisstudio

# pkg install developer/solarisstudio-123/*

If your network requires a proxy, you will need set the proxy before starting Solaris Studio:


Start Solaris Studio:

$ solstudio

(Notice the alias in .bashrc that adds --jdkhome to the solstudio start up command.)

Go to "Tools -> Plugins.

Click on "Reload Catalog"

Load the Java SE plugins. I ran into a problem when the Maven plug in was installed. Something that I should diagnose at a future date.


Create a New Project:

File -> New Project

Step 1:
- Catagory: Java
- Project: Java Application
- Next


Step 2: Fill in similar to this:


Copy the example source into the project:

$ cp -r \
    $HADOOP_PREFIX/src/examples/org/apache/hadoop/examples/* \

Starting to look like a development environment:


Modify the Project to compile with Hadoop jars. Right-click on the project and select "Properties"


Add in the necessary Hadoop compile jars:


I found that I needed these jars at run time:


Add Program Arguments (2 10):


Now, if you click on the "Run" button. PiEstimators will run inside the IDE:


And the set up behaves as expected if you set a break point and click on "Debug":


Tuesday May 21, 2013

non-interactive zone configuration

When creating new Solaris zones, at initial boot up, the system administrator is prompted for the new hostname, network settings, etc of the new zone. I get tired of the brittle process of manually entering the initial settings and I prefer to be able to automate the process. I had previously figured out the process for Solaris 10, but I've only recently figured out the process for Solaris 11.

As a review, with Solaris 10, use your favorite editor to create a sysidcfg file:

network_interface=primary {
name_service=DNS {

1) Solaris 10 Install: Using sysidcfg to avoid answering the configuration questions in a newly installed zone:

After the "zoneadm -z app-41 install" you can copy the sysidcfg file to "/ZONES/app-41/root/etc/sysidcfg"  (assuming your "zonepath" is "/ZONES/app-41") and the initial boot process will read the settings from the file and not prompt the system administrator to manually enter the settings.

2) Solaris 10 Clone: Using sysidcfg when cloning the zone 

I used a similar trick on Solaris 10 when cloning old zone "app-41 to new zone "app-44":

#  zonecfg -z app-41 export | sed -e 's/app-41/app-44/g' | zonecfg -z app-44
#  zoneadm -z app-44 clone app-41
#  cat
/ZONES/app-41/root/etc/sysidcfg | sed -e 's/app-41/app-44/g' > /ZONES/app-44/root/etc/sysidcfg
#  zoneadm -z app-44 boot

With Solaris 11, instead of a small human readable file which will containing the configuration information, the information is contained in an XML file that would be difficult to create using an editor. Instead, create the initial profile by executing "sysconfig":

# sysconfig create-profile -o sc_profile.xml
# mkdir /root/profiles/app-61
# mv sc_profile.xml /root/profiles/app-6

The new XML format is longer so I won't include it in this blog entry and it is left as an exercise for the reader to review the file that has been created.

1) Solaris 11 Install

# dladm create-vnic -l net0 app_61

# zonecfg -z app-61
Use 'create' to begin configuring a new zone.
zonecfg:p3231-zone61> create
create: Using system default template 'SYSdefault'
zonecfg:app-61> set zonepath=/ZONES/app-61
zonecfg:app-61> add net
zonecfg:app-61:net> set physical=app_61
zonecfg:app-61:net> end
zonecfg:app-61> verify
zonecfg:app-61> commit
zonecfg:app-61> exit

# zoneadm -z app-61 install -c /root/profiles/app-61
# zoneadm -z app-61 boot
# zlogin -C app-61

2) Solaris 11 Clone: If you want to clone app-61 to app-62 and have an existing sc_profile.xml, you can re-use most of the settings and only adjust what has changed:

# dladm create-vnic -l net0 app_62

# zoneadm -z app-61 halt

# mkdir /root/profiles/app-62

# sed \
-e 's/app-61/app-62/g' \
-e 's/app_61/app_62/g' \
-e 's/' \
< /root/profiles/app-61/sc_profile.xml \
> /root/profiles/app-62/sc_profile.xml

# zonecfg -z app-61 export | sed -e 's/61/62/g' | zonecfg -z app-62

# zoneadm -z app-62 clone -c /root/profiles/app-62 app-61
# zoneadm -z app-62 boot
# zlogin -C app-62

I hope this trick saves you some time and makes your process less brittle.

Thursday May 16, 2013

Hadoop Java Error logs

I was having trouble isolating a problem with "reduce" tasks running on Hadoop slave servers. 

After poking around on the Hadoop slave, I found an interesting lead in /var/log/hadoop/userlogs/job_201302111641_0057/attempt_201302111641_0057_r_000001_1/stdout:

$ cat /tmp/hadoop-hadoop/mapred/local/userlogs/job_201302111641_0059/attempt_201302111641_0059_r_000001_1/stdout

# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0xfe67cb31, pid=25828, tid=2
# JRE version: 6.0_35-b10
# Java VM: Java HotSpot(TM) Server VM (20.10-b01 mixed mode solaris-x86 )
# Problematic frame:
# C  []  pthread_mutex_trylock+0x29
# An error report file with more information is saved as:
# /tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201302111641_0059/attempt_201302111641_0059_r_000001_1/work/hs_err_pid25828.log
# If you would like to submit a bug report, please visit:
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

The HotSpot crash log (hs_err_pid25828.log in my case) will be very interesting because it contains information obtained at the time of the fatal error, including the following information, where possible:

  • The operating exception or signal that provoked the fatal error
  • Version and configuration information
  • Details on the thread that provoked the fatal error and thread's stack trace
  • The list of running threads and their state
  • Summary information about the heap
  • The list of native libraries loaded
  • Command line arguments
  • Environment variables
  • Details about the operating system and CPU

Great, but hs_err_pid25654.log had been cleaned up before I could get to it. In fact, I found that the hs_err_pid.log files were available for less than a minute and they were always gone before I could capture one.

To try to retain the Java error log file, my first incorrect guess was:


My next approach was to add "-XX:ErrorFile=/tmp/hs_err_pid%p.log" to the Java command line for the reduce task.

When I tried adding the Java option to HADOOP_OPTS in /usr/local/hadoop/conf/, I realized that this setting isn't applied to the Map and Reduce Task JVMs.

Finally, I found that adding the Java option to the property in mapred-site.xml WORKED!!

$ cat /usr/local/hadoop/conf/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->





Now I can view the Java error logs on my Hadoop slaves:

$ ls -l /tmp/*err*
-rw-r--r--   1 hadoop   hadoop     15626 May 16 15:42 /tmp/hs_err_pid10028.log
-rw-r--r--   1 hadoop   hadoop     15795 May 16 15:43 /tmp/hs_err_pid10232.log

Tuesday Dec 04, 2012

Solaris 11 VNC Server is "blurry" or "smeared"

I've been annoyed by quality of the image that is displayed by my VNC viewer when I visit a Solaris 11 VNC server. How should I describe the image? Blurry? Grainy? Smeared? Low resolution? Compressed? Badly encoded?

This is what I have gotten used to seeing on Solaris 11:

Solaris 11 Blurry VNC image

This is not a problem for me when I view Solaris 10 VNC servers. I've finally taken the time to investigate, and the solution is simple. On the VNC client, don't allow "Tight" encoding.

My VNC Viewer will negotiate to Tight encoding if it is available. When negotiating with the Solaris 10 VNC server, Tight is not a supported option, so the Solaris 10 server and my client will agree on ZRLE. 

Now that I have disabled Tight encoding on my VNC client, the Solaris 11 VNC Servers looks much better:

Solaris 11 crisp VNC image

How should I describe the display when my VNC client is forced to negotiate to ZRLE encoding with the Solaris 11 VNC Server? Crisp? Clear? Higher resolution? Using a lossless compression algorithm?

When I'm on a low bandwidth connection, I may re-enable Tight compression on my laptop. In the mean time, the ZRLE compression is sufficient for a coast-to-coast desktop, through the corporate firewall, encoded with VPN, through my ISP and onto my laptop. YMMV.


Jeff Taylor-Oracle


« May 2015