X

Jeff Taylor's Weblog

CDH 5.5 hive and hadoop start up problems on Solaris

This was a bit of a puzzler that I don't think that many other people
will have to deal with, but I thought that it was interesting:

% hive
/opt/cloudera/cdh5.5.0/hive-1.1.0-cdh5.5.0/bin/hive: line 184: [: too many arguments
Usage: dirname [ path ]

/opt/cloudera/cdh5.5.0/hadoop-2.6.0-cdh5.5.0/bin/hadoop: line 26: /
opt/cloudera/cdh5.5.0/hadoop-2.6.0-cdh5.5.0/libexec/hadoop-config.sh: No such file or directory
/opt/cloudera/cdh5.5.0/hadoop-2.6.0-cdh5.5.0/bin/hadoop: line 144: exec: : not found
Unable to determine Hadoop version information.
'hadoop version' returned:
Usage: dirname [ path ]
/opt/cloudera/cdh5.5.0/hadoop-2.6.0-cdh5.5.0/bin/hadoop: line 26: /opt/
cloudera/cdh5.5.0/hadoop-2.6.0-cdh5.5.0/libexec/hadoop-config.sh: No such file or directory
/opt/cloudera/cdh5.5.0/hadoop-2.6.0-cdh5.5.0/bin/hadoop: line 144: exec: : not found

%


What?? But the hive start up script used to work!! It works when
logged in as another user!! It works when run on Linux!! What went
wrong?


A little detective work showed this:

% bash -x hive

...

'[' -f Running .cshrc /opt/cloudera/cdh5.5.0/hadoop-2.6.0-cdh5.5.0/bin/hadoop ']'

...

OK, so someone was just trying to debug .cshrc. Lets try  running the script from a bash shell ... same problem!! Additionally, the hive startup script should be
running with bash in any case because it has this at the top:

#!/usr/bin/env bash


It turned out that the HADOOP_IN_PATH variable had the Running .cshrc string.

HADOOP_IN_PATH=`which hadoop 2>/dev/null`

if [ -f ${HADOOP_IN_PATH} ]; then

  HADOOP_DIR=`dirname "$HADOOP_IN_PATH"`/..

fi

Because on Solaris "which" is a cshell script.

$ head -3 /bin/which
#! /usr/bin/csh -f
#
# Copyright 2005 Sun Microsystems, Inc.  All rights reserved.


This was OK on Linux because "which" is an executable, not a cshell script:

$ file /usr/bin/which
/usr/bin/which: ELF 64-bit LSB executable, AMD x86-64,
version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared
libs), stripped



One solution on Solaris is to use the "GNU which" program "gwhich" in the hive and hadoop startup scripts.


Or, of course, solve the problem by taking the echos out of .cshrc.

Join the discussion

Comments ( 1 )
  • Mayur Reddy Monday, April 25, 2016

    This always had me baffled! Thanks for taking the time out and chasing it down to its root :)


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.