X

An Oracle blog about Oracle Enterprise Manager and Oracle Management Cloud

Simplify deployment of JVMD Agents to command line Java applications

Contributing
Author: Shiraz Kanga, Consulting Member of Technical Staff, Oracle

Most customers of Oracle Enterprise Manager using JVM Diagnostics use the tool to monitor their Java Applications servers like Weblogic, Websphere, Tomcat, etc. In this environment it is fairly easy to deploy the JVMD Agent. Since it is distributed as a war file, you merely deploy the agent into a running application server using the management GUI or command line tools. Then you can start monitoring with no need for a restart of the app server or for the modification of any startup commands or scripts. However, with other types of Java applications that do not allow for any code deployment at runtime such as AWT/Swing or command line java applications these steps are necessary. Modifying startup scripts is complex because each application comes with its own custom and unique launch script. Additionally, the command that actually launches the runtime needs to have the java command with its related parameters (like -Xmx) the JVMD Agent with its own parameters (like console host/port) and the application itself which may have some more custom parameters. People often get confused due to the complexity that is seen here.

I've recently had customers that needed to monitor Hadoop, HDFS, Zookeeper, Kafka, Cassandra and Solr with JVMD. In order to simplify some of the complexity discussed above, I created a simple script based framework that makes things a bit easier. Feel free to
use my approach to quickly setup JVMD with these or any other command line java programs. You can also use it as the basis for your own modifications. The framework modifies the startup scripts supplied with these tools in order to add the JVMD agent. All the code/scripts are attached in a zip file. Both original and modified versions of all changed scripts are included so you can easily see the modifications I made with a simple diff.

Here's how these scripts are setup. Everything is configured using 4 environment variables as shown
below:

    export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3800
export JVMD_UNIQUE_ID=<unique name for each server process>

where the JVMD_AGENT_HOME must contain the jamagent-env.sh (from the attached zip file) and
jamagent.war (which can be downloaded from your JVMD console). The first three of these are likely to remain unchanged for all the JVMs being monitored so you can easily add them directly into jamagent-env.sh if needed.

The JVMD_UNIQUE_ID will always be unique so it must not be placed there. However it has two other modes where you can use a pointer to the unique ID instead of specifying it directly. You can point to either an environment variable or to a JVM system property that holds the actual unique ID. If you are using these cases then you could add this one to the jamagent-env.sh script too.

If JVMD_UNIQUE_ID starts with the string "sysprop-" then the actual unique
ID will be read from the JVM system property named by the string
following "sysprop-". For example if JVMD_UNIQUE_ID is
"sysprop-server_name" and we have a system property -Dserver_name=MyTestingServer then
JVMD will use MyTestingServer as the JVM unique identifier.

If JVMD_UNIQUE_ID starts with the string "envvar-" then the actual unique
ID will be read from the environment variable named by the string
following "envvar-". For example if JVMD_UNIQUE_ID is
"envvar-server_name" and we have an environment variable called
server_name=MyTestingServer then JVMD will use MyTestingServer as
the JVM unique identifier.

Caution: Do not use dash (minus) character in the environment
variable setup of unique id. Use underscore instead.

Generic Launch Script Modifications

After these four environment variables are set we need to modify our launch scripts. Make sure you have a backup of all files before you proceed. In the main script that you use to launch your java application look for a line that has a format that is similar to the one below: 
    $JAVA $JAVA_OPTS $MAIN_CLASS $MAIN_CLASS_ARGS

and replace it with
    $JAVA $JAVA_OPTS $JVMD_AGENT_INSERT $MAIN_CLASS $MAIN_CLASS_ARGS

So we simply added a $JVMD_AGENT_INSERT just before the name of the Main class. If there are multiple such lines then you should modify them all in the same way. And in order to configure $JVMD_AGENT_INSERT we also need to source
jamagent-env.sh (with some error checking). So we insert a snippet like this in the line just
before the JAVA invocation. 

# add JVMD Agent Env settings
[[ -e "${JVMD_AGENT_HOME}/jamagent-env.sh" ]]
&& source "${JVMD_AGENT_HOME}/jamagent-env.sh" ||
{ echo "ERROR: JVMD_AGENT_HOME undefined or does not contain jamagent-env.sh" 1>&2 ; exit 1; } 

NOTE: Everything after the comment above should in a single line of code in your launch script. This line gets mangled by the blogging software so it is best to cut & paste it from it from one of the scripts in the attached zip file.

We will now look at how I used these techniques to add JVMD monitoring to Kafka, Hadoop, Zookeeper, Cassandra and Solr. 

1) Kafka 2.8.0-0.8.1.1

I used Kafka 2.8.0-0.8.1.1 and downloaded it directly from the Kafka site. In Kafka, ALL processes are initiated through a
common launcher called kafka-run-class.sh in the bin
folder. All the other shell scripts (including the built-in Zookeeper) call this one. So this single insertion point is the only
place that we will need to modify in order to add JVMD monitoring to
Kafka. Pretty simple. Using the modified script (inside the attached zip file) you can run the servers as shown below:

TEST - with mods to use JVMD
cd /home/skanga/servers/kafka_2.8.0-0.8.1.1/bin
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3800
# start a zookeeper server
export JVMD_UNIQUE_ID=zookeeper-server
./zookeeper-server-start.sh ../config/zookeeper.properties
# start a kafka server
export JVMD_UNIQUE_ID=kafka-server
./kafka-server-start.sh ../config/server.properties


2) Hadoop 2.4.1

The scripts called hadoop, hfds, mapred and yarn
in the hadoop bin directory will ALL need to be modified for JVMD
monitoring. Using the modified scripts (inside the attached zip file) you can run all the servers as shown below:


TEST - with mods for hadoop command to use JVMD

cd /home/skanga/servers/hadoop-2.4.1
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3802
# Launch the hdfs nfs gateway
export JVMD_UNIQUE_ID=hdfs-nfs3-gateway
./bin/hdfs nfs3
# Run a mapreduce history server
export JVMD_UNIQUE_ID=mapred-historyserver
./bin/mapred historyserver
# Run a yarn resource manager
export JVMD_UNIQUE_ID=yarn-resourcemanager
./bin/yarn resourcemanager
# Run a hadoop map-reduce job to find the value of PI (QuasiMonteCarlo method)
export JVMD_UNIQUE_ID=hadoop-test-pi-montecarlo
./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar pi 1024 100


3) Zookeeper 3.4.6

The standalone version of zookeeper has a
common environment setup script called zkEnv.sh where most JVMD
setup can be done. After that a minor modification is needed in the java launch command in zkServer.sh after which all JVMD monitoring works fine. The scripts
called zkCleanup.sh and zkCli.sh probably do not need monitoring but
can be easily added if really needed.

TEST - with mods for zkServer.sh command to use JVMD

cd /home/skanga/servers/zookeeper-3.4.6/bin
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3800
export JVMD_UNIQUE_ID=zk-server
# start the zookeeper server
./zkServer.sh start
./zkServer.sh status
./zkServer.sh stop

4) Cassandra 2.0.9

The Apache Cassandra data store has a
common environment setup script called conf/cassandra-env.sh where we can add the command to source our include script. Then a minor modification is needed to the java launch command in
bin/cassandra after which all JVMD monitoring works fine. The other scripts probably do not need monitoring but
can be easily added if really needed. 


TEST - with mods for cassandra command to use JVMD

cd /home/skanga/servers/apache-cassandra-2.0.9/bin
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3800
export JVMD_UNIQUE_ID=cassandra-server
# start cassandra
./cassandra -f

5) Solr 4.9.0

The Solr search server is an interesting case. In production scenarios, users will probably use the Solr war file in their own application server. In this scenario the standard JVMD warfile can be deployed to the same application server and monitored easily. However, the Solr distribution also include an embedded mode which may be used by simply running java -jar start.jar and for this scenario we have converted this java command into a simple script called start.sh and added it to the same folder as start.jar in order to run it. Using this script (inside the attached zip file) you can run a test as shown below:

TEST - with addition of start.sh command to use JVMD with Solr

cd /home/skanga/servers/solr-4.9.0/example
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3800
export JVMD_UNIQUE_ID=solr-server
# start solr
./start.sh

After everything is setup properly for your servers you should see all the relevant JVMs in the default pool with the proper ID as shown in the image below.


JVMs in Default Pool (with hostnames & ip addresses blanked out)
Click image to expand it in a new tab

Remember to be a bit patient and wait a few seconds until the connections are established and the servers appear in the console.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.