Thursday Sep 11, 2014

Simplify deployment of JVMD Agents to command line Java applications

Contributing Author: Shiraz Kanga, Consulting Member of Technical Staff, Oracle

Most customers of Oracle Enterprise Manager using JVM Diagnostics use the tool to monitor their Java Applications servers like Weblogic, Websphere, Tomcat, etc. In this environment it is fairly easy to deploy the JVMD Agent. Since it is distributed as a war file, you merely deploy the agent into a running application server using the management GUI or command line tools. Then you can start monitoring with no need for a restart of the app server or for the modification of any startup commands or scripts. However, with other types of Java applications that do not allow for any code deployment at runtime such as AWT/Swing or command line java applications these steps are necessary. Modifying startup scripts is complex because each application comes with its own custom and unique launch script. Additionally, the command that actually launches the runtime needs to have the java command with its related parameters (like -Xmx) the JVMD Agent with its own parameters (like console host/port) and the application itself which may have some more custom parameters. People often get confused due to the complexity that is seen here.

I've recently had customers that needed to monitor Hadoop, HDFS, Zookeeper, Kafka, Cassandra and Solr with JVMD. In order to simplify some of the complexity discussed above, I created a simple script based framework that makes things a bit easier. Feel free to use my approach to quickly setup JVMD with these or any other command line java programs. You can also use it as the basis for your own modifications. The framework modifies the startup scripts supplied with these tools in order to add the JVMD agent. All the code/scripts are attached in a zip file. Both original and modified versions of all changed scripts are included so you can easily see the modifications I made with a simple diff.

Here's how these scripts are setup. Everything is configured using 4 environment variables as shown below:

    export JVMD_AGENT_HOME=/home/skanga/servers
    export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
    export JVMD_MANAGER_PORT=3800
    export JVMD_UNIQUE_ID=<unique name for each server process>

where the JVMD_AGENT_HOME must contain the jamagent-env.sh (from the attached zip file) and jamagent.war (which can be downloaded from your JVMD console). The first three of these are likely to remain unchanged for all the JVMs being monitored so you can easily add them directly into jamagent-env.sh if needed.

The JVMD_UNIQUE_ID will always be unique so it must not be placed there. However it has two other modes where you can use a pointer to the unique ID instead of specifying it directly. You can point to either an environment variable or to a JVM system property that holds the actual unique ID. If you are using these cases then you could add this one to the jamagent-env.sh script too.

If JVMD_UNIQUE_ID starts with the string "sysprop-" then the actual unique ID will be read from the JVM system property named by the string following "sysprop-". For example if JVMD_UNIQUE_ID is "sysprop-server_name" and we have a system property -Dserver_name=MyTestingServer then JVMD will use MyTestingServer as the JVM unique identifier.

If JVMD_UNIQUE_ID starts with the string "envvar-" then the actual unique ID will be read from the environment variable named by the string following "envvar-". For example if JVMD_UNIQUE_ID is "envvar-server_name" and we have an environment variable called server_name=MyTestingServer then JVMD will use MyTestingServer as the JVM unique identifier.

Caution: Do not use dash (minus) character in the environment variable setup of unique id. Use underscore instead.

Generic Launch Script Modifications

After these four environment variables are set we need to modify our launch scripts. Make sure you have a backup of all files before you proceed. In the main script that you use to launch your java application look for a line that has a format that is similar to the one below: 
    $JAVA $JAVA_OPTS $MAIN_CLASS $MAIN_CLASS_ARGS
and replace it with
    $JAVA $JAVA_OPTS $JVMD_AGENT_INSERT $MAIN_CLASS $MAIN_CLASS_ARGS

So we simply added a $JVMD_AGENT_INSERT just before the name of the Main class. If there are multiple such lines then you should modify them all in the same way. And in order to configure $JVMD_AGENT_INSERT we also need to source jamagent-env.sh (with some error checking). So we insert a snippet like this in the line just before the JAVA invocation. 

# add JVMD Agent Env settings
[[ -e "${JVMD_AGENT_HOME}/jamagent-env.sh" ]] 
&& source "${JVMD_AGENT_HOME}/jamagent-env.sh" ||
{ echo "ERROR: JVMD_AGENT_HOME undefined or does not contain jamagent-env.sh" 1>&2 ; exit 1; } 

NOTE: Everything after the comment above should in a single line of code in your launch script. This line gets mangled by the blogging software so it is best to cut & paste it from it from one of the scripts in the attached zip file.

We will now look at how I used these techniques to add JVMD monitoring to Kafka, Hadoop, Zookeeper, Cassandra and Solr. 

1) Kafka 2.8.0-0.8.1.1

I used Kafka 2.8.0-0.8.1.1 and downloaded it directly from the Kafka site. In Kafka, ALL processes are initiated through a common launcher called kafka-run-class.sh in the bin folder. All the other shell scripts (including the built-in Zookeeper) call this one. So this single insertion point is the only place that we will need to modify in order to add JVMD monitoring to Kafka. Pretty simple. Using the modified script (inside the attached zip file) you can run the servers as shown below:

TEST - with mods to use JVMD
cd /home/skanga/servers/kafka_2.8.0-0.8.1.1/bin
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3800

# start a zookeeper server
export JVMD_UNIQUE_ID=zookeeper-server
./zookeeper-server-start.sh ../config/zookeeper.properties

# start a kafka server
export JVMD_UNIQUE_ID=kafka-server
./kafka-server-start.sh ../config/server.properties

2) Hadoop 2.4.1

The scripts called hadoop, hfds, mapred and yarn in the hadoop bin directory will ALL need to be modified for JVMD monitoring. Using the modified scripts (inside the attached zip file) you can run all the servers as shown below:

TEST - with mods for hadoop command to use JVMD

cd /home/skanga/servers/hadoop-2.4.1
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3802

# Launch the hdfs nfs gateway
export JVMD_UNIQUE_ID=hdfs-nfs3-gateway
./bin/hdfs nfs3

# Run a mapreduce history server
export JVMD_UNIQUE_ID=mapred-historyserver
./bin/mapred historyserver

# Run a yarn resource manager
export JVMD_UNIQUE_ID=yarn-resourcemanager
./bin/yarn resourcemanager

# Run a hadoop map-reduce job to find the value of PI (QuasiMonteCarlo method)
export JVMD_UNIQUE_ID=hadoop-test-pi-montecarlo
./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar pi 1024 100

3) Zookeeper 3.4.6

The standalone version of zookeeper has a common environment setup script called zkEnv.sh where most JVMD setup can be done. After that a minor modification is needed in the java launch command in zkServer.sh after which all JVMD monitoring works fine. The scripts called zkCleanup.sh and zkCli.sh probably do not need monitoring but can be easily added if really needed.

TEST - with mods for zkServer.sh command to use JVMD

cd /home/skanga/servers/zookeeper-3.4.6/bin
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3800
export JVMD_UNIQUE_ID=zk-server

# start the zookeeper server
./zkServer.sh start
./zkServer.sh status
./zkServer.sh stop

4) Cassandra 2.0.9

The Apache Cassandra data store has a common environment setup script called conf/cassandra-env.sh where we can add the command to source our include script. Then a minor modification is needed to the java launch command in bin/cassandra after which all JVMD monitoring works fine. The other scripts probably do not need monitoring but can be easily added if really needed. 

TEST - with mods for cassandra command to use JVMD

cd /home/skanga/servers/apache-cassandra-2.0.9/bin
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3800
export JVMD_UNIQUE_ID=cassandra-server

# start cassandra
./cassandra -f

5) Solr 4.9.0

The Solr search server is an interesting case. In production scenarios, users will probably use the Solr war file in their own application server. In this scenario the standard JVMD warfile can be deployed to the same application server and monitored easily. However, the Solr distribution also include an embedded mode which may be used by simply running java -jar start.jar and for this scenario we have converted this java command into a simple script called start.sh and added it to the same folder as start.jar in order to run it. Using this script (inside the attached zip file) you can run a test as shown below:

TEST - with addition of start.sh command to use JVMD with Solr

cd /home/skanga/servers/solr-4.9.0/example
export JVMD_AGENT_HOME=/home/skanga/servers
export JVMD_MANAGER_HOST=jvmdconsole.us.oracle.com
export JVMD_MANAGER_PORT=3800
export JVMD_UNIQUE_ID=solr-server

# start solr
./start.sh

After everything is setup properly for your servers you should see all the relevant JVMs in the default pool with the proper ID as shown in the image below.


JVMs in Default Pool (with hostnames & ip addresses blanked out)
Click image to expand it in a new tab

Remember to be a bit patient and wait a few seconds until the connections are established and the servers appear in the console.

Monday Mar 17, 2014

Installing a JDBC patch to an EM agent

If you have an Enterprise Manager 12c Release 3 or older agent monitoring database targets, Support may recommend that you install a JDBC (Java database connectivity) patch, such as patch 17591700, to prevent high CPU consumption by the agent.


JDBC patches, including 17591700, have readme files containing instructions for installing the patches to a database, not to an EM agent. This post provides an example of how to install a JDBC patch to an EM agent. It walks through the steps of installing patch 17591700 to an EM12c Release3 agent.


Here are the steps:

1.  Identify the version of the JDBC client in the Agent Binaries home.

  • Set the ORACLE_HOME environment variable to the Agent Binaries home.

$ setenv ORACLE_HOME /u01/em12/core/12.1.0.3.0


Note: One way to find out the Agent Binaries home is to look in file /etc/oragchomelist on the agent host. It should contain an entry for an agent install in the format of:

<Agent Binaries home>:<Agent home>

For example, file /etc/oragchomelist contains:

/u01/em12/core/12.1.0.3.0:/u01/em12/agent_inst



  • Run the following command to identify the version of the JDBC client.

$ $ORACLE_HOME/OPatch/opatch lsinv -details | grep 'Oracle JDBC/OCI Instant Client'
Oracle JDBC/OCI Instant Client           11.1.0.7.0

In this case, the version of the JDBC client is 11.1.0.7.0.


2.  Download the patch from MOS (My Oracle Support) website. The version of the JDBC client should match the version of the database for which the patch is intended. Stage the patch zip file, p17591700_111070_Generic.zip, on the agent host. I stage the patch in directory /u01/stage/jdbc_patch. I will refer to this location as the patch stage directory.


3.  Go to the patch stage directory and extract files from the zip file.

$ cd /u01/stage/jdbc_patch

$ unzip p17591700_111070_Generic.zip

Archive: p17591700_111070_Generic.zip

creating: 17591700/

inflating: 17591700/README.txt

creating: 17591700/files/

creating: 17591700/files/jdbc/

creating: 17591700/files/jdbc/lib/

inflating: 17591700/etc/xml/GenericActions.xml

inflating: 17591700/etc/xml/ShiphomeDirectoryStructure.xml


4.  Stop the agent.

$ /u01/em12/agent_inst/bin/emctl stop agent

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.

Stopping agent ..... stopped.


5.  Install the patch.

  • Go to the <patch stage directory>/<patch number>.

$ cd /u01/stage/jdbc_patch/17591700

  • Run opatch apply command.

$ $ORACLE_HOME/OPatch/opatch apply

Oracle Interim Patch Installer version 11.1.0.10.0

Copyright (c) 2013, Oracle Corporation. All rights reserved.

Oracle Home : /u01/em12/core/12.1.0.3.0

Central Inventory : /u01/app/oraInventory

from : /u01/em12/core/12.1.0.3.0/oraInst.loc

OPatch version : 11.1.0.10.0

OUI version : 11.1.0.11.0

Log file location : /u01/em12/core/12.1.0.3.0/cfgtoollogs/opatch/17591700_Mar_17_2014_12_03_05/apply2014-03-17_12-03-05PM_1.log

Applying interim patch '17591700' to OH '/u01/em12/core/12.1.0.3.0'

Verifying environment and performing prerequisite checks...

Patch 17591700: Optional component(s) missing : [ oracle.dbjava.jdbc, 11.1.0.7.0 ]

Interim patch 17591700 is a superset of the patch(es) [ 16087066 ] in the Oracle Home

OPatch will roll back the subset patches and apply the given patch.

All checks passed.

Backing up files...

Rolling back interim patch '16087066' from OH '/u01/em12/core/12.1.0.3.0'

Patching component oracle.dbjava.ic, 11.1.0.7.0...

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/aq/AQNotificationEvent$EventType.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/aq/AQNotificationEvent.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/dcn/DatabaseChangeEvent$AdditionalEventType.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/dcn/DatabaseChangeEvent$EventType.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/dcn/DatabaseChangeEvent.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/driver/NTFAQEvent.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/driver/NTFConnection.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/driver/NTFDCNEvent.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/driver/NTFManager.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/driver/T4CConnection.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc6.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc6.jar/oracle/jdbc/driver/T4CTTIokpn.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/aq/AQNotificationEvent$EventType.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/aq/AQNotificationEvent.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/dcn/DatabaseChangeEvent$AdditionalEventType.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/dcn/DatabaseChangeEvent$EventType.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/dcn/DatabaseChangeEvent.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/driver/NTFAQEvent.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/driver/NTFConnection.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/driver/NTFDCNEvent.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/driver/NTFManager.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/driver/T4CConnection.class"

Updating jar file "/u01/em12/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar" with "/u01/em12/core/12.1.0.3.0/.patch_storage/16087066_Feb_4_2013_04_52_18/files//jdbc/lib/ojdbc5.jar/oracle/jdbc/driver/T4CTTIokpn.class"

RollbackSession removing interim patch '16087066' from inventory

OPatch back to application of the patch '17591700' after auto-rollback.

Patching component oracle.dbjava.ic, 11.1.0.7.0...

Verifying the update...

Patch 17591700 successfully applied

Log file location: /u01/em12/core/12.1.0.3.0/cfgtoollogs/opatch/17591700_Mar_17_2014_12_03_05/apply2014-03-17_12-03-05PM_1.log

OPatch succeeded


6. Start the agent.

$ /u01/em12/agent_inst/bin/emctl start agent

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.

Starting agent ............. started.


The above step completes the process of installing patch 17591700 to the 12.1.0.3 agent.

Tuesday Sep 17, 2013

Oracle Enterprise Manager 12c Upgrade sessions at Oracle OpenWorld 2013

This year @ Oracle OpenWorld 2013 we have customers Wells Fargo and Colorcon sharing their success stories and real-life lessons about upgrading to Oracle Enterprise Manager 12.1.0.x. then come join us for the below session: 

https://oracleus.activeevents.com/2013/connect/sessionDetail.ww?SESSION_ID=9588&tclass

Stay Connected:

Twitter |  Face book |  You Tube |  Linked in |  Google+ Newsletter

Thursday Aug 22, 2013

Reduce Oracle Enterprise Manager 12c Patch Set Upgrade Downtime

As Product Manager for Oracle Enterprise Manager, one question that is often being asked to me is how much is the upgrade downtime and how it can be reduced? 

One easy way to reduce the downtime while doing patch set upgrades is to perform Software only upgrade and then shutdown the existing OMS for the upgrade. This approach will not completely eliminate the downtime but reduce it to a great extent. 

When I mention patch set upgrade, we cover following upgrade paths:

a) 12.1.0.1 (with Bundle Patches 1) to 12.1.0.2/ 12.1.0.3

b) 12.1.0.2 to 12.1.0.3

Following table compares the steps required for regular installation (OUI) and software only installation.


 As you may notices from the steps listed above, regular OUI upgrade requires stopping OMS before invoking runInstaller and keeping it down until the upgrade is complete. On the other hand Software only upgrade first copies the bits and sets up the environment before stopping OMS, thus reducing the upgrade downtime.

Next comes Agent upgrade. While we recommend Agent Upgrade Console (AUC) to upgrade agents from 12.1.0.1 (with bundle patch1) /12.0.1.02 to 12.0.1.3, sometimes it’s not clear what happens during the Agent upgrade and when is the actual downtime . So here are high level steps describing the complete process: 


The actual downtime while doing the agent upgrade is on step9 where we shut down your old agent. When the agent is in blackout from step 1 to 8 it still collect’s all the monitoring data so there is no loss. 

More Information: It is recommended to go through the below checklist notes before starting your upgrade 

MOS note 1568143.1 - EM 12c R3: Checklist for Upgrading Enterprise Manager Cloud Control from Version 12.1.0.x to 12.1.0.3

MOS Note  1569883.1 EM 12c R3: Checklist for Upgrading Management Agents Version 12.1.0.x to 12.1.0.3

Collateral :

 EM 12.1.0.3 Install and Upgrade collateral page(PPT,Recorded demo,Whitepapers) on OTN:http://www.oracle.com/technetwork/oem/install-upgrade/index.html

Understanding Enterprise Manager 12.1.0.3 Upgrade and Agent Upgrades
http://www.oracle.com/technetwork/oem/install-upgrade/em-12103-upgrade-1967205.pdf

Stay Connected:

Twitter |  Face book |  You Tube |  Linked in |  Google+ Newsletter


Monday Aug 19, 2013

Simplified Agent and Plug-in Deployment

On your site of hundreds or thousands of hosts have you had to patch agents immediately as they get deployed?  For this reason I’ve always been a big fan of cloning an agent that has the required plug-ins and all the recommended core agent and plug-in patches, then using that clone for all new agent deployments. With EM 12c this got even easier as you can now clone the agent using the console “Add Host” method. You still have to rely on the EM users to use the clone. The one problem I have with cloning is that you have to have a reference target for each platform that you support. If you have a consolidated environment and only have Linux x64, this may not be a problem. If you are managing a typical data center with a mixture of platforms, it can become quite the maintenance nightmare just to maintain your golden images. You must update golden image agents whenever you get a new patch (generic or platform specific) for the agent or plug-in, and recreate the clone for each platform. Typically, I find people create a clone for their most common platforms, and forget about the rest. That means, maybe 80% of their agents meet their standard patch requirements and plug-ins upon deployment, but the other 20% have to be patched post-deploy, or worse – never get patched!

While deployed agents and plug-ins can be patched easily using EM Patches & Updates, but what about the agents still getting deployed or upgraded? Wouldn’t it be nice if they got patched as part of the deployment or upgrade? This article will show you two new features in EM 12.1.0.3 (12cR3) that will help you deploy the most current agent and plug-in versions. Whether you have 100s or 1000s of agents to manage, reducing maintenance and keeping the agents up to date is an important task, and being able to deploy or upgrade to a fully patched agent will save you a lot of time and effort.

[Read More]

Tuesday Aug 06, 2013

Answers to Your Common Questions on Enterprise Manager 12.1.0.3 Agent deployment using Add Host Targets Wizard (Agent Push)

Deploying EM agents through Add Host Targets Wizard (Agent Push) is one of the most used deployment method and oracle recommended method for deploying multiple agents in a single go.  In this blog we will address some frequently asked questions:

Question 1) Can I perform mass agent deployment using command line?

Ans) Yes, starting EM 12.1.0.3 you can now perform mass agent deployment using EM CLI. This functionality will allow you to push multiple agents from command line.  We can divide EM CLI verbs into 2 categories:


A. Perform agent deployment

a) submit_add_host -- Submit an Add Host session
b) list_add_host_platforms -- List the supported Add Host platforms
c) retry_add_host -- Retry a failed Add Host session

d) continue_add_host - Resume a failed Add Host session


B. Tracking Add Host verbs

a) get_add_host_status        -- Display the latest status of a submitted Add Host session.
b) list_add_host_sessions         -- List the submitted Add Host sessions

You can watch short video on how to perform agent deployment using EMCLI from here:
Screenwatch: How to perform mass agent deployment using Add host wizard (Agent Push) through EM CLI (New in EM 12.1.0.3)

Question 2) Can I use my SSH-key credential while deploying agent through Add Host Targets Wizard (Agent Push) ? 

 Answer 2) Starting EM 12.1.0.3 named credentials support SSH public key authentication and password based authentication. So you can use an existing SSH public key authentication without exposing your passwords. There are 2 use cases here.

i.  If you have setup SSH between the OMS and the target hosts and has the ssh keys handy the he has to follow only steps 4 and 5

ii. If you have not setup SSH between the OMS and the target hosts and does not have the ssh keys then he has to follow steps 1 to 5.

1. Navigate to the following location in the OMS home: $<OMS_HOME>/oui/prov/resources/scripts 

For example, /home/software/em/middleware/oms/oui/prov/resources/scripts

2. Run the following script, and pass the OMS user name you used for installing the OMS and the fully qualified name of the target hosts.

sshUserSetup.sh -setup -user <user_name> -hosts <target_hosts>

3. The SSH keys are generated under $OMS_USER_HOME/.ssh/id_rsa and $OMS_USER_HOME/.ssh/id_rsa.pub

4. Upload the keys to EM

5. Select the keys during agent deployment

Question 3) In my company we use Tectia instead of SSH, can I still use Add Host Targets Wizard (Agent Push) to deploy agents? 

Answer 3)  Yes, Add Host Targets Wizard (Agent Push ) supports SSH Vendors are OpenSSH (SSH2), Tectia. If you have any of these setup between your OMS and target machine you don’t have to do anything extra, application is intelligent enough to pick up the configuration and perform the agent deployment.

Question 4) Does Add Host Targets Wizard (Agent Push) support RSH?  

Answer 4) We don’t support RSH as we need secure shell to perform remote execution and remote copy. Even if RSH is setup Agent push application will not honor it.

Question 5)  I am not sure on which port my SSH is running and how to change SSH port for Add Host Targets Wizard (Agent Push)?

Answer 5) SSH default port is 22 but if you want to cross check or its running on different port then you can cat /etc/ssh/sshd_config to find out what port ssh is running on. Another way to find the port used by SSH you can run:

netstat -anp | grep -i sshd

Output of this command  will be like- 

tcp      0 0 0.0.0.0:22         0.0.0.0:*         LISTEN         3188/sshd

So in the above output 22 is the ssh port.

Once you know the ssh port used, you can update the SSH_PORT property in $<OMS_HOME>/oui/prov/resources/Paths.properties as below: SSH_PORT=<port number>

Question 6) Cygwin is required for deploying agents on Windows host using Add Host Targets Wizard (Agent Push). Once 12c agent is properly installed, is there a need to maintain a copy of Cygwin on the Windows server?  Can it be deleted?

Answer 6) Cygwin 1.7  is required to just deploy EM 12c agent on remote host and can be remove once agents are deployed. Cygwin is not required for any other EM lifecycle operations. 

Follow the steps to configure cygwin for agent deployment from here: http://docs.oracle.com/cd/E24628_01/install.121/e22624/preinstall_req_cygwin_ssh.htm#CBHCDFCH

Question 7)  What is the purpose of user "cyg_server" getting created, while Installing Cygwin, from 12c Cloud Control side. Is this user used during authentication? What if we provide some other username instead of "cyg_server"?

Answer 7) This user is an internal user created by ssh for privilege delegation and not used for agent install.

Question 8) I don’t want to deploy agents using EM login user: SYSMAN. Can I perform role based agent deployment? 

 Answer 8) Yes you can deploy agent from UI and EM CLI with non-sysman user having CREATE_TARGET privilege. If user tries to access the add host page with a user who does not have this privilege, he will get the below error.

Question 9) What all authentication utilities does Add Host Targets Wizard (Agent Push) support?

Answer 9) We support for all standard pdp utilities, sudo, pbrun, sesu , su. 

Question 10) What are the sudo requirements for deploying agent using Add Host Targets Wizard (Agent Push)?

Answer 10)  If your install user can become root then make sure that the /etc/sudoers file have following permission: 

<install_user> ALL=(root) /usr/bin/id, <agent_home>/*/agentdeployroot.sh

For example, oracle ALL=(root) /usr/bin/id, /home/oracle/agentibd/*/agentdeployroot.sh.

Here, oracle is the installing user, and /home/oracle/agentibd is the Management Agent home, that is, the agent base directory.

Question 11) Why do we need install user to have permission to run  /usr/bin/id command?

Answer 11) We use this command to check if the install user has the privilege to switch to root user using the privilege delegation tool.  If you are have locked account usecase then the user that you will switch to should have the permission to run this command. More details on locked account usecase refer to : http://docs.oracle.com/cd/E24628_01/install.121/e22624/install_agent.htm#CACJEFJI

Question 12)  In EM 12.1.0.3 do we still need to set the visiblepw in the /etc/sudoers file?

Answer 12) From EM 12.1.0.3 its not mandatory to set this parameter in /etc/sudoers file instead user can just  pass –enablePty in the additional parameters and set the global property oracle.sysman.prov.agentpush.enablePty=true in $OMS_HOME/sysman/prov/agentpush/agentpush.properties.

You need to set the above property so that the privilege delegation tools such as pbrun, sesu, and su can use pseudo terminal for remote command execution over SSH.

More Information:

What's New in Enterprise Manager 12.1.0.3 Agent Deployment

http://www.oracle.com/technetwork/oem/install-upgrade/em-12103-agent-deployment-1967206.pdf

EM 12.1.0.3 Add Host Targets Wizard (Agent Push) chapter

http://docs.oracle.com/cd/E24628_01/install.121/e22624/install_agent.htm#CACJEFJI 

Screenwatch: How to perform mass agent deployment using Add host wizard (Agent Push) through EM CLI (New in EM 12.1.0.3)  

https://apex.oracle.com/pls/apex/f?p=44785:24:0:::24:P24_CONTENT_ID,P24_PREV_PAGE:7714,1 

    Wednesday Jul 24, 2013

    Understanding Agent Resynchronization

    Agent Resynchronization (resync) is an important topic but often misunderstood or misused. In this Q&A styled blog, I discuss how and when it is appropriate to use agent resynchronization.

    What is Agent Resynchronization?

    Management Agent can be reconfigured using target information present in the Management Repository. Resynchronization pushes all targets and related information from the Management Repository to the Management Agent and then unblocks the Agent.

     Why do agents need to be re-synchronized?

    Read More

    About

    Latest information and perspectives on Oracle Enterprise Manager.

    Related Blogs




    Search

    Archives
    « May 2015
    SunMonTueWedThuFriSat
         
    1
    2
    3
    4
    5
    6
    7
    9
    10
    11
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
          
    Today