X

Enterprise Manager Incorrectly Showing Managed Server in Down State

Mark Piller
Senior Principal Consultant

Introduction

When setting up an environment to run ADF applications (I was specifically targeting ADF 11.1.2.4.0) I have problems after running the applyJRF() script in WLST. This problem was easily replicated.

Updated 3-2-2015 please see the end of this blog entry.

Problem

WebLogic reports the Managed Server is RUNNING and Health is OK.

Enterprise Manager in Fusion Middleware Control reports the Managed Server is DOWN and no metrics can be seen for that managed server.

Environment and Software Summary

WebLogic Version

10.3.6.0 with Patch ID 12UV (10.3.6.0.10 PSU Patch for BUG19637463)

ADF Runtime Version

tested with 11.1.1.6.0 and 11.1.1.7.0

Environments

Both Windows and Linux

Details

How to create this scenario

  1. Install WebLogic
  2. Install the ADF Runtime Libraries
  3. Create a Domain with at least one Managed Server
  4. Launch WebLogic server, start the Managed Server in the Admin Console
  5. Launch Enterprise Manager and review the Domain status
  6. Navigate to the Managed Server - it is RUNNING but you are instructed to click on the Apply JRF Template button
  7. Click on the Apply JRF Template button [could also do this in WLST with applyJRF()]
  8. Recycle the servers

How it Appears in WebLogic and Enterprise Manager

WebLogic will continue to display the Managed Server as RUNNING. Enterprise Manager (after the Apply JRF Template) displays the Managed Server as DOWN - no metrics can be found.

How it Appears in Error Logs

The Managed Server log will report java.lang.ClassNotFoundException errors.

The missing classes and their library files are displayed below:

Class File not found

Library

Location

oracle.security.jps.wls.JpsWlsStartupClass

jps-wls.jar

<MW_HOME>\oracle_common\modules\oracle.jps_11.1.1

oracle.jrf.wls.JRFStartup

jrf-api.jar

<MW_HOME>\oracle_common\modules\oracle.jrf_11.1.1

oracle.core.ojdl.weblogic.ODLConfiguration

ojdl.jar

<MW_HOME>\oracle_common\modules\oracle.odl_11.1.1

oracle.jrf.AppContextStartup

jrf-api.jar

<MW_HOME>\oracle_common\modules\oracle.jrf_11.1.1

oracle.as.jmx.framework.wls.spi.StartupListener

jmxspi.jar

<MW_HOME>\oracle_common\modules\oracle.jmx_11.1.1

oracle.j2ee.ws.server.WebServiceServerStartup

wsserver.jar

<MW_HOME>\oracle_common\modules\oracle.webservices_11.1.1

oracle.ias.cache.Startup

cache.jar

<MW_HOME>\oracle_common\modules\oracle.javacache_11.1.1

oracle.dms.wls.DMSStartup

dms.jar

<MW_HOME>\oracle_common\modules\oracle.dms_11.1.1

oracle.dms.wls.DMSServletFilter

dms.jar

<MW_HOME>\oracle_common\modules\oracle.dms_11.1.1

Known Resolutions

I have been able to resolve this problem in three separate ways.

Solution 1 - Classpath

Modify the setDomainEnv script by assigning the PRE_CLASSPATH environment variable the paths to the 9 missing libraries.  Now the Managed Server is correctly reported in Enterprise Manager.

Solution 2 - startManagedWebLogic

DO NOT start the Managed Server in either WebLogic Console or in Enterprise Manager. INSTEAD - start the Managed Server with the startManagedWebLogic script.

Solution 3 - Cluster

Create a cluster. Apply the JRF Template to the cluster. There are no problems when you do it this way.

Oracle Support Document Reference

Oracle Support had the following documents for review. None of these proved to be effective in resolving this problem.



  • Enterprise Manager Fusion Middleware Control 11g (fmwc) - Querying FMWC Related MBeans And Metrics (Doc ID 1533771.1)
  • Grid Control 11g: Application Deployments Status is Down even if they are Active in WebLogic Console (Doc ID 1489924.1)
  • How to upgrade a WLS domain extended with ADF libraries (Doc ID 1408663.1)
  • Bug 14562087 : GC 11G SHOWS APP DEPLOYMENTS DOWN AND MANAGED SERVERS PENDING
  • Enterprise Manager Fusion Middleware Control 11g (fmwc) Multiple Targets Showing Incorrect Down Status (Doc ID 1582731.1)
  • How to Apply Oracle Enterprise Manager Fusion Middleware Control and JRF to a WebLogic Domain and Managed Servers ( 11g ) (Doc ID 947043.1)
  • Oracle Enterprise Manager Fusion Middleware Control (FMWC) Incorrectly Showing Managed Server In Down Status (Doc ID 1641332.1)
  • Installing ADF Runtime and Oracle Enterprise Manager on Oracle Fusion Middleware 11g R1 (Doc ID 853789.1)

Final Resolution - Updated 3-2-15

I have a response from Oracle Support and this resolves the problem of Enterprise Manager NOT reporting the metrics or correctly reporting the status of the managed server. The step that resolves this problem is to change a property in the nodemanager.properties file. StartScriptEnabled=true is the correct setting. The default value of this has been false for any WebLogic release before WebLogic 12c. Below is a table of the documentation I found regarding this setting.

Property

Default Value

Documentation Source

StartScriptEnabled

false

Configuring Java
Node Manager 11.1.1.6

false

Configuring Java
Node Manager 11.1.1.7

true

Configuring Java
Node Manager 12.1.2

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.