This blog introduces how to diagnose Agile PLM related issues.

  • August 8, 2014

Set up Node Manager for Agile Cluster Servers

Jie Chen
Senior Principal Technical Support Engineer

This article describes how to create Node Manager for Agile Cluster and manage them remotely. Suppose we have three Linux machines, one for Admin Server, the other two are for Managed Servers. Below is Agile Cluster information.

# Machine slag9310w5c.mycompany.com
Weblogic Server Location: /opt/bea/wlserver_10.3
Admin Server Name: slag9310w5c-AgileServer
Listen Address: slag9310w5c.mycompany.com:7001
Agile Domain location: /opt/agile0/agileDomain
# Machine slag9310w5c-1.mycompany.com
Weblogic Server Location: /opt/bea/wlserver_10.3
Managed Server 1 Name: slag9310w5c-ManagedServer1
Listen Address: slag9310w5c-1.mycompany.com:7001
Agile Domain location: /opt/agile1/agileDomain
# Machine slag9310w5c-2.mycompany.com
Weblogic Server Location: /opt/bea/wlserver_10.3
Managed Server 2 Name: slag9310w5c-ManagedServer2
Listen Address: slag9310w5c-2.mycompany.com:7001
Agile Domain location: /opt/agile2/agileDomain

Add machines to each cluster node

# Node Manager for Admin Server
slag9310w5c --> slag9310w5c.mycompany.com:6666
# Node Manager for Managed Server 1
slag9310w5c-1 --> slag9310w5c-1.mycompany.com:6667
# Node Manager for Managed Server 2
slag9310w5c-2 --> slag9310w5c-2.mycompany.com:6668

This need to be done from Weblogic Admin Console. We create 3 Machines and add listen address/port for each. These are used for Node Managers.

And then assign each machine to corresponding server. For example we assign slag9310w5c machine to server slag9310w5c-AgileServer.

We can review them in config.xml to confirm we are doing right.

Configure Node managers

On each machine, we create its own Node Manager. We create these directories on each.

# Admin Server‘s Node Manager location: /opt/agile0/nodemanager/slag9310w5c
[oracle@slag9310w5c agile0]$ pwd
[oracle@slag9310w5c agile0]$ mkdir -p nodemanager/slag9310w5c
# Managed Server 1‘s Node Manager location: /opt/agile1/nodemanager/slag9310w5c-1
[oracle@slag9310w5c-1 agile1]$ pwd
[oracle@slag9310w5c-1 agile1]$ mkdir -p nodemanager/slag9310w5c-1
# Managed Server 2‘s Node Manager location: /opt/agile2/nodemanager/slag9310w5c-2
[oracle@slag9310w5c-2 agile2]$ pwd
[oracle@slag9310w5c-2 agile2]$ mkdir -p nodemanager/slag9310w5c-2

Copy startNodeManager.sh from Weblogic Server to each Node Manager‘s directory.

# On Admin Server‘s machine
[oracle@slag9310w5c slag9310w5c]$ pwd
[oracle@slag9310w5c slag9310w5c]$ cp /opt/bea/wlserver_10.3/server/bin/startNodeManager.sh startNodeManager.sh

And modify its NODEMGR_HOME to point to its real Node Manager directory, and JAVA_OPTIONS to includes weblogic.nodemanager.ServiceEnabled=true which is to fix a BUG of Weblogic that nmStart command never returns to console.

# On Admin Server‘s Node Manager dirctory, edit /opt/agile0/nodemanager/slag9310w5c/startNodeManager.sh
JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.nodemanager.ServiceEnabled=true"

Do same to 2 Managed Servers to create startNodeManager.sh and edit it. So we will have:

# On Admin Server, /opt/agile0/nodemanager/slag9310w5c/startNodeManager.sh
JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.nodemanager.ServiceEnabled=true"
# On Managed Server 1, /opt/agile1/nodemanager/slag9310w5c-1/startNodeManager.sh
JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.nodemanager.ServiceEnabled=true"
# On Managed Server 2, /opt/agile2/nodemanager/slag9310w5c-2/startNodeManager.sh
JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.nodemanager.ServiceEnabled=true"

Create nodemanager.properties file in each Node Manager‘s directory.

# On Admin Server, /opt/agile0/nodemanager/slag9310w5c/nodemanager.properties
# On Managed Server 1, /opt/agile1/nodemanager/slag9310w5c-1/nodemanager.properties
# On Managed Server 2, /opt/agile2/nodemanager/slag9310w5c-2/nodemanager.properties

By default, Node Manager will start startWeblogic.sh file. So we use StartScriptName to specify our own Agile Shell Script file. Make sure StartScriptEnabled is set to true. NativeVersionEnabled is used to enable emKill command to stop Agile Server remotely. CrashRecoveryEnabled enables Agile server to restart automatically if it crashes.

From above we see NodeManager will execute startServerAdminFromNM.sh for Admin Server, startServerManaged1FromNM.sh for Managed Server 1 and startServerManaged2FromNM.sh for Managed Server 2. These three files do not exist. So we need to create them. Just copy from the Shell Script in agileDomain/bin/ folder.

# Admin Server /opt/agile0/agileDomain/bin/ directory
[oracle@slag9310w5c bin]$ cp startServerAgileAdmin.sh startServerAdminFromNM.sh
# Managed Server 1 /opt/agile1/agileDomain/bin/ directory
[oracle@slag9310w5c-1 bin]$ cp startServerAgileManaged1.sh startServerManaged1FromNM.sh
# Managed Server 2 /opt/agile2/agileDomain/bin/ directory
[oracle@slag9310w5c-2 bin]$ cp startServerAgileManaged2.sh startServerManaged2FromNM.sh

Because Node Manager invokes these SH file from its own process, we need to edit these three files to use absolute Path for setEnv.sh execution.

# Admin Server /opt/agile0/agileDomain/bin/startServerAdminFromNM.sh
#. ./setEnv.sh -- comment it
#cd .. -- comment it
. /opt/agile0/agileDomain/bin/setEnv.sh
# Managed Server 1 /opt/agile1/agileDomain/bin/startServerManaged1FromNM.sh
#. ./setEnv.sh -- comment it
#cd .. -- comment it
. /opt/agile1/agileDomain/bin/setEnv.sh
# Managed Server 2 /opt/agile2/agileDomain/bin/startServerManaged2FromNM.sh
#. ./setEnv.sh -- comment it
#cd .. -- comment it
. /opt/agile2/agileDomain/bin/setEnv.sh

And be sure to add weblogic.nodemanager.ServiceEnabled=true before "weblogic.server". For example:

"$JAVA_HOME/bin/java"  ... ... -Dweblogic.nodemanager.ServiceEnabled=true weblogic.Server

Enroll each node manager to agileDomain

This needs to be done one Admin Server, 2 Managed Servers one by one. Before enroll, the Admin Server must be started manually.

Invoke Agile‘s sentEnv.sh to setup runtime environment

[oracle@slag9310w5c bin]$ pwd
[oracle@slag9310w5c bin]$ source setEnv.sh
Your environment has been set.

Run weblogic.WLST program

[oracle@slag9310w5c bin]$ java weblogic.WLST
Initializing WebLogic Scripting Tool (WLST) ...
Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands

Use connect command to connect to Admin Server

wls:/offline> connect(‘superadmin‘, ‘agile‘, ‘t3://slag9310w5c.mycompany.com:7001‘)
Connecting to t3://slag9310w5c.mycompany.com:7001 with userid superadmin ...
Successfully connected to Admin Server ‘slag9310w5c.mycompany.com-AgileServer‘ that belongs to domain ‘agileDomain‘.
Warning: An insecure protocol was used to connect to the server. To ensure on-the-wire security, the SSL port or Admin port should be used instead.

Enroll Admin Server‘s Node Manager to agileDomain

wls:/agileDomain/serverConfig> nmEnroll(‘/opt/agile0/agileDomain‘, ‘/opt/agile0/nodemanager/slag9310w5c‘)
Enrolling this machine with the domain directory at /opt/agile0/agileDomain ...
Successfully enrolled this machine with the domain directory at /opt/agile0/agileDomain.
wls:/agileDomain/serverConfig> exit()
Exiting WebLogic Scripting Tool.

Now we have Admin Server‘s Node Manager successfully enrolled to agileDomain. Next we need to do same on Managed Server‘s host with same steps.

source setEnv.sh
java weblogic.WLST
connect(‘superadmin‘, ‘agile‘, ‘t3://slag9310w5c.mycompany.com:7001‘)
nmEnroll(‘/opt/agile1/agileDomain‘, ‘/opt/agile1/nodemanager/slag9310w5c-1‘) # on Managed Server 1 host
nmEnroll(‘/opt/agile2/agileDomain‘, ‘/opt/agile2/nodemanager/slag9310w5c-2‘) # on Managed Server 2 host

Start each Node Manager

Logon Admin Server, start its Node Manager.

[oracle@slag9310w5c slag9310w5c]$ pwd
[oracle@slag9310w5c slag9310w5c]$ ./startNodeManager.sh

Jul 24, 2014 11:06:39 PM weblogic.nodemanager.server.Listener run
INFO: Plain socket listener started on port 6,666, host slag9310w5c.mycompany.com

Need to do same on each Managed Server and start their own Node Manager.

Start Agile Server from remote machine

After Node Manager of Admin Server is started successfully, we can start the Admin Server from other machines which has been installed Weblogic Server, using WLST script.

  • nmConnect: connect to remote Node Manager
  • nmStart: start remote Weblogic Server
  • nmServerStatus: check remote server status
  • nmKill: stop remote Weblogic Server

For example I am starting Agile Admin Server from my local Windows machine by connecting to Admin's Node Manasger.

Note: The password of Node Manager is different from Admin Console Logon password. Node Manager password is defined separately.

Your environment has been set.
C:\Oracle\Middleware\wlserver_12.1\server\bin>java weblogic.WLST
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
wls:/offline> nmConnect(‘superadmin‘, ‘agile‘, ‘slag9310w5c.mycompany.com‘, ‘6666‘, ‘agileDomain‘,‘/opt/agile0/agileDomain‘,‘plain‘)
Connecting to Node Manager ...
Successfully Connected to Node Manager.
wls:/nm/agileDomain> nmStart(‘slag9310w5c.mycompany.com-AgileServer‘)
Starting server slag9310w5c.mycompany.com-AgileServer ...
Successfully started server slag9310w5c.mycompany.com-AgileServer ...
wls:/nm/agileDomain> nmServerStatus(‘slag9310w5c.mycompany.com-AgileServer‘)
wls:/nm/agileDomain> nmKill(‘slag9310w5c.mycompany.com-AgileServer‘)
Killing server slag9310w5c.mycompany.com-AgileServer ...
Successfully killed server slag9310w5c.mycompany.com-AgileServer ...

We can see the execution from Node Manager log

INFO: Starting WebLogic server with command line: /opt/agile0/agileDomain/bin/startServerAdminFromNM.sh

Jul 24, 2014 11:07:15 PM weblogic.nodemanager.server.ServerManager log
INFO: Server output log file is ‘/opt/agile0/agileDomain/servers/slag9310w5c.mycompany.com-AgileServer/logs/slag9310w5c.mycompany.com-AgileServer.out‘

startServerAdminFromNM.sh is called by Node Manager and server log is written to slag9310w5c.mycompany.com-AgileServer.out file.

To start Agile Managed Servers from remote machine, use the same command with corresponding parameters. For example starting Managed Server 1, we have commands:

java weblogic.WLST
nmConnect(‘superadmin‘, ‘agile‘, ‘slag9310w5c-1.mycompany.com‘,‘6667‘, ‘agileDomain‘,‘/opt/agile1/agileDomain‘,‘plain‘)

Another way to start Managed Servers, we can logon Weblogic Console web page to start them from Start button, showing in below screenshot.

Join the discussion

Comments ( 7 )
  • guest Wednesday, December 17, 2014


    we tried to setup nodemanager in our environment by following your post and succeeded as well. But I see strange problem in my case. When I try to start any of the managed servers through WLST command nmStart or through the weblogic.Admin utility the control never returns back though the server starts successfully. I mean everytime I had to do this , I need to issue CTRL+C to exit the command. How can I overcome this ?


    Vidhya Shankar

  • Jie Chen Thursday, December 18, 2014

    Thank you Vidhya for your comment.

    This is a known issue I ever met before. You can add below parameter to your shell script to start nodemanager.


    I also wrote this in this blog.

  • guest Tuesday, June 9, 2015

    Do you have any plans to publish this as a Oracle support document ? If not it would be great if you can point me to any oracle document which explains about the nodemanager setup for agile.

  • Paritosh Thursday, October 8, 2015

    I am getting authentication failure while starting server

    main> <<WLS Kernel>> <> <> <1444304757070> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user denied

    weblogic.security.SecurityInitializationException: Authentication for user denied

    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:966)

    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)

    at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)

    at weblogic.security.SecurityService.start(SecurityService.java:148)

    at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

    at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

    Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User javax.security.auth.login.LoginException: [Security:090301]Password Not Supplied

  • Paritosh Monday, October 19, 2015

    how to configure If admin and managed server are on same machine.

    I have created single machine and mapped three servers.

    How to map single node manager to multiple managed server startup.



  • Jie Chen Monday, October 19, 2015

    Hello Paritosh,

    If Admin and Managed Servers are on same box, you still can setup each node manager for them because each bind with different port number and different startup script file.


  • paritosh Monday, October 19, 2015

    Hi Jie,

    I have tried as suggested 3 node manager,



    Your environment has been set.

    Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 9899; nested exception is:

    java.net.BindException: Address already in use

    <Oct 18, 2015 11:18:38 PM> <FINEST> <NodeManager> <Waiting for the process to die: 34709>

    <Oct 18, 2015 11:18:38 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

    <Oct 18, 2015 11:18:38 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

    Do you have any documents to publish? or steps to configure.

    it would help lot.



Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.