Wednesday Aug 17, 2016

Bash script to deploy webapp to Weblogic server domain via WLST

I wrote a bash script to deploy a war file to ALL managed servers in a Weblogic domain. You should be able to modify it to work for your needs.

#!/bin/bash HOST_NAME=${1-localhost} HOST_USER=${2-weblogic} HOST_PASS=${3-welcome1} HOST_PORT=${4-7001} JAVA_BIN="/usr/bin/java" SCRIPT_HOME="$(dirname $(readlink -f $0))" WL_JAR="${SCRIPT_HOME}/middleware/wlserver_10.3/server/lib/weblogic.jar" JAM_JAR="myapp.war" function getServerList() { WLST_SERVER_LIST=$(${JAVA_BIN} -cp ${WL_JAR} weblogic.WLST <<EOF connect("${HOST_USER}", "${HOST_PASS}", "t3://${HOST_NAME}:${HOST_PORT}") cd('Servers') ls('a') EOF ) SERVER_LIST="" while read -r CURR_LINE; do if [[ ${CURR_LINE} =~ drw-* ]] then SERVER_LIST+=$(printf ${CURR_LINE#*drw-},) fi done <<< "${WLST_SERVER_LIST}" SERVER_LIST=${SERVER_LIST%,*} #echo $SERVER_LIST } function agentDeployer() { ${JAVA_BIN} -cp ${WL_JAR} weblogic.Deployer -adminurl t3://${HOST_NAME}:${HOST_PORT} -user ${HOST_USER} -password ${HOST_PASS} -targets ${SERVER_LIST} -deploy -upload ${JAM_JAR} } echo "Deploying" getServerList agentDeployer

Monday May 12, 2014

Quickly Diagnose the Root Cause of Stuck Threads using Oracle Enterprise Manager 12c JVM Diagnostics

Note: Clicking on any image will open the same image in full size in a new window

One of the hidden gems in Oracle Enterprise Manager 12c is JVM Diagnostics. If you purchased the Weblogic Management Pack license then you already own it. JVMD allows administrators to diagnose performance problems in production Java applications. By eliminating the need to reproduce these “production only” problems in QA, it reduces the time required to resolve them. It does not require complex instrumentation or restarting of the application to get in-depth application details. Application administrators will be able to identify Java problems or database issues that are causing application downtime without any detailed knowledge of the application internals. It is also very well suited to diagnosing issues with “Stuck Threads” which will be the focus of this blog.

What is a [STUCK] Thread

In a Weblogic server, all incoming requests are handled by a thread pool which is controlled by a work manager. Worker threads that are taken out of the pool and not returned after a specified time period are marked as [STUCK] by the work manager. This time period is 10 minutes by default but it is configurable on a per work manager basis using the "StuckThreadMaxTime" parameter (default is set to 600 seconds).

Note that it is possible that some of your threads are doing legitimate work for over 10 min with no issues. If you have such threads then you should consider placing them in a another work manager with proper setting for the "StuckThreadMaxTime" parameter

Why JVMD is Well Suited to Diagnosing [STUCK] Threads

Traditionally, developers will use a stack trace generated by jstack or kill -3 and try to determine the cause of a stuck thread. However, in my experience a majority of the time this stack is not even the culprit. The problem often lies in another tier of the application or even in another thread of the same application. JVMD has the ability to provide additional context such as the name of the request and which tier it called out to Eg: RDBMS servers, LDAP servers, Web servers, RMI servers, etc. Using fine grained thread states (i.e. DB, Network, IO, CPU, RMI, Lock, etc) and the ability to see additional details about the thread, JVMD users can quickly pinpoint the root cause of the problem. Since JVMD is always on, it can also debug these issues that happened in the past and can proactively notify you about stuck threads Eg: Get an email at 1am when you had stuck threads. And lastly, sometimes developers have no access to the target host due to lack of credentials needed to run command line applications.

On several occasions, the thread may be stuck but is doing legitimate work. In such scenarios JVMD allows you to scan back and forth through a large number of samples to see what work is being processed by the thread. In addition, you can take a look at other threads that were serviced the same request to see if they behaved similarly or not. This will allow you to quickly determine whether there is really a problem or not.

Real-Time [STUCK] Thread Analysis

With JVMD there are two real use cases for stuck thread analysis. If you get notified about a stuck thread in real-time (via email, etc) then you can perform a real-time stuck thread analysis. Alternatively, if you are investigating a thread that was stuck in the past but is not present any more, then you can perform a historical stuck thread analysis. In either case the first thing to do is to navigate to the JVM (or JVM pool) where the thread is stuck. We do this by clicking on Targets -> Middleware as shown

From here we can filter the list of targets by target type or by target name. Your most recent filter request will be remembered the next time you visit the page. Select the Target Type of JVM to see all of the JVM targets.

Pick the JVM for the Weblogic server which is having the stuck thread issues and click on it. This will take you to the target home page. Click the button at the top that says “Live Thread Analysis”. Type the word "stuck" into the thread name search box and click on the arrow to filter the table. Now you should see all the stuck threads. In this case we can see a thread that is stuck in the “Network Wait” state. It is stuck on line 358 in function writeBuffer() of which is in package com.sun.ssl.internal.ssl which makes it clear that this stuck thread has made an SSL call and the remote server has not responded in a reasonable amount of time so the client thread is stuck.

Here is another example of stuck threads, this time in the “DB Wait” state. Notice how the tool tip over the SQL ID field shows the SQL being executed. Click on it to view longer SQL statements. Also try clicking on the DB Wait link which takes you directly to this specific database session in the Oracle Database Diagnostics section of EM for further analysis. The columns displayed are controlled by “View” drop-down menu. Here we added the “User” column to show the logged in user who executed the request.

Historical [STUCK] Thread Analysis

In order to start historical stuck thread analysis you need to navigate to the JVM target home page in the same way as discussed in the real-time section. From the target home page clicking on the “JVM Performance Diagnostics” button at the top of the page. On the performance diagnostics page you can filter the data to make it more relevant to your task. The first filter to apply is of time. If you know the exact time you can use the “Edit Date and Time” button to specify it as shown. Otherwise use the handy shortcut links for Day, 1 Hours, 1 Hour or 15 Minutes as needed.

The next thing to filter is the Thread Name. Expand the filter options region if necessary and add the Thread Name filter to be “[STUCK]*” so you only see threads whose name starts with [STUCK]

Below the filter region the “General” tab will show you the Thread States, Top Requests, Top Methods, Top SQLs, Top DB Wait Events and Top Databases – only for the filtered data i.e. for only threads that are stuck. Try clicking on method names to see the call stack for the method. The charts are all interactive and fetch additional data about the item clicked.

If you want to find a specific thread move from the “General” tab over to the “Threads” tab. This is fine grained data with each sample and state transition visible. You can click on any sample to view it in the sample analyzer which should look familiar to you if you saw the threads in real-time. Details about SQLs, Wait states, etc are all available here also along with the complete call stack which can also be exported to a CSV file.

In conclusion, we can see the JVMD provides a rich set of additional details which are only a mouse click away that help you to diagnose the root causes of your stuck threads.

NOTE: Many of the screen shots taken here are using testing & debug code, which deliberately tries to create stuck threads. This does not and should not reflect on the nature of any Oracle products being shipped to customers.

Friday Mar 22, 2013

Cookbook : Middleware as a Service using Oracle Enterprise Manager 12c

Oracle's Middleware as a Service (MWaaS) solution for enterprise private cloud provides a complete application development and deployment environment. It includes a complete runtime environment comprised of all services necessary to deploy and run an enterprise-class application, including services such as application hosting, persistence store, application integration and APIs that enable programmatic access to additional computing services that might be required by an application. Identity services are an example of APIs available within a PaaS environment. MWaaS facilitates cheaper and faster deployment of applications as developers need not deal with the complexities of the underlying hardware and software components.

Oracle recently published a cookbook for Middleware as a Service using Oracle Enterprise Manager 12c. This document explains the step-by-step instructions in provisioning a WebLogic domain using Oracle Enterprise Manager 12c . These steps include:   

  • Security Configuration for Named Credentials, Roles and Accounts for Cloud Management
  • Using Out of Box WebLogic profiles shipped with Oracle Enterprise Manager 12c R2
  • Customizing WebLogic domain creation procedures to your environment and business requirements
  • Setting up the Middleware Zones
  • Setting quota limits for each cloud management roles
  • Definition of Service Templates to be used for middleware domain creation
  • Configuring chargeback policies for your middleware cloud infrastructure

Friday Feb 01, 2013

Network Ports Used in Oracle Enterprise Manager 12c

When planning and configuring your Oracle Enterprise Manager 12c implementation, you will have many infrastructure considerations. One of the most often discussed pieces is the network ports that are used and how to configure load balancers, firewalls and ACLs for communication.

This blog post will help identify the typical default port and range for each component, how to identify it and how to modify the port usage.

[Read More]

Friday Aug 24, 2012

Managing Oracle Exalogic Elastic Cloud with Oracle Enterprise Manager Ops Center

Oracle Enterprise Manager Ops Center 12c now comes out-of-the-box  with the latest release of Oracle Exalogic Elastic Cloud 2.0.1 software. It allows Customer to manage and monitor all components inside the Exalogic rack, including provisioning and management of physical and virtualized server. Ops Center will allow Customers to easily get started with creating and managing Private Clouds using the Exalogic components.

Here is a snaphot of the Assets view showing the managable components of a Quarter Rack with 8 Compute Nodes:

A colleague has recently posted an interesting series of "Exalogic 2.0.1 Tea Break Snippets" which will guide you through the initial steps to get started with setting up your Exalogic environment:

Exalogic 2.0.1 Tea Break Snippets - Creating Cloud Users

Exalogic 2.0.1 Tea Break Snippets - Creating Networks

Exalogic 2.0.1 Tea Break Snippets - Allocating Static IP Addresses

Exalogic 2.0.1 Tea Break Snippets - Creating Accounts

Exalogic 2.0.1 Tea Break Snippets - Importing Public Server Template

Have fun reading these very useful postings !

Dr. Jürgen Fleischer , Oracle Enterprise Manager Ops Center Engineering

Stay Connected:

Twitter |  Face book |  You Tube |  Linked in |  Newsletter

Thursday Mar 08, 2012

Looking for an executive overview of Oracle Enterprise Manager 12c ?

IT professionals are excited by the technical advantages of cloud computing, and Line of Business managers look to the cloud as a driver of business growth, efficiency, and productivity. By transforming IT into a business-centric provider of services that users can access from anywhere, Oracle Enterprise Manager 12c helps you build a more agile, efficient, and  innovative enterprise.

You can get an executive overview of the Oracle Enterprise Manager 12c in this recently published executive brief

For more information, please go to Oracle Enterprise Manager  web page or  follow us at : 

Twitter | Facebook | YouTube | Linkedin | Newsletter

Latest information on Oracle Enterprise Manager and Oracle Management Cloud.

Related Blogs


« April 2017