Monday Sep 30, 2013

Miscellaneous Tips: Solaris, Oracle Database, Java, FMW

[Solaris] Cleanup all IPC resources

Run the following wrapper script with root user privileges.

for i in `ipcs -a | awk '{ print $2 }'`
	ipcrm -m $i 2> /dev/null
	ipcrm -q $i 2> /dev/null
	ipcrm -s $i 2> /dev/null

[Java, WebLogic] Find the process id (pid) of a WebLogic managed server instance

Run the following as the user who owns the process, or with root user privileges.

/usr/java/bin/jps -v | grep <WLS_server_name> | awk '{ print $1 }'

I think this tip is applicable on all supported platforms.

Finding the pid of a managed server, bi_server1.

# /usr/java/bin/jps -v | grep bi_server1 | awk '{ print $1 }'

# pargs 18659 | grep weblogic.Name
argv[7]: -Dweblogic.Name=bi_server1

[Oracle Database] Make Oracle ignore hints

Set the following hidden parameter.


(in general, Oracle does not recommend playing with hidden parameters. Check with Oracle support when in doubt).

[Oracle Database] Data Pump Export in a RAC environment fails with ORA-31693, ORA-31617, ORA-19505, ORA-27037 errors


ORA-31693: Table data object "<SCHEMA>"."<TABLE>":"P_1147" failed to load/unload \
        and is being skipped due to error:
ORA-31617: unable to open dump file "<FILE>" for write
ORA-19505: failed to identify file "<FILE>"
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information: 3


Add CLUSTER=N to the list of existing expdp options.

[Solaris, ZFS] Check the current ARC size and its breakdown

kstat -m zfs | grep size 		(any user)  - OR -
echo ::arc | mdb -k | grep size 	(root user)

echo ::memstat | mdb -k		(root user)


# echo ::arc | mdb -k | grep size
size                      =       259391 MB
buf_size                  =         3218 MB
data_size                 =       249309 MB
other_size                =         6863 MB
l2_hdr_size               =            0 MB

# kstat -m zfs | grep size
        buf_size                        3375105344
        data_size                       261419672320
        l2_hdr_size                     0
        other_size                      7197048560
        size                            271991826224

# echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   14646326            114424    4%
ZFS File Data            31948806            249600   10%
Anon                     24660113            192657    7%
Exec and libs                8912                69    0%
Page cache                 126052               984    0%
Free (cachelist)            24517               191    0%
Free (freelist)         263965754           2062232   79%
Total                   335380480           2620160

[Fusion Middleware] Disable Fusion Middleware Diagnostic Framework (DFW) Dump Sampling

The Diagnostic Framework in FMW 11g environments detect, diagnose and resolve critical errors such as uncaught exceptions, deadlocked threads and out of memory errors. It is enabled by default.

Though DFW is supposed to diagnose and fix some of the issues transparently, due to the inevitable bugs in [all kinds of] software and misconfigurations, sometimes DFW itself may become a major issue. For instance, there is a bug that reported very high system CPU time on a SPARC server where FMW 11g was running. Per the bug description, the system CPU utilization spikes every minute exactly at 00s of a minute, CPU utilization goes down within few seconds - but the pattern persists and the spiky behavior returns within a minute. Another symptom was the sudden drop in available swap space from tens of giga bytes to a few mega bytes when the CPU spike occurs. Upon close examination, it was found out that DFW in FMW is forking tens of jstack processes to collect the thread dumps from an equal number of java processes running in that FMW environment, causing the sudden spike in CPU (each process is busy gathering thread dumps at the same time) and a steep drop in swap space (each jstack process forked a jmap process. both jstack and jmap processes consume some virtual memory just like any other process). All this happened because DFW thought it found a critical issue, and it wasn't noticed or addressed by anyone including the administrators (DFW couldn't fix this particular issue on its own) - so, it kept gathering the diagnostic data continuously. In this example, DFW did the right thing but the diagnostic data collection frequency was too short - only one minute, that diminished the value of DFW and made it a liability. In such dire situations, probably it is best to disable the dump sampling feature of Diagnostic Framework tentatively while the underlying original issue is being fixed in that application environment. It can be enabled again when the critical issue was fixed, and no longer an issue.

Steps to disable Fusion Middleware Diagnostic Framework (DFW) Dump Sampling: (courtesy: Shashidhara Varamballi)

Method (1) Using WLST:

  1. run
  2. connect to the AdminServer
  3. execute command: enableDumpSampling (enable=0, server='<server_name>')

Method (2) Manual editing of config file:

  1. Edit $DOMAIN_HOME/config/fmwconfig/servers/<server_name>/dfw_config.xml
  2. Change the "enabled" attribute from "true" to "false".
    <dumpSampling enabled="false">
  3. Change the "useExternalCommands" attribute from "true" to "false".
    <threadDump useExternalCommands="false"/>
  4. Save the changes


Fusion Middleware Diagnostics weblog

[Solaris 11] Virtual-to-physical link (NIC) mapping

Check the output of /sbin/dladm show-phys (any user). By default, only those physical links that are available on the running system are displayed. Option -P shows the physical device and attributes of all physical links.


$ /sbin/dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      ixgbe0
net5              Infiniband           down       0      unknown   ibp2
net1              Ethernet             up         1000   full      ixgbe1
net6              Infiniband           down       0      unknown   ibp3
net4              Ethernet             up         10     full      usbecm2

$ /sbin/dladm show-phys -P
LINK              DEVICE       MEDIA                FLAGS
net8              ibp1         Infiniband           r----
net0              ixgbe0       Ethernet             -----
net7              ibp0         Infiniband           r----
net3              ixgbe3       Ethernet             r----
net5              ibp2         Infiniband           -----
net1              ixgbe1       Ethernet             -----
net6              ibp3         Infiniband           -----
net4              usbecm2      Ethernet             -----
net2              vsw0         Ethernet             r----

Sunday May 17, 2009

Installing Siebel Web Extension (SWE) on top of Sun Java System Web Server 7.0

As of today, Sun Java System Web Server 7.0 is not a certified platform to deploy Siebel 8.x enterprise on. We are working with Oracle Corporation to make this certification happen so our customers can take advantage of the performance optimizations that went into the web server release 7.0.

Meanwhile those who want to give it a try can do so with little effort. In release SJSWS 7.0, the start/stop/restart/.. scripts were appropriately relocated to bin directory under the virtual web server instance. The installer for Siebel 8.x Web Server Extension looks for the start script [of the web server] under the home directory of the virtual web server instance. (because it was the default location until the release of SJSWS 7.0). The installation fails if the installer cannot find the start script in the location it is expecting it to be.

Due to the relocation mentioned above, installation of the Siebel Web Server Extension fails at the very last step where it tries to modify the start script with a bunch of LD_PRELOADs so the Siebel Web Extension loads up and runs on the Sun Java System Web Server. To get around this failure, all you have to do is to create a symbolic link in the home directory of the virtual web server instance pointing to the startserv script residing in the bin directory.

The following example shows the necessary steps.

% pwd

% ln -s bin/startserv start

% ls -l start
lrwxrwxrwx   1 pspp     dba           13 May 17 17:01 start -> bin/startserv

Install Siebel Web Extension in the normal way. No other changes are required.


% ls -l start\*
-rwxr-xr-x   1 pspp     dba         4157 May 17 17:38 start
-rwxr-xr-x   1 pspp     dba         3456 May 17 17:38 start_.bak

% mv bin/startserv bin/startserv.orig
% mv start bin/startserv

Notice that the Siebel installer actually made two copies of the startup script from the symbolic link. The original bin/startserv remained intact after the SWE installation.

Finally start the Web Server instance by running the startserv script. It should start with no issues.

% pwd

% ./startserv
Sun Java System Web Server 7.0U5 B03/10/2009 16:38
info: swe_init reports: SWE plug-in log file
info: CORE5076: Using [Java HotSpot(TM) Server VM, Version 1.5.0_15] from [Sun Microsystems Inc.]
info: HTTP3072: http-listener-1: http://siebel-pspp:8000 ready to accept requests
info: CORE3274: successful server startup

Before we conclude, do not forget the fact that Sun Java System Web Server 7.0 is not yet certified with Siebel 8.x release. Use the instructions mentioned in this blog post at your own risk. However if you do like to take that risk, consider installing the latest release of Sun Java System Web Server, which is SJSWS 7.0 Update 5 as of this writing.

Stay tuned for the certification news though.

Thursday Feb 26, 2009

Accessing MySQL Database(s) with JDBC

A new technical article entitled "Using the MySQL Connector/J JDBC Driver With the Java SE Platform", has been posted on at:

This article explains the essential steps involved in accessing/manipulating the data in a MySQL database from a Java application. MySQL Connector/J JDBC driver was used in the example code to show the database connectivity, data manipulation steps. Application developers who are new to Java programming language [but not to MySQL database] are the target audience of this article.

Stay tuned for the next article in this series "Using MySQL with PHP" ..

Sunday May 25, 2008

Deploying TWiki 4.2.0 on Sun Java Web Server 7.0

(The following instructions are based on Manish Kapur's Installing TWiki on Sun Java System Web Server. These instructions complement Manish's 2005 blog post)

The goal is to get the TWiki up and running on Sun Java Web Server (formerly known as Sun ONE Web Server / iPlanet Web Server). The assumption is that Sun Java Web Server is running on the Solaris operating system.

  1. Create 'twiki' user.
    % mkdir /export/twiki
    % useradd -d /export/twiki -s /bin/bash twiki
    % cd /export
    % chown twiki:other twiki
    % passwd twiki

  2. Install Sun Java Web Server 7.0 Update 2 in /export/twiki/sjws7u2 directory. Select 'Custom' installation; and choose port '8080' to run the web server.

    Sun Java Web Server 7.0 Update 2 is available for free at Sun Downloads web site.
            View by Category -> Web & Proxy Servers -> Web Servers -> Web Server 7.0 Update 2.

  3. Install TWiki 4.2.0

    Download TWiki tgz file
    % mkdir /export/twiki/sjws7u2/https-<hostname>/docs/twiki
    % cp TWiki-4.2.0.tgz /export/twiki/sjws7u2/https-<hostname>/docs/twiki
    % cd /export/twiki/sjws7u2/https-<hostname>/docs/twiki
    % gunzip TWiki-4.2.0.tgz
    % tar xf TWiki-4.2.0.tar

  4. Enable CGI on the web server

    • cd /export/twiki/sjws7u2/https-<hostname>/config

    • Edit the 'default' section of obj.conf file to include the following two lines

      NameTrans fn="pfx2dir" from="/twiki/view" dir="/export/twiki/sjws7u2/<hostname>/docs/twiki/bin/view" name="cgi"

      Service fn="send-cgi" type="magnus-internal/cgi" nice="10"

      Hopefully the following example gives an idea on where to insert those two lines. In the example, 't2000-240-08' is the hostname.

       % diff -C 3 obj.conf obj.conf.orig
      \*\*\* obj.conf    Wed May 14 01:36:34 2008
      --- obj.conf.orig       Wed May 14 01:02:52 2008
      \*\*\* 10,17 \*\*\*\*
        AuthTrans fn="match-browser" browser="\*MSIE\*" ssl-unclean-shutdown="true"
        NameTrans fn="ntrans-j2ee" name="j2ee"
        NameTrans fn="pfx2dir" from="/mc-icons" dir="/export/twiki/sjws7u2/lib/icons" name="es-internal"
      - # Consider anything in the directory /export/twiki/sjws7u2/https-t2000-240-08/docs/twiki/bin to be a CGI
      - NameTrans fn="pfx2dir" from="/twiki/view" dir="/export/twiki/sjws7u2/https-t2000-240-08/docs/twiki/bin/view" name="cgi"
        PathCheck fn="uri-clean"
        PathCheck fn="check-acl" acl="default"
        PathCheck fn="find-pathinfo"
      --- 10,15 ----
      \*\*\* 20,27 \*\*\*\*
        ObjectType fn="type-j2ee"
        ObjectType fn="type-by-extension"
        ObjectType fn="force-type" type="text/plain"
      - # Run CGI processes with a "Nice" level of 10
      - Service fn="send-cgi" type="magnus-internal/cgi" nice="10"
        Service method="(GET|HEAD)" type="magnus-internal/directory" fn="index-common"
        Service method="(GET|HEAD|POST)" type="\*~magnus-internal/\*" fn="send-file"
        Service method="TRACE" fn="service-trace"
      --- 18,23 ----

  5. Install Revision Control System (RCS), GNU diff utilities (diffutils) and update the PATH variable in user's .profile.

    Those packages are available in 'ready to install' form at and web sites. Perhaps the easiest way is to use pkg-get to pull those packages with the required dependencies from Blastwave.

    Assuming RCS and diffutils are available under /usr/csw/bin directory,
    % cat ~/.profile | grep PATH
    export PATH=/usr/bin:/usr/sbin:/usr/local/bin:/usr/sfw/bin:/usr/csw/bin:$PATH

  6. Configure the TWiki. Edit TWiki's \*.cfg files, that is.

    1. Create the config file LocalLib.cfg.

      1. There is a template for this config file in twiki/bin/LocalLib.cfg.txt. Copy LocalLib.cfg.txt to LocalLib.cfg.
        % cd /export/twiki/sjws7u2/https-<hostname>/docs/twiki/bin
        % cp LocalLib.cfg.txt LocalLib.cfg
      2. Edit LocalLib.cfg to update the $twikiLibPath variable.
        $twikiLibPath = "/export/twiki/sjws7u2/https-<hostname>/docs/twiki/lib";

    2. Manually create the config file LocalSite.cfg.
      % cd /export/twiki/sjws7u2/https-<hostname>/docs/twiki/lib
      % cp TWiki.spec LocalSite.cfg

    3. Update the following variables with appropriate values in LocalSite.cfg.

      $TWiki::cfg{DefaultUrlHost}, $TWiki::cfg{ScriptUrlPath}, $TWiki::cfg{PubUrlPath}, $TWiki::cfg{PubDir}, $TWiki::cfg{TemplateDir}, $TWiki::cfg{DataDir}, $TWiki::cfg{LocalesDir}, and $TWiki::cfg{OS}, $TWiki::cfg{RCS}{EgrepCmd} and $TWiki::cfg{RCS}{FgrepCmd}.

      This diff output shows how they were configured in the demo TWiki that I used for the Proof-Of-Concept (POC). The hardware it was deployed on is a Sun Fire T2000. So Cool Stack was installed on the server to take advantage of the optimized Perl.

  7. Exit the current shell; and reconnect.

  8. Restart the Sun Java Web Server.
    % cd /export/twiki/sjws7u2/https-<hostname>/bin
    % ./stopserv
    % ./startserv
  9. Manually execute the 'view' script from the command line.
    % cd /export/twiki/sjws7u2/https-/docs/twiki/bin
    % ./view
    You should see valid HTML (not errors) on the stdout. If you see errors, check /export/twiki/sjws7u2/https-<hostname>/logs/errors for the error messages. Fix the issues and re-run 'view' from the command line. Continue this exercise until the script returns valid HTML.

  10. Finally verify the TWiki installation through the web browser.
            Open http://<webhost>:<port>/twiki/view in a web browser.
  1. Installing TWiki on Sun Java System Web Server by Manish Kapur, Sun Microsystems
  2. Using PHP on Sun Java System Web Server by Joe McCabe, Sun Microsystems

Copy of these instructions are also available at my other weblog:

Benchmark announcements, HOW-TOs, Tips and Troubleshooting


« April 2014