Sunday May 17, 2009

Installing Siebel Web Extension (SWE) on top of Sun Java System Web Server 7.0

As of today, Sun Java System Web Server 7.0 is not a certified platform to deploy Siebel 8.x enterprise on. We are working with Oracle Corporation to make this certification happen so our customers can take advantage of the performance optimizations that went into the web server release 7.0.

Meanwhile those who want to give it a try can do so with little effort. In release SJSWS 7.0, the start/stop/restart/.. scripts were appropriately relocated to bin directory under the virtual web server instance. The installer for Siebel 8.x Web Server Extension looks for the start script [of the web server] under the home directory of the virtual web server instance. (because it was the default location until the release of SJSWS 7.0). The installation fails if the installer cannot find the start script in the location it is expecting it to be.

Due to the relocation mentioned above, installation of the Siebel Web Server Extension fails at the very last step where it tries to modify the start script with a bunch of LD_PRELOADs so the Siebel Web Extension loads up and runs on the Sun Java System Web Server. To get around this failure, all you have to do is to create a symbolic link in the home directory of the virtual web server instance pointing to the startserv script residing in the bin directory.

The following example shows the necessary steps.

% pwd

% ln -s bin/startserv start

% ls -l start
lrwxrwxrwx   1 pspp     dba           13 May 17 17:01 start -> bin/startserv

Install Siebel Web Extension in the normal way. No other changes are required.


% ls -l start\*
-rwxr-xr-x   1 pspp     dba         4157 May 17 17:38 start
-rwxr-xr-x   1 pspp     dba         3456 May 17 17:38 start_.bak

% mv bin/startserv bin/startserv.orig
% mv start bin/startserv

Notice that the Siebel installer actually made two copies of the startup script from the symbolic link. The original bin/startserv remained intact after the SWE installation.

Finally start the Web Server instance by running the startserv script. It should start with no issues.

% pwd

% ./startserv
Sun Java System Web Server 7.0U5 B03/10/2009 16:38
info: swe_init reports: SWE plug-in log file
info: CORE5076: Using [Java HotSpot(TM) Server VM, Version 1.5.0_15] from [Sun Microsystems Inc.]
info: HTTP3072: http-listener-1: http://siebel-pspp:8000 ready to accept requests
info: CORE3274: successful server startup

Before we conclude, do not forget the fact that Sun Java System Web Server 7.0 is not yet certified with Siebel 8.x release. Use the instructions mentioned in this blog post at your own risk. However if you do like to take that risk, consider installing the latest release of Sun Java System Web Server, which is SJSWS 7.0 Update 5 as of this writing.

Stay tuned for the certification news though.

Saturday Nov 01, 2008

Ramifications of the Solaris 10 kernel patch 137111


A recent code change in Solaris 10 inadvertantly exposed an inherent bug in some of the 32-bit applications that rely on their own memory allocators. Due to this, some of the 3rd party applications which were working earlier without the KU 137111 may crash on Solaris 10/SPARC with the KU 137111 (any revision).

Symptoms & the Cause

It was identified that majority of such application failures are mainly due to the applications' custom memory allocator that incorrectly returns 4-byte aligned mutexes in place of the required 8-byte aligned mutexes. In Solaris, mutex_t and pthread_mutex_t structures have been defined to be aligned on an 8-byte boundary. Both of those structures contain the upad64_t member, which is a double even for the 32-bit applications. The natural alignment of a double is 8 bytes; and per the SPARC Compliance Definition 2.4, the structures must be aligned according to their strictest member. That is, applications which create 4-byte aligned mutexes are technically non-compliant on Solaris/SPARC (for the sake of simplicity, such code will be referred to as the non-complying code for the remainder of this blog entry).

Due to a change in the implementation of the userland mutexes introduced by CR 6296770 in KU 137111-01, objects of type mutex_t and pthread_mutex_t must start at 8-byte aligned addresses. If this requirement is not satisfied, all non-compliant applications on Solaris/SPARC may fail with the signal SEGV with a callstack similar to the following one or with similar callstacks containing the function mutex_trylock_process.

  \*_atomic_cas_64(0x141f2c, 0x0, 0xff000000, 0x1651, 0xff000000, 0x466d90)
  set_lock_byte64(0x0, 0x1651, 0xff000000, 0x0, 0xfec82a00, 0x0)
  fast_process_lock(0x141f24, 0x0, 0x1, 0x1, 0x0, 0xfeae5780)

Patches & the Next Steps

Note that only non-compliant 32-bit applications will be affected by the KU 137111. All other complying 32-bit applications continue to run as expected even with the KU 137111 - hence the customers, partners, ISVs and the other software vendors must understand the fact that it is not a Solaris issue. Customers running into this issue must work with the respective software vendors to obtain a patch/fix. We suggest the ISVs and the rest of the software vendors to pro-actively check their 32-bit native code for any discrepancies like the one mentioned in this blog entry.

In our testing of some of the enterprise applications, we have identified Oracle's Siebel CRM as one of the potential applications that is vulnerable to the KU 137111. It appears that IBM's Lotus Domino Server is also prone to a crash on Solaris 10 with the same kernet patch. Speaking of these two known cases, Oracle/Siebel and IBM/Lotus Domino customers (running Solaris) should approach Oracle and IBM Corporations respectively but not Sun Microsystems for a proper fix.

As it may take some time for the ISVs / software vendors to identify and fix the non-complying code in their applications, Sun is planning to provide an interim fix to the mutex byte alignment issue in the form of a Solaris kernel patch. As of this writing, we expect the fix to be integrated into the KU 137137-07. The fix is already available in the latest update of the Solaris, Solaris 10 10/08. Those who cannot upgrade to Solaris 10 10/08 from the prior versions of Solaris 10 must wait for the patch KU [Updated 12/07/08] 137137-07 137137-09.

One must note that the fix in Solaris is a tentative one that allows the non-complying code to run on SPARC hardware for the time being. There is no guarantee that the non-complying code continues to run 'as is' in the future with new Solaris kernel patches and/or major updates/releases of the Solaris operating system. So the best long term solution is for the software vendors to fix the non-compliant code before it is too late.


Steve S and Roger F of Sun Microsystems.

Sunday May 25, 2008

Deploying TWiki 4.2.0 on Sun Java Web Server 7.0

(The following instructions are based on Manish Kapur's Installing TWiki on Sun Java System Web Server. These instructions complement Manish's 2005 blog post)

The goal is to get the TWiki up and running on Sun Java Web Server (formerly known as Sun ONE Web Server / iPlanet Web Server). The assumption is that Sun Java Web Server is running on the Solaris operating system.

  1. Create 'twiki' user.
    % mkdir /export/twiki
    % useradd -d /export/twiki -s /bin/bash twiki
    % cd /export
    % chown twiki:other twiki
    % passwd twiki

  2. Install Sun Java Web Server 7.0 Update 2 in /export/twiki/sjws7u2 directory. Select 'Custom' installation; and choose port '8080' to run the web server.

    Sun Java Web Server 7.0 Update 2 is available for free at Sun Downloads web site.
            View by Category -> Web & Proxy Servers -> Web Servers -> Web Server 7.0 Update 2.

  3. Install TWiki 4.2.0

    Download TWiki tgz file
    % mkdir /export/twiki/sjws7u2/https-<hostname>/docs/twiki
    % cp TWiki-4.2.0.tgz /export/twiki/sjws7u2/https-<hostname>/docs/twiki
    % cd /export/twiki/sjws7u2/https-<hostname>/docs/twiki
    % gunzip TWiki-4.2.0.tgz
    % tar xf TWiki-4.2.0.tar

  4. Enable CGI on the web server

    • cd /export/twiki/sjws7u2/https-<hostname>/config

    • Edit the 'default' section of obj.conf file to include the following two lines

      NameTrans fn="pfx2dir" from="/twiki/view" dir="/export/twiki/sjws7u2/<hostname>/docs/twiki/bin/view" name="cgi"

      Service fn="send-cgi" type="magnus-internal/cgi" nice="10"

      Hopefully the following example gives an idea on where to insert those two lines. In the example, 't2000-240-08' is the hostname.

       % diff -C 3 obj.conf obj.conf.orig
      \*\*\* obj.conf    Wed May 14 01:36:34 2008
      --- obj.conf.orig       Wed May 14 01:02:52 2008
      \*\*\* 10,17 \*\*\*\*
        AuthTrans fn="match-browser" browser="\*MSIE\*" ssl-unclean-shutdown="true"
        NameTrans fn="ntrans-j2ee" name="j2ee"
        NameTrans fn="pfx2dir" from="/mc-icons" dir="/export/twiki/sjws7u2/lib/icons" name="es-internal"
      - # Consider anything in the directory /export/twiki/sjws7u2/https-t2000-240-08/docs/twiki/bin to be a CGI
      - NameTrans fn="pfx2dir" from="/twiki/view" dir="/export/twiki/sjws7u2/https-t2000-240-08/docs/twiki/bin/view" name="cgi"
        PathCheck fn="uri-clean"
        PathCheck fn="check-acl" acl="default"
        PathCheck fn="find-pathinfo"
      --- 10,15 ----
      \*\*\* 20,27 \*\*\*\*
        ObjectType fn="type-j2ee"
        ObjectType fn="type-by-extension"
        ObjectType fn="force-type" type="text/plain"
      - # Run CGI processes with a "Nice" level of 10
      - Service fn="send-cgi" type="magnus-internal/cgi" nice="10"
        Service method="(GET|HEAD)" type="magnus-internal/directory" fn="index-common"
        Service method="(GET|HEAD|POST)" type="\*~magnus-internal/\*" fn="send-file"
        Service method="TRACE" fn="service-trace"
      --- 18,23 ----

  5. Install Revision Control System (RCS), GNU diff utilities (diffutils) and update the PATH variable in user's .profile.

    Those packages are available in 'ready to install' form at and web sites. Perhaps the easiest way is to use pkg-get to pull those packages with the required dependencies from Blastwave.

    Assuming RCS and diffutils are available under /usr/csw/bin directory,
    % cat ~/.profile | grep PATH
    export PATH=/usr/bin:/usr/sbin:/usr/local/bin:/usr/sfw/bin:/usr/csw/bin:$PATH

  6. Configure the TWiki. Edit TWiki's \*.cfg files, that is.

    1. Create the config file LocalLib.cfg.

      1. There is a template for this config file in twiki/bin/LocalLib.cfg.txt. Copy LocalLib.cfg.txt to LocalLib.cfg.
        % cd /export/twiki/sjws7u2/https-<hostname>/docs/twiki/bin
        % cp LocalLib.cfg.txt LocalLib.cfg
      2. Edit LocalLib.cfg to update the $twikiLibPath variable.
        $twikiLibPath = "/export/twiki/sjws7u2/https-<hostname>/docs/twiki/lib";

    2. Manually create the config file LocalSite.cfg.
      % cd /export/twiki/sjws7u2/https-<hostname>/docs/twiki/lib
      % cp TWiki.spec LocalSite.cfg

    3. Update the following variables with appropriate values in LocalSite.cfg.

      $TWiki::cfg{DefaultUrlHost}, $TWiki::cfg{ScriptUrlPath}, $TWiki::cfg{PubUrlPath}, $TWiki::cfg{PubDir}, $TWiki::cfg{TemplateDir}, $TWiki::cfg{DataDir}, $TWiki::cfg{LocalesDir}, and $TWiki::cfg{OS}, $TWiki::cfg{RCS}{EgrepCmd} and $TWiki::cfg{RCS}{FgrepCmd}.

      This diff output shows how they were configured in the demo TWiki that I used for the Proof-Of-Concept (POC). The hardware it was deployed on is a Sun Fire T2000. So Cool Stack was installed on the server to take advantage of the optimized Perl.

  7. Exit the current shell; and reconnect.

  8. Restart the Sun Java Web Server.
    % cd /export/twiki/sjws7u2/https-<hostname>/bin
    % ./stopserv
    % ./startserv
  9. Manually execute the 'view' script from the command line.
    % cd /export/twiki/sjws7u2/https-/docs/twiki/bin
    % ./view
    You should see valid HTML (not errors) on the stdout. If you see errors, check /export/twiki/sjws7u2/https-<hostname>/logs/errors for the error messages. Fix the issues and re-run 'view' from the command line. Continue this exercise until the script returns valid HTML.

  10. Finally verify the TWiki installation through the web browser.
            Open http://<webhost>:<port>/twiki/view in a web browser.
  1. Installing TWiki on Sun Java System Web Server by Manish Kapur, Sun Microsystems
  2. Using PHP on Sun Java System Web Server by Joe McCabe, Sun Microsystems

Copy of these instructions are also available at my other weblog:

Tuesday Apr 08, 2008

Running Batch Workloads on Sun's CMT Servers

Ever since Sun introduced Chip Multi-Threading (CMT) hardware in the form of UltraSPARC T1's T1000/T2000, our internal mail aliases were inundated with variety of customer stories, majority of those go like 'batch jobs are taking 12+ hours on T2000, where as it takes only 3 or 4 hours on US-IV+ based v490'. Even after two and half years since the introduction of the revolutionary CMT hardware, it appears that majority of Sun customers are still under the impression that Sun's CMT systems like T2000, T5220 are not capable of handling CPU intensive batch workloads. It is not a valid concern. CMT processors like UltraSPARC T1, T2, T2 Plus can handle batch workloads just as well like any other traditional/conventional processor viz. UltraSPARC-IV+, SPARC64-VI, AMD Opteron, Intel Xeon, IBM POWER6. However CMT awareness and little effort are required at the customer end to achieve good throughput on CMT systems.

First of all, the end users must realize the fact that the maximum clock speed of the existing CMT processor line-up (UltraSPARC T1, UltraSPARC T2, UltraSPARC T2 Plus) is only 1.4 GHz; and on top of that each strand (individual hardware thread) within a core shares the CPU cycles with the other strands that operate on the same core (Note: each core operates at the speed of the processor). Based on these facts, it is no surprise to see batch jobs taking longer times to complete when only one or a very few single-threaded batch jobs are submitted to the system. In such cases, the system resources are fairly under-utilized in addition to the longer elapsed times. One possible trick to achieve the required throughput in the expected time frame is to split up the workload into multiple jobs. For example, if an EDU customer needs to generate 1000 transcripts, the customer should consider submitting 4 individual jobs with 250 transcripts each or 8 jobs with 125 transcripts each rather than submitting one job for all 1000 transcripts. Ideally the customer should observe the resource utilization (CPU%, for example); and experiment with the number of jobs to be submitted until the system achieves the desired throughput within the expected time frame.

Case study: Oracle E-Business Suite Payroll 11i workload on Sun SPARC Enterprise T5220

In order to prove that the aforementioned methodology works beyond a reasonable doubt, let's take Oracle's E-Business Suite 11.5.10 Payroll workload as an example. On a single T5220 with one 1.4 GHz UltraSPARC T2 processor, acting as the batch, application and database server, 4 payroll threads generated 5,000 paychecks in 31.53 minutes of time consuming only 6.04% CPU on average. ~9,500 paychecks is the projected hourly throughput. This is a classic example of what majority of Sun's CMT customers are experiencing as of today i.e., longer batch processing times with little resource consumption. Keep in mind that each UltraSPARC T2 and UltraSPARC T2 Plus processors can execute up to 64 jobs in parallel (on a side note, UltraSPARC T1 processor can execute up to 32 jobs in parallel). So to put the idling resources for effective use, there by to improve the elapsed times and the overall throughput, few experiments were conducted with 64 payroll threads and the results are very impressive. With a maximum of 64 payroll threads, it took only 4.63 minutes to process 5,000 paychecks at an average of 40.77% CPU utilization. In other words, similarly configured T5220 can process ~64,700 paychecks at less than half of the available CPU cycles. Here is a word of caution: just because the processor can execute 64 threads in parallel, it doesn't mean it is always optimal to submit 64 parallel jobs on systems like T5220. Very high number of batch jobs (payroll threads in this particular scenario) might be an overkill for simple tasks like NACHA in Payroll process.

The following white paper has more detailed information about the nature of the workload and the results from the experiments with various number of threads for different components of the Oracle Applications' Payroll batch workload. Refer to the same white paper for the exact tuning information as well.

Link to the white paper:
     E-Business Suite Payroll 11i (11.5.10) using Oracle 10g on a Sun SPARC Enterprise T5220

Here is the summary of the results that were extracted from the white paper:

Hardware configuration
          1x Sun SPARC Enterprise T5220 for running the application, batch and the database servers
              Specifications: 1x 1.4 GHz 8-core UltraSPARC T2 processor with 64 GB memory

Software configuration
          Oracle E-Business Suite 11.5.10
          Oracle 10g R1 RDBMS
          Solaris 10 8/07

Oracle E-Business Suite 11i Payroll - Number of employees: 5,000
Component #Threads Time (min) Avg. CPU% Hourly Throughput
Payroll process 64 1.87 90.56 160,714
PrePayments 64 0.20 46.33 1,500,000
Ext. Proc. Archive 64 1.90 90.77 157,895
NACHA 8 0.05 2.52 6,000,000
Check Writer 24 0.38 9 782,609
Costing 48 0.23 32.5 1,285,714
Total or Average NA 4.63 min 40.77% 64,748

It is evident from the average CPU% that the Payroll process and the External Process Archive components are extremely CPU intensive; and hence take longer time to complete. That's the reason 64 threads were configured for those components to run at the full potential of the system. Light-weight components like NACHA need fewer threads to complete the job efficiently. Configuring 64 threads for NACHA will have a negative impact on the throughput. In other words, we would be wasting CPU cycles for no apparent improvement.

It is the responsibility of the customers to tune the application and the workload appropriately. One size doesn't fit all.

The Payroll 11i results on the T5220 demonstrate clearly that Sun's CMT systems are capable of handling batch workloads well. It would be interesting to see how well they perform against other systems equipped with traditional processors with higher clock speeds. For this comparison, we could use couple of results that were published by UNISYS and IBM with the same workload. The following table summarizes the results from the following two white papers. For the sake of completeness, Sun's CMT results were included as well.

Source URLs:
  1. E-Business Suite Payroll 11i (11.5.10) using Oracle 10g on a UNISYS ES7000/one Enterprise Server
  2. E-Business Suite Payroll 11i (11.5.10) using Oracle 10g for Novell SUSE Linux on IBM eServer xSeries 366 Servers
Oracle E-Business Suite 11i Payroll - Number of employees: 5,000
Vendor OS Hardware Config #Threads Time (min) Avg. CPU% Hourly Throughput
UNISYS Linux: RHEL 4 Update 3 DB/App/Batch server: 1x Unisys ES7000/one Enterprise Server (4x 3.0 GHz Dual-Core Intel Xeon 7041 processors, 32 GB memory) 121 5.18 min 53.22% 57,915
IBM Novell SUSE Linux Enterprise Server 9 SP1 DB, App servers: 2x IBM eServer xSeries 366 4-way server (4x 3.66 GHz Intel Xeon MP Processors (EM64T), 32 GB memory) 12 8.42 min 50+%2 35,644
Sun Solaris 10 8/07 DB/App/Batch server: 1x Sun SPARC Enterprise T5220 (1x 1.4 GHz 8-core UltraSPARC T2 processor, 64 GB memory) 8 to 64 4.63 min 40.77% 64,748

Better results were highlighted. The results speak for themselves. One 1.4 GHz UltraSPARC T2 processor outperformed four 3 GHz / 3.66 GHz processors in terms of the average CPU utilization and most importantly in the hourly throughput (Hourly throughput calculation relies on the total elapsed time).

Before we conclude, let us reiterate few things purely based on the factual evidence presented in this blog post:

  • Sun's CMT servers like T2000, T5220, T5240 (two socket system with UltraSPARC T2 Plus processors) are good to run batch workloads like Oracle Applications Payroll 11i

  • Sun's CMT servers like T2000, T5220, T5240 are good to run the Oracle 10g RDBMS when the DML/DDL/SQL statements that make up the majority of the workload are not very complex, and

  • When the application is tuned appropriately, the performance of CMT processors can outperform some of the traditional processors that were touted to deliver the best single thread performance


1. There is a note in the UNISYS/Payroll 11i white paper that says "[...] the gains {from running increased numbers of threads} decline at higher numbers of parallel threads." This is quite contrary to what Sun observed in its Payroll 11i experiments on UltraSPARC T2 based T5220. Higher number of parallel threads (maximum: 64) improved the throughput on T5220, where as UNISYS' observation is based on their experiments with a maximum of 12 parallel threads. Moral of the story: do NOT treat all hardware alike.

2. IBM's Payroll 11i white paper has no references to the average CPU numbers. 50+% was derived from the "Figure 3: Average CPU Utilization".


Benchmark announcements, HOW-TOs, Tips and Troubleshooting


« April 2014