Siebel on Sun CMT hardware : Best Practices
By Giri Mandalika on Oct 13, 2008
The following suggested best practices are applicable to all Siebel deployments on CMT hardware (Tx00, T5x20, T5x40) running Solaris 10 [Note: some of this tuning applies to Siebel running on conventional hardware running Solaris]. These recommendations are based on our observations from the 14,000 user benchmark on Sun SPARC Enterprise T5440. Your mileage may vary.All Tiers
- Ensure that the system's firmware is up-to-date.
- Check the Sun System Firmware Release Hub for the latest firmware.
- Upgrade to the latest update release of Solaris 10.
- Note to the customers running Siebel on Solaris 5/08: apply the kernel patch 137137-07 as soon as it is available on sunsolve.sun.com web site. Patch 137137-07 and later revisions, Solaris 10 10/08 will have the workaround to a critical Siebel specific bug. Oracle Corporation will eventually fix the bug in their codebase - in the meantime Solaris is covering for Siebel and all other 32-bit applications with their own memory allocators that return unaligned mutexes. Check the RFE 6729759 Need to accommodate non-8-byte-aligned mutexes and Oracle's Siebel support document 735451.1 Do NOT apply Kernel Patch 137111-04 on Solaris 10 for more details.
- Enable 256M large pages on all nodes. By default, the latest update of Solaris 10 will use a maximum of 4M pages even when 256M pages are a good fit.
- 256M pages can be enabled with the following
\* 256M pages set max_uheap_lpsize=0x10000000
- Set the following in a shell or add the following lines to the shell's profile (bash/ksh).
ulimit -n 2048 export LD_PRELOAD_32=/usr/lib/extendedFILE.so.1:$LD_PRELOAD_32
Technically the file descriptor limit can be set to as high as 65536. However from the application's perspective, 2048 is a reasonable limit.
To improve the scalability of the multi-threaded workloads, preload MT-hot object-caching memory allocation library like
- eg., To preload the
libumemlibrary, set the LD_PRELOAD_32 environment variable in the shell (bash/ksh) as shown below.
Web and the Application servers in the Siebel Enterprise stack are 32-bit. However Oracle 10g or 11g RDBMS on Solaris 10 SPARC is 64-bit. Hence the path to the
libumem library in the PRELOAD statement differs slightly in the database-tier as shown below.
Be aware that the trade-off is the increase in memory footprint -- you may notice 5 to 20% increase in the memory footprint with one of these MT-hot memory allocation libraries preloaded. Also not every Siebel application module benefits from MT-hot memory allocators. The recommendation is to experiment before implementing in production environments.
Application fared well with the following set of TCP/IP parameters on Solaris 10 5/08.
ndd -set /dev/tcp tcp_time_wait_interval 60000 ndd -set /dev/tcp tcp_conn_req_max_q 1024 ndd -set /dev/tcp tcp_conn_req_max_q0 4096 ndd -set /dev/tcp tcp_ip_abort_interval 60000 ndd -set /dev/tcp tcp_keepalive_interval 900000 ndd -set /dev/tcp tcp_rexmit_interval_initial 3000 ndd -set /dev/tcp tcp_rexmit_interval_max 10000 ndd -set /dev/tcp tcp_rexmit_interval_min 3000 ndd -set /dev/tcp tcp_smallest_anon_port 1024 ndd -set /dev/tcp tcp_slow_start_initial 2 ndd -set /dev/tcp tcp_xmit_hiwat 799744 ndd -set /dev/tcp tcp_recv_hiwat 799744 ndd -set /dev/tcp tcp_max_buf 8388608 ndd -set /dev/tcp tcp_cwnd_max 4194304 ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 67500 ndd -set /dev/udp udp_xmit_hiwat 799744 ndd -set /dev/udp udp_recv_hiwat 799744 ndd -set /dev/udp udp_max_buf 8388608
Siebel Application Tier
- All T-series systems (T1000/T2000, T5120/T5220, T5120/T5240, T5440) support the 256M page size. However Siebel's
siebmtshwscript restricts the page size to 4M. Comment out the following lines in
# This will set 4M page size for Heap and 64 KB for stack MPSSHEAP=4M MPSSSTACK=64K MPSSERRFILE=/tmp/mpsserr LD_PRELOAD=/usr/lib/mpss.so.1 export MPSSHEAP MPSSSTACK MPSSERRFILE LD_PRELOAD
- Configure the Object Managers in such a way that each OM will be handling at least 200 active users. Siebel's standard recommendation of 100 or less users per Object Manager is suitable for conventional systems but not ideal for CMT systems like Tx000, T5x20, T5x40, T5440. Sun's CMT systems are ideal for running multi-threaded processes with tons of LWPs per process. Besides, there will be significant improvement in the overall memory footprint with less number of Siebel Object Managers.
- Oracle 10g R2 10.2.0.4 32-bit client is supposed to have a fix for the process crash issue - however it wasn't verified in our test environment.
Siebel Database Tier
- Eliminate double buffering by forcing the file system to use direct I/O.
Oracle database caches the data in its own cache within the shared global area (SGA) known as the database block buffer cache. Database reads and writes are cached in block buffer cache so the subsequent accesses for the same blocks do not need to re-read the data from the operating system. On the other hand, file systems on Solaris default to reading the data though the global file system cache for improved I/O operations. That is, by default each read is cached potentially twice - one copy in the operating system's file system cache, and the other copy in Oracle's block buffer cache. In addition to double caching, there is also some extra CPU overhead for the code which manages the operating system's file system cache. The solution is to eliminate double caching by forcing the file system to bypass the OS file system cache when reading and writing to the disk.
- In the 14,000 user benchmark setup, the UFS file systems (holding the data files and the redo logs) were mounted with the
mount -o forcedirectio /dev/dsk/<partition> <mountpoint>
- In the 14,0000 user benchmark setup, there are two Sun StorateTek 2540 arrays connected to the T5440 - one array was holding the data files, where as the other was holding the Oracle redo log files.
- In the 14,0000 user benchmark setup, two online redo logs were configured each with 10 GB disk space. When all 14,000 concurrent users are on-line, there is only one log switch in a 60 minute period.
Oracle Database Initialization Parameters
DB_FILE_MULTIBLOCK_READ_COUNTto appropriate value.
DB_FILE_MULTIBLOCK_READ_COUNTparameter specifies the maximum number of blocks read in one I/O operation during a sequential scan.
- In the 14,0000 user benchmark configuration,
DB_BLOCK_SIZEwas set to 8 kB. During the benchmark run, the average reads are around 18.5 kB per second. Hence setting
DB_FILE_MULTIBLOCK_READ_COUNTto a high value does not necessarily improve the I/O performance. A value of 8 for the database init parameter
DB_FILE_MULTIBLOCK_READ_COUNTseems to perform better.
CPU_COUNTto 64. Otherwise, by default Oracle RDBMS assumes 128 and 256 for the
CPU_COUNTon T5240 and T5440 respectively. Oracle's optimizer might use a completely different execution plan when it notices such a large number for the
CPU_COUNT; and the resulting execution plan need not necessarily be an optimal one. In the 14,000 user benchmark, setting
CPU_COUNTto 64 produced optimal execution plans.
_enable_NUMA_optimizationto FALSE. On these multi-socket servers,
_enable_NUMA_optimizationwill be set to TRUE by default. During the 14,000 user benchmark run, we noticed intermittent shadow process crashes with the default behavior. We didn't realize any additional gains either with the default NUMA optimizations.
Siebel Web Tier
- Upgrade to the latest service pack of Sun Java Web Server 6.1 (32-bit).
- Run the Sun Java Web Server in multi-process mode by setting the
magnus.confto a value that is greater than 1. In the multi-process mode, the web server can handle requests using multiple processes with multiple threads in each process.
- When you specify a value greater than 1 for the
MaxProcs, the web server relies on the operating system to distribute connections among/between multiple web server processes. However many modern operating systems including Solaris do not distribute connections evenly, particularly when there are a small number of concurrent connections.
- Tune the maximum simultaneous requests by setting the
RqThrottleparameter in the
magnus.conffile to appropriate value. A value of 1024 was used in the 14,000 user benchmark.