Friday Nov 18, 2011

Siebel Troubleshooting : An ODBC error occurred; SBL-GEN-03006: Error calling function: DICFindTable m_pReqTbl


A newly installed Siebel application server fails to start despite successful ODBC connectivity to the database. SRProc process logs ODBC error messages similar to the following:

Message: GEN-13,
 Additional Message: dict-ERR-1109: 
       Unable to read value from export file (Data length (32) > Column definition (3)).

Message: GEN-13,
 Additional Message: dict-ERR-1107: Unable to read row 0 from export file (UTLDataValRead pBuf, col 4 ).

GenericLog  GenericError  1     0002157..  11-11-18 13:28  Message: Generated SQL statement:,
 Additional Message: SQLFetch:

Message: Error: An ODBC error occurred,
 Additional Message: Function: DICGetRDObjects; ODBC operation: SQLFetch

Message: GEN-13,
 Additional Message: dict-ERR-1109: Unable to read value from export file (UTLCompressFRead (fseek)).

Message: GEN-13,
 Additional Message: dict-ERR-1107: Unable to read row 0 from export file (UTLDataValRead pBuf, col 0 ).

Message: GEN-10,
 Additional Message: Calling Function: DICLoadDObjectInfo; Called Function: Calling DICGetRDObjects

Message: GEN-10,
 Additional Message: Calling Function: DICLoadDict; Called Function: DICLoadDObjectInfo

(srpdb.cpp (860) err=3006 sys=2) SBL-GEN-03006: Error calling function: DICFindTable m_pReqTbl
(srpsmech.cpp (74) err=3006 sys=0) SBL-GEN-03006: Error calling function: DICFindTable m_pReqTbl
(srpmtsrv.cpp (107) err=3006 sys=0) SBL-GEN-03006: Error calling function: DICFindTable m_pReqTbl
(smimtsrv.cpp (1203) err=3006 sys=0) SBL-GEN-03006: Error calling function: DICFindTable m_pReqTbl
SmiLayerLog Error       Terminate process due to unrecoverable error: 3006. (Main Thread)

An inconsistent or corrupted dictionary file "diccache.dat" is likely the cause.


  • Stop the application server and manually kill the remaining Siebel application specific processes


    stop_server all
    pkill siebmtsh
    pkill siebproc
  • Remove $SIEBEL_HOME/bin/diccache.dat file. It will be re-generated during the application server startup

  • Start the application server
    start_server all

Thursday Oct 06, 2011

Siebel Connection Broker Load Balancing Algorithm

Siebel server architecture supports spawning multiple application object manager processes. The Siebel Connection Broker, SCBroker, tries to balance the load (incoming requests) across different object manager processes running in a single Siebel server.

Least Loaded or Round Robin?

By default, SCBroker forwards the incoming request to any object manager process that is least loaded - meaning the process with the least number of running tasks. In Siebel terminology, this behavior is referred as "least-loaded" or "LL" connection forwarding algorithm. While the default LL algorithm provides the optimal behavior in the best case scenarios, it may lead to serious availability problems if one of several object manager prcesses running in a Siebel server stops responding in a timely fashion [for some reason]. Such an object manager may still accept requests though it may timeout. At some point, the unresponsive/hung or erroneous object manager will have the least number of tasks that may prompt SCBroker component to forward new incoming requests to that object manager process - which in turn leads to a stalemate. To avoid such situations, it is recommended to configure "round-robin" or "RR" algorithm in SCBroker component. When round-robin algorithm is configured, SCBroker ignores the number of running tasks per object manager process and routes all requests to all object managers in a round robin fashion.

While both algorithms have their strengths and weaknesses, customers must weigh both options and choose the one that fits best in their deployment.


Find the current load balancing algorithm:

srvrmgr>  list advanced param ConnForwardAlgorithm for comp SCBroker \
             show PA_ALIAS, PA_VALUE, PA_NAME

PA_ALIAS              PA_VALUE  PA_NAME                                    
--------------------  --------  -----------------------------------------  
ConnForwardAlgorithm  LL        Connection Forward algorithm for SCBroker

Configure SCBroker to use round-robin algorithm:

srvrmgr> change param ConnForwardAlgorithm=RR for comp SCBroker server SERVER_NAME
Command completed successfully.

srvrmgr> list advanced param ConnForwardAlgorithm for comp SCBroker \
            show PA_ALIAS, PA_VALUE, PA_NAME

PA_ALIAS              PA_VALUE  PA_NAME                                    
--------------------  --------  -----------------------------------------  
ConnForwardAlgorithm  RR        Connection Forward algorithm for SCBroker

Other SCBroker parameters of interest: ConnForwardTimeout and ConnRequestTimeout

Saturday Dec 04, 2010

Oracle's Optimized Solution for Siebel CRM 8.1.1

A brief explanation of what an optimized solution is and what it is not can be found in the previous blog entry Oracle's Optimized Solution for PeopleSoft HCM 9.0. We went through a similar exercise to publish another optimized solution around Siebel CRM 8.1.1.

The Siebel solution implements Oracle Siebel CRM using a unique combination of SPARC servers, Sun storage, Solaris OS virtualization, Oracle application middleware and Oracle database products.

URLs to the Siebel CRM white papers:

White you are at it, do not forget to check the 13,000 user Siebel CRM benchmark on the latest SPARC T3 platform.

Sunday May 17, 2009

Installing Siebel Web Extension (SWE) on top of Sun Java System Web Server 7.0

As of today, Sun Java System Web Server 7.0 is not a certified platform to deploy Siebel 8.x enterprise on. We are working with Oracle Corporation to make this certification happen so our customers can take advantage of the performance optimizations that went into the web server release 7.0.

Meanwhile those who want to give it a try can do so with little effort. In release SJSWS 7.0, the start/stop/restart/.. scripts were appropriately relocated to bin directory under the virtual web server instance. The installer for Siebel 8.x Web Server Extension looks for the start script [of the web server] under the home directory of the virtual web server instance. (because it was the default location until the release of SJSWS 7.0). The installation fails if the installer cannot find the start script in the location it is expecting it to be.

Due to the relocation mentioned above, installation of the Siebel Web Server Extension fails at the very last step where it tries to modify the start script with a bunch of LD_PRELOADs so the Siebel Web Extension loads up and runs on the Sun Java System Web Server. To get around this failure, all you have to do is to create a symbolic link in the home directory of the virtual web server instance pointing to the startserv script residing in the bin directory.

The following example shows the necessary steps.

% pwd

% ln -s bin/startserv start

% ls -l start
lrwxrwxrwx   1 pspp     dba           13 May 17 17:01 start -> bin/startserv

Install Siebel Web Extension in the normal way. No other changes are required.


% ls -l start\*
-rwxr-xr-x   1 pspp     dba         4157 May 17 17:38 start
-rwxr-xr-x   1 pspp     dba         3456 May 17 17:38 start_.bak

% mv bin/startserv bin/startserv.orig
% mv start bin/startserv

Notice that the Siebel installer actually made two copies of the startup script from the symbolic link. The original bin/startserv remained intact after the SWE installation.

Finally start the Web Server instance by running the startserv script. It should start with no issues.

% pwd

% ./startserv
Sun Java System Web Server 7.0U5 B03/10/2009 16:38
info: swe_init reports: SWE plug-in log file
info: CORE5076: Using [Java HotSpot(TM) Server VM, Version 1.5.0_15] from [Sun Microsystems Inc.]
info: HTTP3072: http-listener-1: http://siebel-pspp:8000 ready to accept requests
info: CORE3274: successful server startup

Before we conclude, do not forget the fact that Sun Java System Web Server 7.0 is not yet certified with Siebel 8.x release. Use the instructions mentioned in this blog post at your own risk. However if you do like to take that risk, consider installing the latest release of Sun Java System Web Server, which is SJSWS 7.0 Update 5 as of this writing.

Stay tuned for the certification news though.

Saturday Dec 20, 2008

Siebel on Sun Solaris: More Performance with Less Number of mprotect() Calls

By default each transaction in Siebel CRM application makes a large number of serialized mprotect() calls which in turn may degrade the performance of Siebel. When the load is very high on the Siebel application servers, the mprotect() calls are serialized by the operating system kernel resulting in high number of context switches and low CPU utilization.

If a Siebel deployment exhibits the above mentioned pathological conditions, performance / scalability of the application can be improved by limiting the number of mprotect() calls from the application server processes during the run-time. To achieve this behavior, set the value of Siebel CRM's AOM tunable parameter MemProtection to FALSE. Starting with the release of Siebel 7.7, the parameter MemProtection is a hidden one with the default value of TRUE. To set its value to FALSE, run the following command from the CLI version of Siebel Server Manager - srvrmgr.

change param MemProtection=False for comp <component_alias_name> server <siebel_server_name>


component_alias_name is the alias name of the AOM component to be configured. eg., SCCObjMgr_enu is the alias for the Call Center Object Manager, and

siebel_server_name is the name of the Siebel Server for which the component being configured.

Note that this parameter is not a dynamic one - hence the Siebel application server(s) must be restarted for this parameter to be effective.

Run truss -c -p <pid_of_any_busy_siebmtshmw_process> before and after the change to see how the mprotect system call count varies.

For more information about this tunable on Solaris platform, check Siebel Performance Tuning Guide Version 7.7 or later in Siebel Bookshelf.

See Also:
Siebel on Sun CMT hardware : Best Practices

(Originally posted on blogger at:
Siebel on Sun Solaris: More Performance with Less mprotect() Calls)

Monday Dec 08, 2008

Consolidating Siebel CRM 8.0 on a Single Sun SPARC Enterprise Server, T5440

.. blueprint document is now available on Here is the direct link to the blueprint:
            Consolidating Oracle Siebel CRM 8 on a Single Sun SPARC Enterprise Server.

Siebel 8.0 Platform Sizing and Performance Program (PSPP) benchmark workload was used in running all the performance tests using Solaris Containers and Logical Domains (LDoms) on a single Sun SPARC Enterprise T5440 server running Solaris 10 5/08 (containers) and 10/08 (LDoms). The focus of the blueprint is around 3 major configurations -- performance numbers at 3500 users (small configuration), 7000 users (medium configuration) and 14000 user (large configuration) loads. Hence this blueprint document complements the 14,000 user Siebel 8.0 benchmark that we published earlier back in October 2008.

The blueprint has details around the resource allocations for all the tiers of a typical Siebel deployment to support 3500, 7000 and 14000 concurrent users, offers performance tuning tips that are specific to Solaris and Sun CMT systems, and shows the results from the 3500, 7000 and 14000 user performance tests using Solaris 10's virtualization technologies - Solaris Containers and the Logical Domains.


  1. All the performance tests were conducted either with Solaris Containers or with the Logical Domains, but not with a mix of both of those technologies.

  2. Resource allocations were identical in both cases -- that is, with the Solaris Containers and the Logical Domains.

Saturday Nov 15, 2008

Yet Another Siebel 8.0 PSPP Benchmark on Sun CMT Hardware ..

.. Sun SPARC Enterprise T5240.

(This blog entry also serves as a summary page for all the Siebel 8.0 benchmarks that Sun published so far.)

Yesterday Sun published a brand new 10,000 user Siebel 8.0 benchmark result using a combination of T5240 and T5120 servers. In this benchmark, a Sun SPARC Enterprise T5240 server equipped with two 1.2 GHz, 8-Core UltraSPARC T2 Plus processors served as the system under test on which we ran the Siebel Gateway and Enterprise application servers. Two Sun SPARC Enterprise T5120 servers equipped with one 1.2 GHz, 8-Core UltraSPARC T2 processor were configured to run the Oracle database and the Sun Java System Web servers.

A copy of the latest benchmark publication is available on Oracle Applications' benchmark web site at:
        Siebel CRM Release 8.0 Industry Applications and Oracle 10g R2 DB on Sun SPARC Enterprise T5120 & T5240 servers running Solaris 10

For some reason, the topology diagram in the benchmark publication document was messed up esp. the fonts -- probably the odt -> doc -> pdf conversion. The clean copy of the diagram is shown below.

Significance of the Siebel 8.0 on T5240 benchmark

In case if anyone wonder why do we need another Siebel 8.0 benchmark on CMT hardware esp. when we already published couple of Siebel 8.0 benchmarks on T5220 and T5440 systems -- 10,000 users on T5120/T5220 and 14,000 users on T5440, the answer is simple: to show linear scalability.

In the first benchmark that Sun published in January 2008, we showed the scalability of the application, Siebel, on T5220 systems. We were able to scale up to 5,000 concurrent users on a single T5220 system (running the Siebel application servers) with 1.2 GHz, 8-Core US-T2 processor. We've used two such systems to publish the 10,000 user benchmark in the first installment.

The goal of the second benchmark that we published in October 2008 during the T5440 server launch showcases how to consolidate multiple workloads on a T5440 server. We demonstrated it by deploying the whole Siebel Enterprise -- Sun Java System Web Server along with the Siebel Web Server plug-in Siebel Web Engine (SWE), Siebel Gateway Server, Siebel Application Server and the Oracle Database Server -- on a single Sun SPARC Enterprise T5440 server equipped with four 1.4 GHz, 8-Core UltraSPARC T2 Plus processors. We ran 14,000 concurrent virtual users against this setup to make it a benchmark publication. Since we ran all tiers of Siebel Enterprise on the same box, it is hard to compare the scalability numbers from this benchmark against the numbers that we published in the 10,000 user benchmark on T5120/T5220 servers.

In April 2008, Sun has launched the first multi-processor CMT system, Sun SPARC Enterprise T5240. T5240 holds two UltraSPARC T2 Plus processors, and is supposed to exhibit 2x performance[1] as that of a T5220. In other words, two T5220 servers can be consolidated onto a single T5240. To prove this, we re-ran the 10,000 user benchmark that we published back in January 2008 by replacing the two T5220 servers in the application tier with a T5240 server, and keeping the remaining hardware configuration for the web and database servers intact. The results from this benchmark speak for themselves - but for your convenience here is the quick summary of the results.

#users #units x Server Model Business Transactions
Benchmark Publication
URL & Date
10,000 2 x T5220 142,061 1,136,488 10K/T5220, 01/2008
10,000 1 x T5240 141,205 1,129,640 10K/T5240, 11/2008

If you are a Sun-Oracle customer, make sure to check the Siebel on Sun CMT hardware : Best Practices web page for some useful tips.

Related entries:

  1. Siebel 8.0 on Sun SPARC Enterprise T5440 - More Bang for the Buck!!
  2. Sun publishes 10,000 user Siebel 8.0 PSPP benchmark on Niagara 2 systems
  3. Siebel CRM 8.0 PSPP UltraSPARC T2 beats POWER6 and sets World Record

[1] There is no unique definition for the word 'performance' -- it has different meanings based on the context.

Benchmark announcements, HOW-TOs, Tips and Troubleshooting


« April 2014