Friday Aug 10, 2007

Oracle 10gR2/Solaris x64: Must Have Patches for E-Business Suite 11.5.10

If you have an Oracle E-Business Suite 11.5.10 database running on Oracle 10gR2 (10.2.0.2) / Solaris x86-64 platform, make sure you have the following two Oracle patches to avoid concurrency issues and intermittent Oracle shadow process crashes.

Oracle patches

1) 4770693 BUG: Intel Solaris: Unnecessary latch sleeps by default
2) 5666714 BUG: ORA-7445 ON DELETE

Symptoms

1) If the top 5 database timed events look something similar to the following in AWR, it is very likely that the database is running into the bug 4770693 Intel Solaris: Unnecessary latch sleeps by default.

Top 5 Timed Events

EventWaitsTime(s)Avg Wait(ms)% Total Call TimeWait Class
latch: cache buffers chains 94,301 169,403 1,796 86.1Concurrency
CPU time-- 5,478-- 2.8--
wait list latch free 247,466 4,756 19 2.4Other
buffer busy waits 14,928 1,382 93 .7Concurrency
db file sequential read 98,750 552 6 .3User I/O

Apply Oracle server patch 4770693 to get rid of the concurrency issue(s). Note that the fix will be part of the release 10.2.0.3.

2) If the application becomes unstable and if you notice core dumps in cdump directory, have a look at the corresponding stack traces generated in udump directory. If the call stack looks similar to the following stack, apply Oracle server patch 5666714 to overcome this problem.

Alert log will have the following errors:

Errors in file /opt/oracle/admin/VIS/udump/vis_ora_1040.trc:
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [2168] [] [] []
Fri Jun 15 01:30:38 2007
Errors in file /opt/oracle/admin/VIS/udump/vis_ora_1040.trc:
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [9] [] [] []
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [2168] [] [] []
Fri Jun 15 01:30:38 2007
Errors in file /opt/oracle/admin/VIS/udump/vis_ora_1040.trc:
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [9] [] [] []
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [9] [] [] []
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [2168] [] [] [] 

% more vis_ora_1040.trc
...
...
\*\*\* 2007-06-15 01:30:38.403
\*\*\* SERVICE NAME:(VIS) 2007-06-15 01:30:38.402
\*\*\* SESSION ID:(1111.4) 2007-06-15 01:30:38.402
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x878
\*\*\* 2007-06-15 01:30:38.403
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [SIGSEGV] [Address not mapped to object] [2168] [] [] []
Current SQL statement for this session:
INSERT /\*+ IDX(0) \*/ INTO "INV"."MLOG$_MTL_SUPPLY" (dmltype$$,old_new$$,snaptime$$,change_vector$$,m_row$$) 
VALUES (:d,:o,to_date('4000-01-01:00:00:00','YYYY-MM-DD:HH24:MI:SS'),:c,:m)
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst()+23          ?        0000000000000001     00177A9EC 000000000 0061D0A60
                                                   000000000
ksedmp()+636         ?        0000000000000001     001779481 000000000 00000000B
                                                   000000000
ssexhd()+729         ?        0000000000000001     000E753CE 000000000 0061D0B90
                                                   000000000
_sigsetjmp()+25      ?        0000000000000001     0FDDCB7E6 0FFFFFD7F 0061D0B50
                                                   000000000
call_user_handler()  ?        0000000000000001     0FDDC0BA2 0FFFFFD7F 0061D0EF0
+589                                               000000000
sigacthandler()+163  ?        0000000000000001     0FDDC0D88 0FFFFFD7F 000000002
                                                   000000000
kglsim_pin_simhp()+  ?        0000000000000001     0FFFFFFFF 0FFFFFFFF 00000000B
173                                                000000000
kxsGetRuntimeLock()  ?        0000000000000001     001EBF830 000000000 005E5D868
+683                                               000000000
kksfbc()+7361        ?        0000000000000001     001FB60A6 000000000 005E5D868
                                                   000000000
opiexe()+1691        ?        0000000000000001     0029045D0 000000000 0FFDF9250
                                                   0FFFFFD7F
opiall0()+1316       ?        0000000000000001     0028E9FB9 000000000 000000001
                                                   000000000
opikpr()+536         ?        0000000000000001     00290B2DD 000000000 0000000B7
                                                   000000000
opiodr()+1087        ?        0000000000000001     000E7BE1C 000000000 000000001
                                                   000000000
rpidrus()+217        ?        0000000000000001     000E8058E 000000000 0FFDFA6B8
                                                   0FFFFFD7F
skgmstack()+163      ?        0000000000000001     003F611D0 000000000 005E5D868
                                                   000000000
rpidru()+129         ?        0000000000000001     000E808A6 000000000 005E6FAD0
                                                   000000000
rpiswu2()+431        ?        0000000000000001     000E7FD8C 000000000 0FFDFB278
                                                   0FFFFFD7F
kprball()+1189       ?        0000000000000001     000E86E6A 000000000 0FFDFB278
                                                   0FFFFFD7F
kntxslt()+3150       ?        0000000000000001     0030601F3 000000000 005F7C538
                                                   000000000
kntxit()+998         ?        0000000000000001     003058EBB 000000000 005F7C538
                                                   000000000
0000000001E4866E     ?        0000000000000001     001E4864B 000000000 000000000
                                                   000000000
delrow()+9170        ?        0000000000000001     0032020B7 000000000 000000002
                                                   000000000
qerdlFetch()+640     ?        0000000000000001     0033545F5 000000000 0EF38B020
                                                   000000003
delexe()+909         ?        0000000000000001     0032034EA 000000000 005E6FC50
                                                   000000000
opiexe()+9267        ?        0000000000000001     002906368 000000000 000000001
                                                   000000000
opiodr()+1087        ?        0000000000000001     000E7BE1C 000000000 0FFDFCD10
                                                   0FFFFFD7F
ttcpip()+1168        ?        0000000000000001     003D031AD 000000000 0FFDFEDF4
                                                   0FFFFFD7F
opitsk()+1212        ?        0000000000000001     000E77C41 000000000 000E7BA00
                                                   000000000
opiino()+931         ?        0000000000000001     000E7B0D8 000000000 005E5B8F0
                                                   000000000
opiodr()+1087        ?        0000000000000001     000E7BE1C 000000000 000000000
                                                   000000000
opidrv()+748         ?        0000000000000001     000E76A11 000000000 0FFDFF6D8
                                                   0FFFFFD7F
sou2o()+86           ?        0000000000000001     000E73E6B 000000000 000000000
                                                   000000000
opimai_real()+127    ?        0000000000000001     000E3A7C4 000000000 000000000
                                                   000000000
main()+95            ?        0000000000000001     000E3A694 000000000 000000000
                                                   000000000
0000000000E3A4D7     ?        0000000000000001     000E3A4DC 000000000 000000000
                                                   000000000
 
--------------------- Binary Stack Dump ---------------------
 
========== FRAME [1] (ksedst()+23 -> 0000000000000001) ==========
Dump of memory from 0x00000000061D0910 to 0x00000000061D0920
0061D0910 061D0920 00000000 0177A9EC 00000000  [ .........w.....]
========== FRAME [2] (ksedmp()+636 -> 0000000000000001) ==========
Dump of memory from 0x00000000061D0920 to 0x00000000061D0A60
0061D0920 061D0A60 00000000 01779481 00000000  [`.........w.....]
0061D0930 0000000B 00000000 061D0EF0 00000000  [................]
0061D0940 05E5B96C 00000000 05E5C930 00000000  [l.......0.......]
0061D0950 05E5C930 00000000 FE0D2000 FFFFFD7F  [0........ ......]
0061D0960 061D0A40 00000000 00000000 00000000  [@...............]
0061D0970 00000000 00000000 00000000 00000000  [................]
...
...

After installing the Oracle database patches, check the installed patches by running opatch lsinventory on your database server.

% opatch lsinventory
Invoking OPatch 10.2.0.1.0

Oracle interim Patch Installer version 10.2.0.1.0
Copyright (c) 2005, Oracle Corporation.  All rights reserved..


Oracle Home       : /oracle/product/10.1.0
Central Inventory : /export/home/oracle/oraInventory
   from           : /oracle/product/10.1.0/oraInst.loc
OPatch version    : 10.2.0.1.0
OUI version       : 10.2.0.1.0
OUI location      : /oracle/product/10.1.0/oui
Log file location : /oracle/product/10.1.0/cfgtoollogs/opatch/opatch-2007_Aug_10_21-56-03-PDT_Fri.log

Lsinventory Output file location : 
/oracle/product/10.1.0/cfgtoollogs/opatch/lsinv/lsinventory-2007_Aug_10_21-56-03-PDT_Fri.txt

--------------------------------------------------------------------------------
Installed Top-level Products (3): 

Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Products                                         10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 1                            10.2.0.2.0
There are 3 products installed in this Oracle Home.


Interim patches (2) :

Patch  4770693      : applied on Thu Aug 02 15:27:23 PDT 2007
   Created on 12 Jul 2006, 11:52:39 hrs US/Pacific
   Bugs fixed:
     4770693

Patch  5666714      : applied on Fri Jul 20 10:21:33 PDT 2007
   Created on 29 Nov 2006, 04:52:58 hrs US/Pacific
   Bugs fixed:
     5666714

--------------------------------------------------------------------------------

OPatch succeeded.
About

Benchmark announcements, HOW-TOs, Tips and Troubleshooting

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today