Murali Sriram's Blog

Hi Folks!! Well its been really a long time since i posted on my blog. Today i would like to discuss on Implementing PCP (Parallel Concurrent Processing) in a RAC/NON RAC Environment. There are plenty of articles on metalink on Implementing PCP but i will be discussing the fine grain details of PCP in this article.

In a PCP Implementation as we all know we have an Internal Monitor running on each Concurrent Manager Node, Internal Monitor is like a local police monitoring the process on the respective concurrent tier. Oracle introduced a new process called "REVIVER" post ATG_PF.H RUP4 and Release 12 onwards. The job of reviver to restart Internal Concurrent Manager in case of a lost of database connection. You can enable the reviver by turning the context variable flag to enabled in Autoconfig XML File "s_cp_reviver". Review Metalink Note 466752.1 for more information on FND Reviver.

In PCP Implementation when your concurrent tier and database tier are different (which is many RAC Implementations Yes) then you have two options for providing High Availability for the Concurrent Processes

1. If you turn Profile Option "Concurrent: PCP Instance Check" to ON then when your Database goes down then the concurrent managers connecting to the database node which went down will failover to the failover node defined at the concurrent manager level

2. If you turn Profile Option "Concurrent: PCP Instance Check" to OFF then when your Database goes down then the Reviver Process on the Middle Tier tries to restart the Internal Concurrent Manager by connecting to the next surviving database node

Similarly if your Concurrent Tier goes down the concurrent manager should failover after 2-3 PMON Cycles, however if you notice that managers didnt failover then Login to OAM and make the apps tier which went down "Offline"

Login -> OAM -> Click on the Apps Tier which went down -> Turn it "Offline"

• Implementing PCP with RAC
  
• What is FNDREVIVER?

Its been a while since i posted on my blog, Since now my other Project went live and we had a smooth first month end close, I am going to post a series of articles on learnings, new tools and tips and tricks on troubleshooting and tuning which i learnt.

I had this situation many times where i try to start a process ex: database listener, apache process etc... the process errors out because of some permissions problem or port conflict issue. I have been working on environments where we have several applications running on the same server and its pretty hard to identify what port/subport or directory is causing the issue.

After some research i came across a different usage of tool strace (Linux) or truss (UNIX),

    strace lsnrctl start DEV

References

• strace on WIKI

Its been long time since i posted on my blog, i have been very busy at the new project. I recently came across a new unix utility called Netcat. Netcat allows you send and receive data across the machines using TCP/UDP Protocols.

For example if i need to tar an oracle home and and untar on the target server i can use the following commands

On Source

• cd $ORACLE_HOME


• tar cvf - * | netcat <TARGET host><SOME port>

On Target

• cd to destination ORACLE_HOME


• netcat -l -p <PORT source in used>| tar xvf -

References

• http://netcat.sourceforge.net/


• http://en.wikipedia.org/wiki/Netcat

In this article i will show how we leveraged DB Cache Advice Statistics in tuning our Buffer Pool. We had to size the SGA/Buffer Pool for a huge conversion activity which we had to perform, based on buffer pool stats you can make a decision on sizing(approx) your buffer pool.

Below is an example of DB_CACHE_ADVICE data

select size_for_estimate,
       estd_physical_reads
from v$db_cache_advice

In the above example you can notice that by increasing the Buffer Pool size from 1.2G to 2.6G the Estimated Physical reads reduced from 91M to 49M which is approx 50%. However note that this is the not the end of tuning this is one of the step in tuning your instance.

References

• V$DB_CACHE_ADVICE View Description
• DB_CACHE_ADVICE Init.ora Parameter Details
• Jonathan Lewis article on db cache advice

In this article i will show you how to setup Cygwin as a Display Server which can be used for Installing Oracle Softwares (i.e to run runInstaller).

Cygwin is a free software available at www.cygwin.com. It provides an UNIX kind of environment on Windows where you can execute your shell scripts etc... It can also be used as Display Server when installed with X11 Packages. Below i will explain step by step on installing cygwin and configuring it as a display server

Installation Instructions

• Download Cygwin Setup utility from http://www.cygwin.com/
• Install Cygwin (during install in Select Packages screen make sure you select Utils and X11 Packages, ensure all the libraries under these two packages are selected, you need to manually select each package)
• Verify the Install (installation usually takes around 30 mins when you do internet install) by clicking on the cygwin icon on your desktop.

Configuring it as Display Server

• Start cygwin on your desktop/latop
• Type "startx" command without quotes on the cygwin window
• Once you see an xterm window popup type "xhost +" without quotes (note the space between plus sign and xhost). You will see a message as "access control disabled, clients can connect from any host"
• Type "ipconfig" and get the IP Address of your laptop/desktop
• Login to your unix server using putty or any other utility, export DISPLAY variable to your "<ipaddress>:0.0"
• Verify the display server by running xclock from putty session this will popup an xclock on your laptop.

Troubleshooting

• Many times you cannot ping your laptop/desktop from UNIX server this could be due to Windows Firewall, disable and try it.
• Sometimes TCP/IP Reset on Windows has helped in UNIX server able to ping your laptop/desktop (Note: execute this step with caution it might cause your LAN or Wireless Adapters not to work sometimes)

References

• Cygwin on Wikipedia
• On Solaris VNC Package is available on SUN OS Companion CD from which you can install SunVNC Package and use VNC Viewer for your Display server requirements.

In this article i would like to discuss couple of options we have in migrating Database storage to ASM (Automatic Storage Management). Recently we had a to migrate a moderately sized database to ASM storage and found migrating to ASM through enterprise manager was the Best Option, its was just few clicks and instance was migrated.

• Migrating using Enterprise Manager
• Migrating using Recovery Manager (RMAN)
• Migrating using Oracle DataGuard

With Enterprise Manager migration was pretty smooth, if you would like to try the RMAN approach then refer Metalink Note 312731.1 (section 3.8) for a step by step instructions on migrating

References

• PLSQL Scripts to Migrate from ASM to Non-ASM
• ASM Technical Whitepapers and Best Practices
• DBMS_FILE_TRANSFER API for transfer to and from ASM
• Disk Rebalancing in ASM using EM

IPMP is something new i learnt in this project. To avoid single point of failure to your Public(& Virtual) IP/Private IP in a RAC environment you can Implement Multipathing of IP Address. In this configuration you configure a set of Network Inteface Cards (NIC's) on each cluster node to support an IP Address. Here is how it works

One of the NIC will be used as the Primary link for all communications. Upon failure of the Primary NIC, IPMP will automatically fail the physical(& virtual) IP to the standby NIC

Reference on Configuring IPMP

• Metalink Note on configuring IPMP 283107.1
• Using Multipathing in Solaris 10 Operating System
• Solaris 10 Sysadmin Guide on IPMP

Recently of my one old customer mailed me asking why this particular odf running forever during the upgrade, after analysis found that the odf was creating an index on one of the key Account Receivables tables which had millions of records. This one odf was skewing the upgrade timings a lot even though the boxes were powerful.

Solution was to use adpatch parallel_index_threshold=<higher value>, i had used this option in my previous upgrade and made a big difference (however it depends on how many parallel_max_servers init.ora parameter you have set)

References

• AD Utilities Guide explaining various adpatch options

Its been a while since i posted something on my blog, now that we went Live i would like to share some of my recent project experiences in next few blog articles. We wanted to use Oracle CRS to provide high availability for EBS Databases because VCS has some issue in managing ASM Instance. After some analysis i found this whitepaper on Implementing CRS to manage Oracle Single Instance Database, the same can be found here

We Implemented this solution, we also extended the solution to shutdown and startup middle tier services incase of a failover. I will post some Test Cases on what we performed to validate the Cluster.

an interesting information how google make a difference BLACKLE

In this article i would like to discuss in brief about DMZ Implementations. Many times Customers have a requirement to setup DMZ Environment for Externally Visible Applications such as iStore, iRecruitment etc...but the next question they have is can they share the File System (i.e. appltop, comntop, iAS/806) with the external DMZ Server. Technically sharing the file system between the Internal Server and External Server is possible but it defeats the whole purpose of Setting up the DMZ Server, here is the explanation

             ||                            ||

Internet  || External Web Tier   || Internal Web Tier/Database Tier

             ||                            ||

            FW1                         FW2

If you share an appltop/comntop between External and Internal Server you are violating the concept of DMZ. A hacker playing with External Filesystem is also playing with your Internal Filesystem making your second Firewall useless.

It is also documented in Metalink Note 287176.1, However if you are having one or more servers inside the DMZ then you can Implement Shared Appltop among all the External Middle Tiers.

Folks, some good news from myside, I am applying the biggest patchset in my life yes i am getting married.I invite your gracious presence on our wedding in Bangalore, India on April 26, 2007, for more details please visit

• Our Wedding Site
  

What is the equivalent command for on Linux for lsps on AIX?? Well most of the times we struggle to find the equivalent commands on other Operating Systems from Solaris to Linux, I found the following link to be of great help to all the folks out on the field.

• Commands Guide
• Unix Guide

• Sizing Lounge
• BenchMark Results

Recently we had a challenge where staging table drop and truncates were causing "RO Enqueue" wait events, the problem was magnified by a huge buffer cache on a 3 node RAC Instance. We worked with Support for several weeks and later found that in 9i Instance when you do a drop/truncate of a table which is one of extent and the extent size is not more than half the size of buffer pool then drop/truncate does by flush of objectd, if the number of extents is more than one then it does a full buffer pool scan.

The concurrent program was using a tablespace to create staging tables which had low initial extent storage parameters causing the staging table created to be a multi extent table. by chaging to a tablespace which had bigger initial extent and could accomodate the staging table in one extent, the difference was unbelievable,

Try it out on your development instance by droping/truncating a multi extent table and a single extent table with huge buffer pool (50Gig??)