Is It Safe to Use SANs for EBS R12 Instance Tops?

Our documentation about sharing filesystems between multiple Oracle E-Business Suite Release 12 application servers recommends that you install the Instance Top (INST_TOP) on a local filesystem. This has prompted an interesting discussion about whether this is really mandatory, or whether it's technically feasible to put the Instance Top on, say, a dedicated fibre-attached SAN. 

Release 12 shared filesystem:

Our guidance on the INST_TOP being installed on a local file system is based on three major considerations:

  1. Separation of duties and security implications
  2. Impact of SAN performance on Apache
  3. Additional troubleshooting complexity

1. Separation of Duties & Security Implications

Our recommended configuration allows for different file system privileges and ownership between the Instance Top (INST_TOP) and the Code Top (ORACLE_HOMEs & APPL_TOP). This allows for the segregation of duties between administrators for the respective servers. Patching can be done on the Code Top by central system administrators who own the central shared portion of the file system. Instance Tops can be owned by instance sysadmins, who usually already own the CPU box with local storage.

Some instance-specific, run-time-generated files (e.g. reports, temp files) can include unencrypted data. Contrast those with database files (DBFs), which can be self-encrypted or contain encrypted data. Even with encrypted file system solution in place, there is less depth in defenses around some of the INST_TOP files.

2. Impact of SAN Performance on Apache

Apache performance is highly sensitive to mutex file access latency, and at higher loads is also sensitive to I/Os per second.  We tried using a central SAN for INST_TOPs in our internal EBS development environments but found the performance to be unacceptable.  However, not all SANs are created equal, and depending on the SAN, it might be good for even production use.

A very good article on this point is available from the SQLTeam web site:

3. Additional Troubleshooting Complexity

A network storage access problem can have a spectrum of symptoms, including performance slow downs and even as intermittent end-user session failures. Some of the affected code paths were made more resilient over the years, but we still prefer to err on the side of prudence and not potentially cause these (hard to diagnose) problems.

Your Mileage May Vary

All that said, you might decide that your testing of SAN performance demonstrates that its latency and I/O transaction throughput are good enough for your requirements. 

Our Support and Development teams will attempt to reproduce any reported issues in a multinode environment where the INST_TOPs are stored on a local filesystem.  If the issues are isolated to the external placement of the Instance Tops, our recommendations would be to either revert back to local storage, or to work with your SAN vendor to optimize the SAN's performance.

References

Related Articles

Comments:

Hi Ivo,

One of our recent experiences may give you another reason why the INST_TOP should be installed on a local file system.

We are running 11i shared file system architect on 4 app servers. Recently one of our ADMINs transferred our admin scripts from app1 to app2,3,4 in an attempt to keep the scripts and scheduled cron in sync. So, if one app server dies, others still function without impacting operations. But one of the scripts invoking 11i startup/shutdown scripts is server specific and 11i $ADMIN_SCRIPTS_HOME is on the $COMMON_TOP/admin/scripts/${CONTEXT_NAME} directory:

$COMMON_TOP/admin/scripts/xxxxxprd_xxxxx-app1
$COMMON_TOP/admin/scripts/xxxxxprd_xxxxx-app2
$COMMON_TOP/admin/scripts/xxxxxprd_xxxxx-app3
$COMMON_TOP/admin/scripts/xxxxxprd_xxxxx-app4

Because $COMMON_TOP resides on the shared storage, basically it allows the server specific script to run on different servers, e.g. we can stop all app2 services from app2 and start all app2 services from app3 at the same time. That was the accident that happened to us and caused a great deal of headache and required intensive troubleshooting (SR: 7774130.992 - I also escalated the issue to Steven Chan).

Apparently, the R12 architects are doing a better job in permitting apps DBAs to define the environment variable $ADMIN_SCRIPTS_HOME which points to the "$INST_TOP/admin/scripts" directory and it is recommended that this resides on the local disk so that sever specific scripts are truly segregated.

Hope this helps.
Jennifer.

Posted by Jennifer Chen on October 01, 2009 at 04:12 AM PDT #

Hi Steven,
We are in the process of setting up an Advanced Supply Chain Planning environment with R12 and looking into implementing it with the Shared Application FS architecture. After I saw the recommendation on the INST_TOP in the manual and on Metalink, I decided to open an SR (3-2111204121) for clarification because it did not make much sense to me. We currently have an existing 11.5.10.2 environment, which we implemented with the shared architecture about two years ago and used NFS for that purpose. In that environment, we directed the Apache Lock file to the local FS but the rest remained on NFS and it has been working fine for us. INST_TOP contains configuration files and log files (Apache, OPMN, etc). With the exception of Apache log file, the rest are not written to with frequent interval. So, the question for you is:
- If the only issue is (in terms of performance) with the Apache Lock file which that could be pointed to a local file system and the rest of the INST_TOP could be left on NFS. What issue do you see with this configuration?
- If you look at the Enterprise Deployment Guide for Oracle SOA Suite 11g, the recommended architecture from Oracle is to implement the entire suite on NFS. SOA 11g also uses Apache from OHS 10.1.3.x to front end the WebLogic Server. Your recommendation seems to be in conflict with what the Oracle SOA 11g team is suggesting?

Thanks
Amir

Posted by Amir Hameed on September 14, 2010 at 09:23 AM PDT #

Hi, Amir,

Ivo may choose to answer, but I suspect that OpenWorld-related activities may delay his response.

As Ivo notes in his article above, our recommendations are based upon our internal tests, but that all SANs differ in their performance characteristics. We have deliberately recommended the most conservative architecture that will work for everyone.

Depending upon your SAN, you may be able to adopt a more-aggressive stance. Your mileage may vary, so you should conduct careful performance and scalability tests to validate that your target architecture works as intended.

If your tests conclude that your planned configuration works to your satisfaction, then you have likely answered your own question. We cannot answer it, however, since we can't replicate your transactional mix, load, nor hardware architecture. Only you can judge whether this more-aggressive strategy will work for you.

Questions about SOA 11g might be better-answered by that team; I have no insight into their recommendations, which are (obviously) made generically without reference to EBS environments.

Regards,
Steven

Posted by Steven Chan on September 15, 2010 at 03:28 AM PDT #

Hi Steven,
My mistake for not realizing that this was not written by you. I would like to make the following points:
- When a statement is made in a manual, it would help if a background is also provided for that statement. In this case, the reason for suggesting a local file system for INST_TOP related files should have been stated or a reference should have been provided for the suggestion.
- This architecture can make the environment configuration quite complex. For example, In my case, I am using EMC DMX SAN storage for the database, EMC NAS (Celera/Clariion) storage for the Shared Application stack and if I were to use a local file system for INST_TOP then I will need DAS for that. Considering the way we backup our environments today where different technologies are used to backup storage on different frames (EMC's BCV technology to backup storage on DMX SAN, EMC's snap technology to backup storage on the clariion storage), the use of DAS in the mix would further complicate the backup process because we wil need to use a different technology to back it up.
- Finally, can Ivo share his tests with us so that we know how he tested it and whether he ever tried putting just the Apache lock files on DAS as opposed to putting the entire INST_TOP.

Thanks
Amir

Posted by Amir Hameed on September 29, 2010 at 09:14 AM PDT #

Steven,

(thanks in adv. helping on previous Q's i may not have thanked you for)

I am busy singing the virtues of OracleVM to my current client and trying to migrate their over complicated infrastructure to OracleVM.
I see all oracle products are now supported on it including R12.
However I want to take advantage of the HA features of OVM.
surely putting the INST_TOP on local storage means you cannot HA enable your midtier VM's. Can you let me know if the oracle advice is that you cannot use OVM HA clusters with R12? (seems odd given the praises Wim et al were just singing about it at the OVM3.0 release). Or can you tell me how this can be achieved.

Thanks again
Martin Brambley

Posted by martin brambley on August 30, 2011 at 03:31 AM PDT #

Martin,

Please excuse if point 2. from the article is not clear enough, but it is meant to answer exactly your question. E-Business Suite does not require Instance Tops be placed on local storage; we instead require Instance Tops be placed on low latency storage.

If your OVM HA cluster provides low-latencies, you can place your Instance Tops there without any known issues. However, if your storage latency (used for Instance Tops) deteriorates, you will likely experience Apache performance problems.

Hope this clarifies the point.
Thanks,
Ivo

Posted by Ivo Dujmovic on August 30, 2011 at 08:48 AM PDT #

Ivo,

We upgraded to 12.1.3 in July 2011 from 11.5.10.2 with $INST_TOP residing on Sun 7210 SAN as an iSCSI target. We rarely experienced issues in the 11.5.10 environment related to Apache. We encountered period failures (weekly) with Apache and core services that were directly related to have the $INST_TOP on the SAN.

To complicate the picture, our entire environment is VM based i.e., all Linux servers are running as guests under Oracle VM with guests using iSCSI targets. We had no physical servers running Oracle EBS and no local disks that were "physical."

To solve the issue, we created a small partion on the underlying VM server and made it available to the production guest as a physical disk. Once we moved the instance top to that local disk, the number of failures was reduced immediately and the instability we were seeing was eliminated. I'm a believer--at least with low-end SAN solutions.

Riley Carrier

Posted by Riley Carrier on January 24, 2012 at 04:12 AM PST #

Hi, Riley,

Thanks for letting us know about your experiences with that configuration. This is very helpful.

Regards,
Steven

Posted by Steven Chan on January 25, 2012 at 09:52 AM PST #

Hello Steve,

Please correct if my understanding is right or wrong article "Is It Safe to Use SANs for EBS R12 Instance Tops?" is this something to do with Shared Application tech stack / Shared applicaiton tier / Shared appl_top?

If I use SAN for having separate application tiers without configuring Shared Application tech stack / Shared applicaiton tier / Shared appl_top then will there be any issues? Please advice.

Thanks in advance

Posted by guest on February 02, 2012 at 01:49 PM PST #

Hello, Guest,

You can use SANs for shared application tier file systems. This article goes into more detail about selecting file systems for the E-Business Suite:

Choosing a Shared File System for Oracle E-Business Suite
http://blogs.oracle.com/stevenChan/entry/choosing_an_ebs_shared_file_system

You can use SANs for individual application tier file systems without setting up a shared file system. The drawback of taking that approach would be that you would need to patch each of the individual file systems separately.

Regards,
Steven

Posted by Steven Chan on February 09, 2012 at 10:21 AM PST #

Thanks Steven.

So the conclusion is "we can use SAN for individual application tier file systems without setting up a shared file system and there are no issues with it ?" and there are no issues except for the suggested drawback -patching each of the indvidual f/s separately.

Posted by guest on February 15, 2012 at 10:43 AM PST #

Hi, Guest,

Yes, that would be a generic statement.

All SANs have different performance and I/O characteristics, so I'd recommend that you conduct load-testing to ensure that there aren't any latency-driven side-effects when your individual servers are all accessing the same SAN storage under load. As always, your mileage may vary, so actual performance testing is crucial to validating any storage architecture in real conditions.

Regards,
Steven

Posted by Steven Chan on February 16, 2012 at 08:20 AM PST #

does this still hold true for ebiz 12.2?

Posted by guest on March 04, 2014 at 03:05 AM PST #

Hello, Guest,

Yes, this article still applies to EBS 12.2. We do not certify individual storage systems or third-party hardware with the E-Business Suite. These systems are assumed to be compatible without requiring application-level changes to E-Business Suite code.

As always, you should test your hardware with the E-Business Suite before deploying it into production.

Regards,
Steven

Posted by Steven Chan on March 04, 2014 at 09:08 AM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
4
5
6
7
8
9
10
11
12
13
14
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today