It's possible to scale up your E-Business Suite environment with multiple application tier servers to improve fault tolerance and performance. It's also possible to share a single filesystem between them: all application tier files are installed on a single shared disk resource that's mounted from each application tier node. In Release 12, that would look like this:
This allows you to apply patches once to the central filesystem, rather than maintaining each application tier server node individually. We recommend this approach; it reduces maintenance overheads for those multiple servers and shortens your patching downtimes.
Beginning with Oracle E-Business Suite Release 12, we also allow you to share an applications tier file system between multiple E-Business Suite database instances, too. For more details about this advanced option, see this article.
Customers embarking upon this path inevitably ask, "Which shared filesystem do you recommend?" The short answer is that we don't recommend any specific filesystem, but there's more to it than just that.
Does Oracle Certify Storage Systems?
Not any more. Our Server Technologies division used to run an Oracle Storage Compatibility Program (OSCP) to validate specialized storage products. At one time, Oracle and its partners worked together to validate specialized storage technology including NFS file servers, remote mirroring, and snapshot products. The storage industry matured over time, and this program was ended in January, 2007.
The successor to this program is the Oracle Certification Environment (OCE) group. This group provides resources for third-party vendors to certify their own products with Oracle technology. The OCE team works with Oracle Partner Management and third party vendors for approving support statements published by third party vendors with respect to certification projects with Oracle.
It's important to note that these certifications are performed by the third-party vendors themselves and not the E-Business Suite Development division. Certification statements made by third-party vendors partnering with the Oracle Certification Environment group are not reviewed or endorsed by the E-Business Suite division.
Does the E-Business Suite Division Certify Storage Systems?
No, I'm afraid not. EBS Development doesn't have the resources to certify or compare even a subset of the leading filesystems. Since we don't have hands-on experience with different filesystems in a controlled test environment, we can't make any informed recommendations for a given product. We generally suggest that customers either perform their own product testing or consult a trusted consultancy that compares the relative merits of each product against a consistent set of criteria.
What are the EBS Requirements for a Shared Filesystem?
Shared filesystems must be transparent to the calling application, in this case, the E-Business Suite. In other words, no modifications to the E-Business Suite should be necessary to ensure compatibility with the shared filesystem.
Our Frequently Asked Questions: Sharing the Application Tier File System in Oracle Applications 11i (Note 243880.1) states:
... your shared application tier file system can reside on any type of shared disk resource. Examples of shared disk resources include an NFS mounted disk or a disk array. The shared disk resource does not have to be local to the machine, and it can also be a standalone disk array. Usual tuning considerations apply.
The same thing applies to Oracle E-Business Suite Release 12, too.
What About OCFS2 or GFS?
There are many different shared filesystems out there, too many to list here. The general statements about EBS requirements for a shared filesystem above apply to all third-party file system products.
However, we get a lot of questions about two specific products due to their close relationship and packaging with Oracle's own operating system releases:
- Oracle Clustered File System (OCFS2)
- Red Hat Global File System (GFS)
Here's the E-Business Suite position on these two shared file systems:
Oracle Clustered File System (OCFS2)
The E-Business Suite's database tier is built on the Oracle Database. The Oracle Database is certified with OCFS2. Therefore, OCFS2 is supported for the E-Business Suite database tier, too.
The E-Business Suite's application tier is built on Oracle Application Server. Oracle Application Server is not yet certified to run on OCFS2.
If our Fusion Middleware group ever certifies Oracle9i Application Server 1.0.2.2.2 (used by Apps 11i) or Oracle Application Server 10g (used by Apps 12) to run on OCFS2, then the E-Business Suite's application tier will be certified on that file system.
Red Hat Global File System (GFS)
Specific versions of the Oracle Database are certified with GFS running on specific Red Hat and Oracle Enterprise Linux releases. Therefore, those GFS combinations are supported for the E-Business Suite database tier, too.
Sadly, I haven't been able to locate any externally-published statements about Oracle Application Server compatibility with GFS. This usually means that these two products haven't been tested together. If you want an explicit statement of support for GFS for Fusion Middleware products, your best bet would be to log a Service Request against the Oracle Application Server product in question.
Back to the database and GFS: there are some special support provisions for this database configuration. See the "Support Process for GFS 6.0 and 6.1" section of Using Redhat Global File System (GFS) as shared storage for RAC (Note 329530.1), which states:
Oracle's product support teams will not take support calls on Red Hat GFS. All issues known to be related to Red Hat GFS must be opened with Red Hat directly. When an Oracle SR is opened for an Oracle product or a Red Hat Enterprise Linux issue in a configuration that includes GFS, Oracle Support will do their best effort to determine if the issue is GFS software related. In that case, Oracle will hand-off the GFS related issue to Red Hat Support.
It's important to note that the E-Business Suite division does not test the E-Business Suite with GFS. We haven't performed any certification or compatibility tests with that filesystem and don't have any empirical data about how well this particular combination will work.
What Does EBS Development Use Internally?
We're in Development, not marketing, and we're expressly not able to endorse third-party products. What we can do is give you a glimpse of what we use internally within Oracle for the E-Business Suite Development division.
At any given time we have hundreds of E-Business Suite environments running simultaneously within the EBS Development division. These are centrally managed by our terrific EBS/Fusion Operations group. This internal Oracle group has has created some really interesting infrastructure over the years. One of the most useful custom solutions allows developers to get a new EBS environment on demand. Shortly after their request, an automated process instantiates a new Apps environment and the developer is off to the races.
The underpinnings of this are Network File System (NFS) mounted filesystems running on NetApp. Our Operations group has tested ZFS-based filers, which are also NFS-mounted filesystems.
In practical terms, this means that nearly all of our development, testing, and certification environments for the E-Business Suite are all running on NFS mounts. We explicitly assume that our use of NFS generalizes to all shared file systems.
What Does Oracle Use Internally for its Production Global Single Instance?
Our EBS development use of NFS is paralleled by Oracle's own global single instance deployment of Apps 12. Our production EBS instance connects via Gigabit Ethernet to a shared NFS (NAS) NetApp FAS960 clustered storage system running NetApp 7.2.4.
Our four production Sun F25K database servers are equipped with 44 dual core CPUs and over 176 GB RAM, Sun Solaris 9, Sun Cluster 3.1, and Veritas VxVM/VxFS 4.0* mp02. Each of these database nodes has three GigE cards connecting them to the backend database storage, an EMC Symmetrix DMX3000 storage system.
Remember, this isn't an endorsement or a recommendation; it's merely a peek into what we use here internally at Oracle.
References
- Using Redhat Global File System (GFS) as shared storage for RAC (Note 329530.1)
- OCFS2: Supportability as a general purpose filesystem (Note 421640.1)
Related Articles
Comments (12)
Great post, as usual, Steven. It's nice to have all of this information in one place. Clears up a lot of ambiguity. Now when shared file system questions crop up on the OTN forums, we have another resource to which to direct posters. :-)
Regards,
John P.
Posted by John Piwowar | July 11, 2009 9:46 AM
Posted on July 11, 2009 09:46
Steven,
Thanks for the post. One question -- the recommendation in the documentation is that the INST_TOP is located on each host and not shared. Yet it appears from the diagram that our internal apps are sharing the instance_top. Is it safe to presume in this scenario the only thing on the local file system are the lock and pid files?
Tom
Posted by Tom Mullen | July 12, 2009 7:25 PM
Posted on July 12, 2009 19:25
Hello, Tom,
I can't say for certain, but that would be a reasonable assumption. I think the diagram was busy enough that the GSI team was using a bit of shorthand in that tiny little box for the INST_TOP details.
You're right about our standing recommendation: Note 384248.1 states, "... for optimum performance, it is advisable to use a local file system for the Instance Home."
Regards,
Steven
Posted by Steven Chan | July 15, 2009 9:41 AM
Posted on July 15, 2009 09:41
Hello, John,
My pleasure -- and thanks for your recent help on our OTN forums. I saw that you'd fielded questions about this lately. That, along with some discussions on Oracle's internal forums, spurred me to pull all of this together in one place at last.
Regards,
Steven
Posted by Steven Chan | July 15, 2009 9:44 AM
Posted on July 15, 2009 09:44
Steven
We are currently going through an upgrade to R12 and 10g of the database. The Systems Integrator has said that the local disk used to store the INST_TOP directory must be on local INTERNAL disk to the machine (we're using Sun M4000 and M5000) servers. We asked if it would be possible to use dedicated fibre attached SAN and were told that it wasn't supported and had to be internal disk. Is this correct? It seems very unusual that Oracle would insist on using internal disk rather than dedicated SAN storage. I can understand not using shared storage. This will present us with an even bigger problem soon as we are moving everything to SAN including the boot partition. Any advice on this would be appreciated. I guess the question is does local disk really have to be local to the machine or is SAN ok if it's dedicated to that machine.
Many thanks
William
Posted by william butcher | July 29, 2009 1:20 AM
Posted on July 29, 2009 01:20
Hello, William,
Thanks for the insight into your deployment architecture.
My understanding is that our guidance on the INST_TOP being installed on a local filesystem is to avoid issues with aliases, collisions, or conflicts that might occur in a shared filesystem.
I don't think our architects have any opinions about dedicated fibre-attached SANs, since we don't do any explicit testing with that particular configuration. From a support perspective, if the fibre-attached SAN is transparent to the application server (i.e. appears to the server as local storage), I can't see any issues with this configuration.
I'll run this by our architects, just to be sure. I'll post any follow-ups here, if I learn something new that contradicts what I just outlined.
Regards,
Steven
Posted by Steven Chan | August 3, 2009 11:28 AM
Posted on August 3, 2009 11:28
Steven
Thanks for your reply, this is very useful. I have been having a chat with a colleague at a different company and they have tried to address this issue with support. A SR was logged (7473036.994) in which support stated the following
QUESTION #1
===========
Implemented a shared filesystem but have noticed errors in the logs that
are consistent with incorrect disk usage. it is needed to get verification and clarification of what is meant by local disk in this context - does it
exclude any network/san sourced disk even when used exclusively for the
server to which it is made available? Does it expected to be "internal"
server disk storage?
ANSWER #1
==========
Local is internal disk storage in this context. See also the following
from the R12 SHARED APPL_TOP NOTE:
* Oracle HTTP Server Lock Files
Oracle HTTP Server processes create temporary lock files for its internal operation. The location of these lock files is specified in httpd.conf by
the LockFile and OpmMtxFile respectively. You must ensure that value of
the AutoConfig variable s_lock_pid_dir is set to a location on the local
file system to avoid file locking issues on the network file system.
So this suggests that support require the lock and pid files to be on internal disk rather than SAN which does seem rather strange.
thanks for your help in resolving this issue it has given us a real headache with disk configuration.
William
Posted by william butcher | August 4, 2009 8:10 AM
Posted on August 4, 2009 08:10
Hi, William,
One of our EBS architects was inspired by your question to write the following blog article:
Is It Safe to Use SANs for EBS R12 Instance Tops? - http://blogs.oracle.com/stevenChan/2009/08/local_disk_vs_san_for_inst_tops.html
Let me know if you still have any outstanding questions after reviewing that new article.
Regards,
Steven
Posted by Steven Chan | August 5, 2009 11:59 AM
Posted on August 5, 2009 11:59
Hi steven,
do you know if the application techstack team will be in a position to pursue certification of shraed appl_top and ASm cluster filesystems ? I notice that certification against 11g R2 is already underway and I am very interested in whether this includes certification against the new cluster filesystem for sahring the application top...
Thanks
Posted by Robin Chatterjee | September 10, 2009 6:07 AM
Posted on September 10, 2009 06:07
Hi, Robin,
I'm looking into this. I'll post an update here as soon as I get a confirmation about Oracle Application Server's certification on ASM.
Regards,
Steven
Posted by Steven Chan
|
September 11, 2009 2:26 PM
Posted on September 11, 2009 14:26
Hi, Robin,
I have gotten some weak confirmation that the Fusion Middleware group was planning to certify Oracle Application Server with ASM at some point. I haven't been able to get details on where they stand on that effort, though.
Your best bet would be to log a formal Service Request via My Oracle Support (formerly Metalink) to get a definitive answer from one of the Fusion Middleware specialists.
Regards,
Steven
Posted by Steven Chan | October 1, 2009 9:35 AM
Posted on October 1, 2009 09:35
Hi Williams,
I am designing the architecture which looks more or less similar to yourr but i will be using 2 node Sun T-series servers for shared application tier, and we have SAN storage.
I have not yet finalised on the use of shared filesystem, as per oracle doc and stevens blog we can use any NFS mounted disk or disk array, could you please share if you have used NFS or disk array(please explain) .
thanks
Pravin
Posted by Pravin | November 19, 2009 4:58 PM
Posted on November 19, 2009 16:58