Thursday Mar 01, 2007

Latest JVM Tuning Recommendations for Apps 11i

In my previous article "Configuring Middle-Tier JVMs for Applications 11i" I provided some suggestions in the paragraph "Rough Guidelines for JVMs" as to how many JVMs could be implemented as an initial starting point.  Subsequent to that blog article,  Guidelines to setup the JVM in Apps 11i"
(Metalink Note 362851.1) has been released which includes the ATG Performance Group's official recommendations, some of which are different than stated in my original article. 

JConsole_heap:

In this article I would like to go into a little bit more depth about the newly published recommendations.  I will focus only on the OACoreGroup JVM and not consider the Forms Servlet, Discoverer or XML Service JVM settings.  I will also assume the reader has some familiarity with Java and eBiz 11i already and is also using eBusiness version 11.5.10 with JDK 1.4 or 1.5.   


Background

From a CPU utilization perspective, there is no need to run more than 1 JVM per CPU because of the following reasons:

  • The JVM uses native threads and the native threads can be scheduled on any of the available cores/CPUs. 
  • Multiple parallel Garbage Collection (GC)  threads -- introduced in 1.4 -- are spawned.  Running multiple JVMs per CPU could result in an excessive number of background and GC threads. 
  • Multiple JVM increases the number of JDBC connections required and there is a memory and CPU overhead for each JVM process. 

For the majority of our web applications, more than 90% of the memory allocations are transient (for the life of the request or page), so new space GCs dominate much more than old generation GCs.  Old generation GCs are usually only a problem with older versions of Applications code or modules such as Configurator which load a large amount of objects (e.g. config model) in the old space.


Why increase the number of JVMs

There are several reasons why you would want to increase the number of JVMs:

  1. Minimize GC pause overhead
  2. Reduce the JVM heap size
  3. Minimize contention due to hot synchronized methods (i.e. monitor contention).
Based on Oracle's performance testing, benchmarking, and our experience with working directly with enterprise-class customers, we have determined that 100 users per OACore JVM has generally proven to be a practical and performant number.   Increasing the session.timeout parameter in zone.properties above the 30 minute default may increase JVM memory requirements.

Recomendations in summary

Guideline #1:   For the OACoreGroup JVMs start with the lower of the following two values:

  1. Number of cores on the middle tier
  2. Peak Concurrent users / 100
For example: 

If you have 2 x Dual Core CPUs in your server and have 500 peak users, then 4 JVMs is the recommended starting point, since the number of cores is the lower number.  However, if you only had 300 peak users, then you would configure 3 JVMs as a starting point as this is now the lower figure.


Guideline #2:  Size your maximum heap size as either 512 MB or 1 GB.  If you start with 512 MB and find that more memory is required, increase to a maximum of 1 GB.   If more memory than 1 GB is required, then add another JVM instead (free physical memory allowing) to increase the total memory available overall. 

For example:

You are using 1 x JVM with 1 GB heap size and find you need to increase memory.  Configure your system for 2 JVMs, each with 750 MB heap size, thus providing 1.5 GB in total.


Your mileage will vary


As always, these recommendations are generalised and can only act as a suggested starting point for your system.  As mentioned in my previous blog article, there are many factors that effect how the JVM is utilized, so you will need to undertake representative testing to prove that the number of JVMs you select is suitable for your specific user load and profile.

Related


Friday Feb 16, 2007

Tuning JVMs for Release 11i

Some people consider tuning the performance of Java Virtual Machines (JVMs) in an E-Business Suite environment to be more of an art than science.  These comments usually result because of the large number of factors that may be considered, factors that are sometimes difficult to quantify.

JConsole_heap:

Adding to the existing material available to help you perfect your art, our E-Business Suite Performance Group released a short whitepaper late with some tips:
This short but lucid Note covers topics such as:
  • The number of recommended users per JVM
  • The number of recommended JVMs per CPU
  • Heap configuration tips for OACoreGroup, with guidelines for Forms Servlet mode and Configurator use
  • Recommended techstack patches
Related Articles

Thursday Dec 21, 2006

New AutoConfig Rollup Patch O Now Available

As befits the holiday season, here's a little gift to you from the Applications Technology Group:  the latest AutoConfig Rollup Patch O (that's the letter O and not the number zero) is now available for download.


This new TXK (FND & ADX) AutoConfig Template Rollup Patch O (Patch 5478710)  for December 2006 is a cumulative patch.  Includes all E-Business Suite technology stack configuration fixes that we've previously released either individually or in patchsets such as Rollup N.  This Rollup fixes over three dozen issues that affect areas such as:
  • Real Application Clusters (RAC) configurations
  • Cloning, including cloning fixes for UTF-8 environments
  • Java Virtual Machine (JVM) configuration and stability
  • Platform-specific issues for HP Tru64 and Sun Solaris
  • JMailer + Oracle Application Manager integration
  • Context file generation and variable population
  • JDK 5.0 configurations
  • 10gR2 database configurations
  • Forms configurations
For complete details about the patch's contents, see the patch's README.

Related

Friday Dec 15, 2006

Patch-Related Enhancements in Release 12

Patching is an integral part of an Apps DBA's operational duties, and I've previously written about some of the things we're working on to improve the whole Lifecycle Management experience. 

R12 Apps Dashboard File History Screenshot:

In his OpenWorld 2006 session, Kenneth Baxter took the wraps off some of the new patching and maintenance features coming in Release 12.  His crisp presentation is full of interesting details about the changes to the maintenance capabilities in R12's Oracle Applications Manager, notably:
  • New Patch Wizard functionality, improving the process of identifying, downloading, and analyzing recommended patches
  • Registering flagged files to understand a patch's impact on customizations
  • Improvements in determining whether given patches have been applied
  • Manual Steps Infrastructure (MSI) functionality, reducing the number and complexity of manual steps
  • Support for checking codelevels and baselines, reducing the risk of a patch "breaking" other functionality
  • Patch filters to tailor patch searches to your specific environment
  • Central access point for all patch details, including applied patch reports, timing reports, and log files
As an added bonus, his presentation is packed with R12 screenshots; you can download it here:
Related
The above is intended to outline our general product direction.  It is intended for information purposes only, and may not be incorporated into any contract.   It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision.  The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle. 

Wednesday Dec 06, 2006

Master Class in Apps 11i Performance Tuning

Ahmed Alomari is one of Oracle's foremost experts in optimizing performance for the E-Business Suite.  His OpenWorld sessions are always filled with DBAs looking for practical tips on getting more out of their existing hardware, and this year's session was packed with more practical pointers than ever before. 

If you're looking for a master-class in Apps performance tuning, it doesn't get better than this.  Better yet, it's all contained in a single free download.

Gather Schema Statistics Screenshot:

In his presentation, Ahmed covers:

  • Optimization of Forms and Reports
    • Via socket mode, Forms Dead Client detection, and the latest performance-related Forms patches
    • Costs of using Cancel Query
    • Things that your end-users can do to improve performance
    • Tracing Forms performance via SQL and FRD tracing 
    • Optimization of Report via SQL tracing, or by enabling the built-in Reports Trace feature

  • Optimization of Concurrent Manager:
    • Using specialization rules and work shifts to bind jobs to specific time windows
    • Conflict Resolution Manager sleep time guidelines for parallel workers
    • Tips for using Transaction Managers for Inventory Transactions and other synchronous online processing
    • Pipes be gone!  Avoiding transaction manager instances per database instance for RAC
    • Tracing and debug options at the program and request level
  • Optimization of the Application Tier
    • Tuning Apache processes and clients via httpds.conf settings
    • Caching of non-HTML resources such as images, style sheets, and Java scripts via httpd.conf or apps.conf settings
    • Configuring dedicated mod_plsql listeners
    • Apache Jserv / JVM logging and auto load-balancing
    • Tuning Apache Hotspot and garbage collection settings, and interpreting GC output to tune JVM heaps
    • Reducing Java memory and connection leaks
    • Tracking JDBC connections
    • Performance benchmarks after upgrading to JDK 5.0:  ~ 15%!
    • New JDK 5.0 performance management tools, including Jconsole, jps, jstat, visualgc
    • Optimizing Java Server Pages via precompilation
    • Tuning Portal-related settings
    • Discoverer 10g performance-related options
  • Tuning Network performance
    • Testing latency with via ping and optimal packet sizes
    • Performance benchmarks for using Web Cache for compression
  • Tuning Database tier performance
    • Optimal buffer cache and shared pool settings
    • Use of asynchronous I/O, Quick I/O, and locally managed temp files
    • Use of Stats Pack and AWR to take performance snapshots, and things to look for in those snapshots
    • Database performance metrics to monitor
    • Allocating CPU resources via Database Resource Manager
    • Tips for minimizing the impact of gathering statistics, and how to interpret the results
    • Benefits of the new OA Tablespace Model, and before-and-after benchmark comparisons (e.g. a 1.4 TB database shrunk to 850 GB!)

  • And much, much more...
Even if you're not having performance issues now, this is worth a look.  If you are having performance issues, reviewing this presentation should be your top priority today:
Related

Thursday Nov 30, 2006

Inside Oracle's Own Global Operations Data Center

[Oct. 27, 2008 Update:  The latest version of this popular presentation from OpenWorld 2008 is now available for download.  For links to the latest version, see this article.]

Every year at OpenWorld, Bret Fuller discusses how Oracle consolidated over 70 individual E-Business Suite instances into a single instance serving all global operations.  The statistics for Oracle's own Global Single Instance keep rising impressively every year:

Oracle Global Single Instance Statistics:

Bret's presentation this year covers:
  • The business metrics before and after the consolidation
  • Savings estimates and a discussion of business benefits
  • Oracle's Global Single Instance architecture, including a glimpse of the off-site production and disaster-recovery setups in our Austin and Colorado Springs data centers
  • Before and after hardware comparisons
  • Tips on project management best practices
  • Detailed view of the change management processes around the Global Single Instance
  • Critical success factors and tips
You can download this year's presentation here:
Related

Tuesday Nov 14, 2006

Summary of Oracle E-Business Suite Technology Updates

If you're a regular of this site, you might be wondering how to convey all of the latest Apps techstack news to your management.  If so, you'll find Lisa Parekh's OpenWorld 2006 presentation useful. 

Release 12 ORACLE_HOMEs:

I report to Lisa, so her presentation covers all of the breaking news reported here on this site, and more, including:
  • Release 12 technology stack changes, such as the use of Oracle Application Server 10g 10.1.2 and 10.1.3 in different ORACLE_HOMEs and the latest 10gR2 database
  • Release 12 user interface screenshots
  • Release 12 supported browsers, including IE 6.0, Firefox 1.5, and Safari 2.0
  • OA Framework Personalization enhancements
  • Oracle Portal and WSRP support for E-Business Suite portlets
  • XML Publisher enhancements in Release 12, including the new Data Extraction Engine, Bursting Engine, and other new RTF template features
  • Workflow enhancements, including Workflow Flexfields and Workflow Bulk Interface
  • Release 12 Integration Repository previews
  • Upcoming technology stack certifications for the E-Business Suite Release 11i application and database tiers, such as:
You can download her presentation here:
Related
The above is intended to outline our general product direction.  It is intended for information purposes only, and may not be incorporated into any contract.   It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision.  The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle. 

Monday Nov 13, 2006

New Systems Management Capabilities for Release 12

The latest details about the upcoming Release 12 Oracle Application Management Pack for Oracle Enterprise Manager 10g were unveiled at OpenWorld.  This new product replaces the older "Grid Control Plug-in for E-Business Suite," and many of the new features are backwards-compatible with Release 11i, too.

OAM R12 screenshot:

This latest plug-in will extend the Enterprise Manager 10g Grid Control capabilities for monitoring and managing E-Business Suite environments.  It will include a number of cool new features, including:
  • User interface enhancements, including an enhanced Home and Topology views

  • Automatic discovery of E-Business Suite environments

  • New configuration management capabilities, including views, comparisons, and searchs of configuration information across multiple Apps systems

  • Application Service Level management for Concurrent Porcessing, Forms, Workflow, and Self-Service, with configurable service level alerts

  • JVM usage monitoring, including monitoring of the Application Module pool, locked AOL/J connections, and OA Framework cache components

  • Enhanced cloning and data scrambling capabilities
For a more in-depth discussion of these new features and capabilities -- and lots of screenshots -- check out Biju Mohan's OpenWorld presentation:
Related
The above is intended to outline our general product direction.  It is intended for information purposes only, and may not be incorporated into any contract.   It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision.  The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.

Wednesday Nov 08, 2006

Mod_plsql Not Supported in E-Business Suite Release 12

[July 11, 2008 Update:  Many readers have raised questions about why mod_plsql was removed from Release 12. Others have asked what to do about their mod_plsql-based Apps 11i customizations and extensions when upgrading to R12.

George Buzsaki,our preeminent E-Business Suite architect, has put together an excellent new whitepaper that addresses these topics.  For a pointer to that whitepaper, see this article.]

[June 4, 2008 Update:  It should be stressed that the Oracle E-Business Suite is distinct from Oracle Application Server.  Oracle E-Business Suite Development chooses to use specific Oracle Application Server components in the E-Business Suite technology stack.  These decisions by E-Business Suite Development should not be interpreted to represent the release policies or plans for Oracle Application Server. 

It must be emphasized that Oracle is fully committed to supporting
mod_plsql as part of Oracle Application Server and as part of the Oracle Database distribution into the indefinite future.]

One of my degrees is in psychology -- cognitive science, to be exact -- so it's interesting when that world intersects with my present work in the IT industry.  A landmark text in grief counseling is Elizabeth Kubler-Ross' On Death and Dying, in which she described the general grieving process as proceeding in roughly five stages:
  • Denial and isolation
  • Anger
  • Bargaining
  • Depression
  • Acceptance
Floyd Teter has astutely pointed out that OpenWorld's news of mod_plsql's demise in Oracle E-Business Suite Release 12 may trigger this process for some of you.  So, to help you through the first stage of denial, on behalf of the Applications Technology Group, I'll officially reiterate that here:
  • Release 12 will not support the use mod_plsql
Switching to Oracle Application Framework in Release 12

If you're planning on upgrading to R12 and you currently have Release 11i customizations that depend on mod_plsql, you would be well-advised to do the following things immediately:
  1. Inform your management about this upcoming change

  2. Prepare a comprehensive inventory of all of your existing 11i mod_plsql customizations and dependencies

  3. Review R12 capabilities with your business users and stakeholders, to see if your customizations are still needed

  4. If yes, budget time in your R12 upgrade cycle for:
  • A test install of an R12 demonstration environment to support a functional review with your business users, to see if the customizations can be further weeded out by relying on native R12 functionality

  • Training yourself and your developers in the Oracle Application Framework (OAF)

  • Doing the actual conversion from mod_plsql to the OA Framework model
For obvious reasons, the Release 12 Oracle Application Framework documentation is not available publicly yet.  In the meantime, it can't hurt to review the existing OA Framework documentation for Release 11i, which will help you get a general sense of the migration process. 

We plan to release some specific technical guidance on preparing your custom development for R12 soon; watch this space for that announcement.

Related
The above is intended to outline our general product direction.  It is intended for information purposes only, and may not be incorporated into any contract.   It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision.  The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle. 

Tuesday Nov 07, 2006

Diving Deep Into the Release 12 Techstack

Ivo Dujmovic is an architect in our hard-working Applications Technology Integration team.  We were lucky to get some of his time at OpenWorld 2006 this year, where he discussed the deep innards of the Release 12 technology stack. 

Release 12 Techstack Overview: Overview of three-tier logical architecture for E-Business Suite Release 12, including the database, application server, and desktop tiers. 

For the most comprehensive preview released so far on how the Release 12 technology stack will be architected and work, take a look at his presentation.  He covers:
  • Highlights of new R12 techstack components
  • New Application Server 10g (10.1.2 & 10.1.3) ORACLE_HOMEs replacing the 9iAS 1.0.2.2.2 O_HOMEs from 11i
  • Switch from JServ to Oracle Containers for J2EE (OC4J) for running servlets, Java Server Pages (JSP), and Enterprise Java Beans (EJB)
  • OC4J Deployment details, including replacements for jserv.conf and jserv.properties configuration files
  • New R12 filesystem layout, including the introduction of an Instance Home (INST_TOP)
  • New use of Oracle Process Manager & Notification Server (OPMN)
  • New Forms 10g deployment
  • Comparison of 11i and R12 environment variables
  • Discussion of relative benefits of Forms Servlet vs. Server (socket) mode deployments
  • Disabled mod_plsql in R12 (more about this in a later article)
  • Things you can do today to prepare for R12
Related
The above is intended to outline our general product direction.  It is intended for information purposes only, and may not be incorporated into any contract.   It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision.  The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle. 

Thursday Oct 19, 2006

Investigating java.lang.OutOfMemoryError with Apps 11i Middle Tier JVMs

If you had to guess what the error "java.lang.OutOfMemoryError" indicates when seen in your Apache log files, you would probably be right...

In most cases this message is telling you that the Java process has exhausted available memory and cannot process some requests successfully.   In many cases additional symptoms will be:

  • Poor web page performance
  • User requests being timed out
  • Java processes taking 100% CPU on your server.  
The Java Virtual Machine (JVM) will sometimes be restarted by oprocmgr as performance is so poor it presumes the JVM has died.

There can be other causes.  For example, this message can sometimes be given where there is the lack of some other resource such as threads per process (kernel parameter "max_thread_proc" on HP-UX) but this case should be easily identifiable from the exact message written out.

Where has all the memory gone?


Every object created in Java takes some memory.  Once all java code that references the java object has completed, then the Garbage Collector (GC) automatically removes the object and claims back the memory.

You can run out of memory due to:



  • Insufficient memory allocated to the JVM to cope with the amount of work
  • Suboptimal GC processing
  • A memory leak -- Java code is not releasing an object when it should
  • A Java code or operating system defect
How are JVMs configured with Apps 11i

Two important configuration files for Java in an Apps environment are:
  • jserv.properties
    This file controls the JVM parameters in the "wrapper.bin.parameters=" entry.  When you install Apps originally or update the JDK version, the default settings are updated in your CONTEXT.xml file ("s_jvm_options" for the OACoreGroup JVM) and controlled by AutoConfig thereafter.

What can I do about OutOfMemoryError?


  • Are you using Oracle Configurator?
Configurator can sometimes require huge chunks of memory for a relatively long period of time, so Oracle now recommends setting up separate JVMs specifically for Configurator.  For more details see Setting Up a Dedicated JServ for Running Only Oracle Configurator" (Metalink Note 343677.1).

  • Increase available memory
An obvious choice, you may think, and this will often at least delay the effect of the issue, but may not always solve it.   You can achieve this by either increasing the number of JVMs or the memory settings ("-Xmx512M" for example).  In general, I would not recommend going much over 640MB per JVM, in order to keep the Garbage Collection time to a reasonable level.

  • Tuning GC

You can influence how Java performs Garbage Collection.   I would normally recomend reviewing and possibly backing out any non-default JVM parameters first, as these are sometimes just carried forward because they were used with previous JDK versions.  As always, any tuning should be undertaken in a TEST environment to ensure the changes have a positive impact.   


  • Using Concurrent Mark Sweep collector

With multi-CPU servers using JDK 1.4 and higher, the following parameters have been seen to improve performance and GC throughput on some environments:



-XX:+UseConcMarkSweepGC -XX:+UseParNewGC  



  • Identify the components that are using memory

Not always as easy as it sounds.  Generating class histograms or java profiling may be the only options. 


What data should I gather to investigate further ?

The initial investigation should focus on trying to identify the pattern of memory usage building up and how frequent GC is occurring.  The tools and information described below may help to achieve this



  • Review stdout files

Basic Garbage Collection information is normally written to the $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm/*.stdout file for each JVM.  Additional Garbage Collection information can be logged, for example:



-XX:+PrintGCTimeStamps -XX:+PrintGCDetails



  • Add more verbose GC logging


-Xverbosegc


This allows you to see how much of each memory pool is being used.  For example you may find it is the "Permanent Generation" space that is running out of memory, rather than the "Tenured Generation".  If this turns out to be the case, the "-XX:MaxPermSize" parameter could be increased to provide more memory.



  • Collect class histograms


-XX:+PrintClassHistogram


Implemented in JDK 1.4 and higher by some vendors.  Can be very useful if available.  Beware of using this on HP-UX platforms running JDK 1.4 as it seems to crash the JVM rather than output the data!



  • Use JConsole

Discussed in a previous article: Using JConsole to monitor Apps 11i JVMs



  • Java Profiling


-Xrunhprof


This should tell you exactly where the memory is being used, but has a huge impact on performance and is therefore not likely to be possible to gather this data, except for a lightly used test environment.



  • Other clues

Does the DMS output for the JVM show increasing active connections or threads?


Do JDBC connections to the database keep increasing? What module names do they relate to?


Does temporarily disabling "AM Pool" reduce memory consumption (Profile option "FND: Application Module Pool Enabled")?


Are you seeing PL/SQL "cursor leaks"?  See, "Understanding and Tuning the Shared Pool" (Metalink Note 62143.1) in section "Checking hash chain lengths" for SQL to check this.


  • Any Java Thread Deadlocks or Database Deadlocks?

Are Java processes spinning to 100% CPU or is CPU use very low when the error occurs?


  • Look for platform references

Review your operating system vendor's web site to look for known issues or required platform patches for your Java version.


My Production system is failing every few hours, what can I do right now?

A possible quick fix would be to increase memory and/or number of JVMs.  Be sure to ensure that this won't cause the operating system to start swapping!


You could also consider a scheduled bounce of Apache every 12 or 24 hours if possible. 


These steps may allow you to minimise the immediate impact on the business so you can investigate and implement changes in a methodical, safe, controlled and tested manner.


Where can I read more about Java memory and tuning?

There are many resources on the Internet to describe how Java uses memory and what steps to take for performance tuning.  The best place to start is your hardware vendors web site as the options available depend on their Java implementation and also the specific Java version you are using.  


For example:



Conclusion

Dealing with Java processes running out of memory can sometimes be as simple as increasing the memory available, or may require detailed investigation to identify the root cause.     


Although each case will have its own unique considerations, I hope that the information in this article has given you some ideas as to the general approach to take should the need arise.

Additional References


Wednesday Oct 11, 2006

Security Fixes in ATG Rollup 4

A wave of security-related questions have followed the recent release of Oracle Applications Technology 11i.ATG_PF.H Rollup 4 (RUP4) (Patch 4676589).  Our initial release of the documentation accompanying this patch was a bit ambiguous about which security patches were included in this Rollup.


The following documents have just been updated with more detailed listings of security-related patches included in Rollup 4:
What's particularly important about these updates is that the documents now list those security patches that are not included in Rollup 4.  Those patches must be applied on top of Rollup 4.

Related

Thursday Aug 31, 2006

New Techstack Baseline for Release 11i, Redux

Our recently-announced new techstack baseline for Release 11i has triggered a lot of questions, particularly around desupport timing for older configurations, certification baselines for new patches, and the availability of old patches.

In response, we've just published a new version of the document that clarifies things:  Rebaselined Oracle Applications Technology Components for Releases 11.5.7, 11.5.8, 11.5.9, and 11.5.10 (Metalink Note 363827.1).

I strongly encourage a careful review of this latest version.  A few highlights:

  • All customers should upgrade to the latest Oracle Applications Technology product family rollup, currently Oracle Applications Technology 11i.ATG_PF.H Rollup 4 (4676589).

  • If you can't do that, you should have these patches installed, at minimum.  All new Applications Technology patches -- starting with the July 2006 Critical Patch Update (CPUJul2006) -- require this minimum baseline.
  • All patches released prior to the July 2006 Critical Patch Update will continue to be available for download.
  • Beginning with the July 2007 Critical Patch Update (CPUJul2007), new ATG techstack patches will be certified only with the current and previous rollup patches.
Related

Monday Aug 07, 2006

Latest E-Business Suite Technology Stack (ATG RUP 4) Now Available

[Editor May 14, 2007 Update:  ATG RUP 5 is now available.  See this article:  Latest E-Business Suite Technology Stack (ATG RUP 5) Now Available]

[Editor Aug 24, 2006 Update:  The SSO 10g Integration Patch for the E-Business Suite is now available -- see this article:
DMZs, SSL & RAC for OracleAS 10g + Release 11i]

This is significant news:  the latest Family Pack Rollup for the E-Business Suite Technology Stack is now available for download.  

11i Architecture:

The Applications Technology Group (ATG) Family Pack H Rollup 4 (11i.ATG_PF.H RUP 4, Patch 4676589) is a collection of technology-stack patches that can safely be applied on top of the ATG Family Pack H. 

This Rollup patch is cumulative:  all previous patches released for Family Pack H since the initial 11.5.10 release, including Rollup 3, are included in this latest patch.

Why Is This Rollup Important?

We've spent a huge amount of effort both within the Applications Technology Group and across the entire Applications Suite to test and certify this Rollup patch.

Our focus for ATG Rollup 4 has been to consolidate all known safe defect fixes, performance improvements and security enhancements into one well integrated and fully certified update.  We also focused on eliminating as many "co-requisite" patches as possible (application product patches required to interoperate with new ATG Rollups), by working to restore backward compatibility with earlier application levels wherever possible. 

We recommend that all customers make plans to move to Rollup 4 by the end of the year, especially any customers who are not already at ATG Rollup 3.

Changes and Fixes in Rollup 4

ATG Family Pack H Rollup 4 (11i.ATG_PF.H RUP 4, Patch 4676589) includes a range of new features and bug fixes for:
  • Alerts
  • Workflow
  • Oracle Applications Framework (OAF)
  • Forms Personalization
  • XML Gateway
  • Password management
  • Flexfields
  • Oracle User Management (UMX)
  • Oracle Applications Tablespace Migration (OATM)
  • XML Publisher
  • Concurrent Processing
Prerequisites

Oracle Applications Technology ATG_PF.H RUP4 can be applied only to an existing Oracle Applications Release 11i system running Oracle Applications Technology Family Pack 11i.ATG_PF.H, which is included in Oracle Applications Release 11.5.10. If you don't already have a running Release 11.5.10 system, then you must first install the Oracle E-Business Suite 11.5.10 or patch up to Applications Technology Family Pack 11i.ATG_PF.H.

Special Advisory for Single Sign-On 10g Users

The SSO 10g Integration patch for Rollup 4 (Build 4.0) is still undergoing certification and will be made available in late August 2006. 

If you've already integrated your E-Business Suite environment with Single Sign-On 10g by following Metalink Note 233436.1, you should postpone your implementation of 11i.ATG_PF.H RUP 4 until Build 4.0 is released.  As usual, I'll announce the availability of that patch here when it's released publicly.

Related

Friday Aug 04, 2006

AutoConfig Template Rollup Patch N Now Available

Editor Dec 21 Update:  AutoConfig Rollup O is now available for download.  It includes AutoConfig Rollup N.

AutoConfig continues to be updated, so those of you who are using it in your E-Business Suite environments will be pleased to hear that the latest AutoConfig Template Rollup Patch has been released.


The TXK (FND & ADX) AutoConfig Template Rollup Patch N (Patch 5107107)  for August 2006 has just been released to Metalink.

This patch contains additional content on top of TXK AutoConfig Template Rollup Patch M (Patch 4709948).  This latest patch contains the latest enhancements and fixes to the actual AutoConfig engine, also.

For complete details about the patch's contents, see the patch's README.

References:

Tuesday Aug 01, 2006

Configuring Middle-Tier JVMs for Applications 11i

When you call Oracle Support with a problem like apj12 errors in your mod_jserv.log, middle-tier Java Virtual Machines (JVMs) crashing, or poor middle-tier performance, then it will often be suggested to increase the number of JVM processes.  So, the key question that likely occurs to you is, "How many JVMs are required for my system?"

Processing Java Traffic in Groups

First, some quick background:  web requests received by Oracle HTTP Server (Apache) to process Java code is sent to one of four different types of JVM groups to be processed.   You can see this in the jserv.conf file:

 ApJServGroup OACoreGroup 2 1 /usr/.../jserv.properties
 ApJServGroup DiscoGroup  1 1 /usr/.../viewer4i.properties
 ApJServGroup FormsGroup  1 1 /usr/.../forms.properties
 ApJServGroup XmlSvcsGrp  1 1 /usr/.../xmlsvcs.properties

The number of JVMs for each group is signified by the first number on each line.
  • OACoreGroup is the default group.  This is where most Java requests will be serviced
  • DiscoGroup is only used for Discoverer 4i requests
  • FormsGroup is only used for Forms Servlet requests
  • XmlSvcsGrp is for XML Gateway, Web Services, and SOAP requests

In the example above, I have two JVMs configured for OACoreGroup and one JVM configured for each of the other groups.

Factors Affecting Number of JVMs Required

Determining how many JVMs to configure is a complex approximation, as many factors need to be taken into account.  These include:
  • Hardware specification and current utilization levels
  • Operating system patches and kernel settings
  • JDK version and tuning
  • Applications code versions, especially JDBC and oJSP
  • JServ configuration file tuning (jserv.properties and zone.properties)
  • Applications modules being used
  • How many active users
  • User behaviour

Rough Guidelines for JVMs

Luckily, Oracle Development have undertaken various performance tests to establish some rules of thumb that can be used to configure the initial number of JVMs for your system.

OACoreGroup
    •    1 JVM per 100 active users
DiscoGroup
    • Use the capacity planning guide from Note 236124.1 "Oracle 9iAS 1.0.2.2 Discoverer 4i: A Capacity Planning Guide"
FormsGroup
    •    1 JVM per 50 active forms users
XmlSvcsGrp
    •    1 JVM is generally sufficient

In addition to this, Oracle generally recommends no more than 2 JVMs per CPU.    You also need to confirm there are enough operating system resources (e.g. physical memory) to cope with any additional JVMs.

Your Mileage Will Vary

The general guidelines above are just that -- they're very broad estimates, and your mileage will vary.  As I write this, the Applications Technology Group is working on a JVM Sizing whitepaper that will provide guidelines based on whether your E-Business Suite deployment is small, medium, or large.  I'll profile this whitepaper here as soon as it's released publicly.

Until then, it's critical that you test your environment under load, using transactional tests that closely mirror what your users will be doing.  It's useful to use automated testing tools for this, as you create your benchmarks. 

Here are a couple of quick-and-dirty tools that might be useful in sizing your JVMs.

Script to determine "active users" for OACoreGroup

REM
REM SQL to count number of Apps 11i users
REM Run as APPS user
REM
select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions
from icx_sessions icx
where disabled_flag != 'Y'
and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate   
and counter < limit_connects;
REM
REM END OF SQL
REM

How to determine "active forms users" for FormsGroup

Check the number of f60webmx processes on the Middle Tier server.  For example:

ps -ef | grep f60webmx | wc -l

Conclusion

  • The number of required JVMs is extremely site-specific and can be complex to predict
  • Use the rules of thumb as a starting point, but benchmark your environment carefully to see if they're adequate
  • Proactively monitor your environment to determine the efficiency of the existing settings and reevaluate if required

More on Java Memory Tuning Later

Once you've established the right number of JVMs to use, it's then time to optimize them.  I'm intending to discuss Java memory tuning and OutOfMemory issues in a future article.  Stay tuned.

Related


Wednesday Jul 26, 2006

Release 12 Patch Wizard Improvements

A big part of an Apps DBA's job is patching, so I know the Patch Wizard in Oracle Applications Manager is something about which you have a few opinions.  These opinions are sometimes expressed in energetic and colorful language.  It's clear that some of you feel strongly about this area. 
OAM Patch Wizard screenshot:

Some of you have recently provided feedback on the Patch Wizard, which I forwarded to our development team.  They reviewed your comments and provided some glimpses of functionality to come in future versions such as Release 12.  Here's a sampling of your comments and some selected responses from the team:
  1. Patch Wizard downloads patches which have already been downloaded

    This is by design in the current Release 11i version.  Release 12 will download a patch from Metalink only if it doesn't match an existing patch in your staging directory.

  2. Patch Wizard gets some patch prerequisites wrong

    This is a known issue, complicated by the interrelationships between patches, Family Packs, and the wide array of bundles that patches can be delivered in.  The issue of handling patch prerequisites more sensibly is being revisited in Release 12.
  3. Patch Wizard recommends patches included in already-installed patches

    This shouldn't happen for patches that are explicitly included, i.e. patches whose definitions directly include specific sub-patches.  If you encounter this behaviour, file a bug with Oracle Support and we'll investigate this.

    If a patch is implicitly included (i.e. all files in a patch are included in an already-applied patch), this can happen.
  4. Patch impact analysis doesn't note when a patch is significantly changing your installed patch level, e.g. from Minipack H to Minipack J.

    This is true.  This is functionality that is slated to be improved in Release 12.
  5. Patch Wizard should show analysis results even when no patches are recommended.

    This is a commonly requested enhancement request.  This functionality is slated to be delivered in Release 12.
  6. There should be better correlation between analysis runs and concurrent requests.

    The technology used for concurrent requests places some restrictions on this; we're looking at what's feasible for future releases.
  7. Analysis runs may take some time to complete.

    This is true.  Some of the current analysis being performed by the Patch Wizard take some serious computation time.

    It's expected that Release 12 will include some performance gains over earlier releases, partially resulting from avoiding downloads of existing patches, changes in .LDT file packaging, and changes in the logic used by the prerequisite analysis engine.
  8. Large patches may trigger "out of memory" errors in some rare cases.

    If you encounter this behaviour, please file a bug and we'll investigate this. 
Stay tuned to this website for more patching updates for Release 12.  Thanks for your comments -- keep them coming, and I'll do my best to route them to the right teams.

Related
The above is intended to outline our general product direction.  It is intended for information purposes only, and may not be incorporated into any contract.   It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision.  The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle. 

Friday Jun 16, 2006

In-Depth: Load-Balancing E-Business Suite Environments

As I watched blood drain from my arm, my thoughts turned to load-balancing and system redundancy.  My phlebotomist had just cheerfully informed me the lab's primary testing PC had failed that morning, so there would be an unusual delay in getting blood test results until a replacement arrived.

Increasing Fault Tolerance at Lower Cost

You can use load-balancing routers (LBRs) to protect your E-Business Suite from similar types of system failures.  Load-balancers increase your environment's fault-tolerance and scalability by distributing load across a pool of application servers like this:

Generic Apps Load-balancing:

Besides fault-tolerance and scalability, another appealing benefit is that you can use load-balancing to substitute expensive SMP boxes with clusters of inexpensive Linux-based commodity servers. 

Linux Load-Balancing on Oracle's Global Single Instance

In fact, this is what we've done at Oracle:  our own E-Business Suite environment is a Global Single Instance running on about 58 Linux-based application servers.  This has reportedly saved us millions in operating costs. 

If you're joining us at this year's OpenWorld conference, make sure you attend Bret Fuller's ever-popular session on how we run the E-Business Suite internally; some of the statistics he shows on our transactional volumes are mind-blowing.

Supported Load-Balancing Methods

The E-Business Suite supports the following types of load-balancing:
I'll cover only the first two methods in this article.

HTTP Layer Load-Balancing

HTTP Layer load-balancing is the most common method used in E-Business Suite environments. 

HTTP Layer Load-Balancing:

In this configuration, end-users navigate to a specific Web Entry Point that represents your E-Business Suite's domain name.  An HTTP Layer load-balancer routes all subsequent traffic for a specific user to a specific Web Node.

HTTP Layer load-balancers may use heartbeat checks for node death detection and restart, and sophisticated algorithms for load-balancing.

DNS-Based Load-Balancing

When an end-user's browser attempts to access your E-Business Suite environment, your local Domain Name Server (DNS) can direct that user to a specific application server in a pool based on available capacity:

DNS-Based Load-balancing:

Traffic for that user's session will be handled by the application server 10.10.10.10, while other users' traffic may be directed to other application servers in the pool.  Like HTTP layer load-balancers, many DNS-based load-balancers use heartbeat checks against nodes and sophisticated algorithms for load-balancing.

Business Continuity ("Disaster Recovery")

Our larger enterprise-class customers combine DNS-based and HTTP layer load-balancers to support their business continuity plans.  In the event of a disaster, end-users are directed via a DNS-based load-balancer from the primary E-Business Suite environment to an offsite standby site.

Disaster Recovery using DNS + HTTP LBRs:

Minimum Requirement:  Session Persistence

Remember that although Oracle doesn't certify specific load-balancers or networking hardware with the E-Business Suite, we do support their use generically.  In other words, we've designed the E-Business Suite to be able to use load-balancers in general.

The minimum requirement is that a load-balancer support session persistence, where a client's initial HTTP connection is directed to a particular application server, then subsequent HTTP requests from that client are directed to the same server.  As long as a load-balancer is able to handle session persistence (also referred to as "stickiness"), it's likely to work with the E-Business Suite.

Related

Wednesday Jun 14, 2006

Using Third-Party Networking Hardware with Oracle Application Server 10g

If you recall, the E-Business Suite isn't explicitly certified with third-party networking components but is expected to work with them. 

Load-balancer with OracleAS 10g Identity Management:

If you're integrating the E-Business Suite with Oracle Application Server 10g, there's another piece that you can add to the puzzle.

Oracle Application Server 10g isn't explicitly certified with third-party networking components, either, but some testing has been performed with specific vendor products.

These tests cover load-balancers, firewalls, and SSL accelerators.  Some of the load-balancer vendors and products include F5's BIG-IP, Foundry, Citrix's NetScaler, Nortel, and Radware.  The firewall and SSL vendors include Check Point, Cisco, Sonic Wall, and Ingrian.

Remember that Oracle testing doesn't equate to certification.  It's the responsibility of the third-party vendor to certify their hardware with Oracle Application Server 10g.  Regardless of that, you might find it reassuring to know that Oracle's tried some of these combinations in the Oracle Application Server 10g labs. 

Even if your networking vendor hasn't certified their hardware explicitly Oracle Application Server (or even the E-Business Suite), it's generally expected that their products will work if they're standards-compliant.

References

Friday Jun 09, 2006

New Minimum Release 11i Techstack Standards

With the upcoming desupport of E-Business Suite Releases 11.5.1 through 11.5.6, the minimum technology stack configuration baselines that we support have been revised.

OAM Patch Wizard:

What's a "Baseline"?

Recall the key point made in my Blue Bridge of Death article:  it's impossible to test all possible combinations of Apps patches and configurations.

So, all E-Business Suite technology stack patches are regression-tested against a fixed set of standard 11i configurations and patches for each given release. 

These standard configurations are called our testing baseline.  There are different baseline configurations of technology stack patches for 11.5.1, 11.5.2, and so on.

New Minimum Technology Stack Baselines for 11.5.7 through 11.5.10

As of July, 2006, we're raising the bar for Releases 11.5.7 through 11.5.10.  This means that all new technology stack patches will be tested only against the new baselines listed here:
Procrastination and Risk

A surprising number of problems disappear if you wait long enough, although, strangely, yard work and pruning don't fall into this category. 

Like landscaping, E-Business Suite technology stack patching gets more arduous and risky the more you procrastinate. 

If your E-Business Suite technology stack doesn't meet the new baseline minimums, your risk of applying new patches to your environment goes up.  After all, those patches will not have been tested with your configuration, so it's much harder for us to guarantee that they won't have any adverse effects.

Highly recommended:  Use the new published Apps baselines to convince your management to support an upgrade to the new baseline configuration for your release as soon as possible. 

Better yet, use the Oracle Applications Manager Patch Wizard to generate a full list of recommended patches for your environment, and schedule all upgrades at the same time.

Related Articles:

Tuesday Jun 06, 2006

Update: AutoConfig and Rollup Patches

[Editor:  This is an a elaborated and expanded version of a comment from one of our Applications Technology Group architects from the now-massive AutoConfig Comment thread]

AutoConfig Flow:

Just to update folks and clarify some delivery questions around AutoConfig and its templates:

In previous patches: 
  • The AutoConfig Engine and its associated utilities were released via patches bundled in the ADX product ID.
  • The E-Business Suite AutoConfig templates were released via patches bundled in the TXK product ID.
  • To get an up-to-date AutoConfig setup, you needed to download and install both sets of patches individually.
AutoConfig Engine and Templates in a Single Patch

We have now combined the AutoConfig Engine and the E-Business Suite Templates into a single patch for easier management. They are now being combined into TXK rollup patches, the first combined one being:
The Applications Technology Group produces Rollup patches for AutoConfig, Advanced Configurations, and Rehosted Technologies. These are released via patches in the TXK product ID.

These rollup patches get packaged together into a Minipack.

The last TXK Minipack (3219567 Patch 11i.TXK.B Technology Stack Minipack B) was released in November, 2004.  This was part of 11.5.10 and contained TXK AutoConfig Rollup I (Oct 2004) Patch 3594604.

Each subsequent Rollup patch (RUP) includes the previous one.

Hopefully this clarifies the delivery of AutoConfig code.

Enhancements for AutoConfig

On a separate note, I would like to thank everyone who posted comments about AutoConfig features. It took us a long time to get to this point and there are probably a lot more features you want. So if you can continue submitting here your favorite enhancement, we will do our best to include the most desirable ones.

Thanks,
Ivo Dujmovic
Director of TXK Development Team

Related Articles:

Monday Jun 05, 2006

Java 5 and E-Business Suite Release 12

The E-Business Suite Release 12 Rapid Install is expected to include Java 5 (J2SE) for the application tier.  In addition, given that AutoConfig will be compiled as Java 5 objects, we'll be delivering Java 5 on the database tier, too.


Sun J2SE 5 Download Screenshot:

Release 12's run time components will include Java 5 for the web tier (including JSP compilation), the Concurrent Processing tier, and for the AD utilities.  We're still working out the exact Java version now; feel free to subscribe to this blog if you'd like updates when more details get posted.

Related:

The above is intended to outline our general product direction.  It is intended for information purposes only, and may not be incorporated into any contract.   It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision.  The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle. 

Wednesday Apr 26, 2006

Why Use Two ORACLE_HOMEs for Release 12's Application Tier?

Why use two different OracleAS 10g releases in two different ORACLE_HOMEs for the upcoming Oracle E-Business Suite Release 12?  Because you asked for it, of course.  Well, indirectly, perhaps.  You've been asking us when E-Business Suite users can take advantage of the latest Oracle technologies.  So, we're biting the bullet and putting them into our Release technology stack.

Release 12 Application Tier ORACLE_HOMEs:

The latest version of Forms and Reports available today is included in Oracle Application Server 10g 10.1.2.0.2, so that release, at minimum, will be part of the E-Business Suite Release 12.

You've been clamoring for the ability to use Oracle Containers for Java (OC4J), the next-generation successor to JServ.  The latest version of Oracle Containers for Java available today is included in Oracle Application Server 10g 10.1.3, so that release, at minimum, will be part of the E-Business Suite Release 12, too.

The slightly tricky thing is using these two releases together.  In a nutshell, all major services will be started out of the 10.1.3 ORACLE_HOME.  The E-Business Suite modules (packaged in formsapp.eap) will be loaded into the OC4J-Forms instance running out of the 10.1.3 ORACLE_HOME, and frmweb will be invoked out of the 10.1.2 ORACLE_HOME.

The above is intended to outline our general product direction.  It is intended for information purposes only, and may not be incorporated into any contract.   It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision.  The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.

Friday Apr 21, 2006

Managing E-Business Suite Configurations with AutoConfig

Work is its own reward.  You can expect great rewards this year.

~ Paraphrased from Dilbert, Scott Adams

Historically, one of the biggest challenges facing any E-Business Suite system administrator was managing the countless configuration files for technology stack components. 

In the past, different E-Business Suite products would each have their own technology stack configuration recommendations.  For example, iProcurement might recommend that a certain parameter be set to a given value in httpd.conf.  Naturally, it's inevitable that a different product family would come along and recommend a completely contradictory setting for the same parameter. 

Further complicating things:  if the hapless system administrator chose to follow either recommendation, there was no guarantee that the new setting wouldn't break a third unrelated product.  It was enough to make grown sysadmins weep.

Enter stage left, AutoConfig

AutoConfig is a tool that automates the management of all configuration files for all E-Business Suite Release 11i technology stack components.  The Applications Technology Group now centrally controls all parameter settings for all configuration files for the E-Business Suite. 

You might reasonably have expected from the start, but there are over 200 products within the E-Business Suite, and gaining agreement from all development groups to centralize this kind of control within the Applications Technology Group was about as simple a political process as nominating a presidential candidate.  This took years.

That's all behind us now, and today, individual product families are no longer permitted to make recommendations for technology stack configurations of any kind, and any changes to known-good parameter settings are now centrally tested to ensure that they work with all 200 or so E-Business Suite products.  It's about time, and you're the main beneficiary.

Beneath the Hood:  AutoConfig

All of the information required for configuring an Applications instance is collected into two local XML repositories called the Applications Context and the Database Context.  This information describes your instance name, location of servers, and so on.  With the latest Rapid Installs, the information you originally provided at install time is the basis for the Applications and Database Context files.

When AutoConfig runs on the Application tier, it merges information from the Applications Context with presupplied configuration file templates to generate all application-tier configuration files and update database profiles.

When AutoConfig runs on the Database tier, it uses information from the Database Context file to generate all configuration files used on the Database tier.

If you're updating the configuration for an existing instance with AutoConfig, it will take a snapshot of your current configuration before installing the new configuration files.  You can roll back your configuration to any snapshot taken at any given date.  This allows you to experiment safely with different configuration options.  Didn't like the effect of the last change?  Just roll back to the previous AutoConfig snapshot.

But wait... there's more.  AutoConfig can also start and stop all technology stack components that it manages, and there are additional options for pregenerating test configuration files and examining the differences with your existing configuration.

AutoConfig Now Preserves Customizations

With all of these great features, as well as AutoConfig's devotion to motherhood and apple pie, the reluctance of sysadmins to use AutoConfig has been a source of some... ahh... perplexity within our team. 

After all, who wouldn't want to use a tool that guarantees a known-good configuration that works for all E-Business Suite products?  Who could possibly want the burden of managing configuration files themselves, with this as an alternative?

Well, a lot of you, as it turns out.  More than we expected, in fact. 

With some digging, we learned that you've invested in building your own custom configurations and don't want us overwriting your hard-earned changes.   Your customizations might address the need to:
  • Start additional services or processes when you start Oracle Applications services
  • Define and add zones to your JServ configuration
  • Extend Forms to integrate with a third party Java version
  • Develop customer applications that are maintained by AutoConfig
That's understandable, so we enhanced AutoConfig.  Your customizations are now preserved after running AutoConfig, and persist even after new AutoConfig templates are installed. 

So, What Did We Miss?

Despite all this, we still have the nagging impression that the majority of E-Business Suite system administrators still don't use AutoConfig.  We really don't know why.

Assuming that you're all rational, there must be good reasons.  Clearly, you have requirements that AutoConfig doesn't meet yet.

If you haven't switched over to AutoConfig yet, I would appreciate your posting a comment about new features that would encourage you to make the switch.

References:

Monday Apr 17, 2006

AutoConfig Template Rollup Patch M Now Available

This one's hot off the presses, also.  One of the major products from our team is AutoConfig, which I plan to devote a few future posts to.  For now, those of you who are already using AutoConfig in your E-Business Suite environments will be pleased to hear that the latest AutoConfig Template Rollup Patch has been released.


The TXK (FND) AutoConfig Template Rollup Patch M (Patch 4709948)  for April 2006 has just been released to Metalink.

This patch contains additional content on top of TXK AutoConfig Template Rollup Patch L (Patch 4489303).  This latest patch contains the latest enhancements and fixes to the actual AutoConfig engine, also.

For complete details about the patch's contents, see the README.

For details about AutoConfig, see, "Using AutoConfig to Manage System Configurations with Oracle Applications 11i" (Metalink Note 165195.1).

Related Article:

About

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
4
5
6
7
8
9
10
11
12
13
14
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today