Troubleshooting in a Nutshell
By Acshorten-Oracle on Feb 06, 2012
When a product goes into production sometimes it may run into performance issues due a number of issues such as traffic increases and uses of the product it was not designed for originally (as more user discover more and more about using the product in their daily work).
The Performance Troubleshooting Guideline Series (Doc Id: 560382.1) series of whitepapers, from My Oracle Support, covers the techniques (by tier) to help detect and address performance issues for products using the Oracle Utilities Application Framework. The whitepaper covers all the versions of the Oracle Utilities Application Framework so covers a lot of versions of various products.
While the whitepaper is comprehensive and includes a process for diagnosing performance issues there are some general advice that can help you address performance issues and even prevent them. Here is a summary of advice that you should consider when configuring your product as well as diagnosing performance issues:
- Enable automatic caching on browser - One of the common issues I see with new implementations is that they setup their browsers for development and testing with settings that are not appropriate for production. One of the most common ones is the cache setting for Internet Explorer. When the browser gets a page it looks in its cache for the page and its static elements to save on redownloading them for the current session. If the elements are in the cache and they are still active (we assign an "expiry" date and time to each element when they are downloaded) then the browser will use them to render the screen. If they are not there or expired new versions are downloaded prior to rendering a screen. This has a slight delay to the loading of the screen till the cache is populated. If the browser cache parameter is set ot an inefficient value such as "Every Visit to Every Page" then this slight delay can result in large delays to transactions as the screen is loaded each time it is called. This also means bandwidth is greatly increased. By setting the value to "Automatic", the recommended setting for production, optimizes the transmission and rendering of the screens. I have seen performance increase up to 10 fold and bandwidth drop a similar amounf by ensuring this setting is set to "Automatic". Now this setting must be global, across all your users, to have an positive result in terms of bandwidth reduction.
- Ensure you machine meets the minimal requirements for the software - The rendering speed of pages depends on the power available to the browser to render them. Now the Oracle Utilities Application Framework simplifies the rendering somewhat, by offloading some of the complex rendering to the server, but I have seen customers use underpowered client machines that are not even designed to run the underlying operating system at its optimal level let alone a browser application. The Oracle Utilities Application Framework product does not require a powerful machine by no means but a reasonable one (as outlined in the Installation Guide for the product) would greatly benefit performance of rendering.
- Do not underestimate firewalls and virus protection - I saw less and less of this issue over time but not all sites have implemented firewalls or even virus protection on their computers. This is bandwidth and machine killer for the client machines. If you are a site that uses firewalls and virus protection then also be aware that these exact a performance premuim on all processes on a machine so performance can be affected. I am not suggesting that you not install these protections, just be aware of their impact on performance when you test to factor them in.
- Web/Business Application Server optimization settings - There are a number of settings in the J2EE Web Application Server that can determine the performance of your product. These typically are
- Thread pool sizing - Most J2EE server have dynamic sizing but it there is no harm in checking that you can support the number of connections to an individual server. Too few and you have delays in processing transactions.
- Internal limits - Internal tolerances of the Web Application Server. For example, WebLogic uses Socket Readers to control internal management of work and if you do not have enough then there are delays in processing calls.
- JVM memory parameters - These parameters control the amount of memory for the product and the frequency of garbage collection. If they are not set appropriately for your traffic volumes then garbage collection happens too often and that can cause delays in the processing of transactions.
- Child JVM optimizations - If you are using Oracle Utilities Customer Care and Billing and Oracle Enterprise Taxation and Policy Management then you should consider looking at optimizing the number of child JVMs and the recycling time for effective memory management of these JVMs. If there are too few child JVM's or they are running low on memory they can cause delays in transactions.
- Update statistics often - One of the most common remedies for performance for online as well as batch is to ensure the database statistics for your site are up to date. I cannot stress this more than any remedy. In fact any customer I talk to this is one of the things I point out and when they are updated, performance problems seem to disappear more often than not. To explain, Oracle (as well as other databases in the marketplace) use Cost based optimization to determine the optimal path. When executing a new SQL statement, as paths are cached for reuse, Oracle quickly assesses the available access paths based upon the statistics in the database and chooses the lowest cost one to execute. As with humans, if you have the wrong information in front of you, then you can make potentially the wrong decision. Updating the statistics, using dbms_stats, on a regular basis gives the latest information to the optimizer to help it make the best decision it can at execution time.
The advice above is merely a summary of the advice in the whitepapers and the whitepaper document techniques used by customers to address performance issues in their implementations. Some customers incorporate this advice in their monitoring regimes to help identity issues before they become problems.
I strongly advise customers to take a look at the whitepapers to find advice that may assist. It saves time and helps keep your system healthy.