The Art of the Possible: Real Life Case Study in Oracle IAM 11gR2 Performance Tuning by Alex Bolante (Accenture)
By Greg Jensen on Jul 16, 2013
In our last post, we walked through a handful of practical tips and tricks to fine tune your Oracle Identity Management 11gR2 deployment. This week we look at a real life case study, focused on Oracle Directory Services, where we applied our pragmatic approach and solutions.
Case study: a multinational financial services corporation. With presence in over 200 countries, this financial services company enables consumers, businesses, financial institutions and governments to use digital currency instead of cash and checks through one of the world’s most advanced processing networks, capable of handling more than 20,000 transactions per second. Like many legacy customers, the company sought Accenture’s help to strategically plan, design and upgrade to an improved version of Oracle Directory Services that provided:
• Improved directory services performance
• Multi-user topology support
• Enhanced replication
• Increased security
The implementation comprised of approximately 50 servers located across multiple, geographically distributed data centers supporting over 100 applications and more than 250,000 users – included financial institutions, payment product processors and others doing business with this financial services company.
Environment design specification
Our environment design specification was initially developed to support legacy applications, but given a new set of business and technical requirements, we needed to modify and scale the solution to support future business services with enough capacity to grow up to 40% year over year. Key performance requirements included:
• Optimized for reads, writes and replication across data centers located across the globe
• Performs 1000 operations per second
• Supports response time of 0.05 milliseconds for single user id searches
• Supports response time of 0.15 milliseconds for single user attribute writes
• Supports 200 concurrent searches
• Supports growth rate of 10,000 objects per month over the next 5 years
• Provides real time password replication using prioritization
Modifying and scaling the solution:
Our process for modifying and scaling the solution included engaging Oracle product managers and engineers directly to validate our hardware configuration.
Product: Oracle Directory Services
Operating System: 64-bit Solaris 10 Update 10 or higher
Hardware: SPARC T-series
Memory: 64 GB
Disk Space: 270 GB
Swap Space: 15 GB
Tmp Space: 10 GB
File Descriptor Limit: 8192
Replication Topology: Multi-master with no restrictions on the number of masters
We made several recommended configuration changes and tuned the Operating System, Database Cache, Entry Cache, Import Cache, File System Cache and Indexes.
Disable schema check for fast replication
$dsconfpath/dsconf set-server-prop -p portNum check-schema-enabled:off
Set DB cache size to 1000M
$dsconfpath/dsconf set-server-prop -p portNum db-cache-size:1000M
Set entry cache size to 1000M
$dsconfpath/dsconf set-suffix-prop -p portNum suffixDN entry-cache-size:1000M
$dsconfpath/dsconf set-server-prop -p portNum import-cache-size:200M
$dsconfpath/dsconf set-server-prop -p portNum all-ids-threshold:8000
Set repl-purge-delay to 1 days
$dsconfpath/dsconf set-server-prop -p portNum repl-purge-delay:1d
Change log path
dsconf set-log-prop -p portNum ACCESS path:/var/ldaplogs/access
dsconf set-log-prop -p portNum AUDIT path:/var/ldaplogs/audit
dsconf set-log-prop -p portNum ERROR path:/var/ldaplogs/error
Enable Audit log
dscond f set-log-prop -p portNum AUDIT enabled:on
After we applied our performance tunings, we performed our tests in production-like environments, verified and documented our results, profiled and monitored our solution, tweaked and tuned our environment and cycled through this step-by-step process until we were satisfied that we had met all requirements. We shared the results with our Oracle peers to validate – including our testing approach which included search rates and modification rates based on 100 users and 200 users connecting concurrently – and the numbers were right on point with our expectations from the Directory Services upgrade.
How can you apply this to your environment?
Talk to Oracle Product Management, Development and Engineering directly,get them involved in your project as early as possible and keep them engaged throughout your project. It helps to have knowledgeable subject matter experts who can bring your implementation up to par with leading implementations. Some guidelines for checkpoints include:
Checkpoint 1: Before statement of work (SOW) is signed:
• Is the SOW clearly defined?
• Is the described product functionality feasible?
• Are measurable and achievable success criteria defined?
Checkpoint 2: Before requirements, architecture and project plan are delivered:
• Can the product fulfill the defined requirements?
• Is the architecture and solution design sound and scalable?
• Is the customer's environment ready?
Checkpoint 3: Before the design is delivered:
• Is the design technically sound?
• Can the design be implemented, migrated and supported?
• Are the test plans and approach reasonable?
Define specific, measurable objectives for performance tunings based on your requirements. To start with, you can use Accenture’s predefined set of key attributes for developing “good” requirements that are measurable.
• Necessary – an important capability or element of a solution which cannot be compensated for if absent
• Understandable – stated in a context which conveys the essence of what is needed
• Complete – stated in a standalone context which does not rely upon supplemental and/or assumed definitions
• Consistent – does not contradict by context or terminology nor is contradicted by other statements (e.g. is not mutually exclusive)
• Unambiguous – cannot have more than one interpretation
• Attainable – a capability which can be implemented within the constraints of available resources and technology (e.g. product, cost, schedule)
• Verifiable – can establish that the statement has been satisfied through specific measurements, test, demonstration, inspection, and/or analysis
Determine how you plan to implement performance tunings. There is more than one way to skin a cat. In addition to the tuning configuration changes made to the environment, you also have to consider hardware sizing and configurations, middleware technologies, application and data samples used for testing and how you measure/analyze results. For example, hardware sizing guides are meant to provide you with a baseline for your deployment, but they are not exact specifications for your Oracle Identity & Access Management deployment.
The same applies for a vendor certification matrix – while Oracle’s Identity & Access Management product might be certified or supported on another vendor’s middleware or platform stack, that does not automatically imply it is the ‘optimal’ configuration for your deployment. Most organizations already have infrastructure standards (e.g. we use WebSphere Application Server for our J2EE apps), but you need to carefully consider that your Oracle Identity & Access Management deployment may be harder to tweak and tune if implemented on top of multiple vendor stacks. In fact, the more unique your configuration design is, the more challenging it will be to support and the less likely your deployment will be up to par with common practices.
Apply your performance tunings, perform your tests, verify and document your results, profile and monitor your solution, tweak and tune it – wash, rinse and repeat. Consider the testing tools you will use to conduct your performance tests and their limitations. We used both SLAMD and HP LoadRunner for our Directory Services deployment. SLAMD had resource limitations on the number of connections and threads we could test, especially if it was not running off a dedicated server. HP LoadRunner had a limitation with testing multiple attribute updates until we applied a hot fix that the vendor eventually provided.
Also, most deployments are two- to three-tier architectures, so you have to tune the database/directory server, middleware/application server, web servers and every component in between each tier (e.g. load balancers for SSL acceleration). In fact, each tier requires its own performance tuning, pruning, cleaning, care, feeding and regular maintenance. At its core, there are several performance bottlenecks to consider:
• Start with your server or system resources (e.g. over clocked CPU, maxed out memory, resource contention, insufficient space)
• Tune your way up from data tier to application/web tier (e.g. database/directory servers typically require specific optimizer tunings, predefined indexes and table pruning while application servers typically require proper JVM heap size allocation, connection pooling and message queue thresholds)
Share your experiences with the Oracle Security community at large. By now, your Oracle Identity & Access Management solution should be designed to support not only your legacy applications, but also scaled to support future business services!
Stay tuned for our next post on No Where to go but up: Extending the benefits of accelerated IAM to enable new solutions and features where we highlight interesting trends in Security and Identity & Access Management.
Oracle Directory Services: Overview
Oracle Directory Services: Discussion Forums https://forums.oracle.com/community/developer/english/fusion_middleware/identity_management/oracle_directory_server_enterprise_edition_sun_dsee/content?start=0