Ensuring Critical Patch Update Quality
By Eric P. Maurice-Oracle on Jul 24, 2009
Hi, this is Eric Maurice again.
For the last three years, as we have released each Critical Patch Update (CPU), we have been simultaneously posting a summary of the CPU on this blog (see for example, the last blog entry discussing CPUJul2009). I was recently reviewing past blog entries and realized that one topic -- “How CPUs are tested by Oracle before their release” -- continues to be of great interest to customers, so I thought I’d discuss it here.
The topic of CPU testing by Oracle is relevant to customers because of the potentially significant impact of faulty patches. The most obvious impact of faulty patches is that they may fail to properly resolve the vulnerabilities they were intended to close. More importantly, faulty patches may cause regression issues resulting in data corruption, system outages or other effects that cause applications to execute improperly. (Regression issues occur when a change in the code results in introducing unexpected consequences such “breaking” a functionality that was present in previous versions of the code.) Ultimately, faulty patches almost always result in re-issuance by their author, and costly reapplication by customers. But perhaps the least desirable impact of faulty patches is that customers may delay or defer applying future patches because of a fear that “the security patches may create more problems than they solve.”
For all of these reasons, Critical Patch Update quality has been the highest priority for Oracle. Under the leadership of Mary Ann Davidson and her team, all parties involved in the production of Critical Patch Updates are told in no uncertain terms that faulty patches will not be tolerated.
So, how does Oracle help make sure that CPUs don’t create issues in customers’ environment? Let’s look at the process Oracle follows with Database Server CPUs, keeping in mind that Oracle’s objectives with CPU testing are twofold: (1) prevent undue consequences resulting from the CPU patches (regression issues), and (2) ensure that the vulnerabilities are addressed effectively (preventing incorrect or incomplete fixes).
The first step in the remediation process for any vulnerability is the discovery or reporting of the vulnerability to Oracle. Security vulnerabilities can be reported to Oracle by external sources such as customers and security researchers. Vulnerabilities can also (and often will) be reported by internal sources including: Oracle’s own assessment team, developers themselves, and results of automated tests used throughout development.
Once a vulnerability has been discovered or reported to Oracle, it is analyzed to understand -- for example -- what components or interfaces are affected, which components of the code are vulnerable, how the vulnerability can be exploited, how it can be eliminated, what other mitigation measures are possible, etc. (This analysis phase is so important and complex that it could be, by itself, a topic for a future blog entry.)
At the end of the analysis phase, a fix for the vulnerability is produced, and regression tests are conducted to verify that the fix will not result in unintended consequences. Developers and security experts then review this fix before being included in the main code line (that is, if the vulnerability is not limited to previous versions of the affected component) where full regression tests are run. Once the inclusion in the main code line is complete, the fix is ported to future patchsets. In addition, the fix can be scheduled for inclusion in a future Critical Patch Update.
Let’s now look at the process Oracle follows for testing Database Server security fixes specifically included in Critical Patch Updates. To understand the level of effort required with each CPU release, one has first to go back to understanding the scope of the Critical Patch Update program:
- CPU patches are released for ALL Oracle products every quarter, on the Tuesday closest to the 15th of the months of January, April, July, and October;
- On these dates, CPU patches are published for all supported versions on all their respective platforms (with the exception of On-Request versions)
- CPUs for many Oracle products are cumulative (i.e. CPUs include new fixes as well as previously-released fixes).
In other words, on the release day of the CPU, Oracle releases a large number of patch sets for each platform-version combinations affected by the CPU. For example, the April 2009 Critical Patch Update included fixes for seven versions of Oracle Database Server (18.104.22.168; 22.214.171.124; 10.2.0.3; 10.2.0.4; 10.1.0.5; 126.96.36.199; 188.8.131.52DV) on over 20 different platforms!
In order to provide enough time for thoroughly testing the fixes included in CPUs, the initial selection of the new fixes to be included in each CPU takes place 15 weeks before the CPU release date. These fixes are then backported to all supported version-platform combinations with a few exceptions for things like On-Request versions. Next, Oracle runs a full set of regression tests against the aggregated patches for each of these version-platform combinations.
Six weeks before the CPU release date, CPUs are assembled in what we call “early combinations.” These “early combinations” are provided to other development organizations to verify that CPU’s don’t create problems across the Oracle stack. For example, throughout this six weeks period, the development organizations for the various product lines such as E-Business Suite, PeopleSoft, and Siebel, perform tests with the database CPU early combinations to ensure that database fixes will not result in problems with the applications and to catch regressions that might occur with customer applications. During this period, Oracle performs additional comprehensive vulnerability fix testing to ensure that all newly addressed vulnerabilities are effectively fixed in all version-platform combinations of the CPU.
In addition to the steps above, during the four final weeks before the release of each CPU, CPU early combinations are provided to the support and other organizations to perform installation testing. It is during that period that Oracle ensures that documentation is clear and accurate and that the CPU installs as expected. It is also during this period that the support organization grows familiar with the CPU in order to provide support to customers who may call for CPU-related issues once the CPU has been published.
Note that under Oracle Software Security Assurance disclosure policies, Oracle doesn’t share CPU early combinations or other information associated to the upcoming CPU with third party organizations. This is because Oracle is very concerned about treating each customer equally, and ensuring that the content of the upcoming CPU not be leaked out, possibly aiding in the development of exploit code. However, during the entire process leading to the release of the CPU, Oracle works with the individuals who have submitted the vulnerabilities fixed in the CPU, keeping them informed of progress, and sometimes discussing the nature of the fix implemented by the CPU with them. These individuals, many of them security researchers, are listed in the credit section of the CPU Release Note.
This commitment to CPU quality under Oracle Software Security Assurance has eliminated many issues that customers may have previously encountered with Critical Patch Updates, leading a prominent security researcher to recently declare: “The database patches are very easy to apply and have very few problems. You get a huge benefit without a lot of effort.” This type of endorsement by folks in the security researchers and analysts’ communities, as well as positive feedback from customers, are great rewards for the people involved in the production, testing, and support of the Critical Patch Updates.