Friday Feb 07, 2014

Heads up! Regression in Solaris 10 Kernel patch 15040[01]-0[67] - now fixed in 15040[01]-09

Update:  Bug 17628036 is fixed in Solaris 10 patch 15040[01]-09 and Solaris 11.1 SRU16, both of which are now available on MOS.  Customers with 15040[01]-0[78] installed are recommended to install 15040[01]-09.  The Solaris 10 January CPU (Critical Patch Update) has been respun to included 15040[01]-09 and is available from MOS:

https://updates.oracle.com/download/18074667.html (SPARC)
https://updates.oracle.com/download/18074668.html (x86)
The issue is far less common on Solaris 11, so you only need to update to SRU16 if you experience the issue.  There will be a further "mop-up" fix in 15040[01]-10 and a later SRU, but 15040[01]-09 resolves the main issue and should be sufficient for most customers (addresses 95%+ of the instances reported).

Original post:

Please note that there's an issue with revs -06 and -07 of the Solaris 10 Kernel patches 15040[01].

Please see Sun Alert 1619580.1 on MOS for further details.  A number of Solaris 10 customers have hit the 2nd of the reported issues. 

We've respun the Solaris 10 January CPU (Critical Patch Update) to revert to rev-05 (now available from MOS), we're expediting a fixed rev-09 (rev-08 won't be released), and have withdrawn revs -06 and -07 from release.  Update: The Solaris 10 January CPU and Recommended patchset have been respun to include the fixed rev-09.

15040[01]-09 will address Bug 17628036 and the current ETA for expedited release is Feb 21.

The Solaris 11 fix is also being expedited in Solaris 11.1 SRU16, but seems to be less prone to the issue.

I apologize for the inconvenience caused.

Thursday Jun 06, 2013

Next Solaris 10 Kernel PatchIDs, 150400 (SPARC) & 150401 (x86)

As I've noted in an update to my previous blog posting, Murphy's Law strikes again!

No sooner had I written that Solaris 10 Kernel PatchIDs 148888-xx (SPARC) and 148889-xx (x86) were here to stay for the foreseeable future, than the integration of the SR-IOV feature into rev-04 of these patches made it prudent to rejuvenate them. 

So from July 2013, the Solaris 10 Kernel PatchIDs will change to be 150400-xx (SPARC) and 150401-xx (x86).

See here for the full Solaris 10 Kernel PatchID sequence.

Friday Mar 22, 2013

Solaris 10 1/13 patchset released and latest Solaris 10 Kernel PatchIDs

Posting updated June 6, 2013, with new Solaris 10 Kernel PatchIDs 150400-xx (SPARC) and 150401-xx (x86):

As usual, we've released a patchset of all the patches contained in Solaris 10 1/13 (Update 11):

We've also included an important post-S10U11 patch - 150125-01 (SPARC) / 149637-02 (x86) - in this patchset, which fixes ZFS Bug 15809921.  See Doc 1535270.1.

This patchset can be applied to any existing Solaris 10 system to bring all pre-existing packages up to the same software level as Solaris 10 1/13.

It is not the same as upgrading to Solaris 10 1/13 (available here), as upgrading will additionally install any new packages delivered in the Update. 

I've also updated my Solaris 10 Kernel PatchID sequence posting with the latest Solaris 10 Kernel PatchIDs, namely: 

  • The Solaris 10 1/13 Kernel patch, 147147-26 (SPARC) / 147148-26 (x86)
  • Post Solaris 10 1/13 Kernel patches have the PatchIDs 148888-xx (SPARC) / 148889-xx (x86)

Please note that there are no more planned updates to Solaris 10, so these latest Kernel PatchIDs - 148888-xx (SPARC) / 148889-xx (x86) - will continue to be used for the foreseeable future.

Murphy's Law strikes again!

No sooner had I written that Solaris 10 Kernel PatchIDs 148888-xx (SPARC) and 148889-xx (x86) were here to stay for the foreseeable future, than the integration of the SR-IOV feature into rev-04 of these patches made it prudent to rejuvenate them. 

So from July 2013, the Solaris 10 Kernel PatchIDs will change to be 150400-xx (SPARC) and 150401-xx (x86).

Dare I tempt fate again by saying these Solaris 10 PatchIDs are likely to remain the same for the foreseeable future ?

I've also updated my Useful Patch Related Downloads posting with links to the Solaris 10 1/13, Jan 2013 CPU, and latest Recommended patchsets.

Thursday Sep 09, 2010

Solaris 10 9/10 (Update 9) released

Solaris 10 9/10 (Update 9) has been released.  See here for information and here for the download (remember to accept the license agreement at the top).  There's also a podcast and a dedicated Solaris blog.

A number of technical articles have been released, including George Wilson's video overview of ZFS enhancements in Solaris 10 9/10.

As with all Solaris Updates, Solaris 10 9/10 contains all available bug fixes which were available at the time that its contents were finalized, pre-applied into the Solaris Update image. 

It also contains a significant number of feature enhancements as described in the above links.

The corresponding Solaris Update Patch Bundle is currently in test and I expect that it should be released in a similar timeframe to previous Updates.  See http://blogs.sun.com/patch/entry/solaris_10_10_08_patch  for information on Solaris Update Patch Bundles.

All standard patches in Update 9 have already been released to SunSolve and My Oracle Support (MOS).  I've updated the Solaris 10 Kernel PatchID Sequence entry below with the Kernel PatchIDs for Solaris 10 9/10 (Update 9).

As with previous Updates, there are a small number of "special" or "script" patches whose sole purpose is to correct issues in the pre-application of patches to the Solaris Update release image.  Since these patches have no purpose whatsoever outside of the Solaris Update build process, they are not released to SunSolve/MOS.   Newer "special" patches have PatchIDs of the format 800xxx to make them easily identifiable, but old "special"/"script" patches are identifable by the words "SPECIAL PATCH" and/or "script patch" in the patch synopsis.  See the SPARC and x86 patch lists. 

<pet peeve>

Please note it is incorrect to refer to Kernel Patch 142909-17 (SPARC) / 142910-17 (x86) as the "Update 9 Kernel patch".  It is the latest Kernel Patch included in Update 9, but this Kernel patch can equally be applied to all previous Solaris 10 releases.   Solaris Updates are built from patches (and a few new packages), patches are not built from Solaris Updates.

</pet peeve>

Wednesday Oct 07, 2009

Latest Solaris 10 Kernel PatchIDs

I've updated the Solaris 10 Kernel PatchID sequence table in http://blogs.sun.com/patch/entry/solaris_10_kernel_patchid_progression with the latest Kernel PatchIDs for Solaris 10 10/09 (Update 8) and Solaris 10 Update 9.

Thursday Jun 25, 2009

Heads up on Kernel patch installation issues with jumpstart or ZFS Root

I'd like to give you a heads-up on a couple of Kernel patch installation issues:

1. There was a bug (since fixed) in the Deferred Activation Patching functionality in a ZFS Root environment on x86 only.  See Sun Alert 263928.  An error message to the effect that a Class Action Script has failed to complete and failure to set up environment for Deferred Activation Patching may be seen.   The relevant CR is 6850329: "KU 139556-08 fails to apply on x86 systems that have ZFS root filesystems and corrupts the OS".    SPARC systems are similarly affected.  The following error message is returned:
mv: cannot rename /var/run/.patchSafeMode/root/lib/libc.so.1.20102 to /lib/libc.so.1: Device busy
ERROR: Move of /var/run/.patchSafeMode/root/lib/libc.so.1.20102 to dstActual failed
usage: puttext [-r rmarg] [-l lmarg] string
pkgadd: ERROR: class action script did not complete successfully

Installation of <SUNWcslr> failed.

This issue is fixed in patch in the Patch Utilities patch 119255-70 or later revision.

BTW: The principal reason ZFS Root support was implemented in Live Upgrade is so that patch application like this to the live boot environment would not be necessary.   With ZFS Root, creating a clone Boot Environment is so easy that there's no good reason not to.   This avoids the need to use technologies such as Deferred Activation Patching which attempt to make it safer to apply arbitrary change to a live boot environment, which is an inherently risky process.

2. There are reproducible issues using jumpstart finish scripts and other scenarios to install Kernel patch 137137-09 followed by Kernel patch 139555-08.   Here's the gist of the issue which I've pulled from an engineering email thread on the subject:

Issue 1: I have a customer whose system is not booting after applying the patch cluster with Live Upgrade (LU).

Solution 1: If using 'luupgrade -t', then you must ensure that latest version of LU patch is installed first, currently 121430-36 is currently the latest revision on SPARC, 121431-37 on x86. Once these patches are installed, LU will automatically handle the build of the boot archive when 'luactivate' is called, thus avoiding the problem.

Issue 2: There are other ways to get oneself into situations where a boot archive is out of sync: e.g. If using jumpstart finish scripts to apply patches that include 137137-09.  Basically any operation that involves patching to an ABE outside of 'luupgrade' will involve a manual build of boot-archive.

Solution 2: One must manually rebuild the boot-archive on the /a partition after applying the patches.  Otherwise once the system boots, the boot-archive will be out of sync.

Here's some more detail on the jumpstart finish script version of this: 

We've seen the same panic a few times when the latest patch cluster is applied via a finish script to a boot environment prior to  s10u6 via a jumpstart installation. It appears that the boot archive is out of sync with the kernel on the system. The boot archive was created from the 137137-09 patch and not updated after the 139555-08 kernel was applied, therefore the mismatch between the kernel and the boot archive.

In these instances updating the boot archive allows the system to boot successfully. Boot failsafe (ok boot -F failsafe) will detect an out of sync boot archive.  Execute the automated update then reboot.  This will now boot from the later kernel (139555-08) which successfully installed from the finish script.

I reproduced the problem in a jumpstart installation environment applying the latest 10_Recommended patch cluster from a finish script. The initial installation was S10U5 which is deployed from a miniroot that has no knowledge of a boot archive (my theory anyway).  This is similar to a live upgrade environment if the boot environment doing the patching is also boot archive unaware (meaning the kernel is pre 137137-09).

In the jumpstart scenario the immediate problem was solved by updating the boot archive by booting failsafe as previously described.  The Solution was to update the boot archive from the finish script after the patch cluster installation completed.  BTW, all patches in the patch cluster installed successfully per the /var/sadm/system/logs.finish.log.

In a standard jumpstart the boot device (install target) is mounted to /a, therefore adding the following entry to the finish script solved the problem:

/a/boot/solaris/bin/create_ramdisk -R /a

Depending on the finish script configuration, and variables the following would also work:

$ROOTDIR/boot/solaris/bin/create_ramdisk -R $ROOTDIR
Issue 3: This above issues are sometimes mis-diagnosed as CR 6850202: "bootadm fails to build bootarchive in certain configurations leading to unbootable system".

But CR 6850202 will only be encountered in very specific circumstances, all of which must occur in order to hit this specific bug, namely:

1. Install u6 SUNWCreq - there's  no mkisofs so we build ufs boot archive

2. Limit /tmp to 512M - thus forcing the ufs build to happen in /var/run

3. Have a separate /var - bootadm.c only lofs nosub mounts "/" when creating the alt root for DAP patching build of boot archive

4. Install 139555-08

You must have all 4 of above in order to hit this, i.e. step 4 must be installing a DAP patch such as a Kernel patch associated with a Solaris 10 Update such as 139555-08. 

Solution 3: Removing the 512MB limit (or whatever limit has been imposed) to /tmp in /etc/vfstab and/or adding SUNWmkcd (and probably SUNWmkcdS) so that mkisofs is available on the system is sufficient to avoid the code path that fails this way.

Booting failsafe and recreating the boot archive will successfully recreate the boot archive.

Here's further input from one of my senior engineers, Enda O'Connor:

If using Live Upgrade (LU), and LU on the live partition is up to date in terms of latest revision of the LU patch, 121430 (SPARC) and 121431 (x86), the boot-archive will be built automatically once users runs shutdown ( after luactivate to activate the new BE ).  This is done from a kill script in rcd.0.

If using a jumpstart finish script, or jumpstart profile to patch a pre-U6 image with latest kernel patches, then you need to run create_ramdisk from the finish script after all patching/packaging operations have been finished.  Alternatively, you can patch your pre-U6 miniroot to the U6 SPARC NewBoot level (137137-09), at which point the modified miniroot will handle the build of the boot_archive after the finish script has run.

If patching U6 and upwards from jumpstart, the boot_archive will get built automatically after finish script has run, so there's no issue in this scenario.

If using any home grown technology to patch or install/modify software on an Alternate Boot Environment ( ABE ), such as ufsrestore/cpio/tar for example, you must always run create_ramdisk manually before booting to said ABE.

Best Wishes,

Gerry.

Friday May 08, 2009

Patch News Update

Solaris 10 5/09 (Update 7) and subsequent Kernel PatchIDs 

The Kernel patch included in Solaris 10 5/09 (Update 7) has now been released to SunSolve.  The PatchIDs are 139555-08 (SPARC) and 139556-08 (x86).  The rest of the patches included in Solaris 10 5/09 are either released or in the process of being released over the next week or so. 

I've updated the Solaris Kernel PatchID sequence listed in http://blogs.sun.com/patch/date/20080416 to reflect this, including the PatchIDs of the post Solaris 10 5/09 (Update 7) sustaining Kernel PatchID and the Solaris 10 Update 8 Kernel PatchID.

We will be releasing a patch bundle containing the set of patches included in Solaris 10 5/09 in the next couple of weeks.  This will be available from the "Solaris Updates Patch Bundle" section on the new look SunSolve Patch Cluster and Patch Bundle page, http://sunsolve.sun.com/show.do?target=patches/patch-access, which now includes a description of the purpose, contents, and update frequency of each patch cluster and bundle.

As always, customers need to have a valid support contract in order to download Solaris patch clusters and bundles.

New PatchFinder tool coming

We plan to release a new PatchFinder tool on SunSolve at the end of May.   This leverages the patch metadata in our release database to provide a much richer customer experience to search for patches.  Further enhancements are planned after the initial deployment.  The old PatchFinder tool will remain available for a transition period.

Zones Parallel Patching Performance Enhancement

The Zones Parallel Patching performance enhancement continues on schedule.  It has been successfully beta tested by a number of key customers who confirm a 3x performance improvement patching zones.   It is on schedule to be released in a revision of the patch utilities patch (119254 SPARC / 119255 x86) in June.

Solaris 10 "Recommended" and Sun Alert Patch Clusters

Improvements to the Solaris 10 "Recommended" and Sun Alert Patch Clusters are on schedule to be deployed before the end of June.  The improvements include an improved install_cluster script (currently available in the Solaris 10 Live Upgrade Zones Starter Patch Bundle and the Solaris 10 Updates Patch Bundles) and other process improvements designed to improve quality.

Solaris 8 Vintage Patch Support Program

The Solaris 8 Vintage Patch Support Program is now up and running.  The first Vintage Solaris 8 patches have been released.

Patches which fix SunAlerts which were issued prior to the April 1 start date of the Solaris 8 Vintage Patch Support program will be released as "normal" (i.e. non-vintage) Solaris 8 patches. 

But all other Solaris 8 patches created after April 1, including patches which fix security issues, require customers to have a Solaris 8 Vintage Patch Support contract to use them.

See http://www.sun.com/bigadmin/topics/vintagepatch/  for further information.

Ooops! Please ignore empty feature kernel patches accidentally released

A software glitch caused a number of empty patches to be accidentally released last week.   The PatchIDs are:

  • 139555, revisions -01 to -07: Solaris 10: Kernel Patch
  • 139556, revisions -01 to -07: Solaris 10_x86: Kernel Patch
  • 139981-01: Solaris 10_x86: md patch

The above patches have been removed from SunSolve.   All of these patches were empty (i.e. they contained no payload packages), so they are incapable of being installed or causing any problems on a customer system.   This notice is simply to clear up any confusion.

The correct Kernel patch revisions are now available.  These are:

  • 139555-08, which is the Solaris 10 Kernel patch included in Solaris 10 5/09 (Update 7)
  • 139556-08, which is the Solaris 10 Kernel patch included in Solaris 10_x86 5/09 (Update 7)

The correct "md" patch revision will be released shortly.  This is:

  • 139981-03: Solaris 10_x86: md patch

On behalf of Sun, I apologize most sincerely for any confusion or inconvenience caused. 

Tuesday May 13, 2008

Solaris 8 and Solaris 9 Kernel PatchID Sequence

As mentioned in a previous posting, the practice of patch "rejuvenation" to break out large complex patches (typically Kernel patches) into smaller, simpler components going forward has a side effect of making it difficult to follow the sequence of PatchIDs.  If you have the parent patch (e.g. an old Kernel patch), it's not obvious which child patches supercede the parent (e.g. what's the latest Kernel PatchID) as the parent isn't obsoleted by rejuvenation.  Instead, the children of the rejuvenation each specify a Requirement on the parent patch from which they were rejuvenated.

I've listed the Solaris 10 Kernel PatchID Sequence in a previous posting.  For the sake of completeness, here's the Solaris 8 and Solaris 9 Kernel PatchID Sequences (with the most current PatchID top of the list):

Solaris 8 Kernel PatchID Sequence

 SPARCx86
117350-01 to -xx
117351-01 to -xx
requires
requires
117000-01 to -05
117001-01 to -05
requires
requires
108528-01 to -29
108529-01 to 29

Solaris 9 Kernel PatchID Sequence

 

 SPARCx86
122300-02 to -xx
122301-02 to -xx
requires
requires
118558-01 to -39
118559-01 to -39
requires
requires
117171-01 to -17
117172-17 only
requires
obsoletes
112233-01 to -12
112234-04 to -11

Wednesday Apr 16, 2008

Solaris 10 Kernel PatchID Sequence

Entry updated June 6, 2013, with new Solaris 10 Kernel PatchIDs, 150400-xx (SPARC) and 150401-xx (x86):

Patch Rejuvenation

When a patch becomes complex and unwieldy, it is "Rejuvenated".   That is, no more revisions of the patch are created.  Instead, further code changes to objects contained in the patch are delivered in a series of smaller, simpler, new child PatchIDs, each of which declares a dependency upon (i.e. requires) the parent Patch.

This process is known as Patch Rejuvenation and is typically performed on the Kernel patches associated with Solaris Update releases.

Customers still need to install the large parent patch once, but subsequent bug fixes are delivered in smaller, simpler patches.

The parent patches effectively provide "stepping-stones" to reach certain key functionality levels, with rejuvenation enabling smaller incremental change in between Update releases.

If a child patch itself becomes complex and unwieldy over time, it too may be Rejuvenated, so we end up with a "family tree" of PatchIDs providing a lineage of bug fixes for particular code areas such as the Kernel.

See https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1006717.1 for further information.

Effect of Solaris Update "SplitGate" process on PatchIDs

Starting in February 2007, a new, improved, source code "Gate" management process was introduced for core Solaris.  This is known as the "SplitGate" process.  This replaced the old "Feature Foldback" Gate management model. "SplitGate" provides much better separation during the development process between feature code destined for release as part of a Solaris Update release and sustaining (bug fix) patches.  This addresses the problem with earlier Solaris 10 Update releases where issues with features destined for an Update release was adversely impacting the releasability of sustaining (bug fix) patches for customers in production on earlier Solaris 10 releases.

Note, as described in earlier blog postings, any change to pre-existing packages, whether as the result of new feature code or bug fixes, is always delivered in a patch.  Therefore, the Kernel patches released at the end of each Update do contain a mixture of feature and bug fix code.  What "SplitGate" provides is better separation of the feature code from bug fix patches until the Update is ready for release.

The improvement in Solaris 10 Kernel patch releasability has been dramatic:

Releasable Solaris 10 Kernel Patches using “Feature Foldback” model:

    SPARC:      21 out of 66 = 32%    (from March 2005 to February 2007)
    x86:            12 out of 66 = 18%    (from March 2005 to February 2007)

Releasable Solaris 10 Kernel Patches using “SplitGate” model:

    SPARC:      105 out of 121 =  87%    (from February 2007 to current [Sept 10th, 2013])
    x86:            108 out of 120 =  90%    (from February 2007 to current [Sept 10th, 2013])

A side effect of the "SplitGate" process, is that each Solaris 10 Update release, starting with Solaris 10 8/07 (Update 4) introduces a new set of PatchIDs which accumulate and obsolete the preceding set of PatchIDs. 

So, for example, a single new Kernel PatchID revision will appear at the end of each Solaris 10 Update release. For instance, 120011-14 (SPARC) and 120012-14 (x86) is the Kernel PatchID associated with Solaris 10 8/07 (Update 4).  Revisions -01 to -13 of this patch are not released to customers as these are purely for the interim internal builds of the Update.  Therefore, 120011-14 (SPARC) and 120012-14 (x86) are the only revisions of these PatchIDs to be released to customers.  This Kernel patch associated with the Update release is then Rejuvenated, so subsequent bug fixes will appear in a new set of PatchIDs, each of which will depend upon (i.e. require) the parent PatchID from which they were rejuvenated.

Solaris 10 Kernel Patch Lineage 

The impact of Patch Rejuvenation and the "SplitGate" process results in the following sequence of Solaris 10 Kernel PatchIDs, starting with the youngest (newest) child PatchID.  The install order of Kernel patches is starting from the bottom of the table upwards:

Solaris 10 SPARC Kernel PatchIDs
Description
Solaris 10 x86 Kernel PatchIDs

150400-01 to 150400-xx

Kernel Bug Fixes
from July 2013

 150401-01 to 150401-xx

  148888-01 to 148888-05

Kernel Bug Fixes
post Solaris 10 1/13 (Update 11) to June 2013

 148889-01 to 148889-05

  147147-26 only

Solaris 10 1/13 (Update 11) Kernel PatchID

 147148-26 only

  147440-01 to 147440-27

Kernel Bug Fixes
post Solaris 10 8/11 (Update 10)

 147441-01 to 147441-27

  144500-19 only

Solaris 10 8/11 (Update 10) Kernel PatchID
 144501-19 only
 144488-01 to 144488-17

Kernel Bug Fixes
post Solaris 10 9/10 (Update 9)

 144489-01 to 144489-17

  142909-17 only

Solaris 10 9/10 (Update 9) Kernel PatchID
 142910-17 only
 142900-01 to 142900-15

Kernel Bug Fixes
post Solaris 10 10/09 (Update 8)

 142901-01 to 142901-15
 141444-09 only
Solaris 10 10/09 (Update 8) Kernel PatchID   141445-09 only
 141414-01 to 141414-10

Kernel Bug Fixes
post Solaris 10 5/09 (Update 7)

 141415-01 to 141415-10
139555-08 only
Solaris 10 5/09 (Update 7) Kernel PatchID 139556-08 only
138888-01 to 138888-08

Kernel Bug Fixes
post Solaris 10 10/08 (Update 6)

138889-01 to 138889-08
 137137-09 only

Solaris 10 10/08 (Update 6) Kernel PatchID

 137138-09 only
137111-01 to 137111-08

Kernel Bug Fixes
post Solaris 10 5/08 (Update 5)

137112-01 to 137112-08
 127127-11 only
Solaris 10 5/08 (Update 5) Kernel PatchID
 127128-11 only
 127111-01 to 127111-11

Kernel Bug Fixes
post Solaris 10 8/07 (Update 4)

 127112-01 to 127112-11
 120011-14 only
Solaris 10 8/07 (Update 4) Kernel PatchID
 120012-14 only
 125100-04 to 125100-10

Kernel Bug Fixes
post Solaris 10 11/06 (Update 3)

125101-01 to 125101-10
118833-02 to 118833-36

118833-33 (SPARC) / 118855-33 (x86) is the Kernel patch included in Solaris 10 11/06 (Update 3) but these patches were not releasable as "standalone" patches to SunSolve.

118833-17 (SPARC) / 118855-14 (x86) is the Kernel patch included in Solaris 10 6/06 (Update 2). 118855-14 was not releasable as a "standalone" patch to SunSolve.

118855-01 to 118855-36
118822-01 to 118822-30
118822-25 (SPARC) / 118844-26 (x86) is the Kernel patch included in Solaris 10 1/06 (Update 1). 118844-26 was not releasable as a "standalone" patch to SunSolve.

118844-01 to 118844-30



About

This blog is to inform customers about patching best practice, feature enhancements, and key issues. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle. The Documents contained within this site may include statements about Oracle's product development plans. Many factors can materially affect these plans and the nature and timing of future product releases. Accordingly, this Information is provided to you solely for information only, is not a commitment to deliver any material code, or functionality, and SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle. THIS INFORMATION MAY NOT BE INCORPORATED INTO ANY CONTRACTUAL AGREEMENT WITH ORACLE OR ITS SUBSIDIARIES OR AFFILIATES. ORACLE SPECIFICALLY DISCLAIMS ANY LIABILITY WITH RESPECT TO THIS INFORMATION. ~~~~~~~~~~~~ Gerry Haskins, Director, Software Lifecycle Engineer

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today