Thursday Jan 16, 2014

Updating database servers using - part 3

When running for updating a db server the "-l" argument is always required to specify the source location of the new release/update. From here we will now call this  'location' the repository'.  The repository comes in two different flavors: as an ISO and as Oracle ULN channel. Let's start with the ULN channel.  

Exadata ULN Channels

For the different Exadata releases starting a corresponding 'base channel' is available on (ULN). Exadata customers can register to ULN with their CSI, subscribe to the channel (e.g. patch/release) they require and then synchronize one or more channels to with the local (in house) repository. This local repository can then be made available to the internal data-center by publishing it via a web server. 

Note: it is not recommended to use an Exadata db server as local YUM repository

Instructions how to subscribe a system to ULN and synchronize to a local repository are the same as for Oracle Linux and Oracle VM, so generic instructions can be found on OTN here. The README of a specific Exadata release will always mention what channels are made available for that release. You can also find this information via MOS 888828.1 and also in 1473002.1.

Additional to the 'base channel', there is also a 'latest channel'. Currently for Exadata there is a latest channel for 11.2 and 12.1 releases. The content of the 'latest' channel will never remain the same (unlike the 'base' channel) as long as there will be updates for that 11.2 or 12.1 release. When for example a new Exadata 12 release will be published this will be added to the existing latest channel (in addition to a 'base channel' also made available). This is the primary reason for the 'latest' channel being much larger (and taking more time to synchonize) than a base channel.

For Exadata installations on release later than, the 'latest' channel brings additional options. With the latest channel it's now possible to specify what release you want to update to. For example when on planning to update to a later (but not the latest 12.1.release, just as an example) you can use the 'latest' channel and specify the "-t" flag to specify what release you want to update to. 

Note that this can only be done with the 'latest' channel and that without specifying the "-t" argument by default the db server will be updated to the most recent release it can find in that channel. Of course there is also the option to just grab the 'base' channel and update without specifying any "-t' option.


  • updating with a latest channel specifying no argument (latest release in the channel will be used) here
  • updating with the latest channel to a specific release that not exists (a typo) here
  • updating to a specific release here

Exadata channel as ISO 

For those not able or willing to synchronize repositories with Oracle ULN there is also an ISO image available. The ISO image is built (and zipped) by Oracle and is only available for  'base' channel content. An ISO is ~1GB and about the same size as the sum of all packages of the corresponding base channel on ULN.

Using ISO or ULN

From an 'update' perspective there isn't much difference between using ISO or a http repository, only the location (-l) changes:

For local YUM repositories (synchronized from Oracle ULN):

./ -u -l http://myrepo/yum/unknown/EXADATA/dbserver/

For using an ISO (example with the iso):

./ -u -l ./ 

The ISO file should not be unzipped and there is no need to make an local 'loop mount' to use the iso - this is all done by the script


For each type of repository some validation checks will be done to see if it a usable is a repository, checks are done for expected files and also if the available Exadata release in the repository is a later release than the one currently installed - because if not, an update would not be possible. 

When specifying an http repository it's required to specify the top level directory containing the 'YUM metadata repository directory'. Typically this is the directory that has the 'repodir' directory in it. (see example here). When an http location cannot be identified as a valid repository an you would see a suggestion how to locate the right url.

Rene Kundersma

Tuesday Jan 14, 2014

Updating database servers using - part 1

In this and future posts I am planning to describe some new functionality and background of starting with Oracle Exadata Database Machine release Some of this functionality will be directly available to the operator via the interface and can actually be used via an argument, however, some of the recent changes are made to make patching even easier, reduce human error and downtime.

You may also find some of the 'new features' described in MOS 1553103.1 'Exadata Database Server Patching using the DB Node Update Utility'

Exclusion/Obsolete list 

With updates to Exadata or later some packages on the database server will become obsolete. When updating a db server the script will mention an 'RPM exclusion list' and an 'RPM obsolete list' in it's confirmation screen. The 'RPM obsolete list' will list the all packages that will be removed by default during the update to (or later) when no action is taken by the operator.

As an example - click here

If you would like to find out first what obsolete packages will be removed you have to choose 'n' when prompted to 'Continue ? [Y/n]'. This will stop your current patching session. Then look at the contents of the freshly created 'obsolete list' (it should have the runid of your dbnodeupdate session in it's header). Example here

All the packages listed in the '/etc/exadata/yum/obsolete.lst' file will be removed by default - this has multiple reasons, mainly this is because these packages are not required anymore for Exadata functioning or they are considered a security risk. In case you would like to keep for example the 'java' package, you should create an 'exclusion file'  which is '/etc/exadata/yum/exclusion.lst' and put the 'java*openjdk' rpm name (or wildcard) in it.


[root@mynode u01]# cat /etc/exadata/yum/exclusion.lst

When  is restarted you would see that the 'exclusion file' is detected (an example here).

All packages you have put in the 'exclusion file' will still be listed in the obsolete file, but will not be removed when the confirmation screen says 'RPM exclusion list: In use (rpms listed in /etc/exadata/yum/exclusion.lst)'  with the update to or later.

Frequent releases of - keep an eye on it: and it's MOS note were designed / made to be released frequently and quickly when needed.This way can provide workarounds for known issues, emergency fixes, new features and best practices to the operator relatively quick.This reduces risk of people not keeping up to date with 'known issues' or not being sure it applies to them. Basically the same idea as with the patchmgr plugins.

Also, unlike to the storage servers some customization can be done on the db servers - best practices and lessons learned from this in regards to patching may also benefit your environment. For those engineers involved with upgrading Exadata database nodes, I'd like to emphasis to always check for the most recent release of when doing a db server update. I have already seen some people watching the releases closely, which is definitely a good thing.

Rene Kundersma

Thursday Jun 28, 2012

Using ISO Image with a Local Repository for updating Exadata Compute nodes

For systems that cannot connect directly to Oracle ULN to build a local repository an ISO image file is made available by Oracle. This ISO image can be mounted and used as a local repository. The ISO image contains a file system that contains only the latest (x86_64) ULN channel and cannot be used to update the database servers to any other release than release ISO and instructions can be found here

 Rene Kundersma

Thursday Jun 21, 2012

New channels for Exadata

With the release of Exadata back in April 2012 Oracle has deprecated the minimal pack for the Exadata Database Servers (compute nodes). From that release the Linux Database Server updates will be done using ULN and YUM. For the release the ULN exadata_dbserver_11. channel was made available and Exadata operators could subscribe their system to it via

With the new release two additional channels are added:
  1. a 'latest' channel (exadata_dbserver_11.2_x86_64_latest)
  2. a 'patch' channel (exadata_dbserver_11.

The patch channel has the new or updated packages updated in from the base channel. The latest channel has all the packages from base and patch channels combined. 

From here there are three possible situations a Database Server can be in before it can be updated to
  1. Database Server is on Exadata release <
  2. Database Server is patched to
  3. Database Server is freshly imaged to
In order to bring a Database Server to for all three cases the same approach for updating can be used (using YUM), but there are some minor differences:

For Database Servers on a release < the following high-level steps need to be performed:
  • Subscribe to el5_x86_64_addons, ol5_x86_64_latest and  exadata_dbserver_11.2_x86_64_latest
  • Create local repository
  • Point Database Server to the local repository*
  • install the update
* during this process a one-time action needs to be done (details in the README)

For Database Servers patched to
  • Subscribe to patch channel  exadata_dbserver_11.
  • Create local repository
  • Point Database Server to the local repository
  • Update the system
For Database Servers freshly imaged to
  • Subscribe to patch channel  exadata_dbserver_11.
  • Create local  repository
  • Point Database Server to the local repository
  • Update the system
The difference between 'situation 2' (Database Server is patched to and 'situation 3' (Database Server is freshly imaged to is that in situation 2 the existing Exadata-computenode.repo file needs to be edited while in situation 3 this file is not existing  and needs to be created or copied. Another difference is that you will end up with more OFA packages installed in situation 2. This is because none are removed during the updating process. 

The YUM update functionality with the new channels is a great enhancements to the Database Server update procedure. As usual, the updates can be done in a rolling fashion so no database service downtime is required. 

For detailed and up-to-date instructions always see the patch README's
Rene Kundersma

Monday Apr 09, 2012

Updating Exadata Compute Nodes using ULN and YUM starting

With this post an update on Exadata Planned Maintenance - The new ULN Updating Procedure for Exadata Compute Nodes.
As you may already know, starting with Oracle Exadata Storage Server release the 'minimal pack' is deprecated.
From now on the (Linux) Computes Node will use Yellowdog Updater (Modified) (YUM) to apply the new updates as rpm packages. 
These RPM packages that come from ULN may also contain firmware updates which will be applied in the same updating procedure.

In order to update the Exadata Compute Nodes via YUM, you need a repository to download the software from. Of course Oracle provides its customers ULN access for this to make this really easy. It may however happen this requirement for http access to a web location from the Compute Nodes is not possible for some reason. For these situations Oracle worked out an alternative solution.

In this post I like to discuss the solution to setup a YUM repository located in the customers Data Center: This is a local system, that can download the updates from ULN and act as a YUM repository for the Compute Nodes as a 'man in the middle' bertween ULN and the Compute Nodes.

For installations planning to setup an internal/local YUM repository and currently not on there are two notes that need to be reviewed carefully before the patching starts:

README for patch 13741363 : Especially chapter 3 "Performing One-Time Setup on Database Servers of Oracle Exadata Database Machine"

The steps described here are one time only steps to prepare and populate the local YUM repository server that will be used for ALL the Compute Nodes. Best is to install the YUM repository server on a separate Linux machine running OEL 4 or OEL 5. Basically the steps are: go to, use your Hardware CSI for the registration steps, register the YUM repository server and subscribe to the right channels and populate the repository. After the download is finished, the repository is 'build' and now 'local' it needs to be made available by http for the Compute Nodes to download from on the local network.

After the setup of the repository V2/X2-2 and X2-8 Compute Nodes require a One-Time setup so they are able to use YUM from now on. The One-Time steps remove a set of packages that can cause the further setup to fail, but also add some packages to support the installations to be done using YUM onwards.

Please pay close attention to one of the most important steps of this One-Time setup: the editing of the repository configuration file /etc/yum.repos.d/Exadata-computenode.repo and making sure it points to your local YUM repository if you don't have direct ULN access.

README for 13536739: Especially chapter 6 "Updating Oracle Linux Database Servers in Oracle Exadata Database Machine" 

After setting up the repository and enabling the Compute Nodes to use YUM there is one thing to do: the update itself. Key step in this process is enabling each Compute Node to use the new repository. After this some  packages (ofed1) may need to be downgraded depending on your installation and some checks for kernel versions need to be done. After this, from now on the the system can be updated using a simple 'yum install' to install the main Exadata database server rpm with all dependent packages.
At the end of the installation the Compute Node is rebooted automatically.

At this moment I have to make some remarks/disclaimers:
  • Please see the notes for 13741363  and 13536739 for exact steps, I am only highlighting to explain the overall procedure of setting up a local repository and configuring the Compute Nodes using it.
  • If you are able to connect your Compute Nodes to ULN directly there is no need to setup a repository and the related steps can be skipped.
  • For X2-2 (and earlier) and X2-8 the 'updating Oracle Linux Database Server' steps are slightly different.
  • Oracle has provided 'helper' scripts to automated the steps described above making it even more easy
  • The YUM Updating procedure only applies to Linux Compute Nodes having images > (for updates to versions lower than you still need to use the minimal pack)
Rene Kundersma

Blog of Rene Kundersma, Principal Member of Technical Staff at Oracle Development USA. I am designing and evaluating solutions and best practices around database MAA focused on Exadata. This involves HA, backup/recovery, migration and database consolidation and upgrades on Exadata. Opinions are my own and not necessarily those of Oracle Corporation. See


« April 2014