Monday Mar 10, 2014

Prereq Dependency Check Added for Exadata DB Server Updating

With this blog update some background on new functionality added to the Exadata utility. 

The recently added 'Prereq Dependency Check' feature in eliminates possible unexpected package dependency problems that occur during update by detecting dependency issues during the prerequisite check phase that could occur when the update is performed. (Linux package dependency issues sometimes happen after operators customizing the db server by installing additional software packages or newer releases of packages). The recommended use of this feature is to run with the –v flag (to verify prereqs only) days before the scheduled maintenance. will report dependency problems that must be resolved before proceeding with the update.

Note that the dependency check functionality is also run by default prior to performing the update. will now also list what packages will be removed for your update.

Some details:
  • Updates starting from - to any release earlier than
    • Dependency check is validated against 'standard' dependencies.
  • Updates starting from - to any release equal to or later than
    • Dependency check is first validated against 'exact' RPM dependency.
    • If 'exact' RPM dependency check passes it is assumed 'minimum' RPM dependency check will also pass.
    • If 'exact' RPM dependency check fails then 'minimum' RPM dependency check is run.
  • Updates starting from release do not have Prereq Dependency Check functionality.
  • will report what checks were executed for your update and which of them did 'pass' or 'fail'
    • If the dependency check is executed as part of –u and only 'minimum' RPM dependency check passes, then the new target will implicitly be changed to 'minimum' (which is equal to -m). 
    • If the dependency check is executed as part of –u and both 'exact' and ’minimum’ RPM dependency checks fail, then the operator will not be able to proceed with the update. For dependency checks that fail a separate report is generated.This report highlights to the failing package. The operator can then decide to either remove/install/update the failing package depending on what works best for that particular server.


  • Prereq run here -this is a prereq only run. Notice the ''Exact' package dependency check failed' and the ''Minimum' package dependency check succeeded'
    • Failing dependencies here  - for more details on what package cause the problem and what can be done to resolve it.
  • Update scenario here - see the same dependency checks and notice 'Update target switched to 'minimum''
Existing backups of the current image overwritten by default:

  • Existing backups of the current image on the inactive lvm will be overwritten by default. You can decide to skip (and retain) the backup by using the "-n" flag.

 Rene Kundersma 

Thursday Jan 16, 2014

Updating database servers using - part 3

When running for updating a db server the "-l" argument is always required to specify the source location of the new release/update. From here we will now call this  'location' the repository'.  The repository comes in two different flavors: as an ISO and as Oracle ULN channel. Let's start with the ULN channel.  

Exadata ULN Channels

For the different Exadata releases starting a corresponding 'base channel' is available on (ULN). Exadata customers can register to ULN with their CSI, subscribe to the channel (e.g. patch/release) they require and then synchronize one or more channels to with the local (in house) repository. This local repository can then be made available to the internal data-center by publishing it via a web server. 

Note: it is not recommended to use an Exadata db server as local YUM repository

Instructions how to subscribe a system to ULN and synchronize to a local repository are the same as for Oracle Linux and Oracle VM, so generic instructions can be found on OTN here. The README of a specific Exadata release will always mention what channels are made available for that release. You can also find this information via MOS 888828.1 and also in 1473002.1.

Additional to the 'base channel', there is also a 'latest channel'. Currently for Exadata there is a latest channel for 11.2 and 12.1 releases. The content of the 'latest' channel will never remain the same (unlike the 'base' channel) as long as there will be updates for that 11.2 or 12.1 release. When for example a new Exadata 12 release will be published this will be added to the existing latest channel (in addition to a 'base channel' also made available). This is the primary reason for the 'latest' channel being much larger (and taking more time to synchonize) than a base channel.

For Exadata installations on release later than, the 'latest' channel brings additional options. With the latest channel it's now possible to specify what release you want to update to. For example when on planning to update to a later (but not the latest 12.1.release, just as an example) you can use the 'latest' channel and specify the "-t" flag to specify what release you want to update to. 

Note that this can only be done with the 'latest' channel and that without specifying the "-t" argument by default the db server will be updated to the most recent release it can find in that channel. Of course there is also the option to just grab the 'base' channel and update without specifying any "-t' option.


  • updating with a latest channel specifying no argument (latest release in the channel will be used) here
  • updating with the latest channel to a specific release that not exists (a typo) here
  • updating to a specific release here

Exadata channel as ISO 

For those not able or willing to synchronize repositories with Oracle ULN there is also an ISO image available. The ISO image is built (and zipped) by Oracle and is only available for  'base' channel content. An ISO is ~1GB and about the same size as the sum of all packages of the corresponding base channel on ULN.

Using ISO or ULN

From an 'update' perspective there isn't much difference between using ISO or a http repository, only the location (-l) changes:

For local YUM repositories (synchronized from Oracle ULN):

./ -u -l http://myrepo/yum/unknown/EXADATA/dbserver/

For using an ISO (example with the iso):

./ -u -l ./ 

The ISO file should not be unzipped and there is no need to make an local 'loop mount' to use the iso - this is all done by the script


For each type of repository some validation checks will be done to see if it a usable is a repository, checks are done for expected files and also if the available Exadata release in the repository is a later release than the one currently installed - because if not, an update would not be possible. 

When specifying an http repository it's required to specify the top level directory containing the 'YUM metadata repository directory'. Typically this is the directory that has the 'repodir' directory in it. (see example here). When an http location cannot be identified as a valid repository an you would see a suggestion how to locate the right url.

Rene Kundersma

Wednesday Jan 15, 2014

Updating database servers using - part 2

Within the Oracle Exadata Database Machine documentation and README's you will generally find two types of backups for database server OS backup:
  • The Exadata Database Machine Owners Guide (Chapter 7) has instructions to create backups stored outside of the dbserver, for example on an NFS mount (see 'Creating a Snapshot-Based Backup of Oracle Linux Database Server')
  • - which creates a local copy of your active lvm.

In this post I will explain background and usage for both backups and how they integrate with

For backing-up and rolling-back Exadata dbserver OS updates  the script is used by For each upgrade by default the script is executed. When executed (either manually or via dbnodeupdate), the script creates a small snapshot of the 'active' sys lvm. The active sys lvm is the primary lvm that your current OS image is running on. For example:

[root@mynode ~]# imageinfo

Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
Image version:
Image activated: 2014-01-13 13:20:52 -0700
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys2

In the above example the active lvm is /dev/mapper/VGExaDb-LVDbSys2.The snapshot is created to have a 'consistent' 'view' of the root filesystem while the backup is made. After the snapshot is created, it's mounted by the same script and then it's contents are copied over to the inactive lvm. For lvm enabled systems, there are always 2 'sys' lvm's "VGExaDb-LVDbSys1" and "VGExaDb-LVDbSys2". VGExaDb-LVDbSys2 will automatically be created (on lvm enabled system) if not existing yet. For the example above, the 'inactive' lvm will be VGExaDb-LVDbSys1

Now, depending on how many files there are in the root (/) filesystem (based on your active sys lvm) the backup times may vary. Previous Grid and Database home installation zip files in /opt/oracle.SupportTools/onecommand will make the backup take longer (not the restore, which I will explain why later). Same for those who have many small files (like mail messages in /var/spool) - the backup may take longer. 

One of the first steps the script will doing when executed is making a backup with this script. Now, if you want to shorten your downtime and make this backup before the start of your 'planned maintenance window' you have 2 options: Either execute the script yourself or use with the "-b" flag to make a backup only before hand.

Example making a backup with here (see 'Backup only' for 'Action')

When you then have the downtime for planned maintenance and already have the backup you can then let dbnodeupdate skip the backup using the "-n" flag.

Example skipping a backup with here (See 'Create a backup: No')

Both Sys lvm's are 30GB each. The snapshot that will be created is ~1GB. It is recommended to keep this in mind when claiming the free space in the volume group to make your /u01 filesystem as big as possible. (the script checks for 2 GB free space in the volume group)

Now, when the update proceeds, the current active lvm will remain the active lvm. This is different than what happens on the cells where the active lvm becomes inactive with an update.  Typically you will only switch active sys lvm's when a rollback needs to be done on a db server, for example, an upgrade from to needs to be rolled-back. What happens then is nothing more than 'switching' the filesystem label of the sys lvm's, updating grub (the bootloader) and restoring the /boot directory (backed up earlier also by Then, a next boot will now have the previous inactive lvm as active lvm.

Rolling back with as in the example here (a rollback from to 

After booting the node, it's recommended to run again with the "-c" flag to relink the oracle home's again.


  • It's important, to make a new backup before attempting a new update.
  • In the above example, there is only talk about the sys lvm's. This means custom partitions including /u01 are not backed up. For regular node updates this is enough to rollback to a previous release but it's recommended to also have a backup of other filesystems inline to your requirements.
  • Nodes deployed without lvm will not have this option available
  • Rolling back db servers to previous Exadata releases with this procedure does not rollback the firmware
Backup / restore procedure owners guide chapter 7 

The backup made with the procedure in chapter 7 of the Oracle Exadatabase Database owners guide covers total node recovery.  Like the procedure a snapshot is used for a consistent view, then in this scenario a copy is placed outside of the db server (via NFS in this example).  This procedure allow you to backup every filesystem you require. In case of emergency - such as a non-bootable system, the node can be booted with the diagnostic iso. For non-customized partitions an interactive script will then question you to provide backup details and recover the node completely. For customized partitions steps (which are almost the same) can also be found in the owners guide.

Advantages /  Disadvantages

Both type of backups serve another goal. Also, these are just examples - of course customized backup and restore scenario's are also possible.The procedure as described in the owners guide requires external storage, while the script uses space on the node - but that is also where the risk is. The backup made with works well for the purpose of rolling back upgrades. With the automation of rollbacks can be done simple and quickly.

However - loss of critical partitions and/or filesystems will not be covered with this type of backup - so you may want to combine both types of OS backup. The general recommendation is to use the default built-in backup procedure when running dbnodeupdate to make easy rollback possible. But also backup the entire OS and customized filesystems outside of the database server with an interval based on your own requirements.

Rene Kundersma

Tuesday Jan 14, 2014

Updating database servers using - part 1

In this and future posts I am planning to describe some new functionality and background of starting with Oracle Exadata Database Machine release Some of this functionality will be directly available to the operator via the interface and can actually be used via an argument, however, some of the recent changes are made to make patching even easier, reduce human error and downtime.

You may also find some of the 'new features' described in MOS 1553103.1 'Exadata Database Server Patching using the DB Node Update Utility'

Exclusion/Obsolete list 

With updates to Exadata or later some packages on the database server will become obsolete. When updating a db server the script will mention an 'RPM exclusion list' and an 'RPM obsolete list' in it's confirmation screen. The 'RPM obsolete list' will list the all packages that will be removed by default during the update to (or later) when no action is taken by the operator.

As an example - click here

If you would like to find out first what obsolete packages will be removed you have to choose 'n' when prompted to 'Continue ? [Y/n]'. This will stop your current patching session. Then look at the contents of the freshly created 'obsolete list' (it should have the runid of your dbnodeupdate session in it's header). Example here

All the packages listed in the '/etc/exadata/yum/obsolete.lst' file will be removed by default - this has multiple reasons, mainly this is because these packages are not required anymore for Exadata functioning or they are considered a security risk. In case you would like to keep for example the 'java' package, you should create an 'exclusion file'  which is '/etc/exadata/yum/exclusion.lst' and put the 'java*openjdk' rpm name (or wildcard) in it.


[root@mynode u01]# cat /etc/exadata/yum/exclusion.lst

When  is restarted you would see that the 'exclusion file' is detected (an example here).

All packages you have put in the 'exclusion file' will still be listed in the obsolete file, but will not be removed when the confirmation screen says 'RPM exclusion list: In use (rpms listed in /etc/exadata/yum/exclusion.lst)'  with the update to or later.

Frequent releases of - keep an eye on it: and it's MOS note were designed / made to be released frequently and quickly when needed.This way can provide workarounds for known issues, emergency fixes, new features and best practices to the operator relatively quick.This reduces risk of people not keeping up to date with 'known issues' or not being sure it applies to them. Basically the same idea as with the patchmgr plugins.

Also, unlike to the storage servers some customization can be done on the db servers - best practices and lessons learned from this in regards to patching may also benefit your environment. For those engineers involved with upgrading Exadata database nodes, I'd like to emphasis to always check for the most recent release of when doing a db server update. I have already seen some people watching the releases closely, which is definitely a good thing.

Rene Kundersma

Thursday Dec 12, 2013

Enhanced lights-out patching for Exadata Storage Cells

The recently released Oracle Exadata version comes with multiple enhancements in the patchmgr utility. For those who don't know what the patchmgr utility is: the patchmgr utility is a tool Exadata Database Administrators use to apply (or rollback) an update to the Oracle Exadata Storage Cells. The enhanced patchmgr will have an option to send the operator an email for the most significant patch and rollback state changes. This eliminates the need to monitor the screen while the update or rollback is in progress.

In order to send the email you need to specify values for the '-smtp_from' and '-smtp_to' flags. Example as follows:

./patchmgr -cells ~/cell_group -patch -rolling \
           -smtp_from  \

Patchmgr will use the sendmail package which  is installed by default on the Exadata Database Server. It will start sendmail if it's not already started and assumes it's configured to deliver email for the domains you specify. You will recognize the format of the alerts patchmgr sends as they have the same formatting as the ASR emails.

The email you will receive when you enable this option can have the following end states for a cell: 

  • Started
  • Start Failed
  • Waiting
  • Patching
  • Rolling Back
  • Failed
  • Succeeded. 
  • Not Attempted 

The majority of the above states probably is self explanatory, however explaining 'Not Attempted' may help: 'Not Attempted' will be the final state of a cell when patching or rolling back (in a rolling fashion) of the current cell has failed. In that case the patching will stop and the remaining cells (which are not touched yet) are in the state 'Not Attempted'. So for example: imagine you are patching cel1,cel2 and cel3. When cel1 fails patching, then the end state for cel2 and cel3 will be 'Not Attempted'.

The following example will give you the idea: Patching starts in a rolling fashion from to for cel1,cel2 and cel3. 

Note: this is an example of the type of state changes you will see, actual patch timings are not correct and not relevant for this example. Also actual states or formatting may change in your release.

1. patchmgr was started in a rolling fashion. Initial state for all cells is 'Started'. This should be seen as this initialization phase. You will receive an email as follows:

2. The actual patching has begun since this is in rolling fashion, patchmgr starts with the first cel listed, which is cel1. The other two cells are waiting until cel1 finishes.

3. After some time patching of cel1 has completed. You will receive another status update in your inbox stating cel1 was updated successfully.

4. patchmgr continues with the next cell.

5. When succeeded (or failed) you will receive a state update via mail.

6. Last cell to be patched is cel3:

7. The last email you will receive will give an update stating patching was completed and lists the end state for all cells.

In case patching or rolling back would fail, you would receive a pointer to a release specific MOS note where you may find additional information.

Rene Kundersma 

Tuesday Sep 03, 2013

Exadata Database Machine Grid Infrastructure and Database Upgrade

Today on My Oracle Support two notes are made available for upgrading your 11.2 Grid Infrastructure and Databases running on the Oracle Exadata Database Machine to 

  • The first note is 1565291.1 This note has the steps for upgrading Grid Infrastructure and Database on BP12 and later to". 
  • The other note is 1555036.1 and has the steps for upgrading Grid Infrastructure and Database on BP11 and earlier to".

Rene Kundersma

Sunday May 26, 2013

Exadata Database Server Patching using the DB Node Update Utility

Today the 'DB Node Update Utility' ( utility) has been made available. The 'DB Node Update Utility' automates all the steps and checks to upgrade Oracle Exadata database servers to a new Exadata release and replaces the manual steps. The utility includes the latest best practices and workarounds for known issues. Where appropriate, the script uses existing 'One-Time Helper scripts' or 'yum update commands'. The script uses the 'build-in' '' script to perform backups before patching where possible. 

Four typical use cases for the utility are :

  • One-Time Setup  (updating procedure for Exadata releases running Oracle Linux 5.5 or later)
  • Updating database servers running Exadata releases later than on Oracle Linux 5.5 or later
  • Rolling back updates
  • Post-Patching (or Post-Rollback steps) (relinking the Oracle homes, enabling Grid Infrastructure to start)

For more details, demo's and downloads see My Oracle Support Note 1553103.1

Rene Kundersma

Wednesday Apr 10, 2013

Exadata Maximum Availability Tests

Recently Oracle published the below video created by our group (MAA  team) which demonstrates Exadata's High Availability in the face of hardware and software failures.  This video is a great showcase on how Exadata handles hardware and software failures. These are tests Oracle does on a daily basis for Engineered systems so every Exadata customer benefits  !

Exadata Maximum Availability Tests from ESG Media on Vimeo.

Tuesday Feb 19, 2013

Oracle Certification: Oracle Exadata Database Machine Administration

For those working on Exadata: Oracle University recently released a new test called "Oracle Exadata Database Machine Administration". This OU exam is very relevant for those designing and administrating Exadata environments.

Passing this test would give you the "Oracle Certified Expert" credential and prove you are able to work out the following key area's:
  • Architecture
  • Exadata Key Capabilities
  • Initial Configuration
  • Storage Server Configuration
  • I/O Resource Management
  • Consolidation
  • Migration
  • Bulk Data Loading
  • Monitoring & ASR
  • Backup and Recovery
  • Planned Maintenance
  • QOS

The test has been developed by Oracle Universities best instructors and I would really recommend taking the test if this is your field of expertise. It's a good opportunity to prove your skills. And for those looking for skilled Exadata specialists, I would recommend asking for people who passed this OU test additional to traditional 11g OCP and some basic Linux & Networking skills.

See: Oracle Certification: Oracle Exadata Database Machine Administration, Software Rel. 11.x

The above link also has references to recommend Exam preparation

Rene Kundersma

Sunday Sep 30, 2012

Exadata X3, and Oracle Platinum Services

Oracle recently announced an Exadata Hardware Update. The overall architecture will remain the same, however some interesting hardware refreshes are done especially for the storage server. Each cell will now have 1600GB of flash, this means an X3-2 full rack will have 20.3 TB of total flash ! For all the details I would like to refer to the Oracle Exadata product page:

Together with the announcement of the X3 generation. A new Exadata release, is made available. New Exadata systems will be shipped with this release and existing installations can be updated to that release. As always there is a storage cell patch and a patch for the compute node, which again needs to be applied using YUM.

Instructions and requirements for patching existing Exadata compute nodes to using YUM can be found in the patch README. Depending on the release you have installed on your compute nodes the README will direct you to a particular section in MOS note 1473002.1.

MOS 1473002.1 should only be followed with the instructions from the patch README. Like with and instructions are added to prepare your systems to use YUM for the first time in case you are still on release and earlier. You will also find these One Time Setup instructions in MOS note 1473002.1

By default all compute nodes that will be updated to will now have the UEK kernel. Before the 'compatible kernel' was used for all compute nodes other than X2-8. For customer will have the choice to replace the UEK kernel with the Exadata standard 'compatible kernel' which is also in the ULN channel. Recommended is to use the kernel that is installed by default.

One of the other great new things brings is Writeback Flashcache (wbfc). By default wbfc is disabled after the upgrade to Enable wbfc after patching on the storage servers of your test environment and see the improvements this brings for your applications.

Writeback FlashCache can be enabled  by dropping the existing FlashCache, stopping the cellsrv process and changing the FlashCacheMode attribute of the cell. Of course stopping cellsrv can only be done in a controlled manner.


  1. drop flashcache
  2. alter cell shutdown services cellsrv
    • again, cellsrv can only be stopped in a controlled manner
  3. alter cell flashCacheMode = WriteBack
  4. alter cell startup services cellsrv
  5. create flashcache all

Going back to WriteThrough FlashCache is also possible, but only after flushing the FlashCache:

  • alter cell flashcache all flush

Last item I like to highlight in particular is already from a while ago, but a great thing to emphasis: Oracle Platinum Services. On top of the remote fault monitoring with faster response times Oracle has included update and patch deployment services.These services are delivered by Oracle Advanced Customer Support at no additional costs for qualified Oracle Premier Support customers.


René Kundersma

Thursday Jun 28, 2012

Using ISO Image with a Local Repository for updating Exadata Compute nodes

For systems that cannot connect directly to Oracle ULN to build a local repository an ISO image file is made available by Oracle. This ISO image can be mounted and used as a local repository. The ISO image contains a file system that contains only the latest (x86_64) ULN channel and cannot be used to update the database servers to any other release than release ISO and instructions can be found here

 Rene Kundersma

Thursday Jun 21, 2012

New channels for Exadata

With the release of Exadata back in April 2012 Oracle has deprecated the minimal pack for the Exadata Database Servers (compute nodes). From that release the Linux Database Server updates will be done using ULN and YUM. For the release the ULN exadata_dbserver_11. channel was made available and Exadata operators could subscribe their system to it via

With the new release two additional channels are added:
  1. a 'latest' channel (exadata_dbserver_11.2_x86_64_latest)
  2. a 'patch' channel (exadata_dbserver_11.

The patch channel has the new or updated packages updated in from the base channel. The latest channel has all the packages from base and patch channels combined. 

From here there are three possible situations a Database Server can be in before it can be updated to
  1. Database Server is on Exadata release <
  2. Database Server is patched to
  3. Database Server is freshly imaged to
In order to bring a Database Server to for all three cases the same approach for updating can be used (using YUM), but there are some minor differences:

For Database Servers on a release < the following high-level steps need to be performed:
  • Subscribe to el5_x86_64_addons, ol5_x86_64_latest and  exadata_dbserver_11.2_x86_64_latest
  • Create local repository
  • Point Database Server to the local repository*
  • install the update
* during this process a one-time action needs to be done (details in the README)

For Database Servers patched to
  • Subscribe to patch channel  exadata_dbserver_11.
  • Create local repository
  • Point Database Server to the local repository
  • Update the system
For Database Servers freshly imaged to
  • Subscribe to patch channel  exadata_dbserver_11.
  • Create local  repository
  • Point Database Server to the local repository
  • Update the system
The difference between 'situation 2' (Database Server is patched to and 'situation 3' (Database Server is freshly imaged to is that in situation 2 the existing Exadata-computenode.repo file needs to be edited while in situation 3 this file is not existing  and needs to be created or copied. Another difference is that you will end up with more OFA packages installed in situation 2. This is because none are removed during the updating process. 

The YUM update functionality with the new channels is a great enhancements to the Database Server update procedure. As usual, the updates can be done in a rolling fashion so no database service downtime is required. 

For detailed and up-to-date instructions always see the patch README's
Rene Kundersma

Monday Apr 09, 2012

Updating Exadata Compute Nodes using ULN and YUM starting

With this post an update on Exadata Planned Maintenance - The new ULN Updating Procedure for Exadata Compute Nodes.
As you may already know, starting with Oracle Exadata Storage Server release the 'minimal pack' is deprecated.
From now on the (Linux) Computes Node will use Yellowdog Updater (Modified) (YUM) to apply the new updates as rpm packages. 
These RPM packages that come from ULN may also contain firmware updates which will be applied in the same updating procedure.

In order to update the Exadata Compute Nodes via YUM, you need a repository to download the software from. Of course Oracle provides its customers ULN access for this to make this really easy. It may however happen this requirement for http access to a web location from the Compute Nodes is not possible for some reason. For these situations Oracle worked out an alternative solution.

In this post I like to discuss the solution to setup a YUM repository located in the customers Data Center: This is a local system, that can download the updates from ULN and act as a YUM repository for the Compute Nodes as a 'man in the middle' bertween ULN and the Compute Nodes.

For installations planning to setup an internal/local YUM repository and currently not on there are two notes that need to be reviewed carefully before the patching starts:

README for patch 13741363 : Especially chapter 3 "Performing One-Time Setup on Database Servers of Oracle Exadata Database Machine"

The steps described here are one time only steps to prepare and populate the local YUM repository server that will be used for ALL the Compute Nodes. Best is to install the YUM repository server on a separate Linux machine running OEL 4 or OEL 5. Basically the steps are: go to, use your Hardware CSI for the registration steps, register the YUM repository server and subscribe to the right channels and populate the repository. After the download is finished, the repository is 'build' and now 'local' it needs to be made available by http for the Compute Nodes to download from on the local network.

After the setup of the repository V2/X2-2 and X2-8 Compute Nodes require a One-Time setup so they are able to use YUM from now on. The One-Time steps remove a set of packages that can cause the further setup to fail, but also add some packages to support the installations to be done using YUM onwards.

Please pay close attention to one of the most important steps of this One-Time setup: the editing of the repository configuration file /etc/yum.repos.d/Exadata-computenode.repo and making sure it points to your local YUM repository if you don't have direct ULN access.

README for 13536739: Especially chapter 6 "Updating Oracle Linux Database Servers in Oracle Exadata Database Machine" 

After setting up the repository and enabling the Compute Nodes to use YUM there is one thing to do: the update itself. Key step in this process is enabling each Compute Node to use the new repository. After this some  packages (ofed1) may need to be downgraded depending on your installation and some checks for kernel versions need to be done. After this, from now on the the system can be updated using a simple 'yum install' to install the main Exadata database server rpm with all dependent packages.
At the end of the installation the Compute Node is rebooted automatically.

At this moment I have to make some remarks/disclaimers:
  • Please see the notes for 13741363  and 13536739 for exact steps, I am only highlighting to explain the overall procedure of setting up a local repository and configuring the Compute Nodes using it.
  • If you are able to connect your Compute Nodes to ULN directly there is no need to setup a repository and the related steps can be skipped.
  • For X2-2 (and earlier) and X2-8 the 'updating Oracle Linux Database Server' steps are slightly different.
  • Oracle has provided 'helper' scripts to automated the steps described above making it even more easy
  • The YUM Updating procedure only applies to Linux Compute Nodes having images > (for updates to versions lower than you still need to use the minimal pack)
Rene Kundersma

Wednesday Jan 18, 2012

First quarterly full stack download patch for Exadata available

Today Oracle released the first quarterly full stack download patch for Exadata a.k.a QFSDP

This patch is available for both Linux and Solaris and contains three primary components:
  1. Infrastructure (storage server, infiniband switch and PDU)
  2. Database (Database, Clusterware but also Opatch and Oplan)
  3. System management (agent, OMS and EM plugins)

The patch is available on MOS for Linux and Solaris and can be used for those customers who want to download the recommended releases for all components for the Oracle Exadata Database Machine in one go.

In this QFSDP you will find the Quarterly Database Patch for Exadata a.k.a QDPE. The current version is Jan 2012.

This is part of a new strategy for patching Exadata that reduces the number and frequency of patches released to customers even more.

For the database: while also continuing to make important fixes available in monthly or bimonthly interim bundle patches, Oracle recommends most customers only applying the QDPE. The currenly recommened QDPE is available via MOS patch 13513783.

MOS note 888828.1 is still the recommend place to look first for the latest information regarding recommend/supported releases.

Rene Kundersma

Saturday Oct 29, 2011

Oracle Enterprise Manager Cloud Control 12c Setup Automation kit for Exadata

With this post I like to mention the availability of the Enterprise Manager 12c setup automation kit for Exadata. I also like to explain how easy it is to deploy the agent for a complete Exadata rack of any size with all it's components. After the discovery the system can managed by EM, the deployment I did took only 30 minutes for a quarter rack with all components.

You can obtain the "Oracle Enterprise Manager Cloud Control 12c Setup Automation kit for Exadata" from MOS by searching for patch 12960596

The deployment of the agent starts with the Exadata configurator sheet. This is the sheet ACS already used for the Exadata deployment. The sheet now also creates a configuration output file for automatic agent and OMS deployment. The file I am talking about is called em.params. This file can be used as input for the OMS or EM 12c deployment scripts. In this posting I will only discuss the agent deployment.

This em.params file will have the following information:

# This is em.param file
# Written : 26-10-2011


EM_CELLS=(cel12 cel13 cel14)

EM_COMPUTE_ILOM_NAME=(db07-c db08-c)
EM_COMPUTE_ILOM_IP=(a.b.c.d a.b.c.e)

machinemodel="X2-2 Quarter rack"



Of course, the names and ipnumbers are changed in this example to hide any specific information.
When you install the kit the configuration file is expected in /opt/oracle.SupportTools/onecommand, this is where your OneCommand deployment files will be anyway.

During installation some additional checks will be done like:
Trying to ping Oracle Management Server Host .
Checking oracle user info
Searching for KVM Switch information by key swikvmname from /opt/oracle.SupportTools/onecommand/em.param
Searching for KVM Switch IP information by key swikvmip from /opt/oracle.SupportTools/onecommand/em.param
Searching for Cisco Switch information by key swiipname from /opt/oracle.SupportTools/onecommand/em.param
Searching for Cisco Switch IP information by key swiipip from /opt/oracle.SupportTools/onecommand/em.param
Searching for IB2 Switch information by key swiib2name from /opt/oracle.SupportTools/onecommand/em.param
Searching for IB2 Switch IP information by key swiib2ip from /opt/oracle.SupportTools/onecommand/em.param
Searching for IB3 Switch information by key swiib3name from /opt/oracle.SupportTools/onecommand/em.param
Searching for IB3 Switch IP information by key swiib3ip from /opt/oracle.SupportTools/onecommand/em.param
Searching for PDUA Name information by key pduaname from /opt/oracle.SupportTools/onecommand/em.param
Searching for PDUA IP information by key pduaip from /opt/oracle.SupportTools/onecommand/em.param
Searching for PDUA Name information by key pdubname from /opt/oracle.SupportTools/onecommand/em.param
Searching for PDUB IP information by key pdubip from /opt/oracle.SupportTools/onecommand/em.param
Searching for ILON Names information by key EM_COMPUTE_ILOM_NAME from /opt/oracle.SupportTools/onecommand/em.param
Searching for ILOM IPS information by key EM_COMPUTE_ILOM_IP from /opt/oracle.SupportTools/onecommand/em.param
Verifying ssh setup..
Verifying SSH setup for root

When the validations are done, the file will be transferred to the compute nodes and the 12c agent will be installed.
Like the 11.1 installation, still no agents will be installed on the storage cells, EM will use the compute nodes as proxy to the cells, it's up to you if you want EM to choose which nodes or choose yourself.

After deployment, there are three easy steps left in the EM GUI:
  • Exadata discovery
  • GI(Cluster) discovery
  • RAC discovery

The discovery steps above are guided by a clear GUI.
All the discovery process needs is access to a compute node, from there all the Exadata components will be discovered automatically.
The file EM needs to be available for that is  /opt/oracle.SupportTools/onecommand/databasemachine.xml

After installation of the kit and discovery in EM 12c, you have a nice overview of the rack with all the components in it, easy to drill down and manage from there. The example below is a quarter rack.

René Kundersma

Oracle MAA Development


Blog of Rene Kundersma, Principal Member of Technical Staff at Oracle Development USA. I am designing and evaluating solutions and best practices around database MAA focused on Exadata. This involves HA, backup/recovery, migration and database consolidation and upgrades on Exadata. Opinions are my own and not necessarily those of Oracle Corporation. See


« April 2014