Sunday Sep 30, 2012

Exadata X3, and Oracle Platinum Services

Oracle recently announced an Exadata Hardware Update. The overall architecture will remain the same, however some interesting hardware refreshes are done especially for the storage server. Each cell will now have 1600GB of flash, this means an X3-2 full rack will have 20.3 TB of total flash ! For all the details I would like to refer to the Oracle Exadata product page:

Together with the announcement of the X3 generation. A new Exadata release, is made available. New Exadata systems will be shipped with this release and existing installations can be updated to that release. As always there is a storage cell patch and a patch for the compute node, which again needs to be applied using YUM.

Instructions and requirements for patching existing Exadata compute nodes to using YUM can be found in the patch README. Depending on the release you have installed on your compute nodes the README will direct you to a particular section in MOS note 1473002.1.

MOS 1473002.1 should only be followed with the instructions from the patch README. Like with and instructions are added to prepare your systems to use YUM for the first time in case you are still on release and earlier. You will also find these One Time Setup instructions in MOS note 1473002.1

By default all compute nodes that will be updated to will now have the UEK kernel. Before the 'compatible kernel' was used for all compute nodes other than X2-8. For customer will have the choice to replace the UEK kernel with the Exadata standard 'compatible kernel' which is also in the ULN channel. Recommended is to use the kernel that is installed by default.

One of the other great new things brings is Writeback Flashcache (wbfc). By default wbfc is disabled after the upgrade to Enable wbfc after patching on the storage servers of your test environment and see the improvements this brings for your applications.

Writeback FlashCache can be enabled  by dropping the existing FlashCache, stopping the cellsrv process and changing the FlashCacheMode attribute of the cell. Of course stopping cellsrv can only be done in a controlled manner.


  1. drop flashcache
  2. alter cell shutdown services cellsrv
    • again, cellsrv can only be stopped in a controlled manner
  3. alter cell flashCacheMode = WriteBack
  4. alter cell startup services cellsrv
  5. create flashcache all

Going back to WriteThrough FlashCache is also possible, but only after flushing the FlashCache:

  • alter cell flashcache all flush

Last item I like to highlight in particular is already from a while ago, but a great thing to emphasis: Oracle Platinum Services. On top of the remote fault monitoring with faster response times Oracle has included update and patch deployment services.These services are delivered by Oracle Advanced Customer Support at no additional costs for qualified Oracle Premier Support customers.


René Kundersma

Thursday Jun 28, 2012

Using ISO Image with a Local Repository for updating Exadata Compute nodes

For systems that cannot connect directly to Oracle ULN to build a local repository an ISO image file is made available by Oracle. This ISO image can be mounted and used as a local repository. The ISO image contains a file system that contains only the latest (x86_64) ULN channel and cannot be used to update the database servers to any other release than release ISO and instructions can be found here

 Rene Kundersma

Thursday Jun 21, 2012

New channels for Exadata

With the release of Exadata back in April 2012 Oracle has deprecated the minimal pack for the Exadata Database Servers (compute nodes). From that release the Linux Database Server updates will be done using ULN and YUM. For the release the ULN exadata_dbserver_11. channel was made available and Exadata operators could subscribe their system to it via

With the new release two additional channels are added:
  1. a 'latest' channel (exadata_dbserver_11.2_x86_64_latest)
  2. a 'patch' channel (exadata_dbserver_11.

The patch channel has the new or updated packages updated in from the base channel. The latest channel has all the packages from base and patch channels combined. 

From here there are three possible situations a Database Server can be in before it can be updated to
  1. Database Server is on Exadata release <
  2. Database Server is patched to
  3. Database Server is freshly imaged to
In order to bring a Database Server to for all three cases the same approach for updating can be used (using YUM), but there are some minor differences:

For Database Servers on a release < the following high-level steps need to be performed:
  • Subscribe to el5_x86_64_addons, ol5_x86_64_latest and  exadata_dbserver_11.2_x86_64_latest
  • Create local repository
  • Point Database Server to the local repository*
  • install the update
* during this process a one-time action needs to be done (details in the README)

For Database Servers patched to
  • Subscribe to patch channel  exadata_dbserver_11.
  • Create local repository
  • Point Database Server to the local repository
  • Update the system
For Database Servers freshly imaged to
  • Subscribe to patch channel  exadata_dbserver_11.
  • Create local  repository
  • Point Database Server to the local repository
  • Update the system
The difference between 'situation 2' (Database Server is patched to and 'situation 3' (Database Server is freshly imaged to is that in situation 2 the existing Exadata-computenode.repo file needs to be edited while in situation 3 this file is not existing  and needs to be created or copied. Another difference is that you will end up with more OFA packages installed in situation 2. This is because none are removed during the updating process. 

The YUM update functionality with the new channels is a great enhancements to the Database Server update procedure. As usual, the updates can be done in a rolling fashion so no database service downtime is required. 

For detailed and up-to-date instructions always see the patch README's
Rene Kundersma

Monday Apr 09, 2012

Updating Exadata Compute Nodes using ULN and YUM starting

With this post an update on Exadata Planned Maintenance - The new ULN Updating Procedure for Exadata Compute Nodes.
As you may already know, starting with Oracle Exadata Storage Server release the 'minimal pack' is deprecated.
From now on the (Linux) Computes Node will use Yellowdog Updater (Modified) (YUM) to apply the new updates as rpm packages. 
These RPM packages that come from ULN may also contain firmware updates which will be applied in the same updating procedure.

In order to update the Exadata Compute Nodes via YUM, you need a repository to download the software from. Of course Oracle provides its customers ULN access for this to make this really easy. It may however happen this requirement for http access to a web location from the Compute Nodes is not possible for some reason. For these situations Oracle worked out an alternative solution.

In this post I like to discuss the solution to setup a YUM repository located in the customers Data Center: This is a local system, that can download the updates from ULN and act as a YUM repository for the Compute Nodes as a 'man in the middle' bertween ULN and the Compute Nodes.

For installations planning to setup an internal/local YUM repository and currently not on there are two notes that need to be reviewed carefully before the patching starts:

README for patch 13741363 : Especially chapter 3 "Performing One-Time Setup on Database Servers of Oracle Exadata Database Machine"

The steps described here are one time only steps to prepare and populate the local YUM repository server that will be used for ALL the Compute Nodes. Best is to install the YUM repository server on a separate Linux machine running OEL 4 or OEL 5. Basically the steps are: go to, use your Hardware CSI for the registration steps, register the YUM repository server and subscribe to the right channels and populate the repository. After the download is finished, the repository is 'build' and now 'local' it needs to be made available by http for the Compute Nodes to download from on the local network.

After the setup of the repository V2/X2-2 and X2-8 Compute Nodes require a One-Time setup so they are able to use YUM from now on. The One-Time steps remove a set of packages that can cause the further setup to fail, but also add some packages to support the installations to be done using YUM onwards.

Please pay close attention to one of the most important steps of this One-Time setup: the editing of the repository configuration file /etc/yum.repos.d/Exadata-computenode.repo and making sure it points to your local YUM repository if you don't have direct ULN access.

README for 13536739: Especially chapter 6 "Updating Oracle Linux Database Servers in Oracle Exadata Database Machine" 

After setting up the repository and enabling the Compute Nodes to use YUM there is one thing to do: the update itself. Key step in this process is enabling each Compute Node to use the new repository. After this some  packages (ofed1) may need to be downgraded depending on your installation and some checks for kernel versions need to be done. After this, from now on the the system can be updated using a simple 'yum install' to install the main Exadata database server rpm with all dependent packages.
At the end of the installation the Compute Node is rebooted automatically.

At this moment I have to make some remarks/disclaimers:
  • Please see the notes for 13741363  and 13536739 for exact steps, I am only highlighting to explain the overall procedure of setting up a local repository and configuring the Compute Nodes using it.
  • If you are able to connect your Compute Nodes to ULN directly there is no need to setup a repository and the related steps can be skipped.
  • For X2-2 (and earlier) and X2-8 the 'updating Oracle Linux Database Server' steps are slightly different.
  • Oracle has provided 'helper' scripts to automated the steps described above making it even more easy
  • The YUM Updating procedure only applies to Linux Compute Nodes having images > (for updates to versions lower than you still need to use the minimal pack)
Rene Kundersma

Wednesday Jan 18, 2012

First quarterly full stack download patch for Exadata available

Today Oracle released the first quarterly full stack download patch for Exadata a.k.a QFSDP

This patch is available for both Linux and Solaris and contains three primary components:
  1. Infrastructure (storage server, infiniband switch and PDU)
  2. Database (Database, Clusterware but also Opatch and Oplan)
  3. System management (agent, OMS and EM plugins)

The patch is available on MOS for Linux and Solaris and can be used for those customers who want to download the recommended releases for all components for the Oracle Exadata Database Machine in one go.

In this QFSDP you will find the Quarterly Database Patch for Exadata a.k.a QDPE. The current version is Jan 2012.

This is part of a new strategy for patching Exadata that reduces the number and frequency of patches released to customers even more.

For the database: while also continuing to make important fixes available in monthly or bimonthly interim bundle patches, Oracle recommends most customers only applying the QDPE. The currenly recommened QDPE is available via MOS patch 13513783.

MOS note 888828.1 is still the recommend place to look first for the latest information regarding recommend/supported releases.

Rene Kundersma

Monday Jan 02, 2012 available for the Oracle Exadata Database Machine

This post is about upgrades. The first thing I like to mention is the availability of for the Oracle Exadata Database Machine since today (Jan 3rd 2012). With this, it is now possible to upgrade your Oracle Exadata Database Machine to To help you best with the upgrade we have released a MOS note that describes the prerequisites and steps to go from or installations to Please see the MOS note for the best approach on how to apply this upgrade in your environment. The MOS note applies to both Solaris 11 Express and Linux installations and upgrades both the Grid Infrastructure and the Database Home on V2, X2-2 and X2-8.

Please see MOS note " to Database Upgrade on Exadata Database Machine" (Doc ID 1373255.1) for more details.

For V1 users we have recently also published MOS note 888834.1. This document contains the steps to upgrade the HP Oracle Database Machine to Oracle Exadata Storage Server Software 11.2.X.  The steps include upgrading Oracle Cluster Ready Services (CRS) and Oracle Automatic Storage Management (ASM) to Oracle Grid Infrastucture and Oracle RDBMS from to Oracle RDBMS After upgrading to 11.2.0 on V1 hardware using MOS note 888834.1,  MOS note 1373255.1 can be used to upgrade to

Please See MOS note "HP Oracle Database Machine 11.2 Upgrade" (Doc ID 888834.1)

René Kundersma

Saturday Oct 29, 2011

Oracle Enterprise Manager Cloud Control 12c Setup Automation kit for Exadata

With this post I like to mention the availability of the Enterprise Manager 12c setup automation kit for Exadata. I also like to explain how easy it is to deploy the agent for a complete Exadata rack of any size with all it's components. After the discovery the system can managed by EM, the deployment I did took only 30 minutes for a quarter rack with all components.

You can obtain the "Oracle Enterprise Manager Cloud Control 12c Setup Automation kit for Exadata" from MOS by searching for patch 12960596

The deployment of the agent starts with the Exadata configurator sheet. This is the sheet ACS already used for the Exadata deployment. The sheet now also creates a configuration output file for automatic agent and OMS deployment. The file I am talking about is called em.params. This file can be used as input for the OMS or EM 12c deployment scripts. In this posting I will only discuss the agent deployment.

This em.params file will have the following information:

# This is em.param file
# Written : 26-10-2011


EM_CELLS=(cel12 cel13 cel14)

EM_COMPUTE_ILOM_NAME=(db07-c db08-c)
EM_COMPUTE_ILOM_IP=(a.b.c.d a.b.c.e)

machinemodel="X2-2 Quarter rack"



Of course, the names and ipnumbers are changed in this example to hide any specific information.
When you install the kit the configuration file is expected in /opt/oracle.SupportTools/onecommand, this is where your OneCommand deployment files will be anyway.

During installation some additional checks will be done like:
Trying to ping Oracle Management Server Host .
Checking oracle user info
Searching for KVM Switch information by key swikvmname from /opt/oracle.SupportTools/onecommand/em.param
Searching for KVM Switch IP information by key swikvmip from /opt/oracle.SupportTools/onecommand/em.param
Searching for Cisco Switch information by key swiipname from /opt/oracle.SupportTools/onecommand/em.param
Searching for Cisco Switch IP information by key swiipip from /opt/oracle.SupportTools/onecommand/em.param
Searching for IB2 Switch information by key swiib2name from /opt/oracle.SupportTools/onecommand/em.param
Searching for IB2 Switch IP information by key swiib2ip from /opt/oracle.SupportTools/onecommand/em.param
Searching for IB3 Switch information by key swiib3name from /opt/oracle.SupportTools/onecommand/em.param
Searching for IB3 Switch IP information by key swiib3ip from /opt/oracle.SupportTools/onecommand/em.param
Searching for PDUA Name information by key pduaname from /opt/oracle.SupportTools/onecommand/em.param
Searching for PDUA IP information by key pduaip from /opt/oracle.SupportTools/onecommand/em.param
Searching for PDUA Name information by key pdubname from /opt/oracle.SupportTools/onecommand/em.param
Searching for PDUB IP information by key pdubip from /opt/oracle.SupportTools/onecommand/em.param
Searching for ILON Names information by key EM_COMPUTE_ILOM_NAME from /opt/oracle.SupportTools/onecommand/em.param
Searching for ILOM IPS information by key EM_COMPUTE_ILOM_IP from /opt/oracle.SupportTools/onecommand/em.param
Verifying ssh setup..
Verifying SSH setup for root

When the validations are done, the file will be transferred to the compute nodes and the 12c agent will be installed.
Like the 11.1 installation, still no agents will be installed on the storage cells, EM will use the compute nodes as proxy to the cells, it's up to you if you want EM to choose which nodes or choose yourself.

After deployment, there are three easy steps left in the EM GUI:
  • Exadata discovery
  • GI(Cluster) discovery
  • RAC discovery

The discovery steps above are guided by a clear GUI.
All the discovery process needs is access to a compute node, from there all the Exadata components will be discovered automatically.
The file EM needs to be available for that is  /opt/oracle.SupportTools/onecommand/databasemachine.xml

After installation of the kit and discovery in EM 12c, you have a nice overview of the rack with all the components in it, easy to drill down and manage from there. The example below is a quarter rack.

René Kundersma

Oracle MAA Development

Thursday Sep 22, 2011

Update on some interesting new or recently updated Exadata related MOS notes and the Oracle Database Appliance

A quick blog update on some interesting new or recently updated Exadata related MOS notes:
  • Dedicated and Global Hot Spares for Exadata Compute Nodes in (Doc ID 1339647.1)
  • How to Expand Exadata Compute Node File Systems (Doc ID 1357457.1)
  • Tool for Gathering I/O Resource Manager Metrics: (Doc ID 1337265.1)
  • Scripts and Tips for Monitoring CPU Resource Manager (Doc ID 1338988.1)
  • Configuring Resource Manager for Mixed Workloads in a Database (Doc ID 1358709.1)
Also wanted to mention the announcement of the Oracle Database Appliance, checkout this paper

From now on smaller post will primarily be done using my twitter account: @rene_kundersma. The larger items will still be posted to this blog.

Rene Kundersma

Friday Sep 09, 2011

Software Updates, Best Practices and Notes

With a very busy time behind and ahead of me, I still like to make mention of a couple of documents recently we published.

OPlan is now available for Linux as well as Solaris X86-64 for the Oracle Exadata Database Machine. The current version of OPlan is For more details, see my previous post

The two notes mentioned below explain about how OPlan can be used and demo how OPLan works to clone the Grid Infrastructure on the Database Machine, patch the cloned home and switch to it. This is called out-of-place patching.

  • MOS Note 1306814.1 - Oracle Software Patching with OPLAN
  • MOS Note 1136544.1 - Minimal downtime patching via cloning 11gR2 ORACLE_HOME directories

Supported software versions for Exadata are still listed in MOS note 888828.1, this is where you will find a reference to the latest Exadata Storage Server software release:, available via patch 12849110. Please see note 1334254.1 for the details.

For the Database Server, the latest patch made available for is Bundle Patch 10, available via Patch 12773458 (Linux version). Please know that BP's can be applied by EM, which makes patching more easy.

For ASR we now have release version 3.3 Released. Details can be found in Note 1185493.1.
The latest PDU metering unit firmware is 1.04 and available as patch 12871297.

For MAA Best pracitces releated to the database machine we released a real good document called "Oracle Exadata Database Machine Consolidation: Segregating Databases and Roles", which you can find here

Also releated to best practices is the document "MAA Best Practices for Oracle Exadata Database Machine", you can find that here

Rene Kundersma

Wednesday Sep 07, 2011

OOW Sessions Related to Database High Availability

A quick one on OOW 2011: this link lists all database HA related sessions !

Monday Jul 04, 2011

Failover capability for plugins Exadata & EMGC Rapid deployment

With this entry two items around Exadata management:

First: a pointer to My Oracle Support note 1110675.1. This is note is called "Oracle Database Machine Monitoring Best Practices" and has a lot of information on Exadata management. In this note you will read about the plugins we have to monitor the various Exadata components and how to configure them, also you will read about creating a 'Dashboard'.

The topic I like to cover here is about the steps to take to setup automated fail-over of targets. This is, because by default a target monitored using a plug-in in Enterprise Manager Grid Control is bound to the agent used during the initial installation.
So, what you basically want to do is make sure another agent (on another node) is available to take over the monitored targets when the first agents fails; providing HA for your monitoring.

For this I like to mention two items:

  1. "Package to add failover capabiity for Plug-ins"   (bottom of note 1110675.1.)
  2. OEM_Exadata_Dashboard_Deployment_v104.pdf

In the file 'OEM_Exadata_Dashboard_Deployment' you read about the following steps:
  • adding the PL/SQL package into the EM repository (package mentioned above)
  • registering the package
  • setting up of agents the target can use for fail-over
  • create failover method and notification rule
  • create fail-back method and notification rule

Once you have executed the steps above, the second agent should be able to take over the targets initially assigned to the first agent when the first fails.

You can test this by stopping the first agent; in Grid Control you will find the targets under the second agent. Making sure the monitored targets can failover to a surviving agent is recommended, but it going that route, it would make sense to also make sure Grid Control (OMS) and database are both also HA.

Second topic for this entry is the "EM 11GC Setup automatic kit" for Exadata.

With the latest Oracle Exadata Database Machine Configurator a new output file (em.param) is generated. This file can be used as input for installing Oracle Enterprise Manager Grid Control Agents on Oracle Exadata Database Machine

With this kit, Exadata customers, having an existing Oracle Enterprise Manager Grid Control environment can have setup agents rapidly by the Oracle or partner teams doing the Exadata deployment. For the sake of completeness: Oracle Enterprise Manager Grid Control is not a requirement for Exadata but a recommendation.


Rene Kundersma

Thursday Jun 09, 2011

Applying bundle patches on Exadata using Enterprise Manager Grid Control

With this post a small first note on applying Exadata Bundle Patches using Enterprise Manager Grid Control 11g.

Did you know:
  • Exadata BP's can be applied in a rolling fashion using Grid Control comparable to 'opatch auto'
  • The patching procedure always updates Opatch to the latest version automatically
  • To some extend conflicts are checked to prevent you from problems using patching
  • Before applying a patch you could run 'analyze' only and verify if the BP patch can be applied.
  • When a SQL script has to run as part of the BP, that's also taken care off

Interested ? Here some resources to help you forward:

More important, check first for required patches and what BP's are tested:
  • Grid Control OMS and Agent Patches required for setting up Provisioning, Patching and Cloning (Doc ID 427577.1)
  • Patch Oracle Exadata Database Machine via Oracle Enterprise Manager 11gR1 ( (Doc ID 1265998.1)
  • Patch Requirements for Setting up Monitoring and Administration for Exadata (Doc ID 1323298.1)

Still, there's only one single source of truth when it comes to which patches are recommended for Exadata:

  • Database Machine and Exadata Storage Server 11g Release 2 (11.2) Supported Versions (ID 888828.1)

A small demo of a comparable 'RAC Rolling patch' can be found on OTN:

Rene Kundersma

Tuesday May 24, 2011

New papers: DR for Exalogic and Exadata + Oracle GoldenGate on Exadata

Recently an updated version of to the MAA whitepaper "Oracle GoldenGate on Oracle Exadata Database Machine Configuration" has been made available. With this update new information is added regarding:

  • DBFS performance configuration
  • Details of DBFS configuration for node failover
  • GoldenGate Replicat commit behavior configuration
  • The agent failover script for clean start-up of GoldenGate

Also a new document is published called "Disaster Recovery for Oracle Exalogic Elastic Cloud with Oracle Exadata Database Machine". This document discusses :

  • Oracle Fusion Middleware Disaster Recovery architecture and strategy for deployments on Oracle Exalogic with Oracle Exadata Database Machine
  • Detailed deployment and configuration steps for the Oracle Fusion Middleware Disaster Recovery solution on Oracle Exalogic and Oracle Exadata Database Machine.
  • Best practices for the Oracle Fusion Middleware Disaster Recovery solution with Exalogic and Exadata
Rene Kundersma

Thursday May 19, 2011

exachk: healtcheck for Exadata

With this entry I like to handle exachk. Exachk is made available via MOS Doc Id. 1070954.1. The tool is designed to audit various important configuration settings
within an Oracle Database Machine Exadata System - Database Servers, Storage Servers and Infiniband Switches. It verifies key components of the Exadata Database Machine against supported version levels, recommended Oracle RAC settings and Exadata best practices. At the moment we support version 11gR2 for V2 & X2-2.

The tool is non-intrusive, when it completes it's collection and analysis it produces two reports, summary and detailed. If an SR needs to be logged the output can be supplied to Oracle for further analysis . The detailed report will contain Benefit/Impact, Risk and Action/Repair information. In many cases it will also reference publicly available documents with additional information about best practices and how to implement them.

For more and detailed information please see MOS Doc Id. 1070954.1

Rene Kundersma

Sunday May 08, 2011

Exadata Storage Server software released

Since some days we have added Patch 12400152 (Exadata Storage Server software to the list of Database Machine and Exadata Storage Server supported versions. The main enhancement in is support for Solaris 11 Express  for x64 on the compute nodes on both X2-2 and X2-8 systems in addition to Linux. New Exadata Database Machines will ship with both Linux and Solaris installed on the compute nodes by default, and the customer can decide which OS to use at install time.

See note 888828.1 for supported patches

See the patch 12400152 README for more details on the enhancements.

 Rene Kundersma


Blog of Rene Kundersma, Consulting Member of Technical Staff at Oracle Development USA. I am designing and evaluating solutions and best practices around database MAA focused on Exadata. This involves HA, backup/recovery, migration and database consolidation and upgrades on Exadata. Opinions are my own and not necessarily those of Oracle Corporation. See


« June 2016