By Rene Kundersma on Apr 22, 2014
Read this new white paper to learn how to deploy and optimize Database-as-a-Service using Oracle Multitenant and Oracle MAA.
With this blog update some background on new functionality added to the Exadata dbnodeupdate.sh utility.
The recently added 'Prereq Dependency Check' feature in dbnodeupdate.sh eliminates possible unexpected package dependency problems that occur during update by detecting dependency issues during the prerequisite check phase that could occur when the update is performed. (Linux package dependency issues sometimes happen after operators customizing the db server by installing additional software packages or newer releases of packages). The recommended use of this feature is to run dbnodeupdate.sh with the –v flag (to verify prereqs only) days before the scheduled maintenance. dbnodeupdate.sh will report dependency problems that must be resolved before proceeding with the update.
When running dbnodeupdate.sh for updating a db server the "-l" argument is always required to specify the source location of the new release/update. From here we will now call this 'location' the repository'. The repository comes in two different flavors: as an ISO and as Oracle ULN channel. Let's start with the ULN channel.
Exadata ULN Channels
For the different Exadata releases starting 188.8.131.52.0 a corresponding 'base channel' is available on linux.oracle.com (ULN). Exadata customers can register to ULN with their CSI, subscribe to the channel (e.g. patch/release) they require and then synchronize one or more channels to with the local (in house) repository. This local repository can then be made available to the internal data-center by publishing it via a web server.
Note: it is not recommended to use an Exadata db server as local YUM repository
Instructions how to subscribe a system to ULN and synchronize to a local repository are the same as for Oracle Linux and Oracle VM, so generic instructions can be found on OTN here. The README of a specific Exadata release will always mention what channels are made available for that release. You can also find this information via MOS 888828.1 and also in 1473002.1.
Additional to the 'base channel', there is also a 'latest channel'. Currently for Exadata there is a latest channel for 11.2 and 12.1 releases. The content of the 'latest' channel will never remain the same (unlike the 'base' channel) as long as there will be updates for that 11.2 or 12.1 release. When for example a new Exadata 12 release will be published this will be added to the existing latest channel (in addition to a 'base channel' also made available). This is the primary reason for the 'latest' channel being much larger (and taking more time to synchonize) than a base channel.
For Exadata installations on release later than 184.108.40.206.1, the 'latest' channel brings additional options. With the latest channel it's now possible to specify what release you want to update to. For example when on 220.127.116.11.0 planning to update to a later (but not the latest 12.1.release, just as an example) you can use the 'latest' channel and specify the "-t" flag to specify what release you want to update to.
Note that this can only be done with the 'latest' channel and that without specifying the "-t" argument by default the db server will be updated to the most recent release it can find in that channel. Of course there is also the option to just grab the 'base' channel and update without specifying any "-t' option.
Exadata channel as ISO
For those not able or willing to synchronize repositories with Oracle ULN there is also an ISO image available. The ISO image is built (and zipped) by Oracle and is only available for 'base' channel content. An ISO is ~1GB and about the same size as the sum of all packages of the corresponding base channel on ULN.
Using ISO or ULN
For local YUM repositories (synchronized from Oracle ULN):
For using an ISO (example with the 18.104.22.168.0 iso):
./dbnodeupdate.sh -u -l ./p17809253_112330_Linux-x86-64.zip
The ISO file should not be unzipped and there is no need to make an local 'loop mount' to use the iso - this is all done by the dbnodeupdate.sh script
For each type of repository some validation checks will be done to see if it a usable is a repository, checks are done for expected files and also if the available Exadata release in the repository is a later release than the one currently installed - because if not, an update would not be possible.
When specifying an http repository it's required to specify the top level directory containing the 'YUM metadata repository directory'. Typically this is the directory that has the 'repodir' directory in it. (see example here). When an http location cannot be identified as a valid repository an you would see a suggestion how to locate the right url.
In this post I will explain background and usage for both backups and how they integrate with dbnodeupdate.sh
For backing-up and rolling-back Exadata dbserver OS updates the dbserver_backup.sh script is used by dbnodeupdate.sh. For each upgrade by default the dbserver_backup.sh script is executed. When executed (either manually or via dbnodeupdate), the dbserver_backup.sh script creates a small snapshot of the 'active' sys lvm. The active sys lvm is the primary lvm that your current OS image is running on. For example:
[root@mynode ~]# imageinfo
Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
Image version: 22.214.171.124.0.131014.1
Image activated: 2014-01-13 13:20:52 -0700
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys2
In the above example the active lvm is /dev/mapper/VGExaDb-LVDbSys2.The snapshot is created to have a 'consistent' 'view' of the root filesystem while the backup is made. After the snapshot is created, it's mounted by the same script and then it's contents are copied over to the inactive lvm. For lvm enabled systems, there are always 2 'sys' lvm's "VGExaDb-LVDbSys1" and "VGExaDb-LVDbSys2". VGExaDb-LVDbSys2 will automatically be created (on lvm enabled system) if not existing yet. For the example above, the 'inactive' lvm will be VGExaDb-LVDbSys1
Now, depending on how many files there are in the root (/) filesystem (based on your active sys lvm) the backup times may vary. Previous Grid and Database home installation zip files in /opt/oracle.SupportTools/onecommand will make the backup take longer (not the restore, which I will explain why later). Same for those who have many small files (like mail messages in /var/spool) - the backup may take longer.
One of the first steps the dbnodeupdate.sh script will doing when executed is making a backup with this script. Now, if you want to shorten your downtime and make this backup before the start of your 'planned maintenance window' you have 2 options: Either execute the dbserver_backup.sh script yourself or use dbnodeupdate.sh with the "-b" flag to make a backup only before hand.
Example making a backup with dbnodeupdate.sh here (see 'Backup only' for 'Action')
When you then have the downtime for planned maintenance and already have the backup you can then let dbnodeupdate skip the backup using the "-n" flag.
Example skipping a backup with dbnodeupdate.sh here (See 'Create a backup: No')
Both Sys lvm's are 30GB each. The snapshot that will be created is ~1GB. It is recommended to keep this in mind when claiming the free space in the volume group to make your /u01 filesystem as big as possible. (the script checks for 2 GB free space in the volume group)
Now, when the update proceeds, the current active lvm will remain the active lvm. This is different than what happens on the cells where the active lvm becomes inactive with an update. Typically you will only switch active sys lvm's when a rollback needs to be done on a db server, for example, an upgrade from 126.96.36.199.0 to 188.8.131.52.0 needs to be rolled-back. What happens then is nothing more than 'switching' the filesystem label of the sys lvm's, updating grub (the bootloader) and restoring the /boot directory (backed up earlier also by dbnodeupdate.sh). Then, a next boot will now have the previous inactive lvm as active lvm.
Rolling back with dbnodepdate.sh as in the example here (a rollback from 184.108.40.206.1 to 220.127.116.11.2)
After booting the node, it's recommended to run dbnodeupdate.sh again with the "-c" flag to relink the oracle home's again.
In this and future posts I am planning to describe some new
functionality and background of dbnodeupdate.sh starting with Oracle Exadata Database Machine release
18.104.22.168.0. Some of this functionality will be directly available to the operator
via the interface and can actually be used via an argument, however, some of the recent changes are made to make patching even easier, reduce human error and downtime.
You may also find some of the 'new features' described in MOS 1553103.1 'Exadata Database Server Patching using the DB Node Update Utility'
With updates to Exadata 22.214.171.124.0 or later some packages on the database server will become obsolete. When updating a db server the dbnodeupdate.sh script will mention an 'RPM exclusion list' and an 'RPM obsolete list' in it's confirmation screen. The 'RPM obsolete list' will list the all packages that will be removed by default during the update to 126.96.36.199.0 (or later) when no action is taken by the operator.
As an example - click here
If you would like to find out first what obsolete packages will be
removed you have to choose 'n' when prompted to 'Continue ? [Y/n]'. This
will stop your current patching session. Then look at the contents of the freshly created 'obsolete list' (it
should have the runid of your dbnodeupdate session in it's header). Example here
All the packages listed in the '/etc/exadata/yum/obsolete.lst' file will be removed by default - this has multiple reasons, mainly this is because these packages are not required anymore for Exadata functioning or they are considered a security risk. In case you would like to keep for example the 'java' package, you should create an 'exclusion file' which is '/etc/exadata/yum/exclusion.lst' and put the 'java*openjdk' rpm name (or wildcard) in it.
[root@mynode u01]# cat /etc/exadata/yum/exclusion.lst
When dbnodeupdate.sh is restarted you would see that the 'exclusion file' is detected (an example here).
All packages you have put in the 'exclusion file' will still be listed in
the obsolete file, but will not be removed when the confirmation screen says 'RPM exclusion list: In use (rpms listed in /etc/exadata/yum/exclusion.lst)' with the
update to 188.8.131.52.0 or later.
Frequent releases of dbnodeupdate.sh - keep an eye on it:
dbnodeupdates.sh and it's MOS note were designed / made to be released frequently and quickly when needed.This way dbnodeupdate.sh can provide workarounds for known issues, emergency fixes, new features and best practices to the operator relatively quick.This reduces risk of people not keeping up to date with 'known issues' or not being sure it applies to them. Basically the same idea as with the patchmgr plugins.
Also, unlike to the storage servers some customization can be done on the db servers - best practices and lessons learned from this in regards to patching may also benefit your environment. For those engineers involved with upgrading Exadata database nodes, I'd like to emphasis to always check for the most recent release of dbnodeupdate.sh when doing a db server update. I have already seen some people watching the releases closely, which is definitely a good thing.
The recently released 184.108.40.206.0 Oracle Exadata version comes with multiple enhancements in the patchmgr utility. For those who don't know what the patchmgr utility is: the patchmgr utility is a tool Exadata Database Administrators use to apply (or rollback) an update to the Oracle Exadata Storage Cells. The enhanced patchmgr will have an option to send the operator an email for the most significant patch and rollback state changes. This eliminates the need to monitor the screen while the update or rollback is in progress.
In order to send the email you need to specify values for the '-smtp_from' and '-smtp_to' flags. Example as follows:
./patchmgr -cells ~/cell_group -patch -rolling \ -smtp_from firstname.lastname@example.org \ -smtp_to email@example.com
Patchmgr will use the sendmail package which is installed by default on the Exadata Database Server. It will start sendmail if it's not already started and assumes it's configured to deliver email for the domains you specify. You will recognize the format of the alerts patchmgr sends as they have the same formatting as the ASR emails.
The email you will receive when you enable this option can have the following end states for a cell:
The majority of the above states probably is self explanatory, however explaining 'Not Attempted' may help: 'Not Attempted' will be the final state of a cell when patching or rolling back (in a rolling fashion) of the current cell has failed. In that case the patching will stop and the remaining cells (which are not touched yet) are in the state 'Not Attempted'. So for example: imagine you are patching cel1,cel2 and cel3. When cel1 fails patching, then the end state for cel2 and cel3 will be 'Not Attempted'.
The following example will give you the idea: Patching starts in a rolling fashion from 220.127.116.11.0 to 18.104.22.168.0 for cel1,cel2 and cel3.
Note: this is an example of the type of state changes you will see, actual patch timings are not correct and not relevant for this example. Also actual states or formatting may change in your release.
1. patchmgr was started in a rolling fashion. Initial state for all cells is 'Started'. This should be seen as this initialization phase. You will receive an email as follows:
3. After some time patching of cel1 has completed. You will receive another status update in your inbox stating cel1 was updated successfully.
4. patchmgr continues with the next cell.
5. When succeeded (or failed) you will receive a state update via mail.
6. Last cell to be patched is cel3:
7. The last email you will receive will give an update stating patching was completed and lists the end state for all cells.
Join this online forum to hear from analysts and experts on how companies are beginning to transform with DBaaS, and learn the prescriptive steps your organization can take to design, deploy, and deliver DBaaS today.
Monday, October 21, 2013
9 a.m.–12 p.m PT / 12 p.m.–3 p.m. ET
Today on My Oracle Support two notes are made available for upgrading your 11.2 Grid Infrastructure and Databases running on the Oracle Exadata Database Machine to 22.214.171.124.
Together with two colleagues from the MAA team I will be doing a presentation on consolidation for Oracle Exadata:
Title: Consolidation and Virtualization on Oracle Exadata: How, What, Where, and When (CON8530)
Abstract: How is database consolidation done on Oracle Exadata? This technical session looks at the past, present, and future consolidation solutions on Oracle Exadata. It describes all the consolidation options on Oracle Exadata, when to choose which, and the best practices for deploying an Oracle Database instance with each one
When: Monday, Sep 23, 3:15 PM - 4:15 PM - Westin San Francisco - Metropolitan I
Today the 'DB Node Update Utility' (dbnodeupdate.sh utility) has been made available. The 'DB Node Update Utility' automates all the steps and checks to upgrade Oracle Exadata database servers to a new Exadata release and replaces the manual steps. The dbnodeupdate.sh utility includes the latest best practices and workarounds for known issues. Where appropriate, the script uses existing 'One-Time Helper scripts' or 'yum update commands'. The script uses the 'build-in' 'dbserver_backup.sh' script to perform backups before patching where possible.
Four typical use cases for the dbnodeupdate.sh utility are :
For more details, demo's and downloads see My Oracle Support Note 1553103.1
Recently Oracle published the below video created by our group (MAA team) which demonstrates Exadata's High Availability in the face of hardware and software failures. This video is a great showcase on how Exadata handles hardware and software failures. These are tests Oracle does on a daily basis for Engineered systems so every Exadata customer benefits !
The test has been developed by Oracle Universities best instructors and I would really recommend taking the test if this is your field of expertise. It's a good opportunity to prove your skills. And for those looking for skilled Exadata specialists, I would recommend asking for people who passed this OU test additional to traditional 11g OCP and some basic Linux & Networking skills.
The above link also has references to recommend Exam preparation
Oracle recently announced an Exadata Hardware Update. The overall architecture will remain the same, however some interesting
hardware refreshes are done especially for the storage server. Each cell will now have 1600GB of flash, this means an X3-2
full rack will have 20.3 TB of total flash ! For all the details I would
like to refer to the Oracle Exadata product page: www.oracle.com/exadata
Together with the announcement of the X3 generation. A new Exadata release, 126.96.36.199 is made available. New Exadata systems will be shipped with this release and existing installations can be updated to that release. As always there is a storage cell patch and a patch for the compute node, which again needs to be applied using YUM.
Instructions and requirements for patching existing Exadata compute nodes to 188.8.131.52 using YUM can be found in the patch README. Depending on the release you have installed on your compute nodes the README will direct you to a particular section in MOS note 1473002.1.
MOS 1473002.1 should only be followed with the instructions from the 184.108.40.206 patch README. Like with 220.127.116.11.0 and 18.104.22.168.1 instructions are added to prepare your systems to use YUM for the first time in case you are still on release 22.214.171.124.2 and earlier. You will also find these One Time Setup instructions in MOS note 1473002.1
By default all compute nodes that will be updated to 126.96.36.199.0 will now have the UEK kernel. Before 188.8.131.52.0 the 'compatible kernel' was used for all compute nodes other than X2-8. For 184.108.40.206.0 customer will have the choice to replace the UEK kernel with the Exadata standard 'compatible kernel' which is also in the ULN 220.127.116.11 channel. Recommended is to use the kernel that is installed by default.
One of the other great new things 18.104.22.168 brings is Writeback Flashcache (wbfc). By default wbfc is disabled after the upgrade to 22.214.171.124. Enable wbfc after patching on the storage servers of your test environment and see the improvements this brings for your applications.
Writeback FlashCache can be enabled by dropping the existing FlashCache, stopping the cellsrv process and changing the FlashCacheMode
attribute of the cell. Of course stopping cellsrv can only be done in a
Going back to WriteThrough FlashCache is also possible, but only after flushing the FlashCache:
Last item I like to highlight in particular is already from a while ago, but a great thing to emphasis: Oracle Platinum Services. On top of the remote fault monitoring with faster response times Oracle has included update and patch deployment services.These services are delivered by Oracle Advanced Customer Support at no additional costs for qualified Oracle Premier Support customers.
For systems that cannot connect directly to Oracle ULN to build a local repository an ISO image file is made available by Oracle. This ISO image can be mounted and used as a local repository. The ISO image contains a file system that contains only the latest (x86_64) ULN channel and cannot be used to update the database servers to any other release than release 126.96.36.199.1. ISO and instructions can be found here
The patch channel has the new or updated packages updated in 188.8.131.52.1 from the base channel. The latest channel has all the packages from 184.108.40.206.0 base and patch channels combined.
The YUM update functionality with the new channels is a great enhancements to the Database Server update procedure. As usual, the updates can be done in a rolling fashion so no database service downtime is required.
Blog of Rene Kundersma, Principal Member of Technical Staff at Oracle Development USA. I am designing and evaluating solutions and best practices around database MAA focused on Exadata. This involves HA, backup/recovery, migration and database consolidation and upgrades on Exadata. Opinions are my own and not necessarily those of Oracle Corporation. See http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm.