Thursday Jul 05, 2012


(updated 3/18/13 to fix disknaming error) Oracle ASMlib on Linux has been a topic of discussion a number of times since it was released way back when in 2004. There is a lot of confusion around it and certainly a lot of misinformation out there for no good reason. Let me try to give a bit of history around Oracle ASMLib.

Oracle ASMLib was introduced at the time Oracle released Oracle Database 10g R1. 10gR1 introduced a very cool important new features called Oracle ASM (Automatic Storage Management). A very simplistic description would be that this is a very sophisticated volume manager for Oracle data. Give your devices directly to the ASM instance and we manage the storage for you, clustered, highly available, redundant, performance, etc, etc... We recommend using Oracle ASM for all database deployments, single instance or clustered (RAC).

The ASM instance manages the storage and every Oracle server process opens and operates on the storage devices like it would open and operate on regular datafiles or raw devices. So by default since 10gR1 up to today, we do not interact differently with ASM managed block devices than we did before with a datafile being mapped to a raw device. All of this is without ASMLib, so ignore that one for now. Standard Oracle on any platform that we support (Linux, Windows, Solaris, AIX, ...) does it the exact same way. You start an ASM instance, it handles storage management, all the database instances use and open that storage and read/write from/to it. There are no extra pieces of software needed, including on Linux. ASM is fully functional and selfcontained without any other components.

In order for the admin to provide a raw device to ASM or to the database, it has to have persistent device naming. If you booted up a server where a raw disk was named /dev/sdf and you give it to ASM (or even just creating a tablespace without asm on that device with datafile '/dev/sdf') and next time you boot up and that device is now /dev/sdg, you end up with an error. Just like you can't just change datafile names, you can't change device filenames without telling the database, or ASM. persistent device naming on Linux, especially back in those days ways to say it bluntly, a nightmare. In fact there were a number of issues (dating back to 2004) :

Correction to the above: ASM can handle device name changes across reboots with the correct ASM_DISKSTRING in the init.ora, it will be able to find the disks even if they changed, however part of device naming and device metadata on reboot is the correct ownership (oracle:dba ....). With ASMLib in place, this is not an issue and it will take care of ownership and permissions of the ASM disk devices.

  • Linux async IO wasn't pretty
  • persistent device naming including permissions (had to be owned by oracle and the dba group) was very, very difficult to manage
  • system resource usage in terms of open file descriptors
  • So given the above, we tried to find a way to make this easier on the admins, in many ways, similar to why we started working on OCFS a few years earlier -> how can we make life easier for the admins on Linux.

    A feature of Oracle ASM is the ability for third parties to write an extension using what's called ASMLib. It is possible for any third party OS or storage vendor to write a library using a specific Oracle defined interface that gets used by the ASM instance and by the database instance when available. This interface offered 2 components :

  • Define an IO interface - allow any IO to the devices to go through ASMLib
  • Define device discovery - implement an external way of discovering, labeling devices to provide to ASM and the Oracle database instance
  • This is similar to a library that a number of companies have implemented over many years called libODM (Oracle Disk Manager). ODM was specified many years before we introduced ASM and allowed third party vendors to implement their own IO routines so that the database would use this library if installed and make use of the library open/read/write/close,.. routines instead of the standard OS interfaces. PolyServe back in the day used this to optimize their storage solution, Veritas used (and I believe still uses) this for their filesystem. It basically allowed, in particular, filesystem vendors to write libraries that could optimize access to their storage or filesystem.. so ASMLib was not something new, it was basically based on the same model. You have libodm for just database access, you have libasm for asm/database access.

    Since this library interface existed, we decided to do a reference implementation on Linux. We wrote an ASMLib for Linux that could be used on any Linux platform and other vendors could see how this worked and potentially implement their own solution. As I mentioned earlier, ASMLib and ODMLib are libraries for third party extensions. ASMLib for Linux, since it was a reference implementation implemented both interfaces, the storage discovery part and the IO part. There are 2 components :

  • Oracle ASMLib - the userspace library with config tools (a shared object and some scripts)
  • oracleasm.ko - a kernel module that implements the asm device for /dev/oracleasm/*
  • The userspace library is a binary-only module since it links with and contains Oracle header files but is generic, we only have one asm library for the various Linux platforms. This library is opened by Oracle ASM and by Oracle database processes and this library interacts with the OS through the asm device (/dev/asm). It can install on Oracle Linux, on SuSE SLES, on Red Hat RHEL,.. The library itself doesn't actually care much about the OS version, the kernel module and device cares. The support tools are simple scripts that allow the admin to label devices and scan for disks and devices. This way you can say create an ASM disk label foo on, currently /dev/sdf... So if /dev/sdf disappears and next time is /dev/sdg, we just scan for the label foo and we discover it as /dev/sdg and life goes on without any worry. Also, when the database needs access to the device, we don't have to worry about file permissions or anything it will be taken care of. So it's a convenience thing.

    Correction: the extra advantage with ASMLib here being the fact that it will take care of the file permissions and ownership of the device.

    The kernel module oracleasm.ko is a Linux kernel module/device driver. It implements a device /dev/oracleasm/* and any and all IO goes through ASMLib -> /dev/oracleasm. This kernel module is obviously a very specific Oracle related device driver but it was released under the GPL v2 so anyone could easily build it for their Linux distribution kernels.

    Advantages for using ASMLib :

  • A good async IO interface for the database, the entire IO interface is based on an optimal ASYNC model for performance
  • A single file descriptor per Oracle process, not one per device or datafile per process reducing # of open filehandles overhead
  • Device scanning and labeling built-in so you do not have to worry about messing with udev or devlabel, permissions or the likes which can be very complex and error prone.
  • Just like with OCFS and OCFS2, each kernel version (major or minor) has to get a new version of the device drivers. We started out building the oracleasm kernel module rpms for many distributions, SLES (in fact in the early days still even for this thing called United Linux) and RHEL. The driver didn't make sense to get pushed into upstream Linux because it's unique and specific to the Oracle database.

    As it takes a huge effort in terms of build infrastructure and QA and release management to build kernel modules for every architecture, every linux distribution and every major and minor version we worked with the vendors to get them to add this tiny kernel module to their infrastructure. (60k source code file). The folks at SuSE understood this was good for them and their customers and us and added it to SLES. So every build coming from SuSE for SLES contains the oracleasm.ko module. We weren't as successful with other vendors so for quite some time we continued to build it for RHEL and of course as we introduced Oracle Linux end of 2006 also for Oracle Linux. With Oracle Linux it became easy for us because we just added the code to our build system and as we churned out Oracle Linux kernels whether it was for a public release or for customers that needed a one off fix where they also used asmlib, we didn't have to do any extra work it was just all nicely integrated.

    With the introduction of Oracle Linux's Unbreakable Enterprise Kernel and our interest in being able to exploit ASMLib more, we started working on a very exciting project called Data Integrity. Oracle (Martin Petersen in particular) worked for many years with the T10 standards committee and storage vendors and implemented Linux kernel support for DIF/DIX, data protection in the Linux kernel, note to those that wonder, yes it's all in mainline Linux and under the GPL. This basically gave us all the features in the Linux kernel to checksum a data block, send it to the storage adapter, which can then validate that block and checksum in firmware before it sends it over the wire to the storage array, which can then do another checksum and to the actual DISK which does a final validation before writing the block to the physical media. So what was missing was the ability for a userspace application (read: Oracle RDBMS) to write a block which then has a checksum and validation all the way down to the disk. application to disk.

    Because we have ASMLib we had an entry into the Linux kernel and Martin added support in ASMLib (kernel driver + userspace) for this functionality. Now, this is all based on relatively current Linux kernels, the oracleasm kernel module depends on the main kernel to have support for it so we can make use of it. Thanks to UEK and us having the ability to ship a more modern, current version of the Linux kernel we were able to introduce this feature into ASMLib for Linux from Oracle. This combined with the fact that we build the asm kernel module when we build every single UEK kernel allowed us to continue improving ASMLib and provide it to our customers.

    So today, we (Oracle) provide Oracle ASMLib for Oracle Linux and in particular on the Unbreakable Enterprise Kernel. We did the build/testing/delivery of ASMLib for RHEL until RHEL5 but since RHEL6 decided that it was too much effort for us to also maintain all the build and test environments for RHEL and we did not have the ability to use the latest kernel features to introduce the Data Integrity features and we didn't want to end up with multiple versions of asmlib as maintained by us. SuSE SLES still builds and comes with the oracleasm module and they do all the work and RHAT it certainly welcome to do the same. They don't have to rebuild the userspace library, it's really about the kernel module.

    And finally to re-iterate a few important things :

  • Oracle ASM does not in any way require ASMLib to function completely. ASMlib is a small set of extensions, in particular to make device management easier but there are no extra features exposed through Oracle ASM with ASMLib enabled or disabled. Often customers confuse ASMLib with ASM. again, ASM exists on every Oracle supported OS and on every supported Linux OS, SLES, RHEL, OL withoutASMLib
  • Oracle ASMLib userspace is available for OTN and the kernel module is shipped along with OL/UEK for every build and by SuSE for SLES for every of their builds
  • ASMLib kernel module was built by us for RHEL4 and RHEL5 but we do not build it for RHEL6, nor for the OL6 RHCK kernel. Only for UEK
  • ASMLib for Linux is/was a reference implementation for any third party vendor to be able to offer, if they want to, their own version for their own OS or storage
  • ASMLib as provided by Oracle for Linux continues to be enhanced and evolve and for the kernel module we use UEK as the base OS kernel
  • hope this helps.

    Wednesday Jul 04, 2012

    What's up with OCFS2?

    On Linux there are many filesystem choices and even from Oracle we provide a number of filesystems, all with their own advantages and use cases. Customers often confuse ACFS with OCFS or OCFS2 which then causes assumptions to be made such as one replacing the other etc... I thought it would be good to write up a summary of how OCFS2 got to where it is, what we're up to still, how it is different from other options and how this really is a cool native Linux cluster filesystem that we worked on for many years and is still widely used.

    Work on a cluster filesystem at Oracle started many years ago, in the early 2000's when the Oracle Database Cluster development team wrote a cluster filesystem for Windows that was primarily focused on providing an alternative to raw disk devices and help customers with the deployment of Oracle Real Application Cluster (RAC). Oracle RAC is a cluster technology that lets us make a cluster of Oracle Database servers look like one big database. The RDBMS runs on many nodes and they all work on the same data. It's a Shared Disk database design. There are many advantages doing this but I will not go into detail as that is not the purpose of my write up. Suffice it to say that Oracle RAC expects all the database data to be visible in a consistent, coherent way, across all the nodes in the cluster. To do that, there were/are a few options : 1) use raw disk devices that are shared, through SCSI, FC, or iSCSI 2) use a network filesystem (NFS) 3) use a cluster filesystem(CFS) which basically gives you a filesystem that's coherent across all nodes using shared disks. It is sort of (but not quite) combining option 1 and 2 except that you don't do network access to the files, the files are effectively locally visible as if it was a local filesystem.

    So OCFS (Oracle Cluster FileSystem) on Windows was born. Since Linux was becoming a very important and popular platform, we decided that we would also make this available on Linux and thus the porting of OCFS/Windows started. The first version of OCFS was really primarily focused on replacing the use of Raw devices with a simple filesystem that lets you create files and provide direct IO to these files to get basically native raw disk performance. The filesystem was not designed to be fully POSIX compliant and it did not have any where near good/decent performance for regular file create/delete/access operations. Cache coherency was easy since it was basically always direct IO down to the disk device and this ensured that any time one issues a write() command it would go directly down to the disk, and not return until the write() was completed. Same for read() any sort of read from a datafile would be a read() operation that went all the way to disk and return. We did not cache any data when it came down to Oracle data files.

    So while OCFS worked well for that, since it did not have much of a normal filesystem feel, it was not something that could be submitted to the kernel mail list for inclusion into Linux as another native linux filesystem (setting aside the Windows porting code ...) it did its job well, it was very easy to configure, node membership was simple, locking was disk based (so very slow but it existed), you could create regular files and do regular filesystem operations to a certain extent but anything that was not database data file related was just not very useful in general. Logfiles ok, standard filesystem use, not so much. Up to this point, all the work was done, at Oracle, by Oracle developers.

    Once OCFS (1) was out for a while and there was a lot of use in the database RAC world, many customers wanted to do more and were asking for features that you'd expect in a normal native filesystem, a real "general purposes cluster filesystem". So the team sat down and basically started from scratch to implement what's now known as OCFS2 (Oracle Cluster FileSystem release 2). Some basic criteria were :

  • Design it with a real Distributed Lock Manager and use the network for lock negotiation instead of the disk
  • Make it a Linux native filesystem instead of a native shim layer and a portable core
  • Support standard Posix compliancy and be fully cache coherent with all operations
  • Support all the filesystem features Linux offers (ACL, extended Attributes, quotas, sparse files,...)
  • Be modern, support large files, 32/64bit, journaling, data ordered journaling, endian neutral, we can mount on both endian /cross architecture,..
  • Needless to say, this was a huge development effort that took many years to complete. A few big milestones happened along the way...

  • OCFS2 was development in the open, we did not have a private tree that we worked on without external code review from the Linux Filesystem maintainers, great folks like Christopher Hellwig reviewed the code regularly to make sure we were not doing anything out of line, we submitted the code for review on lkml a number of times to see if we were getting close for it to be included into the mainline kernel. Using this development model is standard practice for anyone that wants to write code that goes into the kernel and having any chance of doing so without a complete rewrite or.. shall I say flamefest when submitted. It saved us a tremendous amount of time by not having to re-fit code for it to be in a Linus acceptable state. Some other filesystems that were trying to get into the kernel that didn't follow an open development model had a lot harder time and a lot harsher criticism.
  • March 2006, when Linus released 2.6.16, OCFS2 officially became part of the mainline kernel, it was accepted a little earlier in the release candidates but in 2.6.16. OCFS2 became officially part of the mainline Linux kernel tree as one of the many filesystems. It was the first cluster filesystem to make it into the kernel tree. Our hope was that it would then end up getting picked up by the distribution vendors to make it easy for everyone to have access to a CFS. Today the source code for OCFS2 is approximately 85000 lines of code.
  • We made OCFS2 production with full support for customers that ran Oracle database on Linux, no extra or separate support contract needed. OCFS2 1.0.0 started being built for RHEL4 for x86, x86-64, ppc, s390x and ia64. For RHEL5 starting with OCFS2 1.2.
  • SuSE was very interested in high availability and clustering and decided to build and include OCFS2 with SLES9 for their customers and was, next to Oracle, the main contributor to the filesystem for both new features and bug fixes.
  • Source code was always available even prior to inclusion into mainline and as of 2.6.16, source code was just part of a Linux kernel download from, which it still is, today. So the latest OCFS2 code is always the upstream mainline Linux kernel.
  • OCFS2 is the cluster filesystem used in Oracle VM 2 and Oracle VM 3 as the virtual disk repository filesystem.
  • Since the filesystem is in the Linux kernel it's released under the GPL v2
  • The release model has always been that new feature development happened in the mainline kernel and we then built consistent, well tested, snapshots that had versions, 1.2, 1.4, 1.6, 1.8. But these releases were effectively just snapshots in time that were tested for stability and release quality.

    OCFS2 is very easy to use, there's a simple text file that contains the node information (hostname, node number, cluster name) and a file that contains the cluster heartbeat timeouts. It is very small, and very efficient. As Sunil Mushran wrote in the manual :

  • OCFS2 is an efficient, easily configured, quickly installed, fully integrated and compatible, feature-rich, architecture and endian neutral, cache coherent, ordered data journaling, POSIX-compliant, shared disk cluster file system.
  • Here is a list of some of the important features that are included :

  • Variable Block and Cluster sizes Supports block sizes ranging from 512 bytes to 4 KB and cluster sizes ranging from 4 KB to 1 MB (increments in power of 2).
  • Extent-based Allocations Tracks the allocated space in ranges of clusters making it especially efficient for storing very large files.
  • Optimized Allocations Supports sparse files, inline-data, unwritten extents, hole punching and allocation reservation for higher performance and efficient storage.
  • File Cloning/snapshots REFLINK is a feature which introduces copy-on-write clones of files in a cluster coherent way.
  • Indexed Directories Allows efficient access to millions of objects in a directory.
  • Metadata Checksums Detects silent corruption in inodes and directories.
  • Extended Attributes Supports attaching an unlimited number of name:value pairs to the file system objects like regular files, directories, symbolic links, etc.
  • Advanced Security Supports POSIX ACLs and SELinux in addition to the traditional file access permission model.
  • Quotas Supports user and group quotas.
  • Journaling Supports both ordered and writeback data journaling modes to provide file system consistency in the event of power failure or system crash.
  • Endian and Architecture neutral Supports a cluster of nodes with mixed architectures. Allows concurrent mounts on nodes running 32-bit and 64-bit, little-endian (x86, x86_64, ia64) and big-endian (ppc64) architectures.
  • In-built Cluster-stack with DLM Includes an easy to configure, in-kernel cluster-stack with a distributed lock manager.
  • Buffered, Direct, Asynchronous, Splice and Memory Mapped I/Os Supports all modes of I/Os for maximum flexibility and performance.
  • Comprehensive Tools Support Provides a familiar EXT3-style tool-set that uses similar parameters for ease-of-use.
  • The filesystem was distributed for Linux distributions in separate RPM form and this had to be built for every single kernel errata release or every updated kernel provided by the vendor. We provided builds from Oracle for Oracle Linux and all kernels released by Oracle and for Red Hat Enterprise Linux. SuSE provided the modules directly for every kernel they shipped. With the introduction of the Unbreakable Enterprise Kernel for Oracle Linux and our interest in reducing the overhead of building filesystem modules for every minor release, we decide to make OCFS2 available as part of UEK. There was no more need for separate kernel modules, everything was built-in and a kernel upgrade automatically updated the filesystem, as it should. UEK allowed us to not having to backport new upstream filesystem code into an older kernel version, backporting features into older versions introduces risk and requires extra testing because the code is basically partially rewritten. The UEK model works really well for continuing to provide OCFS2 without that extra overhead.

    Because the RHEL kernel did not contain OCFS2 as a kernel module (it is in the source tree but it is not built by the vendor in kernel module form) we stopped adding the extra packages to Oracle Linux and its RHEL compatible kernel and for RHEL. Oracle Linux customers/users obviously get OCFS2 included as part of the Unbreakable Enterprise Kernel, SuSE customers get it by SuSE distributed with SLES and Red Hat can decide to distribute OCFS2 to their customers if they chose to as it's just a matter of compiling the module and making it available.

    OCFS2 today, in the mainline kernel is pretty much feature complete in terms of integration with every filesystem feature Linux offers and it is still actively maintained with Joel Becker being the primary maintainer. Since we use OCFS2 as part of Oracle VM, we continue to look at interesting new functionality to add, REFLINK was a good example, and as such we continue to enhance the filesystem where it makes sense. Bugfixes and any sort of code that goes into the mainline Linux kernel that affects filesystems, automatically also modifies OCFS2 so it's in kernel, actively maintained but not a lot of new development happening at this time. We continue to fully support OCFS2 as part of Oracle Linux and the Unbreakable Enterprise Kernel and other vendors make their own decisions on support as it's really a Linux cluster filesystem now more than something that we provide to customers. It really just is part of Linux like EXT3 or BTRFS etc, the OS distribution vendors decide.

    Do not confuse OCFS2 with ACFS (ASM cluster Filesystem) also known as Oracle Cloud Filesystem. ACFS is a filesystem that's provided by Oracle on various OS platforms and really integrates into Oracle ASM (Automatic Storage Management). It's a very powerful Cluster Filesystem but it's not distributed as part of the Operating System, it's distributed with the Oracle Database product and installs with and lives inside Oracle ASM. ACFS obviously is fully supported on Linux (Oracle Linux, Red Hat Enterprise Linux) but OCFS2 independently as a native Linux filesystem is also, and continues to also be supported. ACFS is very much tied into the Oracle RDBMS, OCFS2 is just a standard native Linux filesystem with no ties into Oracle products. Customers running the Oracle database and ASM really should consider using ACFS as it also provides storage/clustered volume management. Customers wanting to use a simple, easy to use generic Linux cluster filesystem should consider using OCFS2.

    To learn more about OCFS2 in detail, you can find good documentation on in the Documentation area, or get the latest mainline kernel from and read the source.

    One final, unrelated note - since I am not always able to publicly answer or respond to comments, I do not want to selectively publish comments from readers. Sometimes I forget to publish comments, sometime I publish them and sometimes I would publish them but if for some reason I cannot publicly comment on them, it becomes a very one-sided stream. So for now I am going to not publish comments from anyone, to be fair to all sides. You are always welcome to email me and I will do my best to respond to technical questions, questions about strategy or direction are sometimes not possible to answer for obvious reasons.

    Friday Jun 29, 2012

    My own personal use of Oracle Linux

    It always is easier to explain something with examples... Many people still don't seem to understand some of the convenient things around using Oracle Linux and since I personally (surprise!) use it at home, let me give you an idea.

    I have quite a few servers at home and I also have 2 hosted servers with a hosted provider. The servers at home I use mostly to play with random Linux related things, or with Oracle VM or just try out various new Oracle products to learn more. I like the technology, it's like a hobby really. To be able to have a good installation experience and use an officially certified Linux distribution and not waste time trying to find the right libraries, I, of course, use Oracle Linux. Now, at least I can get a copy of Oracle Linux for free (even if I was not working for Oracle) and I can/could use that on as many servers at home (or at my company if I worked elsewhere) for testing, development and production. I just go to and download the version(s) I want and off I go.

    Now, I also have the right (and not because I am an employee) to take those images and put them on my own server and give them to someone else, I in fact, just recently set up my own mirror on my own hosted server. I don't have to remove oracle-logos, I don't have to rebuild the ISO images, I don't have to recompile anything, I can just put the whole binary distribution on my own server without contract. Perfectly free to do so. Of course the source code of all of this is there, I have a copy of the UEK code at home, just cloned from And as you can see, the entire changelog, checkins, merges from Linus's tree, complete overview of everything that got changed from kernel to kernel, from patch to patch, errata to errata. No obfuscating, no tar balls and spending time with diff, or go read bug reports to find out what changed (seems silly to me).

    Some of my servers are on the external network and I need to be current with security errata, but guess what, no problem, my servers are hooked up to which is open, free, and completely up to date, in a consistent, reliable way with any errata, security or bugfix. So I have nothing to worry about. Also, not because I am an employee. Anyone can. And, with this, I also can, and have, set up my own mirror site that hosts these RPMs. both binary and source rpms. Because I am free to get them and distribute them. I am quite capable of supporting my servers on my own, so I don't need to rely on the support organization so I don't need to have a support subscription :-). So I don't need to pay. Neither would you, at least not with Oracle Linux.

    Another cool thing. The hosted servers came (unfortunately) with Centos installed. While Centos works just fine as is, I tend to prefer to be current with my security errata(reliably) and I prefer to just maintain one yum repository instead of 2, I converted them over to Oracle Linux as well (in place) so they happily receive and use the exact same RPMs. Since Oracle Linux is exactly the same from a user/application point of view as RHEL, including files like /etc/redhat-release and no changes from .el. to .centos. I know I have nothing to worry about installing one of the RHEL applications. So, OL everywhere makes my life a lot easier and why not...

    Next! Since I run Oracle VM and I have -tons- of VM's on my machines, in some cases on my big WOPR box I have 15-20 VMs running. Well, no problem, OL is free and I don't have to worry about counting the number of VMs, whether it's 1, or 4, or more than 10 ... like some other alternatives started doing...

    and finally :) I like to try out new stuff, not 3 year old stuff. So with UEK2 as part of OL6 (and 6.3 in particular) I can play with a 3.0.x based kernel and it just installs and runs perfectly clean with OL6, so quite current stuff in an environment that I know works, no need to toy around with an unsupported pre-alpha upstream distribution with libraries and versions that are not compatible with production software (I have nothing against ubuntu or fedora or opensuse... just not what I can rely on or use for what I need, and I don't need a desktop).

    pretty compelling. I say... and again, it doesn't matter that I work for Oracle, if I was working elsewhere, or not at all, all of the above would still apply. Student, teacher, developer, whatever. contrast this with $349 for 2 sockets and oneguest and selfsupport per year to even just get the software bits.

    Wednesday Jun 27, 2012

    Oracle Linux 6 update 3

    Oracle Linux 6.3 channels are now available online

  • repositories. Both base channels and latest channels are available (for free for everyone)
  • repositories. Behind our customer portal but effectively the same content.
  • Source RPMs (.srpm) are being uploaded to

    OL6.3 contains UEK2 kernel-uek-2.6.39-200.24.1. The source rpm is in the above location but our public GIT repository will be synced up shortly as well at;a=summary. Unlike some others, of course, complete source, complete changelog, complete checkin history, both mainline and our own available. No need to go assemble things from a website manually.

    Another cool thing coming up is a boot iso for OL6.3 that boots up uek (2.6.39-200.24.1) as install kernel and uses btrfs as the default filesystem for installation. So latest and greatest direct access to btrfs, a modern well-tested, current kernel, freely available. Enjoy.

    Since it takes a few days for our ISOs to be on, I mirrored them on my own server :

    Saturday May 26, 2012

    from the research department - ramster in Linux 3.4 staging drivers

    Some developers on my team have spent quite a few years on doing research and development on new ways of doing memory management better across servers, both physical and virtual. You often hear me talk about cooperative memory management in the context of Oracle VM. I believe it is important to be able to do memory overcommit the right way and have both the guest OS and the hypervisor know what's going on, instead of having the hypervisor do things behind the guest's back. Transparent memory overcommit causes unpredictable behavior for applications and while this might be of little concern when consolidating lots of small VMs that aren't doing much interesting. When you have applications that are, by nature, memory intensive, such as an Oracle database or any sort of production environment, then there need to be better ways to do this. Better, here, meaning -> more efficient, more predictable and more cooperative between host and guest.

    It's a long road to getting all these stars aligned but we are very close now in terms of Linux kernel features to help with this. Over the last few years Dan Magenheimer has been working hard on what's called transcendent memory, a collection of features that include cleancache, frontswap and ramster. Now with Linux 3.4, we have tmem, cleancache and ramster in kernel upstream. Frontswap is still pending but we're working on it.

    As a refresher, you can read more about cleancache in this writeup I did just bout a year ago. Frontswap is more tricky, because here we can use transcendent memory as a swap device. Pages swapped out cannot just disappear, they have to be persistent until the OS decides to no longer need them. In other words, with cleancache we provide extra memory(extra cache) but with the understanding that that extra memory might just disappear. With frontswap, we provide swap space through the same mechanism, with the understanding that it's not going to just disappear. One place where frontswap is different from regular swap, is its size. A swapdevice is pre-created with a specific size, a file acting as a swap device or a physical/logical volume. They do not grow or shrink dynamically. frontswap is more dynamic. The advantage of frontswap is that you can use ram on a remote server or in the hypervisor as very fast storage. It can also do compression of memory pages using zcache.

    If you want to play with ramster, Dan wrote up a simple how-to.
    Dan also wrote up a presentation which you can find here.

    Looking forward to frontswap getting into the kernel and many of these features get into customers hands through the Unbreakable Enterprise Kernel and as products evolve and features get management also into Oracle VM.

    Thursday May 24, 2012

    The case for Oracle VM

    As I mentioned a few days ago, at a customer visit earlier this month we were discussing Oracle VM and Oracle Linux advantages and the customer was surprised to hear how many features we include in our offerings. My previous write up was about Oracle Linux, this one is about Oracle VM.

    In many ways, the Oracle VM solution has similar advantages. For us to deliver a solution and for our customers and partners to have a stack that's built to work together across products. Because Oracle VM subscriptions include the right to use a number of products, such as Oracle Enterprise Manager 12c and Oracle Enterprise Manager OpsCenter, we end up with an impressive, all-in set of features. Often times however this is not obvious to our users or evaluators and it is important to look at the total package and not just sub components. What I am going to describe is included in our Oracle VM subscription.

    First of all, pricing... our virtualization and infrastructure management solution has a very simple model :

  • Oracle VM premier limited (for 1 or 2 socket servers) : $599 annual support subscription per server
  • It doesn't matter how many VMs you run, it doesn't matter how much RAM is in the box, it doesn't matter how many cores or threads the sockets have. 1 or 2 sockets in the box. That's the price. Annual support subscription fee. No license. Including all the products I will be detailing below.
    So if you have a 2 socket server with 256GB RAM... $599. case closed. 24/7 support.

  • Oracle VM premier (for larger than 2 socket servers) : $1199 annual support subscription per server
  • Same deal. If you have a 4 socket server, or even an 8 socket server with 4 TB of RAM... $1199 subscription. period. No virtual RAM, no per 2 sockets, no multiple options, no starter pack for 2 servers... Simple pricing. 24/7 support.

    Also, as with Oracle Linux, if you purchase an Oracle Server with the Premier support for Systems, it also includes everything I am talking about without the need for extra per server subscriptions.

    Oracle VM support subscriptions for both premier and premier limited comes with the following products and right to use :

  • Oracle VM server
  • Oracle VM server is the product that is installed on each server. It contains the hypervisor and the management console/agent to talk to Oracle VM Manager. This is effectively what you get a support subscription for. A tiny installation of about 200Mb worth of RPMs on a local hard drive or flash drive.

  • Oracle VM Manager
  • Oracle VM Manager is our centralized management console. It is an Oracle WebLogic Server 11g application that uses an Oracle Database as data repository. You typically only need 1 instance to manage a very large set of servers and virtual machines. Oracle VM Manager is the management hub for Oracle VM. It has a web-based admin interface and the management product can be installed on a Linux server, typically outside the Oracle VM server pools. We do not charge extra for this, the use and support of Oracle VM Manager is included in the server subscriptions. More details further down...

  • Oracle Enterprise Manager 12c Cloud Control
  • OEM 12c is our complete management product which can manage from application to disk. It contains management for our various applications, database, middleware, operating systems, virtualization layer, it offers configuration management, lifecycle management, physical and virtual environments, cloud portal and so on. The product consists of many management packs, the components that manage Oracle Linux and Oracle VM (cloud portal, private cloud, self-service portal, VM management, IaaS layer services, cloud API....) are included with Oracle VM subscriptions. These are not extra licensed components, they're part of the package!

  • Oracle Enterprise Manager Ops Center 12c
  • EMOC is a very powerful management product that's focused on infrastructure management, our Oracle systems (X86 and SPARC) (firmware, switches, hardware fault analysis, My Oracle Support integration, bios configuration etc...), our Operating Systems (Oracle Solaris, Oracle Linux), Virtualization products (Solaris Containers, Oracle VM SPARC, Oracle VM x86) and a connector into OEM12c to tie them together. Again, when you have an Oracle VM server subscription, you have the right to use EMOC as part of the stack included with its feature set It also contains an IaaS layer and a cloud API and cli.

    Both EM12c and EMOC connect to Oracle VM Manager instances behind the scenes. They can discover or sync up the information from an existing Oracle VM Manager/Server configuration and you can literally switch back and forth. So you start out installing a bunch of Oracle VM servers, install an Oracle VM manager instance, discover the servers, then install EM12c and/or EMOC, discover Oracle VM Manager and now you have access to their feature sets on top. You can still connect to / log into Oracle VM Manager and so on.

    So when we say "Oracle VM" and when you compare "Oracle VM" without various competitors, we really provide you with all of the above, and our feature checklist and offering is all of the above. Often times people just use Oracle VM Manager/Server without also including the use of EMOC and/or EM12c. Our competitors have many products that are all separately licensed pieces and bits that have to be installed together. Look at the solution bundle. We have no per VM, per 96GB, per socket, per feature pricing... remember, $599 and $1199 subscription.

    Here's a few features that are part of all this :

  • Cloud Infrastructure administrator portal
  • Set up and manage the lowerlevel infrastructure, hypervisor, provisioning of storage...

  • Self service administrator
  • Set up higher-level services, users, roles, access boundaries (assign roles and users to zones)
    Set up charge back plans, publish software components (VMs, full application stacks for self service users)

  • self service user portal
  • end user, developer can log in and see their assigned infrastructure quota and create application setups based on software component from the libraries

  • Consolidation planning physical server assets and P2P or P2V advisories
  • Since EM12c has a view of physical and virtual servers and applications inside, it can provide very detailed planning of which systems are good targets to consolidate, because it has an application view and a complete end to end stack overview, it is much more detailed and capable than simple VM analysis.

  • Zone management (multiple pools of servers into a larger grouping)
  • Depending on administrators needs its possible to configure larger zones (pools) of systems as units

  • Storage management through Oracle VM Storage Connect
  • We can manage iscsi, fibre channel, NFS servers from our management console. It is possible to just provision large chunks of storage and provide credentials to Oracle VM Manager and we can handle the details behind the scenes. Add a new lun? no problem, we will do it. EM12c and EMOC expose this functionality and connect through Oracle VM Manager to the storage.

  • Oracle VM assemblies and templates
  • pre-created virtual machines that include complete Oracle product installations, database, siebel, peoplesoft, ebusiness suite etc... these can eb downloaded from the Oracle website and uploaded into Oracle VM Manager or the EM12c software library for easy deployment of complete solution stacks.

    With assemblies we provide complete multi-tiered environments, including scale-out options. Need more webservers, need more database nodes? We can add/remove them on the fly, based on the application and business need, not just based on what the blackbox VM is doing.

  • Chargeback
  • It is possible to create charge back plans, either your own or using existing models. Whether it's basic charge back for virtualization or depending on management packs and oracle products up the stack for other products as well.

  • build, test, deploy
  • Because of the templates and configuration management for applications support, we can let developers build and create solutions which then get uploaded into the software library for use in product deployments. Since EM12c understands the entire product stack, application to disk, we can automate and reproduce not just simple VM deployments but include application configuration on top as well. Creating a VM with an OS is easy. Deploying a VM with an OS is easy. Deploying and configuring an entire software stack with various applications and database(s) is error prone and can take a lot of time. We can automate it for you, reproduce it, exactly the same steps, every time. Top to bottom.

  • Various abstraction layers
  • Through EM12c, EMOC and Oracle VM Manager you have an easy to use, graphical user interface.
    Through EM12c and EMOC you have access to a complete cloud API.
    EMOC offers an IaaS RESTful interface, a JAVA API library and a complete CLI.
    EM12c offers an IaaS RESTful interface

  • complete monitoring and management
  • EM12c manages and monitors every target in the stack, the applications, the operating systems, the VMs, the hypervisors. You can define jobs, metrics, thresholds, corrective actions. Because we collect information across the entire stack, you can set up auditing rules, you can compete systems top to bottom, you can do change management across the entire infrastructure, physical and virtual. To make it easy, it is possible to create groups of systems to manage as a unit. Configuration management is now more critical than ever. In a cloud infrastructure the number of VMs, and as such, the number of deployed applications, grows exponentially. We keep track of every aspect through a single source/database :) of truth.

  • Roles, LDAP directory services
  • Complete integration of LDAP with EM12c and EMOC. The ability to define fine grained access to all these components. Storage, servers, functions.
  • integration with My Oracle Support
  • Patch advisories, service request management, downloading new templates into the software library

  • standard virtualization features
  • Not to forget the fact that this obviously also includes Live migration, Dynamic Power Management (DPM), Dynamic Resource Management(DRS), storage management, network management(setting up bonding, creating virtual networks, vlan support, built-in HA (VM failover, server failover), cpu pinning, clustered filesystem for storage repositories, direct lun/storage access,...

    Installs on the largest X86 servers available. 4TB RAM, 160 threads, virtual machines up to 2TB of ram and up to 128 virtual CPUs or as many virtual CPUs the guest operating system can handle. Support for Linux, Oracle Solaris, Windows (including Windows logo signed drivers).

    So this is the Oracle VM offering, this is a summary (and I am not even trying to be complete here) of our features. Need an API, need a cli, need a UI, need a cloud portal, need monitoring or management, need roles and responsibilities, need centralized account management, we have it all. So when you compare Oracle VM with our competition out there... compare the included feature set in the subscription.

    A few useful links :

  • EMOC API/CLI reference
  • EM12c API
  • Oracle Virtualization page
  • more Oracle Linux options

    A few days ago I wrote a summary of what you get with Oracle Linux. Because there are so many things, I forgot a few... and this is just a continuation of that previous entry.

    There are 2 more features that I wanted to present :

  • Oracle Database Smart Flash Cache
  • DBSFC is a very cool feature that's available for both Oracle Solaris and Oracle Linux customers with the 11gR2 database. In summary, it allows you to basically extend the Oracle Buffer Cache in memory (SGA) using secondary flash based storage. This flash based storage can be presented to the database through a file on a filesystem on flash storage, a raw disk device (flash-based) or through adding flash storage to Oracle ASM and creating a region inside ASM.

    For the most part this feature is going to help with read-only/read-mostly workloads because DBSFC is a read-only cache extension. It contains clean blocks that got kicked out of the buffercache/sga and now first get placed in this extended cache. A subsequent read can then be from this fast storage instead of from the originating datafiles. When a block gets modified, it's modified in the standard database buffer cache, written to disk and copied over into the flash cache.

    The white paper referenced above provides the details on how to use it and how to configure it in an Oracle Linux environment. You simply specify DB_FLASH_CACHE_FILE and DB_FLASH_CACHE_SIZE and that's it. Any Oracle Database customer using Oracle Linux can make use of this.

  • Assigning a control cgroup (cgroup) in Oracle Linux to an Oracle database instance.
  • Oracle Linux has resource management through a feature called cgroups. cgroups lets you create resource groupings based on cpu, memory or disk parameters (or a combination). cgroups is also the internal set of features that Linux containers (lxc) uses. Basically you put processes (pids/tasks) into a cgroup and then they live within the limits of that cgroup definition. With lxc you basically also get process isolation on top.

    It works like this : as an admin you set up a cgroup, give it a name and set up paremeters around which cpu's to use, how much memory to allocate and so forth. The name is really just a simple mkdir in the cgroups virtual filesystem. You then use a new database init parameter process_group_name and when you start the database it will put its processes and as such its resources into that group. alter system set processor_group_name = 'cgroup_name' scope=spfile;. You can do this for each instance and this way you can isolate cpu and memory resources for each instance on a given OS environment. If you use Oracle Linux 5 with uek/uek2 or Oracle Linux 6 with uek/uek2 you can make use of this.

    cgroups with Oracle Linux 5 and UEK/UEK2 has to happen manually, through mkdir and echo > controlfiles. In Oracle Linux 6 we have cgroup management utilities.

    two more reasons :)...

    Sunday May 20, 2012

    Oracle Linux components

    A few weeks ago I was visiting a customer where we were discussing Oracle Linux and Oracle VM advantages compared to some of our competitors. One of the comments from the customer at some point in the discussion was "And you get -all that- for just -$that-?"... Why, yes, you do, sir. I wanted to take the opportunity to list some of the included advantages of having Oracle Linux support. Next time I will do the same for Oracle VM.

    Oracle Linux is a complete enterprise Linux solution available from Oracle. In fact, it is, we firmly believe, the best enterprise Linux solution available. Both for generic use of Linux and in particular and especially when using any Oracle product on top of it. So it is important to remember and understand that Oracle Linux is not just for running Oracle products on top of Linux. Oracle Linux is a complete generic Linux distribution that's fully supported underneath any Linux application. There are no restrictions from our end as to what you can run on it (as long as they're apps built to run on Linux, of course and any application that runs on Red Hat Enterprise Linux will run the same on the corresponding Oracle Linux version). You can run 3rd party Linux applications just fine. Use it as an NFS server, as a webserver or whatever you want to use the system for. Many customers I talk to somehow believe that it's unique for Oracle products and I have to point out that while we have great, unique advantages and extra benefits running Oracle products on Oracle Linux, it's certainly not in anyway restricted for that. It's the opposite, all you needs is Oracle Linux for any and every Linux server in house.

    Why the best, you ask? Well, we do an incredibly amount of testing on a daily basis (1000's of servers run various types of QA on Oracle Linux amounting to over 100,000 hours of QA happening every single day), we have a huge amount of internal use as consumers (end-user/customer use), we have many years of experience helping enterprise customers in general and we extend that down to Operating System support. We can scale using the greater Oracle infrastructure, etc.

    The support team is a large group of dedicated professionals globally that are trained to handle serious issues, deal with mission critical environments and cases, know their product, have a focus on server use first and foremost, understand, in particular, also the other products around it enough (like, to be able to understand that ASM doesn't stand for -assembler language- (in our customer-base) but stands for automatic storage management (you laugh? I can assure you it's not a joke)) A single phone call: if a database problem is an OS issue, no worries, we deal with it internally, no need to go call someone else that might think a database is a dirty word, or someone that 's an expert in gnome (no offense) but not quite in a complex environment where you have a product use 50GB of RAM and has 2000 processes running and is doing massive amounts of IO and woops something is going wrong. "Reboot?" doesn't cut it.

    When we do Linux development, the focus is basically on two things. 1) find ways to enhance Linux to run our products better (in ways that are generic, and also help other applications). A database does disk/network IO, lots of processes, lots of memory access, if we make our product run better, others also benefit... 2) help make Linux a better OS. improve stability, help add features that an OS needs, unrelated to our own product stack but important for customers to run the operating system, no matter what application. Having access to in-house hardware such as very big x64 servers like the x4800 (4TB ram, 160 threads) and TB and TB of storage allows us to test things at very, very large scale. This allows us to publish benchmark results like the 5M tpc-c result a few months ago. Offering the Unbreakable Enterprise Kernel(UEK) as part of Oracle Linux alongside what we call the Red Hat-Compatible kernel gives us the ability to provide current, modern, tested code to customers without reducing reliable and availability.

    A CTO of another company a little while back implied on twitter that UEK is too current and reduces reliability because the code is new. Can I point something out here? One thing we decided to do, with UEK, was to not backport features but instead just focus on stability. The goal is to stay close (but not too close as we don't ship a development tree) to the "Linus kernel" and focus on testing the heck out of it and fixing any bug that we would find, any regression.. This is good for the Linux community because when we do find a problem, we fix it and it's immediately also relevant to the mainline kernel. This helps us and the kernel community. We spend time testing and fixing, we develop on mainline and we don't waste time backporting new stuff into old versions. It's a symbiotic relationship. The alternative : take a version of Linux that's 3+ years old and take code from a more or less current version, after 100's of 1000's of lines of code changes and architectural changes have happened, and make that new code, somehow work in the old tree. The result : old patched up, totally unique code. Someone spending all their time retrofitting complex features into an old version. Testing is unique, bugs found are mostly only relevant in that version and not so much in newer versions unless they're greater design bugs or one liners. No one else really ever tested it because, well, it's unique, no other company is providing that seriously forked kernel that has 1+M of lines of differences. Somehow, I prefer our approach. At least the folks at SuSE seem to believe the same.

    One advantage we have with our kernel is that we made it possible to install in an existing Oracle Linux 5 environment and you are not required to move to Oracle Linux 6 to get the benefit of new kernel features and new drivers and new performance benefits. Moving from one major version to the next is a very costly effort because you basically always end up doing a re-imaging of the servers. Most customers are driven to new major releases for a hardware refresh. We don't require you to do that, you can run both OL5 and OL6, your choice, your timeframe. You can reboot between 2.6.18 and 2.6.32 or even 2.6.39 (3.0.24) without problem, at any time.

    The complete source code for Oracle Linux is available from (OL5 or OL6). The source code for our UEK and UEK2 kernels is available in a public GIT repository (UEK 2.6.32 or UEK2 2.6.39. Don't worry, it's not just a tar file and you don't have to go browse webpages behind a userid/password to figure out what all changed and try to piece it together yourself. It's all there, both the mainline kernel changelog and our changes on top. Transparent.

    The binaries for Oracle Linux are available from edelivery in the form of ISO images of the installable DVDs for the major releases and all the update releases (OL5.1...5.8, OL6, OL6.1, OL6.2 etc). You are not required to first get a support subscription, you can just go and download the software, no activation keys, no trial period, production use OK, distribute to your friends? no problem, no need to remove logos. Errata/bug updates are available through ULN (ULN is our Oracle Linux customer portal where we have extra channels with extra software bits for subscribers, or from public yum (public yum is an open, free, no registration, no subscription site). All Oracle Linux errata/packages are published on both sites at the same time. If you have a set of servers where support is not needed, just a standard Oracle Linux, and then a set of servers where you do need support because they're production systems etc, you can install OL on both and you subscribe for the systems you need to have support for. One distribution installed across the entire company. No need to find a free distribution and a pay to use distribution. If you no longer need support services for a system, don't worry, we won't ask you to de-install.

    But wait, there's more..

    How about managing Oracle Linux... well, included in Oracle Linux support subscriptions, is the use of Oracle Enterprise Manager 12c and Oracle Enterprise Manager OpsCenter 12. This is management software that can handle 1000's of servers from a single managed web-based interface. It offers features like :

  • bare-metal provisioning of Oracle Linux servers
  • configuration management, collecting and analysis and auditing
  • lights-out or ad hoc patching
  • availability and performance monitoring
  • event management and job systems
  • yum repositories
  • grouping of servers, custom jobs, custom action/corrective action scripts
  • this is all included, for no additional fee. Any server with Oracle Linux support subscriptions include the above. No extra pack, no per server additional fees. Or if you don't really care for that and just want a very simple yum based setup, you can do that as well, just grab the rpms, put them on a server, create a repository and have your servers connect to that. So, very basic and simple, to very complete and comprehensive. Same cost.

    And there is even more...

    Oracle Real Application Clusters makes use of the Oracle Clusterware(CRS). CRS is cluster software that is capable of protecting any kind of application in a failover cluster. Not just Oracle. It is the basis for any clustered environment and provides generic availability eliminating the need for other third party cluster solutions. You can protect your own application using this enterprise grade clustering stack. Without additional fees. It's included in Oracle Linux support subscriptions (basic/premier).. have your own nfs server? webserver? want to monitor it and restart it in a cluster no problem? Go for it. Using the exact same software we use for the most mission critical clustered systems. Want to run Oracle Linux virtualized? 1 guest? 10 guests? 20 guests? You only need one subscription per physical server and we don't charge more if you run 1 or more than 1 guest. And how much for all this?

  • Oracle x86 based systems with Premier support for Systems includes all this, both the Oracle Linux subscription and the use of the management features and Oracle CRS
  • Non-Oracle x86 based systems : 2 socket server $499 for all of the above or 4+ socket server $1199 includes all this, both the Oracle Linux subscription and the use of the management features and Oracle CRS
  • This is 24/7 support, not just office hours. Complete 24/7 support and real indemnification not based on subscription fees paid.

    We also have a premier support subscription which adds the ability to use Oracle ksplice technology to apply your kernel security fixes online, while the system is running at full speed, while the applications are running, while the users are working, without downtime. And I mean apply, not just install. The installed fixes are instantly effective at runtime. zero reboots. zero downtime scheduling. zero administrative program/planning management.. Or premier backport support, critical fixes for a specific bug in the specific version of the rpm you run in a mission critical environment. Or lifetime support, now with virtualization more useful than ever before. Virtual hardware can remain the same for much longer than the support lifecycles.

  • Oracle x86 based systems with Premier support for Systems includes all this, both the Oracle Linux subscription and the use of the management features and Oracle ksplice and premier backports and lifetime support
  • Non-Oracle x86 based systems : 2 socket server $1399 for all of the above or 4+ socket server $2299 includes all this, both the Oracle Linux subscription and the use of the management features and Oracle CRS and Oracle ksplice and premier backports and lifetime support
  • See our store for pricing, and our Oracle Linux site for policies and references to the above.

    Ok there you have it. Almost a novel but we just have that much to offer, and wait 'til I get to Oracle VM.. it just gets even better.

    Wednesday May 16, 2012

    More on templates and first boot configuration

    A little while back I wrote something up about building appliances (physical/virtual) using Oracle Linux. We build a whole set of Oracle VM templates and release them on our edelivery site. But some of what we use inside these templates can be more generically applied... Let me explain..

    A few years ago we created a Linux rpm called oraclevm-template-config. This is a very small package that contains a number of scripts that esssentially do the following:

  • install a service oraclevm-template-config that checks at boot time whether it needs to run or not
  • provides a default first boot configuration service for root password, network setup
  • provides a default cleanup script so that you can re-run this on an existing system, at next boot
  • provides a small set of library calls to use those functions in your own scripts
  • allows you to define your own first boot configuration scripts for your own applications to run
  • regenerates ssh host keys
  • all input was done on the local console or through kernel boot line parameters
  • These things are very straightforward and we wrote it specifically to build Oracle VM templates but there was nothing specific to Oracle VM or virtualization there. The newer version of this, which we released along with Oracle VM 3.x is very much tied into Oracle VM. In the new version (3.x) we integrate the first boot scripts and the configuration scripts with the Oracle VM VM api where it's now possible to send remote VM messages to configure the VM. So in order to automate deployment and have very integrated and fancy assemblies or Oracle VM based templates, this is the way to go. If, however, you want something very simple, with lcoal console (physical or virtual), where it can be on a physical server where you want to reboot and reconfigure and install, whether this is a VirtualBox VM or an Oracle VM VM or, for that matter any other Virtualization technology based VM. the older version is quite usable. I tend to use that one for the VirtualBox templates that I create all the time.

    We just published this rpm on The rpm can be directly downloaded for Oracle Linux 5, here. Or when you have a system registered with public-yum or with ULN it is also in the add-ons channel. If you have a yum repo config then just enable the addon channel and you can simple type yum install oraclevm-template-config.

    So what are some of the basics you can do :

    initial install By default when you install template config, you have a service that provides the basic, configure root password, setup local network at what's called firstboot. To enable this to happen at the next reboot (which one can then call firstboot), just simple type /usr/sbin/oraclevm-template --enable. There is a simple default cleanup script which un-configures the network and removes ssh keys : /usr/sbin/oraclevm-template --cleanup. Keep in mind that when you run this script, the network will reset to dhcp and you will lose your network connection if you are in a remote ssh session.

    So once you do this and reboot your server or your VM. The next reboot will automatically start /usr/sbin/oraclevm-template --config and go through the questionnaire.

    The base configuration script is /etc/sysconfig/oraclevm-template. If you enable the service, you will see RUN_TEMPLATE_CONFIG=YES. This means next boot it will run. There is also a log file /var/log/oraclevm-template.log.

    If all you need is a simple way of packaging a VM or configure a physical server that deals with network setup, the above is enough. If you have your own software installed and it can handle IP changes without reconfiguration, then you can create an appliance and once firstboot happens, the configuration is set, network is setup, your application comes up and you're done. However, in some cases you want to run your own scripts and integrate with this template configuration. We also handle that case...

    If you write your own shell script (or really any script) that you install on the system, you can add that script to /etc/sysconfig/oraclevm-template and instead of doing the default firstboot run, it will at firstboot (when you call --enable) execute your script instead. Simply by setting TEMPLATE_CONFIG_SCRIPT=/ Inside that script you can do whatever you want... You can also provide a cleanup script for --cleanup, using

    Keep in mind that we do not automatically run --cleanup, this is a manual step. I would recommend doing the following :

  • /usr/sbin/oraclevm-template --enable
  • /usr/sbin/oraclevm-template --cleanup
  • history -c
  • shutdown -h now
  • We provide a few default function you can include in your own scripts. These functions are defined in /usr/lib/oraclevm-template/functions just source them in your bash script.

    ovm_configure_network [dhcp [hostname]] | [static] | [ip_adddress netmask gateway dns_server hostname]

    You can call this like ovm_configure_network "dhcp" in your script and it will configure your network that way. If you set it to static then there will be a dialog for entering ip, hostname, gateway, netmask, dns, or if you provide that in your function call it will just set the changes without dialog.

    A simple OS cleanup. This resets the network to dhcp and cleans up ssh keys.

    disable iptables

    Shows user input to enter a key in your script execution flow

    ovm_error_message Displays an error message to the screen and also to /var/log/oraclevm-template.log

    ovm_info_message Displays an informational message to the screen and also to /var/log/oraclevm-template.log

    ovm_warn_message Displays a warning message to the screen and also to /var/log/oraclevm-template.log

    All of the above is very simple, yet very powerful. Like I mentioned in the beginning, I use this all the time when creating both VirtualBox and Oracle VM server templates and it makes my life so much easier. People often write these on their own but I wanted to share how to do this so, when you use Oracle Linux, and/or Oracle VM Server or Oracle VM VirtualBox or anything else, and you're building a cool appliance, physical or virtual, use it and you don't even need a support subscription to download it:)...

    Friday May 04, 2012

    building an appliance? physical ? virtual? production quality? use Oracle Linux

    Many companies are building appliances these days. Whether they are virtual appliance (VMs) or physical appliances, they all have an embedded OS inside. Now, I have looked at many of them, in particular virtual appliances and a few things struck me as interesting :

  • many had a random version of Linux included. Nothing bad about any of the distributions used at all, but just a hodgepodge of debian, fedora, opensuse, centos, etc... Almost all were marked "trial" or "demonstration"
  • a number of these appliances had a known commercial version of Linux, were marked trial edition and had restrictions on use (use for 30 days and so on).
  • So what are some problems with these scenarios?

    Well, first of all, even though there is absolutely nothing wrong at all with using community-based and community-maintained distributions (they do a great job), in a commercial setting, or when you build a product that lives on top of it, you need to have the ability to get formal support. Maybe not everyone needs this, but many companies do not have the knowledge (nor should they necessarily) to support the Linux distribution itself and they would want a reasonable formal service level for when something goes wrong, or when to expect something to get released.

    Secondly, in the case of using a commercial Linux distribution, there are distribution restrictions and even access restrictions. Certain Linux distribution vendors that offer formal support subscriptions will not give you access to the binaries without a subscription for support services and they also do not let you re-distribute without special agreements or making lots of modifications (or even restrict modifications).

    Enter Oracle Linux...

    What do you get?

  • freely downloadable ISO images for all the releases and major updates (Oracle Linux 4(4.4, 4.5, 4.6,4.7,4.8), Oracle Linux 5(5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7,5.8),Oracle Linux 6 (6.1, 6.2)). Both source code and binaries.No support subscription needed to download
  • freely re-distributable Both source code and binaries (no contract or support subscription needed to re-distribute, including logos and trademarks)
  • freely available bugfix errata and security errata, released at same time as we release them to customers with support subscriptions Both source code and binaries. In other words, every RPM released through a free public yum repository
  • the ability to purchase a support subscription for what you build, ship, support to your customers without any change to your appliance. No need to have a trial version or demo version using a community-edition Linux and a production version based on a commercial-edition Linux
  • use it in production, go for it, whatever the use case is, use it, if you want support, get a support subscription, if you don't need support, that's fine.
  • the ability to have a support subscription that offers on-line zero-downtime, rebootless security and bugfix updates as part of Oracle Linux Premier support with Oracle Ksplice technology
  • if you run with Centos or Red Hat Enterprise Linux, just replace the rpm list with the Oracle Linux RPMs and you're done.. so easy to switch
  • heavily tested on the biggest servers and biggest IO intensive, process intensive and network intensive workloads out there
  • do this for installs on physical servers, for Virtual Machines (whether this be Oracle VM Virtual Appliances, VMWare Virtual Appliances or something else, it doesn't matter)
  • One Enterprise Linux distribution, for all of the above. We make it easy for you. Grab the code, binaries and source, use it, distribute it, build your environments with it, freely, no contracts needed. Need our help, get a support subscription. Choice, open. Virtual, physical, cloud. Not just obfuscated tar balls. No license or activation key, good consistent SLAs for releasing security updates, well tested,... Run Oracle Linux in-house in test and development environments, run it in production environments, use it for customer systems, distribute it, any or all of the above. One distribution that you can manage across all the use cases. No need to manage different versions even if they're similar, no need to make different distribution choices based on your use case and pay/not pay.

    Thursday Apr 05, 2012

    Some Oracle VM 3 updates

    Today we did another patch set update for Oracle VM 3 (3.0.3-build 240). This can be downloaded from My Oracle Support as patch ID 13614645. There are quite a few updates in here and I highly recommend any Oracle VM 3 customer or user to install this update.

    This patch can be installed on top of Oracle VM 3.0 versions 3.0.2 and 3.0.3. The patch is cumulative for 3.0.3. So if you already installed patch update 1 (3.0.3-150) then this will just be incremental on top of that and brings you to 3.0.3-build 227. There is a readme file which contains the patchlist in the patch info.

    The following patches are released on ULN for Oracle VM server 3.0 :

  • initscripts-8.45.30-2.100.18.el5.x86_64 The inittab file and the /etc/init.d scripts.
  • kernel-ovs- The Linux kernel
  • kernel-ovs-firmware- Firmware files used by the Linux kernel
  • osc-oracle-ocfs2-0.1.0-35.el5.noarch Oracle Storage Connect ocfs2 Plugin
  • osc-plugin-manager-1.2.8-9.el5.3.noarch Oracle Storage Connect Plugin Infrastructure
  • osc-plugin-manager-devel-1.2.8-9.el5.3.noarch Oracle Storage Connect Plugin Development
  • ovs-agent-3.0.3-41.6.x86_64 Agent for Oracle VM
  • xen-4.0.0-81.el5.1.x86_64 Xen is a virtual machine monitor
  • xen-devel-4.0.0-81.el5.1.x86_64 Development libraries for Xen tools
  • xen-tools-4.0.0-81.el5.1.x86_64 Various tooling for the manipulation of Xen instances
  • Errata emails will be sent in the next few days with details on the above updates. Or you will find them here.

    I also did an update of my Oracle VM utilities to 0.4.0. They are also available from My Oracle Support, patch ID 14736239.
    These utils can be unzipped and installed on the server running Oracle VM Manager. Typically in /u01/app/oracle/ovm-manager-3/ovm_utils. There is a set of man pages in /u01/app/oracle/ovm-manager-3/ovm_utils/man/man8. There now are 6 commands :

  • ovm_vmcontrol : VM level operations
  • ovm_servercontrol : server level operations
  • ovm_vmdisks : virtual disk/physical location mapping for VM disks
  • ovm_vmmessage : message passing utility between the manager and the VM tools (in the Oracle VM templates)
  • ovm_repocontrol : repository level operations
  • ovm_poolcontrol : pool level operations

    Some of the new changes :

  • at a pool level, acknowledge events and cascade to servers and virtual machines with outstanding events
  • at a pool level, do a rescan of the storage for fibrechannel/iscsi disks if you add new devices (it does this operation then on every running server)
  • at a repository level, fixup a device if it had a failed create repository
  • at a repository level, refresh the repository and this will update the free space in the UI for ocfs2 repositories
  • at a server level, acknowledge server events and cascade to virtual machines if needed
  • at a VM level, acknowledge VM events
  • at a VM level, bind vcpus to cores with vcpuset/vcpuget

    Please see the man pages and remember that these tools are just written As Is - no SRs... (per the documentation) Hopefully they are useful.

  • Thursday Mar 29, 2012

    4.8M wasn't enough so we went for 5.055M tpmc with Unbreakable Enterprise Kernel r2 :-)

    We released a new set of benchmarks today. One is an updated tpc-c from a few months ago where we had just over 4.8M tpmc at $0.98 and we just updated it to go to 5.05M and $0.89. The other one is related to Java Middleware performance. You can find the press release here.

    Now, I don't want to talk about the actual relevance of the benchmark numbers, as I am not in the benchmark team. I want to talk about why these numbers and these efforts, unrelated to what they mean to your workload, matter to customers. The actual benchmark effort is a very big, long, expensive undertaking where many groups work together as a big virtual team. Having the virtual team be within a single company of course helps tremendously... We already start with a very big server setup with tons of storage, many disks, lots of ram, lots of cpu's, cores, threads, large database setups. Getting the whole setup going to start tuning, by itself, is no easy task, but then the real fun starts with tuning the system for optimal performance -and- stability. A benchmark is not just revving an engine at high rpm, it's actually hitting the circuit. The tests require long runs, require surviving availability tests, such as surviving crashes -and- recovery under load.

    In the TPC-C example, the x4800 system had 4TB ram, 160 threads (8 sockets, hyperthreaded, 10 cores/socket), tons of storage attached, tons of luns visible to the OS. flash storage, non flash storage... many things at high scale that all have to be perfectly synchronized.

    During this process, we find bugs, we fix bugs, we find performance issues, we fix performance issues, we find interesting potential features to investigate for the future, we start new development projects for future releases and all this goes back into the products. As more and more customers, for Oracle Linux, are running larger and larger, faster and faster, more mission critical, higher available databases..., these things are just absolutely critical. Unrelated to what anyone's specific opinion is about tpc-c or tpc-h or specjenterprise etc, there is a ton of effort that the customer benefits from. All this work makes Oracle Linux and/or Oracle Solaris better platforms. Whether it's faster, more stable, more scalable, more resilient. It helps.

    Another point that I always like to re-iterate around UEK and UEK2 : we have our kernel source git repository online. Complete changelog of the mainline kernel, and our changes, easy to pull, easy to dissect, easy to know what went in when, why and where. No need to go log into a website and manually click through pages to hopefully discover changes or patches. No need to untar 2 tar balls and run a diff.

    Monday Mar 26, 2012

    Using EPEL repos with Oracle Linux

    There's a Fedora project called EPEL which hosts a set of additional packages that can be installed on top of various distributions such as Red Hat Enterprise Linux, CentOS, Scientific Linux and of course also Oracle Linux. These packages are not distributed by the distribution vendor and as such also not supported by the vendors (including Oracle) however for users that want to pick up some extras that are useful, it's very easy to do this.

    All you need to do is download the EPEL RPM from the website, install it on Oracle Linux 5 or Oracle Linux 6 and run yum install or yum search to get the packages.

    example : 
    # wget
    # rpm -ivh epel-release-6-5.noarch.rpm
    # yum repolist
    Loaded plugins: refresh-packagekit, rhnplugin
    repo id                repo name                                          status
    epel                   Extra Packages for Enterprise Linux 6 - x86_64      7,124
    The folks that build these repositories are doing a great job at adding very useful packages. They are free, but also unsupported of course.

    Thursday Mar 22, 2012

    Setting up Oracle Linux 6 with public-yum for all updates

    I just wanted to give you a quick example on how to get started with Oracle Linux 6 and start using the updates we published on

  • Download Oracle Linux (without the requirement of a support subscription) from

  • Install Oracle Linux from the ISO or DVD image

  • Log in as user root
  • Download the yum repo file from

    # cd /etc/yum.repos.d
    # wget

  • If you want, you can edit the repo file and enable other repositories, I enabled [ol6_UEK_latest] by just setting enabled=1 in the file with a text editor.

  • Run yum repolist to show the registered channels and you see we are including everything including the latest published RPMs.

    Now you can just run yum update and any time we release new security errata or bugfix errata for OL6, they will be posted and you will automatically get them. It's very easy, very convenient and actually very cool. We do a lot more than just build OL RPMs and distribute them, we have a very comprehensive test farm where we test the packages extensively.

  • Lots of goodies

    We just issued a press release with a number of very good updates for everyone

    There are a few things of importance :

    1) As of right now, Oracle Linux 6 with the Unbreakable Kernel is certified with a number of Oracle products such as Oracle Database 11gR2 and Oracle Fusion Middleware. The certification pages in the Oracle Support portal will be updated with the latest certification status for the various products.

    As always we have gone through a long period of very comprehensive testing and validation to ensure that the whole stack works really well together, with very large database workloads, middleware application workloads etc.

    2) Standard certification efforts for Oracle Linux 6 with the Red Hat Compatible Kernel are in progress and we expect that to be completed in the next few months. Because of the compatibility between OL6 and RHEL6 we can then also state certification for RHEL6.

    3) Oracle Linux binaries (and of course source code) have been free for download -and- use (including production, not just trial periods) since day one. You can freely redistribute the binaries, unlike many other Linux vendors where you need to pay a support subscription to even get access to the binaries. We offered both the base distribution release DVDs (OL4, OL5, OL6) and the update releases, such as 5.1, 5.2 etc. this way. Today, in this announcement, we also started to make available the bugfix and security updates released in between these update releases. So the errata streams (both binary and source code) for OL4, 5 and 6 are now free for download and use from This includes uek and uek2.

    The nice thing is, if you want a complete up to date system without support, use this, if you then need support, get a support subscription. Simple, convenient, effective. We have great SLA's in producing our update streams, consistency in release timing and testing of all the components.

    Have at it!


    Wim Coekaerts is the Senior Vice President of Linux and Virtualization Engineering for Oracle. He is responsible for Oracle's complete desktop to data center virtualization product line and the Oracle Linux support program.

    You can follow him on Twitter at @wimcoekaerts


    « April 2014