Tuesday Aug 27, 2013

Single Instance/RAC Oracle VM templates update

Superstar Saar just released a new set of Oracle VM templates. We (Oracle) just released 2 patch sets for the Oracle RDBMS - and (x86 and x86_64)

Simultaneously, Saar updated his Oracle VM templates to include these latest patchsets as well for both architectures (x86 and x86_64).

  • with OL5
  • with OL6
  • with OL5
  • with OL6
  • These templates can be deployed on Oracle VM using the DeployCluster tool, all you need to do is create a very simple textfile with the parameters.

    All templates default to UEK2 2.6.39-400. The templates can be used to create Single Instance, Single Instance with HA (Oracle Restart) and Oracle RAC databases.

    The options vary from ASM, NFS, OCFS2 for db files, local filesystem, no DB, Clusterware only etc.

    Full stack, download, deploy. Production RDBMS code, Production Oracle Linux.


    Simple Sample script:

    # cat netconfig.ini 
    DOMAINNAME=wimmekes.net  # May be blank
    DNSIP=  # Starting from 2013 Templates allows multi value
    CLONE_SINGLEINSTANCE=yes  # Setup Single Instance

    and then # deploycluster -u admin -p mypassword -H localhost -M mydbvm1 -> done

    Wednesday Aug 07, 2013

    Oracle VM templates for Database 12c both single instance and rac

    Today we made available a few new Oracle VM templates on edelivery. A set of VM templates for database 12c and another set for database 11g

    You can find more information on the otn pages here.

    A very important new feature added is the ability to deploy single instance database. In the past the database templates were focused on RAC deployments (Real Application Cluster) but because of popular demand, we also added support for Single Instance. With Single Instance you can really create a new VM with the database up and running in a matter of a few (very few) minutes, and with a very simple config file.

    Example config file for single instance :

    $ cat netconfig.ini 
    DOMAINNAME=wimmekes.net  # May be blank
    DNSIP=  # Starting from 2013 Templates allows multi value
    CLONE_SINGLEINSTANCE=yes  # Setup Single Instance

    That's literally it. You don't need to do anything other than run a few Oracle VM CLI or UI commands and run deploycluster and you're all set. After a few minutes, the VM will be pingable and you can run sqlplus against the database running inside the VM.

    If you use the CLI, here is a sample workflow :

  • import the template
  • - importtemplate repository name=[reponame] url=[http://myurl/template.tbz] server=[servername]
  • create vm from template
  • - clone vm name=[templatename] destType=Vm destName=[vmname] serverpool=[serverpoolname]
  • Create new vnic
  • - create vnic name=[macaddress] network=[network] (list network, will show you the various networks)
  • remove old vnics (you could rename one or alter one but to simplify I just remove the old vnics of the cloned vm and add the newly created
  • - remove vnic name=[macaddr] from vm name=[vmname]
    - show vm name=[vmname] to see the attached vnics

    And that's it, now you can use that netconfig.ini example, edit it for your environment and run deploycluster:

    On top of single instance, the templates also expose or give you the ability to easily configure and enable many of the new rdbms 12c functionality :

    - Oracle Flex Cluster and/or Flex ASM, Hub/Leaf nodes
    - Container Database with x number of pluggable databases
    - Database Express
    - ACFS filesystem
    - Oracle Restart (single instance database with HA)
    - local or shared filesystem installs, including OCFS2 and ACFS
    - Admin Managed or Policy managed database creation with serverpools
    - OS kernel updated to the latest uek 2 version 2.6.39-400

    And all of the above are simple parameters in the config files. This can be 100% automated, 100% reproducible and you don't need to know how to configure them all yourself. As always, high quality work by Saar Maoz.

    Production ready, not trial, not using a random OS, all ready to go. Production-ready virtual appliances.

    Sunday Apr 21, 2013

    Importing Oracle VM templates through a proxy

    I am working on a little tool that makes it easy to import an Oracle VM template in a more automated fashion, using python's built-in SimpleHTTPServer. While working on this, I realized that in many environments the Oracle VM Servers might be in an isolated network so that they don't have direct access to the intranet. We're talking about the management network here.

    One simple way around this, is to take one server that's on the same network as the Oracle VM Server's management network, for instance, the Oracle VM Manager system... and install something like TinyProxy on that machine. Then, use that servername as the proxy in Oracle VM Manager when you import a VM, VM Template or VM Assembly.

    TinyProxy can be found in the EPEL repository (http://fedoraproject.org/wiki/EPEL). The tinyproxy RPM will install without issue on Oracle Linux. It is very easy/simple to configure and this can be a good workaround or solution to make it easy to import templates or VMs while the servers are on a more isolated network.

    Sunday Jan 06, 2013

    oracle vm template config script example

    The programmatic way to extend Oracle VM Template Configure is to build your own module.

    To write your own module, you have to build an RPM that contains a configure script in a specific format, let's go through the steps to do this.

    Oracle VM template configure works very similar to the init.d and chkconfig script model. For template config we have the /etc/template.d directory, all the scripts go into /etc/template.d/scripts. Then symlinks are made to other subdirectories based on the type of target the scripts provide. At this point we handle configure and cleanup. When a script/module gets added using ovm-chkconfig, the header of the script is read to verify the name, priority and targets and then a symlink is made to the corresponding subdirectories under /etc/template.d.

    As an example, you have /etc/init.d/sshd which is the main sshd initscript and when sshd is enabled you will find a symlink in /etc/rc3.d/S55sshd to /etc/init.d/sshd. These symlinks are created by chkconfig when you enable or disable a service. The same thing goes for Oracle VM template config and the content of /etc/template.d/scripts. You will see /etc/template.d/scripts/ssh and since ssh (on my system) is enabled for the configure target, I have a symlink to /etc/template.d/configure.d/70ssh.

    Like init.d, the digit in front of the script name specifies the priority at which it should be run.

    The most important and complex part is writing your own script for your own application. Our scripts are in python, theoretically you could write it in a different language, as long as the input, output and argument handling remains the same. The examples here will all be in python. Each script has 2 main part : (1) the script header which contains information like script name, targets, priorities and description and (2) the actual script which has to handle a small set of parameters. You can take a look at the existing scripts for examples.

    (1) script header
    Aside from a copyright header that suits your needs, the script headers require a very specific comment block, here is an example :

    # name: network
    # configure: 50
    # cleanup: 50
    # description: Script to configure template network.

    You have to use the exact same format. Provide your own script name, which will be used when calling ovm-chkconfig, the targets (right now we implement configure and cleanup) and the priority for your script. The priority will specify in what order the scripts get executed. You do not have to implement all targets, if you have a configure target but not cleanup, that is OK, same goes for cleanup versus configure. It is up to you. The configure target gets called when a first boot/initial start of the VM happens, cleanup happens when you manually initiate a cleanup in your VM or when you want to restore the VM to its original state.

    # name: [script name]
    # [target]: [priority]
    # [target]: [priority]
    # description: a description and can
    #   cross multiple lines.

    Now for the body of the script. Basically the main requirement is that it accepts a [target] parameter. Let's say we have script called foo that needs to be run at configure time, then the script (/etc/template.d/scripts) will have to accept and understand handling the parameter configure. If you also want to call it for cleanup, then it has to handle cleanup. You can have your script handle any other arguments, this is totally up to you, they are optional for our purposes. There is one optional parameter which is useful to implement and this is -e or --enumerate. ovm-template-config uses this to be able to enumerate the parameters for a target for your script.

    Here is the firewall example:

    # ovm-template-config --human-readable --enumerate configure --script firewall
      [{u'description': u'Whether to enable network firewall: True or False.',
        u'hidden': True,
        u'key': u'com.oracle.linux.network.firewall'}])]
    and if you run the script manually :

    # ./firewall configure -e
    [{"hidden": true, "description": "Whether to enable network firewall: True or False.", "key": "com.oracle.linux.network.firewall"}]

    In other words, the firewall script lists the parameters it expects when run as a configure target.

    Now here is an example of the script body, in python. It implements the configure and cleanup target and handles the enumerate argument. Part of the magic is handled in templateconfig.cli.

        import json
    except ImportError:
        import simplejson as json
    from templateconfig.cli import main
    def do_enumerate(target):
        param = []
        if target == 'configure':
            param += []
        elif target == 'cleanup':
            param += []
        return json.dumps(param)
    def do_configure(param):
        param = json.loads(param)
        return json.dumps(param)
    def do_unconfigure(param):
        param = json.loads(param)
        return json.dumps(param)
    def do_cleanup(param):
        param = json.loads(param)
        return json.dumps(param)
    if __name__ == '__main__':
        main(do_enumerate, {'configure': do_configure, 'cleanup': do_cleanup})

    So now you can fill this out with your own parameters and code. Again taking the firewall script as an example, to add expected keys :

    def do_enumerate(target):
        param = []
        if target == 'configure':
            param += [{'key': 'com.oracle.linux.network.firewall',
                       'description': 'Whether to enable network firewall: True or False.',
                       'hidden': True}]
        return json.dumps(param)

    The above shows that this script expect the key com.oracle.linux.firewall to be set and what the default is, along with a description. Add this for each key/value pair that you expect for your script and then afterwards it is easy to understand what the input to your script needs to be, again by running ovm-template-config.

    To execute actions at configure time, based on values set, here's a do_configure() example:

    def do_configure(param):
        param = json.loads(param)
        firewall = param.get('com.oracle.linux.network.firewall')
        if firewall == 'True':
            shell_cmd('service iptables start')
            shell_cmd('service ip6tables start')
            shell_cmd('chkconfig --level 2345 iptables on')
            shell_cmd('chkconfig --level 2345 ip6tables on')
        elif firewall == 'False':
            shell_cmd('service iptables stop')
            shell_cmd('service ip6tables stop')
            shell_cmd('chkconfig --level 2345 iptables off')
            shell_cmd('chkconfig --level 2345 ip6tables off')
        return json.dumps(param)

    When the script is called, you can use param.get() to retrieve key/value variables and then just make use of it. Just like in the firewall example, you can do whatever you want, call out other commands, add more python code, it's up to you...

    It is also possible to alter keys or add new keys which then get sent back. So if you want your script to communicate values back which can be retrieved later through the manager API, for instance with ovm_vmmessage -q, you can simply to this :

    param['key'] = 'some value'

    Key can be an existing key, or a new one.

    And that's really it... for the script. Next up is packaging.

    In order to install and configure these template configure scripts, they have to be packaged in an RPM, with a specific naming convention. Package the script(s), there can be more than one, as ovm-template-config-[scriptname]. Ideally in the post install of the RPM you want to add the script automatically. Execute # /usr/sbin/ovm-chkconfig --add [scriptname]. When de-installing a script/RPM, remove it at un-install time, # /usr/sbin/ovm-chkconfig --del [scriptname].

    Here is an example of an RPM spec file that can be used:

    Name: ovm-template-config-example
    Version: 3.0
    Release: 1%{?dist}
    Summary: Oracle VM template example configuration script.
    Group: Applications/System
    License: GPL
    URL: http://www.oracle.com/virtualization
    Source0: %{name}-%{version}.tar.gz
    BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)
    BuildArch: noarch
    Requires: ovm-template-config
    Oracle VM template example configuration script.
    %setup -q
    rm -rf $RPM_BUILD_ROOT
    make install DESTDIR=$RPM_BUILD_ROOT
    rm -rf $RPM_BUILD_ROOT
    if [ $1 = 1 ]; then
        /usr/sbin/ovm-chkconfig --add example
    if [ $1 = 0 ]; then
        /usr/sbin/ovm-chkconfig --del example
    * Tue Mar 22 2011 Zhigang Wang  - 3.0-1
    - Initial build.

    Modify the content to your liking, change the name example to your script name, and add whatever else dependencies you might have or whatever files need to be bundled along with this. If you want to bundle executables or scripts that live in other locations, that's allowed. As you can see from the spec file, it automatically called ovm-chkconfig --add and --del at post-install and pre-uninstall time of the RPM.

    In order to create RPMs, you have to install rpmbuild, # yum install rpm-build.

    To make it easy, here's a Makefile you can use and help automate all of this :

    	@echo 'Commonly used make targets:'
    	@echo '  install    - install program'
    	@echo '  dist       - create a source tarball'
    	@echo '  rpm        - build RPM packages'
    	@echo '  clean      - remove files created by other targets'
    dist: clean
    	mkdir $(PACKAGE)-$(VERSION)
    	tar -cSp --to-stdout --exclude .svn --exclude .hg --exclude .hgignore \
    	    --exclude $(PACKAGE)-$(VERSION) * | tar -x -C $(PACKAGE)-$(VERSION)
    	tar -czSpf $(PACKAGE)-$(VERSION).tar.gz $(PACKAGE)-$(VERSION)
    	rm -rf $(PACKAGE)-$(VERSION)
    	install -D example $(DESTDIR)/etc/template.d/scripts/example
    rpm: dist
    	rpmbuild -ta $(PACKAGE)-$(VERSION).tar.gz
    	rm -fr $(PACKAGE)-$(VERSION)
    	find . -name '*.py[cdo]' -exec rm -f '{}' ';'
    	rm -f *.tar.gz
    .PHONY: dist install rpm clean

    Create a directory, copy over your script, the spec file and this Makefile. Run # make dist, to create a src tarball of your code and then # make rpm. This will generate an RPM in the RPMS/noarch directory. For instance: /root/rpmbuild/RPMS/noarch/ovm-template-config-test-3.0-1.el6.noarch.rpm

    Next you can take this RPM and install it on a target system.

    # rpm -ivh  /root/rpmbuild/RPMS/noarch/ovm-template-config-test-3.0-1.el6.noarch.rpm
    Preparing...                ########################################### [100%]
       1:ovm-template-config-tes########################################### [100%]

    And as you can see, it's added to the ovm-chkconfig list :

    # ovm-chkconfig --list|grep testtest                 on:75       
    off         off         on:25       off         off         off         off        

    One point of caution : the configure scripts get executed very early on in the bootstage. ovmd is executed as S00ovmd. This is well before many other services are (1) configured, (2) running. So if your product requires services like network connectivity or others to be up and running, then you have to split up the configuration into two parts. First, use the above to gather configuration data remotely, store it in a way that you can use it, and then add your own /etc/init.d scripts which can take this data afterwards. So you can have your own init scripts executed at a late stage when the services you depend on are available.

    That's really all there is to it. Thanks to Zhigang for example code I have used here.


    Wim Coekaerts is the Senior Vice President of Linux and Virtualization Engineering for Oracle. He is responsible for Oracle's complete desktop to data center virtualization product line and the Oracle Linux support program.

    You can follow him on Twitter at @wimcoekaerts


    « April 2014