Tuesday May 29, 2012

New Oracles VM RAC template with support for oracle vm 3 built-in

The RAC team did it again (thanks Saar!) - another awesome set of Oracle VM templates published and uploaded to My Oracle Support.

You can find the main page here.

What's special about the latest version of DeployCluster is that it integrates tightly with Oracle VM 3 manager. It basically is an Oracle VM frontend that helps start VMs, pass arguments down automatically and there is absolutely no need to log into the Oracle VM servers or the guests. Once it completes, you have an entire Oracle RAC database setup ready to go.

Here's a short summary of the steps :

  • Set up an Oracle VM 3 server pool
  • Download the Oracle VM RAC template from oracle.com
  • Import the template into Oracle VM using Oracle VM Manager repository -> import
  • Create a public and private network in Oracle VM Manager in the network tab
  • Configure the template with the right public and private virtual networks
  • Create a set of shared disks (physical or virtual) to assign to the VMs you want to create (for ASM/at least 5)
  • Clone a set of VMs from the template (as many RAC nodes as you plan to configure)
  • With Oracle VM 3.1 you can clone with a number so one clone command for, say 8 VMs is easy.
  • Assign the shared devices/disks to the cloned VMs
  • Create a netconfig.ini file on your manager node or a client where you plan to run DeployCluster
  • This little text file just contains the IP addresses, hostnames etc for your cluster. It is a very simple small textfile.
  • Run deploycluster.py with the VM names as argument
  • Done.

    At this point, the tool will connect to Oracle VM Manager, start the VMs and configure each one,

  • Configure the OS (Oracle Linux)
  • Configure the disks with ASM
  • Configure the clusterware (CRS)
  • Configure ASM
  • Create database instances on each node.
  • Now you are ready to log in, and use your x node database cluster. x No need to download various products from various websites, click on trial licenses for the OS, go to a Virtual Machine store with sample and test versions only - this is production ready and supported.

    Software. Complete.

    example netconfig.ini :

    # Node specific information
    NODE1=racnode1
    NODE1VIP=racnode1-vip
    NODE1PRIV=racnode1-priv
    NODE1IP=192.168.1.2
    NODE1VIPIP=192.168.1.22
    NODE1PRIVIP=10.0.0.22
    NODE2=racnode2
    NODE2VIP=racnode2-vip
    NODE2PRIV=racnode2-priv
    NODE2IP=192.168.1.3
    NODE2VIPIP=192.168.1.23
    NODE2PRIVIP=10.0.0.23
    # Common data
    PUBADAP=eth0
    PUBMASK=255.255.255.0
    PUBGW=192.168.1.1
    PRIVADAP=eth1
    PRIVMASK=255.255.255.0
    RACCLUSTERNAME=raccluster
    DOMAINNAME=mydomain.com
    DNSIP=
    # Device used to transfer network information to second node
    # in interview mode
    NETCONFIG_DEV=/dev/xvdc
    # 11gR2 specific data
    SCANNAME=racnode12-scan
    SCANIP=192.168.1.50
    

    Saturday May 26, 2012

    example of transcendent memory and oracle databases

    I did some tests with tmem using an Oracle Database 11gR2 and swingbench setup. You can see a graph below. Let me try to explain what this means.

    Using Oracle VM 3 with some changes booting dom0 (additional parameters at the boot prompt) and with UEK2 as a guest kernel in my VM, I can make use of autoballooning. What you see in the graph below is very simple : it's a timeline (horizontal)of how much actual memory the VM is using/needing. I created 3 16GB VMs that I wanted to run on a 36GB Oracle VM server (so more VM memory than we have physically available in the server). When I start a 16GB VM (vertical) the Linux guest immediately balloons down to about 700Mb in size. It automatically releases pages to the hypervisor that are not needed, it's free/idle memory otherwise. Then I start a database with a 4GB SGA, as you can see, the second I start the DB, the VM grows to just over 4GB in size. Then I start swingbench runs, 25, 50, 100, 500, 1000 users. Every time such a run starts, you can see memory use/grab go up, when swingbench stops it goes down. In the end after the last run with 1000 users I also shut down the database instance and memory drops all the way to 700Mb.

    I ran 3 guests with swingbench and the database in each and through dynamic ballooning and the guests cooperatively working with the hypervisor, I was able to start all 3 16GB VMs and there was no performance impact. When there was free memory in the hypervisor, cleancache kicked in and guests made use of those pages, including deduping and compression of the pages.

    If you want to play with this yourself, you can run this command in dom0 to get decent statistics out of the setup : xm tmem-list --long --all | /usr/sbin/xen-tmem-list-parse. It will show you the compression ratio, the cache ratios etc. I used those statistics to generate the chart below. This yet another example of how, when one can control both the hypervisor and the guest operating system and have things work together, you get better and more interesting results than just black box VM management.

    tmem>

    from the research department - ramster in Linux 3.4 staging drivers

    Some developers on my team have spent quite a few years on doing research and development on new ways of doing memory management better across servers, both physical and virtual. You often hear me talk about cooperative memory management in the context of Oracle VM. I believe it is important to be able to do memory overcommit the right way and have both the guest OS and the hypervisor know what's going on, instead of having the hypervisor do things behind the guest's back. Transparent memory overcommit causes unpredictable behavior for applications and while this might be of little concern when consolidating lots of small VMs that aren't doing much interesting. When you have applications that are, by nature, memory intensive, such as an Oracle database or any sort of production environment, then there need to be better ways to do this. Better, here, meaning -> more efficient, more predictable and more cooperative between host and guest.

    It's a long road to getting all these stars aligned but we are very close now in terms of Linux kernel features to help with this. Over the last few years Dan Magenheimer has been working hard on what's called transcendent memory, a collection of features that include cleancache, frontswap and ramster. Now with Linux 3.4, we have tmem, cleancache and ramster in kernel upstream. Frontswap is still pending but we're working on it.

    As a refresher, you can read more about cleancache in this writeup I did just bout a year ago. Frontswap is more tricky, because here we can use transcendent memory as a swap device. Pages swapped out cannot just disappear, they have to be persistent until the OS decides to no longer need them. In other words, with cleancache we provide extra memory(extra cache) but with the understanding that that extra memory might just disappear. With frontswap, we provide swap space through the same mechanism, with the understanding that it's not going to just disappear. One place where frontswap is different from regular swap, is its size. A swapdevice is pre-created with a specific size, a file acting as a swap device or a physical/logical volume. They do not grow or shrink dynamically. frontswap is more dynamic. The advantage of frontswap is that you can use ram on a remote server or in the hypervisor as very fast storage. It can also do compression of memory pages using zcache.

    If you want to play with ramster, Dan wrote up a simple how-to.
    Dan also wrote up a presentation which you can find here.

    Looking forward to frontswap getting into the kernel and many of these features get into customers hands through the Unbreakable Enterprise Kernel and as products evolve and features get management also into Oracle VM.

    Thursday May 24, 2012

    The case for Oracle VM

    As I mentioned a few days ago, at a customer visit earlier this month we were discussing Oracle VM and Oracle Linux advantages and the customer was surprised to hear how many features we include in our offerings. My previous write up was about Oracle Linux, this one is about Oracle VM.

    In many ways, the Oracle VM solution has similar advantages. For us to deliver a solution and for our customers and partners to have a stack that's built to work together across products. Because Oracle VM subscriptions include the right to use a number of products, such as Oracle Enterprise Manager 12c and Oracle Enterprise Manager OpsCenter, we end up with an impressive, all-in set of features. Often times however this is not obvious to our users or evaluators and it is important to look at the total package and not just sub components. What I am going to describe is included in our Oracle VM subscription.

    First of all, pricing... our virtualization and infrastructure management solution has a very simple model :

  • Oracle VM premier limited (for 1 or 2 socket servers) : $599 annual support subscription per server
  • It doesn't matter how many VMs you run, it doesn't matter how much RAM is in the box, it doesn't matter how many cores or threads the sockets have. 1 or 2 sockets in the box. That's the price. Annual support subscription fee. No license. Including all the products I will be detailing below.
    So if you have a 2 socket server with 256GB RAM... $599. case closed. 24/7 support.

  • Oracle VM premier (for larger than 2 socket servers) : $1199 annual support subscription per server
  • Same deal. If you have a 4 socket server, or even an 8 socket server with 4 TB of RAM... $1199 subscription. period. No virtual RAM, no per 2 sockets, no multiple options, no starter pack for 2 servers... Simple pricing. 24/7 support.

    Also, as with Oracle Linux, if you purchase an Oracle Server with the Premier support for Systems, it also includes everything I am talking about without the need for extra per server subscriptions.

    Oracle VM support subscriptions for both premier and premier limited comes with the following products and right to use :

  • Oracle VM server
  • Oracle VM server is the product that is installed on each server. It contains the hypervisor and the management console/agent to talk to Oracle VM Manager. This is effectively what you get a support subscription for. A tiny installation of about 200Mb worth of RPMs on a local hard drive or flash drive.

  • Oracle VM Manager
  • Oracle VM Manager is our centralized management console. It is an Oracle WebLogic Server 11g application that uses an Oracle Database as data repository. You typically only need 1 instance to manage a very large set of servers and virtual machines. Oracle VM Manager is the management hub for Oracle VM. It has a web-based admin interface and the management product can be installed on a Linux server, typically outside the Oracle VM server pools. We do not charge extra for this, the use and support of Oracle VM Manager is included in the server subscriptions. More details further down...

  • Oracle Enterprise Manager 12c Cloud Control
  • OEM 12c is our complete management product which can manage from application to disk. It contains management for our various applications, database, middleware, operating systems, virtualization layer, it offers configuration management, lifecycle management, physical and virtual environments, cloud portal and so on. The product consists of many management packs, the components that manage Oracle Linux and Oracle VM (cloud portal, private cloud, self-service portal, VM management, IaaS layer services, cloud API....) are included with Oracle VM subscriptions. These are not extra licensed components, they're part of the package!

  • Oracle Enterprise Manager Ops Center 12c
  • EMOC is a very powerful management product that's focused on infrastructure management, our Oracle systems (X86 and SPARC) (firmware, switches, hardware fault analysis, My Oracle Support integration, bios configuration etc...), our Operating Systems (Oracle Solaris, Oracle Linux), Virtualization products (Solaris Containers, Oracle VM SPARC, Oracle VM x86) and a connector into OEM12c to tie them together. Again, when you have an Oracle VM server subscription, you have the right to use EMOC as part of the stack included with its feature set It also contains an IaaS layer and a cloud API and cli.

    Both EM12c and EMOC connect to Oracle VM Manager instances behind the scenes. They can discover or sync up the information from an existing Oracle VM Manager/Server configuration and you can literally switch back and forth. So you start out installing a bunch of Oracle VM servers, install an Oracle VM manager instance, discover the servers, then install EM12c and/or EMOC, discover Oracle VM Manager and now you have access to their feature sets on top. You can still connect to / log into Oracle VM Manager and so on.

    So when we say "Oracle VM" and when you compare "Oracle VM" without various competitors, we really provide you with all of the above, and our feature checklist and offering is all of the above. Often times people just use Oracle VM Manager/Server without also including the use of EMOC and/or EM12c. Our competitors have many products that are all separately licensed pieces and bits that have to be installed together. Look at the solution bundle. We have no per VM, per 96GB, per socket, per feature pricing... remember, $599 and $1199 subscription.

    Here's a few features that are part of all this :

  • Cloud Infrastructure administrator portal
  • Set up and manage the lowerlevel infrastructure, hypervisor, provisioning of storage...

  • Self service administrator
  • Set up higher-level services, users, roles, access boundaries (assign roles and users to zones)
    Set up charge back plans, publish software components (VMs, full application stacks for self service users)

  • self service user portal
  • end user, developer can log in and see their assigned infrastructure quota and create application setups based on software component from the libraries

  • Consolidation planning physical server assets and P2P or P2V advisories
  • Since EM12c has a view of physical and virtual servers and applications inside, it can provide very detailed planning of which systems are good targets to consolidate, because it has an application view and a complete end to end stack overview, it is much more detailed and capable than simple VM analysis.

  • Zone management (multiple pools of servers into a larger grouping)
  • Depending on administrators needs its possible to configure larger zones (pools) of systems as units

  • Storage management through Oracle VM Storage Connect
  • We can manage iscsi, fibre channel, NFS servers from our management console. It is possible to just provision large chunks of storage and provide credentials to Oracle VM Manager and we can handle the details behind the scenes. Add a new lun? no problem, we will do it. EM12c and EMOC expose this functionality and connect through Oracle VM Manager to the storage.

  • Oracle VM assemblies and templates
  • pre-created virtual machines that include complete Oracle product installations, database, siebel, peoplesoft, ebusiness suite etc... these can eb downloaded from the Oracle website and uploaded into Oracle VM Manager or the EM12c software library for easy deployment of complete solution stacks.

    With assemblies we provide complete multi-tiered environments, including scale-out options. Need more webservers, need more database nodes? We can add/remove them on the fly, based on the application and business need, not just based on what the blackbox VM is doing.

  • Chargeback
  • It is possible to create charge back plans, either your own or using existing models. Whether it's basic charge back for virtualization or depending on management packs and oracle products up the stack for other products as well.

  • build, test, deploy
  • Because of the templates and configuration management for applications support, we can let developers build and create solutions which then get uploaded into the software library for use in product deployments. Since EM12c understands the entire product stack, application to disk, we can automate and reproduce not just simple VM deployments but include application configuration on top as well. Creating a VM with an OS is easy. Deploying a VM with an OS is easy. Deploying and configuring an entire software stack with various applications and database(s) is error prone and can take a lot of time. We can automate it for you, reproduce it, exactly the same steps, every time. Top to bottom.

  • Various abstraction layers
  • Through EM12c, EMOC and Oracle VM Manager you have an easy to use, graphical user interface.
    Through EM12c and EMOC you have access to a complete cloud API.
    EMOC offers an IaaS RESTful interface, a JAVA API library and a complete CLI.
    EM12c offers an IaaS RESTful interface

  • complete monitoring and management
  • EM12c manages and monitors every target in the stack, the applications, the operating systems, the VMs, the hypervisors. You can define jobs, metrics, thresholds, corrective actions. Because we collect information across the entire stack, you can set up auditing rules, you can compete systems top to bottom, you can do change management across the entire infrastructure, physical and virtual. To make it easy, it is possible to create groups of systems to manage as a unit. Configuration management is now more critical than ever. In a cloud infrastructure the number of VMs, and as such, the number of deployed applications, grows exponentially. We keep track of every aspect through a single source/database :) of truth.

  • Roles, LDAP directory services
  • Complete integration of LDAP with EM12c and EMOC. The ability to define fine grained access to all these components. Storage, servers, functions.
  • integration with My Oracle Support
  • Patch advisories, service request management, downloading new templates into the software library

  • standard virtualization features
  • Not to forget the fact that this obviously also includes Live migration, Dynamic Power Management (DPM), Dynamic Resource Management(DRS), storage management, network management(setting up bonding, creating virtual networks, vlan support, built-in HA (VM failover, server failover), cpu pinning, clustered filesystem for storage repositories, direct lun/storage access,...

    Installs on the largest X86 servers available. 4TB RAM, 160 threads, virtual machines up to 2TB of ram and up to 128 virtual CPUs or as many virtual CPUs the guest operating system can handle. Support for Linux, Oracle Solaris, Windows (including Windows logo signed drivers).

    So this is the Oracle VM offering, this is a summary (and I am not even trying to be complete here) of our features. Need an API, need a cli, need a UI, need a cloud portal, need monitoring or management, need roles and responsibilities, need centralized account management, we have it all. So when you compare Oracle VM with our competition out there... compare the included feature set in the subscription.

    A few useful links :

  • EMOC API/CLI reference
  • EM12c API
  • Oracle Virtualization page
  • more Oracle Linux options

    A few days ago I wrote a summary of what you get with Oracle Linux. Because there are so many things, I forgot a few... and this is just a continuation of that previous entry.

    There are 2 more features that I wanted to present :

  • Oracle Database Smart Flash Cache
  • DBSFC is a very cool feature that's available for both Oracle Solaris and Oracle Linux customers with the 11gR2 database. In summary, it allows you to basically extend the Oracle Buffer Cache in memory (SGA) using secondary flash based storage. This flash based storage can be presented to the database through a file on a filesystem on flash storage, a raw disk device (flash-based) or through adding flash storage to Oracle ASM and creating a region inside ASM.

    For the most part this feature is going to help with read-only/read-mostly workloads because DBSFC is a read-only cache extension. It contains clean blocks that got kicked out of the buffercache/sga and now first get placed in this extended cache. A subsequent read can then be from this fast storage instead of from the originating datafiles. When a block gets modified, it's modified in the standard database buffer cache, written to disk and copied over into the flash cache.

    The white paper referenced above provides the details on how to use it and how to configure it in an Oracle Linux environment. You simply specify DB_FLASH_CACHE_FILE and DB_FLASH_CACHE_SIZE and that's it. Any Oracle Database customer using Oracle Linux can make use of this.

  • Assigning a control cgroup (cgroup) in Oracle Linux to an Oracle database instance.
  • Oracle Linux has resource management through a feature called cgroups. cgroups lets you create resource groupings based on cpu, memory or disk parameters (or a combination). cgroups is also the internal set of features that Linux containers (lxc) uses. Basically you put processes (pids/tasks) into a cgroup and then they live within the limits of that cgroup definition. With lxc you basically also get process isolation on top.

    It works like this : as an admin you set up a cgroup, give it a name and set up paremeters around which cpu's to use, how much memory to allocate and so forth. The name is really just a simple mkdir in the cgroups virtual filesystem. You then use a new database init parameter process_group_name and when you start the database it will put its processes and as such its resources into that group. alter system set processor_group_name = 'cgroup_name' scope=spfile;. You can do this for each instance and this way you can isolate cpu and memory resources for each instance on a given OS environment. If you use Oracle Linux 5 with uek/uek2 or Oracle Linux 6 with uek/uek2 you can make use of this.

    cgroups with Oracle Linux 5 and UEK/UEK2 has to happen manually, through mkdir and echo > controlfiles. In Oracle Linux 6 we have cgroup management utilities.

    two more reasons :)...

    Sunday May 20, 2012

    Oracle Linux components

    A few weeks ago I was visiting a customer where we were discussing Oracle Linux and Oracle VM advantages compared to some of our competitors. One of the comments from the customer at some point in the discussion was "And you get -all that- for just -$that-?"... Why, yes, you do, sir. I wanted to take the opportunity to list some of the included advantages of having Oracle Linux support. Next time I will do the same for Oracle VM.

    Oracle Linux is a complete enterprise Linux solution available from Oracle. In fact, it is, we firmly believe, the best enterprise Linux solution available. Both for generic use of Linux and in particular and especially when using any Oracle product on top of it. So it is important to remember and understand that Oracle Linux is not just for running Oracle products on top of Linux. Oracle Linux is a complete generic Linux distribution that's fully supported underneath any Linux application. There are no restrictions from our end as to what you can run on it (as long as they're apps built to run on Linux, of course and any application that runs on Red Hat Enterprise Linux will run the same on the corresponding Oracle Linux version). You can run 3rd party Linux applications just fine. Use it as an NFS server, as a webserver or whatever you want to use the system for. Many customers I talk to somehow believe that it's unique for Oracle products and I have to point out that while we have great, unique advantages and extra benefits running Oracle products on Oracle Linux, it's certainly not in anyway restricted for that. It's the opposite, all you needs is Oracle Linux for any and every Linux server in house.

    Why the best, you ask? Well, we do an incredibly amount of testing on a daily basis (1000's of servers run various types of QA on Oracle Linux amounting to over 100,000 hours of QA happening every single day), we have a huge amount of internal use as consumers (end-user/customer use), we have many years of experience helping enterprise customers in general and we extend that down to Operating System support. We can scale using the greater Oracle infrastructure, etc.

    The support team is a large group of dedicated professionals globally that are trained to handle serious issues, deal with mission critical environments and cases, know their product, have a focus on server use first and foremost, understand, in particular, also the other products around it enough (like, to be able to understand that ASM doesn't stand for -assembler language- (in our customer-base) but stands for automatic storage management (you laugh? I can assure you it's not a joke)) A single phone call: if a database problem is an OS issue, no worries, we deal with it internally, no need to go call someone else that might think a database is a dirty word, or someone that 's an expert in gnome (no offense) but not quite in a complex environment where you have a product use 50GB of RAM and has 2000 processes running and is doing massive amounts of IO and woops something is going wrong. "Reboot?" doesn't cut it.

    When we do Linux development, the focus is basically on two things. 1) find ways to enhance Linux to run our products better (in ways that are generic, and also help other applications). A database does disk/network IO, lots of processes, lots of memory access, if we make our product run better, others also benefit... 2) help make Linux a better OS. improve stability, help add features that an OS needs, unrelated to our own product stack but important for customers to run the operating system, no matter what application. Having access to in-house hardware such as very big x64 servers like the x4800 (4TB ram, 160 threads) and TB and TB of storage allows us to test things at very, very large scale. This allows us to publish benchmark results like the 5M tpc-c result a few months ago. Offering the Unbreakable Enterprise Kernel(UEK) as part of Oracle Linux alongside what we call the Red Hat-Compatible kernel gives us the ability to provide current, modern, tested code to customers without reducing reliable and availability.

    A CTO of another company a little while back implied on twitter that UEK is too current and reduces reliability because the code is new. Can I point something out here? One thing we decided to do, with UEK, was to not backport features but instead just focus on stability. The goal is to stay close (but not too close as we don't ship a development tree) to the "Linus kernel" and focus on testing the heck out of it and fixing any bug that we would find, any regression.. This is good for the Linux community because when we do find a problem, we fix it and it's immediately also relevant to the mainline kernel. This helps us and the kernel community. We spend time testing and fixing, we develop on mainline and we don't waste time backporting new stuff into old versions. It's a symbiotic relationship. The alternative : take a version of Linux that's 3+ years old and take code from a more or less current version, after 100's of 1000's of lines of code changes and architectural changes have happened, and make that new code, somehow work in the old tree. The result : old patched up, totally unique code. Someone spending all their time retrofitting complex features into an old version. Testing is unique, bugs found are mostly only relevant in that version and not so much in newer versions unless they're greater design bugs or one liners. No one else really ever tested it because, well, it's unique, no other company is providing that seriously forked kernel that has 1+M of lines of differences. Somehow, I prefer our approach. At least the folks at SuSE seem to believe the same.

    One advantage we have with our kernel is that we made it possible to install in an existing Oracle Linux 5 environment and you are not required to move to Oracle Linux 6 to get the benefit of new kernel features and new drivers and new performance benefits. Moving from one major version to the next is a very costly effort because you basically always end up doing a re-imaging of the servers. Most customers are driven to new major releases for a hardware refresh. We don't require you to do that, you can run both OL5 and OL6, your choice, your timeframe. You can reboot between 2.6.18 and 2.6.32 or even 2.6.39 (3.0.24) without problem, at any time.

    The complete source code for Oracle Linux is available from oss.oracle.com (OL5 or OL6). The source code for our UEK and UEK2 kernels is available in a public GIT repository (UEK 2.6.32 or UEK2 2.6.39. Don't worry, it's not just a tar file and you don't have to go browse webpages behind a userid/password to figure out what all changed and try to piece it together yourself. It's all there, both the mainline kernel changelog and our changes on top. Transparent.

    The binaries for Oracle Linux are available from edelivery in the form of ISO images of the installable DVDs for the major releases and all the update releases (OL5.1...5.8, OL6, OL6.1, OL6.2 etc). You are not required to first get a support subscription, you can just go and download the software, no activation keys, no trial period, production use OK, distribute to your friends? no problem, no need to remove logos. Errata/bug updates are available through ULN (ULN is our Oracle Linux customer portal where we have extra channels with extra software bits for subscribers, or from public yum (public yum is an open, free, no registration, no subscription site). All Oracle Linux errata/packages are published on both sites at the same time. If you have a set of servers where support is not needed, just a standard Oracle Linux, and then a set of servers where you do need support because they're production systems etc, you can install OL on both and you subscribe for the systems you need to have support for. One distribution installed across the entire company. No need to find a free distribution and a pay to use distribution. If you no longer need support services for a system, don't worry, we won't ask you to de-install.

    But wait, there's more..

    How about managing Oracle Linux... well, included in Oracle Linux support subscriptions, is the use of Oracle Enterprise Manager 12c and Oracle Enterprise Manager OpsCenter 12. This is management software that can handle 1000's of servers from a single managed web-based interface. It offers features like :

  • bare-metal provisioning of Oracle Linux servers
  • configuration management, collecting and analysis and auditing
  • lights-out or ad hoc patching
  • availability and performance monitoring
  • event management and job systems
  • yum repositories
  • grouping of servers, custom jobs, custom action/corrective action scripts
  • this is all included, for no additional fee. Any server with Oracle Linux support subscriptions include the above. No extra pack, no per server additional fees. Or if you don't really care for that and just want a very simple yum based setup, you can do that as well, just grab the rpms, put them on a server, create a repository and have your servers connect to that. So, very basic and simple, to very complete and comprehensive. Same cost.

    And there is even more...

    Oracle Real Application Clusters makes use of the Oracle Clusterware(CRS). CRS is cluster software that is capable of protecting any kind of application in a failover cluster. Not just Oracle. It is the basis for any clustered environment and provides generic availability eliminating the need for other third party cluster solutions. You can protect your own application using this enterprise grade clustering stack. Without additional fees. It's included in Oracle Linux support subscriptions (basic/premier).. have your own nfs server? webserver? want to monitor it and restart it in a cluster no problem? Go for it. Using the exact same software we use for the most mission critical clustered systems. Want to run Oracle Linux virtualized? 1 guest? 10 guests? 20 guests? You only need one subscription per physical server and we don't charge more if you run 1 or more than 1 guest. And how much for all this?

  • Oracle x86 based systems with Premier support for Systems includes all this, both the Oracle Linux subscription and the use of the management features and Oracle CRS
  • Non-Oracle x86 based systems : 2 socket server $499 for all of the above or 4+ socket server $1199 includes all this, both the Oracle Linux subscription and the use of the management features and Oracle CRS
  • This is 24/7 support, not just office hours. Complete 24/7 support and real indemnification not based on subscription fees paid.

    We also have a premier support subscription which adds the ability to use Oracle ksplice technology to apply your kernel security fixes online, while the system is running at full speed, while the applications are running, while the users are working, without downtime. And I mean apply, not just install. The installed fixes are instantly effective at runtime. zero reboots. zero downtime scheduling. zero administrative program/planning management.. Or premier backport support, critical fixes for a specific bug in the specific version of the rpm you run in a mission critical environment. Or lifetime support, now with virtualization more useful than ever before. Virtual hardware can remain the same for much longer than the support lifecycles.

  • Oracle x86 based systems with Premier support for Systems includes all this, both the Oracle Linux subscription and the use of the management features and Oracle ksplice and premier backports and lifetime support
  • Non-Oracle x86 based systems : 2 socket server $1399 for all of the above or 4+ socket server $2299 includes all this, both the Oracle Linux subscription and the use of the management features and Oracle CRS and Oracle ksplice and premier backports and lifetime support
  • See our store for pricing, and our Oracle Linux site for policies and references to the above.

    Ok there you have it. Almost a novel but we just have that much to offer, and wait 'til I get to Oracle VM.. it just gets even better.

    Saturday May 19, 2012

    using ovm_utils

    Now that we released Oracle VM 3.1 and I just added ovm_utils 0.5.2 on ARU, it's time to give a few examples/use cases of the utilities :)

    The utilities are available for download from http://support.oracle.com search for patch name 13602094.

    It's a small 2.5Mb zip file which you should install on the manager node and unzip in /u01/app/oracle/ovm-manager-3. There are manpages for the utilities as well. These are installed in /u01/app/oracle/ovm-manager-3/ovm_utils/man/man8. To make it easy, simply add this to your MANPATH : export MANPATH=/u01/app/oracle/ovm-manager-3/ovm_utils/man.

    To use the utilities you need the admin username and password for the manager instance and in general you can just use localhost as the hostname. If you want to be able to connect to a remote manager instance, you have to first enable SSL on the manager node. By default we listen only on 127.0.0.1. There is an option in the utils to make a secure connection, use -X, but again, you first have to enable SSL for this. You can enable secure connections using ovm_utils and the addkeystore command. We have a support note for this.

    So the format is something like : ./ovm_managercontrol -u admin -p mypassword -h localhost -c [options...]..
    There are quite a few commands in the various utilities at this point. I broke the utilities up into categories :br

  • ovm_managercontrol : generic manager level functions
  • yuminfo : display yum configuration
    setupyum : configure yum
    addkeystore : configure the SSL keystore
    keystoreinfo : display keystore file and path
    createcpugroup : create a CPU Compatibility group
    removecpugroup : delete CPU Compatibility group
    addservertocpugroup : add server to CPU Compatibility group
    removeserverfromcpugroup : remove server from CPU Compatibility group
    listcpugroups : list all CPU Compatibility groups and its servers
    setsessiontimeout : set VM Console session timeout (Oracle VM 3.1+)
    getsessiontimeout : display VM Console session timeout(Oracle VM 3.1+)

  • ovm_repocontrol : generic repository level functions
  • list : list all repositories
    status : query repository status
    info : display repository information
    refresh : send a repository refresh command
    fixrepo : fix a failed repository create
    create : create repository on a storage device

  • ovm_poolcontrol : generic server pool level functions
  • status : display the current status of the server pool
    info : display server pool info such as memory, utilization, servers,...
    list : list all server pools registered with this Oracle VM Manager instance
    events : list all latest event types for this server pool
    addserver: add server to this server pool (-S server name)
    removeserver: remove server to this server pool (-S server name)
    refresh : refresh storage in the server pool

  • ovm_servercontrol : generic server level functions
  • start : use IPMI to power-on the server
    stop : shutdown the server
    kill : use IPMI to power-off the server
    mainton : turn on maintenance mode for this server
    maintoff : turn off maintenance mode for this server
    status : display the current status of the server (Running,Stopped,...)
    info : display server info such as memory, utilization, server-pool,...
    lock : temporarily lock the server disallowing any updates to happen
    upgrade : send an upgrade command to the server
    list : list all servers registered with this Oracle VM Manager instance
    events : list all latest event types for this server
    discover : discover server with provided hostname or IP address and Oracle VM agent password (-P)
    listnfsexports : list NFS exports on this server
    deletenfsexport : delete an NFS export on this server
    createnfsexport : create a new NFS export on this server

  • ovm_vmcontrol : generic Virtual Machine level functions
  • start : start the Virtual Machine
    stop : Power-off the Virtual Machine
    suspend : Suspend a running Virtual Machine
    kill : Kill the Virtual Machine
    resume : Resume a suspended Virtual Machine
    restart : Restart a running Virtual Machine (Power-off, start)
    status : display the current status of the Virtual Machine (Running, Stopped,Suspended)
    info : display Virtual Machine info such as memory, utilization, server-pool,...
    lock : temporarily lock the server disallowing any updates to happen
    events : list all latest events for this Virtual Machine requiring acknowledgement
    list : list all Virtual Machines registered with this Oracle VM Manager instance
    vcpuset : bind virtual cpu’s to physical threads for this Virtual Machine
    vcpuget : list bindings of virtual cpus to physical threads for this Virtual Machine
    settags : store a set of comma separate custom tags for a Virtual Machine
    gettags : retrieve list of tags for this Virtual Machine
    fixcfg : fix Virtual Machine Configuration file
    delete : delete Virtual Machine

  • ovm_vmdisks : a utility to list the virtual disk info for a given Virtual Machine
  • ovm_vmmessage : utility to send/receive message to/from a Virtual Machine using the Oracle VM VM API
  • Here a a few (hopefully) useful examples:

    # ovm_managercontrol -u admin -p Manager -h localhost -c getsessiontimeout
    # ovm_managercontrol -u admin -p Manager -h localhost -c setsessiontimeout -T 60

    Session timeout is the Oracle VM Console session timeout (connect to a Virtual Machine console). When you have a slow network, the default timeout of 30 seconds might be too short and this lets you set this to a longer (or shorter value).

    # ovm_servercontrol -u admin -p Manager -h localhost -c listnfsexports -s server1
    Oracle VM Server Control utility 0.5.1.
    Connected.
    Command : listnfsexports
    NFS Exports :
            id         : 0004fb0000230000978a28e2dc85e06b
            client     : ca-srs1
            options    :
            repository : SATA_ST3500320AS_5QM1EYTX
            path       : /OVS/Repositories/0004fb000003000031fac25b24740cca/
    ---
            id         : 0004fb00002300004b161fd1a0d106b7
            client     : ca-vdi1
            options    : rw,no_root_squash
            repository : SATA_ST3500320AS_5QM1EYTX
            path       : /OVS/Repositories/0004fb000003000031fac25b24740cca/
    ---
    Exit... 

    nfs exports are very useful and new in 3.1. It is now possible to export a local ocfs2 repository on a FC, iscsi or local disk device to be exported from an Oracle VM server to the outside world. You define which repository and which client can mount the repository. listnfsexports lists the exports defined on a specific Oracle VM Server.

    # ovm_servercontrol -u admin -p Manager -h localhost -c createnfsexport -s server1 
    -C ca-vdi1 -o rw,no_root_squash -r SATA_ST3500320AS_5QM1EYTX
    Oracle VM Server Control utility 0.5.1.
    Connected.
    Command : createnfsexport
    Creating nfs export for repository 'SATA_ST3500320AS_5QM1EYTX' on server.
    Created repository export
    Exit... 

    Here I have a repository named SATA_ST3500320AS_5QM1EYTX (this happens to be a local disk on server1) and I am exporting that repo through nfs to external server ca-vdi1. So now, on ca-vdi1, I can do mount server1:/OVS/Repositories/0004fb000003000031fac25b24740cca /mnt and I can backup(or restore) files.

    # /ovm_servercontrol -u admin -p Manager -h localhost -c deletenfsexport -s server1 
    -i 0004fb0000230000978a28e2dc85e06b
    Oracle VM Server Control utility 0.5.1.
    Connected.
    Command : deletenfsexport
    Deleting nfs export with ID '0004fb0000230000978a28e2dc85e06b'
    Deleted repository export
    Exit... 

    Simply deleting a repository. You need to specify the UUID of the nfs export, which you can find using listnfsexports.

    # ovm_repocontrol -u admin -p Manager -h localhost -c create -r foo -s wopr5 
    -i 3600144f057ef8a0000004fb3194b0001
    Oracle VM Repository Control utility 0.5.1.
    Connected.
    Command : create
    Found LUN with id '3600144f057ef8a0000004fb3194b0001'
    Creating Filesystem...
    Creating repository... Please wait...
    Create repository completed.
    Exit... 

    Create a repository on the LUN with a specific UUID. This works around slow storage where we hit a 2 minute timeout.

    # ovm_vmcontrol -u admin -p Manager -h ovmm -c vcpuget -v apitest
    Oracle VM VM Control utility 0.3.9.
    Connected.
    Command : vcpuget
    Current pinning of virtual CPUs to physical threads : 2,3
    

    Pinning virtual CPU's to physical threads.

    # ovm_vmcontrol -u admin -p Manager -h ovmm -c vcpuset -v apitest -s 5,6
    Oracle VM VM Control utility 0.3.9.
    Connected.
    Command : vcpuset
    Pinning virtual CPUs
    Pinning of virtual CPUs to physical threads '5,6' 'apitest' completed.
    A restart of Virtual Machine 'apitest' is required. 
    

    # vm_servercontrol  -u admin -p Manager -h localhost -s server -c events -A all
    Oracle VM Server Control utility 0.3.10.
    Connected.
    Command : events
    Server Events
    Acknowledging Virtual Machine events
            Virtual Machines on this server :
    VM : el5u4_32_pvm_1
    VM : el6u1_64_hvm_1
    VM : el5u7_64_hvm_1
    VM : el5u7_64_pvm_1
    VM : el6u1_64_pvm_1
    VM : el6u2bd5_32_pvhvm_1
    VM : el6u1_32_pvm_1
    VM : uek-boot-el5-64
    Exit..

    Acknowledging events on a given server, including any guest events.

    Friday May 18, 2012

    Oracle VM 3.1 released

    Today we formally announced the release of Oracle VM Server for x86 3.1. It is available for download from http://edelivery.oracle.com/linux. The software consists of 2 ISO images, the Oracle VM 3.1 Manager iso and Oracle VM 3.1 server iso... You can find the documentation library on the documentation website.

    Quite a few enhancements went into this release :

  • Oracle VM Manager UI enhancements to improve ease-of-use.
  • We continue to reduce the number of clicks to do actions and also have very easy to use and very streamlined configuration wizards and setup wizards. For instance for create VM or Create Server pool.

  • Storage enhancements for backup and recovery purposes.
  • It is now possible to do an nfs export on an Oracle VM server of a local/shared SAN/iscsi repository. When you have a filesystem based (ocfs2) repository with Oracle VM 3, only servers that are part of the Server pool can access the filesystem, with this new enhancement, you can define an nfs client hostname/IP address which will be allowed to do a remote nfs mount of this volume for backup purposes (or restore).

  • Using Unbreakable Enterprise Kernel 2 (2.6.39+) as the dom0 kernel
  • We now use the exact same Linux kernel for Oracle Linux with 2.6.39 (uek2) and for Oracle VM 3.1. This is possible because of all the work we have done together with Citrix/Xensource people over the last few years to get the necessary support for Xen and Linux merged into Linux. As I have mentioned in a previous blog entry, Linux mainline contains everything needed to have Linux be a complete dom0 management kernel for Xen without any additional patches. Unlike some of the FUD out there about "Xen" not being part of Linux, we have always made clear that Xen is a hypervisor. And Linux runs on top as the management/control domain kernel giving a great level of separation/isolation. In enterprise environments this is a big deal and helps us create solid, stable, secure solutions.

    The advantage for customers is that we do an incredible amount of testing on uek2 in terms of performance, network, disk io and scalability/stability in general and now Oracle VM automatically benefits from this additional testing. On top of that, for our partners, they do not have to worry about testing 2 different products in terms of drivers, since we share the exact same code base we have the exact same drivers for our hardware/storage vendors.

  • Clone many times
  • Simply the ability to say, clone a template X times in one step.

    You can find more information in the documentation set and in the release notes.

    Wednesday May 16, 2012

    More on templates and first boot configuration

    A little while back I wrote something up about building appliances (physical/virtual) using Oracle Linux. We build a whole set of Oracle VM templates and release them on our edelivery site. But some of what we use inside these templates can be more generically applied... Let me explain..

    A few years ago we created a Linux rpm called oraclevm-template-config. This is a very small package that contains a number of scripts that esssentially do the following:

  • install a service oraclevm-template-config that checks at boot time whether it needs to run or not
  • provides a default first boot configuration service for root password, network setup
  • provides a default cleanup script so that you can re-run this on an existing system, at next boot
  • provides a small set of library calls to use those functions in your own scripts
  • allows you to define your own first boot configuration scripts for your own applications to run
  • regenerates ssh host keys
  • all input was done on the local console or through kernel boot line parameters
  • These things are very straightforward and we wrote it specifically to build Oracle VM templates but there was nothing specific to Oracle VM or virtualization there. The newer version of this, which we released along with Oracle VM 3.x is very much tied into Oracle VM. In the new version (3.x) we integrate the first boot scripts and the configuration scripts with the Oracle VM VM api where it's now possible to send remote VM messages to configure the VM. So in order to automate deployment and have very integrated and fancy assemblies or Oracle VM based templates, this is the way to go. If, however, you want something very simple, with lcoal console (physical or virtual), where it can be on a physical server where you want to reboot and reconfigure and install, whether this is a VirtualBox VM or an Oracle VM VM or, for that matter any other Virtualization technology based VM. the older version is quite usable. I tend to use that one for the VirtualBox templates that I create all the time.

    We just published this rpm on http://public-yum.oracle.com. The rpm can be directly downloaded for Oracle Linux 5, here. Or when you have a system registered with public-yum or with ULN it is also in the add-ons channel. If you have a yum repo config then just enable the addon channel and you can simple type yum install oraclevm-template-config.

    So what are some of the basics you can do :

    initial install By default when you install template config, you have a service that provides the basic, configure root password, setup local network at what's called firstboot. To enable this to happen at the next reboot (which one can then call firstboot), just simple type /usr/sbin/oraclevm-template --enable. There is a simple default cleanup script which un-configures the network and removes ssh keys : /usr/sbin/oraclevm-template --cleanup. Keep in mind that when you run this script, the network will reset to dhcp and you will lose your network connection if you are in a remote ssh session.

    So once you do this and reboot your server or your VM. The next reboot will automatically start /usr/sbin/oraclevm-template --config and go through the questionnaire.

    The base configuration script is /etc/sysconfig/oraclevm-template. If you enable the service, you will see RUN_TEMPLATE_CONFIG=YES. This means next boot it will run. There is also a log file /var/log/oraclevm-template.log.

    If all you need is a simple way of packaging a VM or configure a physical server that deals with network setup, the above is enough. If you have your own software installed and it can handle IP changes without reconfiguration, then you can create an appliance and once firstboot happens, the configuration is set, network is setup, your application comes up and you're done. However, in some cases you want to run your own scripts and integrate with this template configuration. We also handle that case...

    If you write your own shell script (or really any script) that you install on the system, you can add that script to /etc/sysconfig/oraclevm-template and instead of doing the default firstboot run, it will at firstboot (when you call --enable) execute your script instead. Simply by setting TEMPLATE_CONFIG_SCRIPT=/myscript.sh. Inside that script you can do whatever you want... You can also provide a cleanup script for --cleanup, using TEMPLATE_CLEANUPS_SCRIPT=mycleanupscript.sh.

    Keep in mind that we do not automatically run --cleanup, this is a manual step. I would recommend doing the following :

  • /usr/sbin/oraclevm-template --enable
  • /usr/sbin/oraclevm-template --cleanup
  • history -c
  • shutdown -h now
  • We provide a few default function you can include in your own scripts. These functions are defined in /usr/lib/oraclevm-template/functions just source them in your bash script.

    ovm_configure_network [dhcp [hostname]] | [static] | [ip_adddress netmask gateway dns_server hostname]

    You can call this like ovm_configure_network "dhcp" in your script and it will configure your network that way. If you set it to static then there will be a dialog for entering ip, hostname, gateway, netmask, dns, or if you provide that in your function call it will just set the changes without dialog.

    ovm_cleanup_os
    A simple OS cleanup. This resets the network to dhcp and cleans up ssh keys.

    ovm_disable_firewall
    disable iptables

    press_anykey
    Shows user input to enter a key in your script execution flow

    ovm_error_message Displays an error message to the screen and also to /var/log/oraclevm-template.log

    ovm_info_message Displays an informational message to the screen and also to /var/log/oraclevm-template.log

    ovm_warn_message Displays a warning message to the screen and also to /var/log/oraclevm-template.log

    All of the above is very simple, yet very powerful. Like I mentioned in the beginning, I use this all the time when creating both VirtualBox and Oracle VM server templates and it makes my life so much easier. People often write these on their own but I wanted to share how to do this so, when you use Oracle Linux, and/or Oracle VM Server or Oracle VM VirtualBox or anything else, and you're building a cool appliance, physical or virtual, use it and you don't even need a support subscription to download it:)...

    Friday May 04, 2012

    building an appliance? physical ? virtual? production quality? use Oracle Linux

    Many companies are building appliances these days. Whether they are virtual appliance (VMs) or physical appliances, they all have an embedded OS inside. Now, I have looked at many of them, in particular virtual appliances and a few things struck me as interesting :

  • many had a random version of Linux included. Nothing bad about any of the distributions used at all, but just a hodgepodge of debian, fedora, opensuse, centos, etc... Almost all were marked "trial" or "demonstration"
  • a number of these appliances had a known commercial version of Linux, were marked trial edition and had restrictions on use (use for 30 days and so on).
  • So what are some problems with these scenarios?

    Well, first of all, even though there is absolutely nothing wrong at all with using community-based and community-maintained distributions (they do a great job), in a commercial setting, or when you build a product that lives on top of it, you need to have the ability to get formal support. Maybe not everyone needs this, but many companies do not have the knowledge (nor should they necessarily) to support the Linux distribution itself and they would want a reasonable formal service level for when something goes wrong, or when to expect something to get released.

    Secondly, in the case of using a commercial Linux distribution, there are distribution restrictions and even access restrictions. Certain Linux distribution vendors that offer formal support subscriptions will not give you access to the binaries without a subscription for support services and they also do not let you re-distribute without special agreements or making lots of modifications (or even restrict modifications).

    Enter Oracle Linux...

    What do you get?

  • freely downloadable ISO images for all the releases and major updates (Oracle Linux 4(4.4, 4.5, 4.6,4.7,4.8), Oracle Linux 5(5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7,5.8),Oracle Linux 6 (6.1, 6.2)). Both source code and binaries.No support subscription needed to download
  • freely re-distributable Both source code and binaries (no contract or support subscription needed to re-distribute, including logos and trademarks)
  • freely available bugfix errata and security errata, released at same time as we release them to customers with support subscriptions Both source code and binaries. In other words, every RPM released through a free public yum repository
  • the ability to purchase a support subscription for what you build, ship, support to your customers without any change to your appliance. No need to have a trial version or demo version using a community-edition Linux and a production version based on a commercial-edition Linux
  • use it in production, go for it, whatever the use case is, use it, if you want support, get a support subscription, if you don't need support, that's fine.
  • the ability to have a support subscription that offers on-line zero-downtime, rebootless security and bugfix updates as part of Oracle Linux Premier support with Oracle Ksplice technology
  • if you run with Centos or Red Hat Enterprise Linux, just replace the rpm list with the Oracle Linux RPMs and you're done.. so easy to switch
  • heavily tested on the biggest servers and biggest IO intensive, process intensive and network intensive workloads out there
  • do this for installs on physical servers, for Virtual Machines (whether this be Oracle VM Virtual Appliances, VMWare Virtual Appliances or something else, it doesn't matter)
  • One Enterprise Linux distribution, for all of the above. We make it easy for you. Grab the code, binaries and source, use it, distribute it, build your environments with it, freely, no contracts needed. Need our help, get a support subscription. Choice, open. Virtual, physical, cloud. Not just obfuscated tar balls. No license or activation key, good consistent SLAs for releasing security updates, well tested,... Run Oracle Linux in-house in test and development environments, run it in production environments, use it for customer systems, distribute it, any or all of the above. One distribution that you can manage across all the use cases. No need to manage different versions even if they're similar, no need to make different distribution choices based on your use case and pay/not pay.

    http://www.oracle.com/linux
    http://public-yum.oracle.com
    http://edelivery.oracle.com/linux

    Thursday May 03, 2012

    understanding memory allocation in oracle vm / xen

    As a follow up to my previous blog about cpu topology, I wanted to add a little bit about memory topology and memory allocation in the hypervisor. Most systems these days that are multi-socket are considered NUMA. Even though over the years, the NUMA-factor has gone down drastically,there still is a small amount of memory locality involved.

    My test setup is a dual socket server with 36GB memory. You can see this in Oracle VM Manager as part of the server info or directly on the server with xm info :

    # xm info 
    ..
    total_memory           : 36852
    free_memory            : 25742
    ..
    

    I have a few VMs running on this server which is why you see memory be lower than total. The 16GB VM is running with tmem enabled and because of that is not using up all memory but only the base memory needed to be active for the workload it's running.

    # xm list
    Name                                        ID   Mem VCPUs      State   Time(s)
    0004fb00000600001668dac79108cb84             2  4096     4     -b----    129.9
    0004fb0000060000804bac06a5087809             1  4096     4     -b----    129.4
    0004fb0000060000db9c71d539c940ed             3 16000     4     -b----     28.3
    Domain-0                                     0  1244    24     r-----    188.0
    

    Let's start with a clean slate and look at some statistics. The following commands will dump detailed memory information on your server :

    # xm debug-key u ; xm dmesg. Basically debug info for NUMA memory info. xm dmesg will show you the debug output.

    
    (XEN) 'u' pressed -> dumping numa info (now-0xFE:A1CFFF69)
    (XEN) idx0 -> NODE0 start->0 size->4980736
    (XEN) phys_to_nid(0000000000001000) -> 0 should be 0
    (XEN) idx1 -> NODE1 start->4980736 size->4718592
    (XEN) phys_to_nid(00000004c0001000) -> 1 should be 1
    (XEN) CPU0 -> NODE0
    (XEN) CPU1 -> NODE0
    (XEN) CPU2 -> NODE0
    (XEN) CPU3 -> NODE0
    (XEN) CPU4 -> NODE0
    (XEN) CPU5 -> NODE0
    (XEN) CPU6 -> NODE0
    (XEN) CPU7 -> NODE0
    (XEN) CPU8 -> NODE0
    (XEN) CPU9 -> NODE0
    (XEN) CPU10 -> NODE0
    (XEN) CPU11 -> NODE0
    (XEN) CPU12 -> NODE1
    (XEN) CPU13 -> NODE1
    (XEN) CPU14 -> NODE1
    (XEN) CPU15 -> NODE1
    (XEN) CPU16 -> NODE1
    (XEN) CPU17 -> NODE1
    (XEN) CPU18 -> NODE1
    (XEN) CPU19 -> NODE1
    (XEN) CPU20 -> NODE1
    (XEN) CPU21 -> NODE1
    (XEN) CPU22 -> NODE1
    (XEN) CPU23 -> NODE1
    (XEN) Memory location of each domain:
    (XEN) Domain 0 (total: 318627):
    (XEN)     Node 0: 282976
    (XEN)     Node 1: 35651
    
    The above output shows that the first 12 cpu's are bound to memory node 0 and the next 12 to memory node 1. The info shows how many pages of RAM are available on each node NODE0 start->0 size->4980736 and NODE1 start->4980736 size->4718592. the Dom0 domain is about 1.2Gb of RAM and it has some memory allocated on each NODE (it also has all of it's 24 vcpu's allocated across all threads in the box). Now let's start a VM.

    # xm list
    Name                                        ID   Mem VCPUs      State   Time(s)
    0004fb0000060000804bac06a5087809             4  4096     4     r-----      8.8
    Domain-0                                     0  1244    24     r-----    240.9
    
    # xm debug-key u ; xm dmesg
    ...
    (XEN) Memory location of each domain:
    (XEN) Domain 0 (total: 318627):
    (XEN)     Node 0: 282976
    (XEN)     Node 1: 35651
    (XEN) Domain 4 (total: 1048576):
    (XEN)     Node 0: 1048576
    (XEN)     Node 1: 0
    
    You can see that the newly started VM (domain 4) has 4Gb allocated on node 0.
    # xm vcpu-list 4
    Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
    0004fb0000060000804bac06a5087809     4     0     0   -b-       4.8 0-3
    0004fb0000060000804bac06a5087809     4     1     3   -b-      26.1 0-3
    0004fb0000060000804bac06a5087809     4     2     2   -b-       3.5 0-3
    0004fb0000060000804bac06a5087809     4     3     1   -b-       2.4 0-3
    
    The VM also has its virtual CPUs bound to node 0. Let's start another VM.

    # xm vcpu-list 6
    Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
    0004fb00000600001668dac79108cb84     6     0    19   r--       2.2 19-23
    0004fb00000600001668dac79108cb84     6     1    23   r--      24.6 19-23
    0004fb00000600001668dac79108cb84     6     2    20   -b-       1.4 19-23
    0004fb00000600001668dac79108cb84     6     3    22   -b-       1.1 19-23
    
    # xm debug-key u ; xm dmesg
    ...
    (XEN) Memory location of each domain:
    (XEN) Domain 0 (total: 318627):
    (XEN)     Node 0: 282976
    (XEN)     Node 1: 35651
    (XEN) Domain 4 (total: 1048576):
    (XEN)     Node 0: 1048576
    (XEN)     Node 1: 0
    (XEN) Domain 6 (total: 1048576):
    (XEN)     Node 0: 0
    (XEN)     Node 1: 1048576
    
    As you can see, this domain 6 has vCPUs bound to node 1, and Xen automatically also allocates memory from node 1. To ensure memory locality. It tries hard to keep memory and CPU as local as possible. Of course when you run with many VMs with many vCPUs then memory allocation will be spread out across multiple nodes.

    After starting a 16Gb VM on this server (domain 7), now that 8Gb is allocated, you will see that this 16Gb VM's memory allocation is across the 2 memory nodes :

    (XEN) Memory location of each domain:
    (XEN) Domain 0 (total: 318627):
    (XEN)     Node 0: 282976
    (XEN)     Node 1: 35651
    (XEN) Domain 4 (total: 1048576):
    (XEN)     Node 0: 1048576
    (XEN)     Node 1: 0
    (XEN) Domain 6 (total: 1048576):
    (XEN)     Node 0: 0
    (XEN)     Node 1: 1048576
    (XEN) Domain 7 (total: 4097012):
    (XEN)     Node 0: 2524148
    (XEN)     Node 1: 1572864
    

    Wednesday May 02, 2012

    figuring out cpu topology in oracle vm

    I have had a few people in the past ask about CPU topology on Oracle VM and how to figure out which CPUs, cores or threads are used by a VM... Here's a quick overview :

    1) get a summary of your server

    The xm info command prints out a whole bunch of information about your system.

    Look for the lines below in the output for detail on your systems CPUs.

    # xm info
    ...
    nr_cpus                : 8
    nr_nodes               : 1
    cores_per_socket       : 4
    threads_per_core       : 2
    cpu_mhz                : 3200
    ...
    
    This server has a single socket with 4 cores and 2 threads per core. Total of 8 "CPUs". so cpu0..7 really is thread 0..7.

    # xm info
    ...
    nr_cpus                : 12
    nr_nodes               : 1
    cores_per_socket       : 6
    threads_per_core       : 2
    ...
    
    This server has a single socket with 6 cores and 2 threads per core. Total of 12 "CPUs".

    2) get the cpu topology

    The xenpm command lets you print out the thread/core/socket topology on a given server :

    # xenpm get-cpu-topology
    CPU     core    socket  node
    CPU0     0       0       0
    CPU1     0       0       0
    CPU2     1       0       0
    CPU3     1       0       0
    
    In the above example, it's a single socket machine with 2 cores and 2 threads per core. cpu 0 is thread 0 of core 0, cpu 1 is thread 1 of core 0, cpu 2 is thread 0 of core 1, cpu 3 is thread 1 of core 1. If you were to have cpus="0,1" in the VM config file, you would be running on core 0. If you were to have cpus="0-3" in the VM config file, you would actually run a VM on both cores.

    # xenpm get-cpu-topology
    CPU     core    socket  node
    CPU0     0       0       0
    CPU1     0       0       0
    CPU2     1       0       0
    CPU3     1       0       0
    CPU4     2       0       0
    CPU5     2       0       0
    CPU6     3       0       0
    CPU7     3       0       0
    
    In the above example, you have a single socket server with 4 cores and 2 threads per core. cpu 0 -> thread 0 of core 0, cpu 1 -> thread 1 of core 0 etc.If you were to have cpus="4-7" in the VM config file, you would run the VM on cores 2 and 3.

    # xenpm get-cpu-topology
    CPU     core    socket  node
    CPU0     0       0       0
    CPU1     0       0       0
    CPU2     1       0       0
    CPU3     1       0       0
    CPU4     2       0       0
    CPU5     2       0       0
    CPU6     8       0       0
    CPU7     8       0       0
    CPU8     9       0       0
    CPU9     9       0       0
    CPU10    10      0       0
    CPU11    10      0       0
    
    In the above example, you see a single socket server with 6 cores hyperthreaded.

    3) Show the guest/VM vcpu bindings to physical threads : Use the command xm vcpu-list to list a summary of which virtual CPU is running on which physical CPU.

    # xm vcpu-list 1
    Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
    0004fb00000600007c351fa24276c63f     1     0     5   -b-    4673.6 5-6
    0004fb00000600007c351fa24276c63f     1     1     5   -b-    4534.0 5-6
    
    If you add the domain ID to the command xm vcpu-list 1, you get the information for just that guest or VM. In the above example, you have a guest with 2 virtual CPUs both running, at this time, on physical cpu (thread in my case) 5. The column CPU Affinity show's 5-6 which means that both virtual CPUs could be running on either thread 5 or 6. This shows that the guest is pinned on those 2 threads. Combined with the information of xenpm get-cpu-topology I can then see that in my case CPU 5 is thread 1 of core 2 and CPU 6 is thread 1 of core 8. So this 2 vcpu guest is pinned to 2 separate physical cores.

    # xm vcpu-list
    Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
    0004fb00000600007c351fa24276c63f     1     0     5   -b-    4676.8 5-6
    0004fb00000600007c351fa24276c63f     1     1     5   -b-    4537.0 5-6
    Domain-0                             0     0     0   -b-     932.1 any cpu
    Domain-0                             0     1     6   -b-    1168.0 any cpu
    Domain-0                             0     2     7   -b-    1010.8 any cpu
    Domain-0                             0     3    11   -b-     903.0 any cpu
    Domain-0                             0     4     8   -b-     494.2 any cpu
    Domain-0                             0     5     9   r--     773.8 any cpu
    Domain-0                             0     6     1   -b-     522.7 any cpu
    Domain-0                             0     7     2   -b-     785.1 any cpu
    Domain-0                             0     8     4   -b-     473.8 any cpu
    Domain-0                             0     9     3   -b-     728.1 any cpu
    Domain-0                             0    10    10   -b-     490.8 any cpu
    Domain-0                             0    11     0   r--    1219.6 any cpu
    
    This is the same system but xm vcpu-list without argument. It also shows the dom0 guest. As you can see in this example, dom0 can run on any physical thread and the CPU Affinity is any cpu which implies any virtual CPU can be scheduled on any physical thread so no pinning or partitioning.

    4) using ovm_utils ovm_vmcontrol lets a user or admin set and get the cpu/vcpu bindings for a VM through Oracle VM Manager in 3.0.

    # ./ovm_vmcontrol -u admin -p Manager1 -h localhost -v apitest -c vcpuget
    Oracle VM VM Control utility 0.4.2.
    Connected.
    Command : vcpuget
    Current pinning of virtual CPUs to physical threads : 5,6
    
    In the above example, I have a VM named apitest and using the action vcpuget it shows me that virtual cpus of this guest are bound to threads 5 and 6. This happens to be the VM in the previous example which showed that this was core 2 and core 8 (3).

    Now let's bind the vcpu's to core 0, I just run the following command :

    # ./ovm_vmcontrol -u admin -p Manager1 -h localhost -v apitest -c vcpuset  -s 0
    Oracle VM VM Control utility 0.4.2.
    Connected.
    Command : vcpuset
    Pinning virtual CPUs
    Pinning of virtual CPUs to physical threads  '0' 'apitest' completed.
    
    Now, running the xm vcpu-list command, I can see that this immediately took effect (and also will on subsequent startups of the VM) :
    # xm vcpu-list 1
    Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
    0004fb00000600007c351fa24276c63f     1     0     0   -b-    4687.6 0
    0004fb00000600007c351fa24276c63f     1     1     0   -b-    4547.2 0
    
    The VM now has CPU Affinity 0 for both virtual CPUs.
    About

    Wim Coekaerts is the Senior Vice President of Linux and Virtualization Engineering for Oracle. He is responsible for Oracle's complete desktop to data center virtualization product line and the Oracle Linux support program.

    You can follow him on Twitter at @wimcoekaerts

    Search

    Categories
    Archives
    « May 2012 »
    SunMonTueWedThuFriSat
      
    1
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    17
    21
    22
    23
    25
    27
    28
    30
    31
      
           
    Today