Wednesday Jan 20, 2010

Logical Domains 1.3 Available

Logical Domains (LDoms) 1.3 is out! Some of the key new features include:
  • Dynamic Resource Management (DRM): policy-based load-balancing of cpu resources between domains
  • With the release of Solaris 10 10/09 (aka update 8), ability to dynamically add & remove crypto devices from domains
  • Significant speedup of warm migration (aka domain mobility) operations through HW crypto & compression algorithms
  • Faster network failover with link-based IPMP support
  • Domain hostid & MAC address properties can now be modified via the ldm set-domain command

DRM is a key new feature for which we've been getting many requests ever since LDoms launched in April 2007. With it, you can specify policies that determine how cpu resources are load-balanced between running logical domains. For each policy, you can specify (among other things) the min & max cpu count desired, the high & low utilization thresholds you'd like to stay within, the speed at which cpus are added or deleted, the time of day during which the policy is active, and a priority. Each specified policy can apply to one or more domains and can be active or inactive. Multiple policies can be active simultaneously, even if they affect the same domain(s). That's where the priority value comes in; lower numbers denote higher priority (valid values are 1-9999).

See complete details on setting up DRM policies in this section of the Logical Domains 1.3 Administration Guide

Further information about LDoms in general, and links to download the LDoms 1.3 components are available (as always) here.

Tuesday Sep 08, 2009

LDoms 1.2 available via IPS

Good news for folks running LDoms under OpenSolaris (either 2009.06 or the latest 2010.02 development builds): the LDoms 1.2 software is now available in the /release and /dev IPS repositories. You can get it by running 'pfexec pkg install ldomsmanager'

Saturday Jul 04, 2009

LDoms 1.2 is out!

LDoms 1.2 is now available. I've included a snippet of the initial marketing announcement below. There will be more official marketing activity in the coming weeks (including the updating of, but the bits are available now.

The LDoms team is pleased to announce the availability of LDoms v1.2.

In this new release you will find the following features:

  • Support for CPU power management
  • Support for jumbo frames
  • Restriction of delayed reconfiguration operations to the control domain
  • Support for configuring domain dependencies
  • Support for autorecovery of configurations
  • Support for physical-to-virtual migration tool
  • Support for configuration assistant tools.
We look forward to you downloading LDoms Manager v1.2 now from here. The full documentation set for LDoms 1.2 can be found here.

In addition, as I mention here, this release fully enables the enhanced FMA IO fault diagnosis changes introduced into the Solaris 10 5/09 release (aka s10u7).

UPDATE: One important thing I forgot to mention is that this release supports OpenSolaris 2009.06 for SPARC. OpenSolaris can be run either as the control/io/service domain or within a guest domain. In addition, the LDoms 1.2 LDom Manager & related files are available as an IPS package in the /dev repository under SUNWldom.
CORRECTION: I jumped the gun: the IPS package with LDoms 1.2 is not quite ready yet, but should be available very soon. When it is, it'll be under "ldomsmanager".


UPDATE 2: LDoms 1.2 is now available via IPS. Details here.

Monday Jun 01, 2009

OpenSolaris & LDoms

OpenSolaris 2009.06 officially launched today. The big news is that it supports SPARC. I was going to post a nice article about how it is also supported under LDoms, but Marcus & Liam did a much better job than I could. See here and here.

Sunday May 03, 2009

Improving integration of LDoms and FMA

Scott Davenport recently posted a blog entry announcing the availability of Solaris 10 Update 7 (aka Solaris 10 5/09 or just s10u7), and touted some of the FMA improvements in the release. Of particular interest for LDoms and its users are improvements in diagnosis of IO faults when one or more IO root complexes are allocated to domains other than the control domain (i.e. so-called root domains). These improvements were designed & developed through a collaborative effort between the LDoms & FMA teams. The collaboration between these teams is nothing new; even before the initial release of Logical Domains technology in April 2007, as well as ever since, there has been tight coordination between LDoms and FMA, both in terms of the technology, and the teams.

The changes needed to resolve the IO diagnosis problems required new interfaces between the FMA and LDoms code, as well as new software on both sides of the interface. As Scott mentioned, the FMA software is now available in the Solaris 10 5/09 release. However, the necessary changes on the LDoms side (in the LDoms Manager) will be available in the upcoming 1.2 release of Logical Domains, currently scheduled for this summer. That's no reason not to install s10u7 in a Logical Domains environment now; all currently supported versions of LDoms will function correctly with s10u7 installed in a control domain, guest domains, or both. And once LDoms 1.2 is released, simply installing the new firmware & LDoms Manager that make up the 1.2 update on a system running s10u7 will automatically enable the improved IO diagnosis features (along with several other new LDoms features, but that's the subject of another post).

Monday Feb 23, 2009

Oracle 10g certification on Logical Domains

Excellent News! Oracle 10g is now certified & supported on LDoms for both single instances and RAC implementations. The details are available here.

Wednesday Dec 24, 2008

LDoms 1.1

Just in time for the holidays, LDoms version 1.1 is now available! This is a major new release of Logical Domains technology, with an extensive list of new features & bugfixes. Here are the highlights:

Major Features Introduced in LDoms version 1.1:

  • Warm and Cold Migration
  • Network NIU Hybrid IO
  • VLAN Support for Virtual Network Interface and Virtual Switch
  • Public XML Interface and XMPP Connection with the Domain Manager
  • Virtual IO DR
  • Virtual Disk Multipathing and Failover
  • Virtual Switch Support for Link Aggregated Interface
  • iostat(1M) Support in Guest Domains

Alex has more details on these features here.

Other Improvements Include:

  • Improved Interrupt Distribution (CR 6671853)
  • Performance improvements for virtual IO (CR 6689871, 6640564)
  • Solaris Installation to Single Slice Disk (CR 6685162)
  • Numerous Improvements and Extension to our Domain Services Infrastructure (CR 6560890)
  • Improved Console Behavior when not using Virtual Console (CR 6581309)
  • VDisk EFI Label Support for Disk Image (CR 6558966)
  • VDisk Support for Disk Managed by Multipathing Software (Veritas DMP, EMC Powerpath) (CR 6694540, 6637560)
  • LDoms Manager Improvements to IO FMA (CR 6463270)
  • ldm list -l now displays MAC assigned to guest (CR 6586046)
  • Improved Error Messages (CR 6741733, 6590124, 6715063)
  • ldm list -o provides fine-grained control of configuration display options (CR 6562222)
  • More accurate utilization percentage reporting (CR 6637955, 6709020)
  • Can now explicitly set a domain's hostid (CR 6670605)
  • Improved persistence of VIO service and VDS volume names (CR 6729544, 6771264)
  • More predictable behavior when deciding which cpus to DR out of a domain (CR 6567372)
  • More accurate annotations in ldm ls-spconfig output (CR 6744046)
  • Better support for large, fragmented memory configs in a domain (CR 6749507)
  • Supports setting persistent WANboot keys from OBP (CR 6510365)
  • Lots of Bug Fixes (over 100)

Get it here!

Monday Oct 13, 2008

New Sun SPARC Enterprise T5440 Server runs LDoms 1.0.3

Today Sun is announcing the latest in our line of sun4v SPARC CMT systems: the Sun SPARC Enterprise T5440 Server. This is a four socket server based on our UltraSPARC T2 Plus processor.

With up to 256 available threads, this is the best platform yet for running our Logical Domains (LDoms) virtualization technology. LDoms 1.0.3, released last May, fully supports all shipping configurations of the T5440. All the necessary firmware & software comes pre-installed. If you need to download any of the LDoms 1.0.3 software, just go here.

In addition, there's a new resource available for helping administrators get the most out of their LDoms installation: the LDoms Community Cookbook. It just went live today; in fact, at the time of this writing, not all sections are live yet. Please check back often, and remember, this is a Wiki & a community resource, so feel free to add content, make corrections, etc.

To read what other Sun engineers have to say about the T5440, see Allan Packer's blog for an updated list of T5400-related blog entries.

Note: the Sun SPARC Enterprise T5440 Server is not to be confused with our recently anounced Sun Netra T5440 Server, which is a two socket carrier-grade server.

Monday May 19, 2008

LDoms 1.0.3

Logical Domains (LDoms) 1.0.3 is now available. This release is mainly intended to enable many new features included in Solaris 10 5/08 (aka update 5). Jason did an excellent job with details & logistics in this blog entry, so there's no need in my repeating that info here.

One thing I do want to mention in terms of LDom Manager functionality is that with this release, the XML format produced by the

ldm ls-constraints -x
command has changed. While the LDom Manager will continue to accept our previous, so-called v2 XML format, as of 1.0.3, it also accepts & produces the new v3 format. This format is designed to closely align with the schema defined as part of the draft Open Virtual Machine Format (OVF) specification.

This is just the tip of the iceberg. Coming in LDoms 1.1 (currently targeted for release in Q4CY08) will be a complete XML based control interface for monitoring & managing Logical Domains, based on this same v3 schema. In addition, it will utilize the XMPP transport, providing secure, standards-based XML messaging between client application & the LDom Manager.

This combination of a standards-based schema over a standard XML transport provides a rich control interface for creating management applications. More details about this new management interface, including detailed specifications, will be forthcoming.

Wednesday Apr 09, 2008

UltraSPARC T2 Plus & LDoms 1.0.2

Today we announced our T5140/T5240 platforms, based around the UltraSPARC T2 Plus processor. This is the first CMT platform which supports multiple processor chips (2), providing up to 128 hardware threads of CMT goodness! And to go along with these platforms, we're releasing Logical Domains version 1.0.2, which fully supports the new T5140/T5240 systems, including the ability to create up to 128 logical domains to match the number of available hardware threads.

Of course, most users probably won't want to run 128 single threaded logical domains, but the fact that you can is a testament to the scalability of the architecture.

Other features of LDoms 1.0.2 include:

  • Support for up to 64 domains on UltraSPARC T2 based systems
  • Libvirt management interface software included
  • A slew of bug fixes

Of course, LDoms 1.0.2 supports all the platforms supported by LDoms 1.0.1 & LDoms 1.0, and like its predecessors, is available at no extra cost. In fact, LDoms 1.0.2 comes factory installed on all T5140/T5240 systems. The associated firmware & LDoms Manager packages are also available from the Sun Download Center.

Saturday Feb 23, 2008

Uber Cool Nerd God says I'm an Uber Cool Nerd God.  What are you?  Click here!

Sunday Dec 02, 2007

LDoms versions & compatibility

With the release of LDoms 1.0.1 last October, along with the release of the first members of our UltraSPARC T2 (US T2) based platform family, issues of version compatibility naturally arise. When I first posted about the availability of LDoms 1.0.1, I included the following caveats:

WARNING: There are two important caveats when upgrading from LDoms 1.0 to LDoms 1.0.1:

  • Configurations saved to the service processor under 1.0 are not usable under 1.0.1. The LDoms 1.0.1 Administration Guide describes the upgrade procedure that needs to be applied to work around this. Part of this procedure needs to be carried out BEFORE performing the actual upgrade!
  • You must upgrade both the firmware and LDom Manager components at the same time.

One of the pillars of Sun's products is our ability to maintain compatibility; we have explicit practices & processes in place to guarantee compatibility between product releases. Unfortunately, we did not apply those to the 1.0.1 LDoms upgrade. This was an omission we do not intend to repeat. So, when upgrading from LDoms 1.0 to any subsequent version, the caveats mentioned above do apply (the current LDoms release is still 1.0.1, but we are planning for two follow-on releases in CY 2008).

As we wrap up the development phase of our upcoming 1.0.2 release, we're re-verifying that existing configurations created under 1.0.1 do continue to work under 1.0.2, and that firmware & LDom Manager upgrades do not need to be synchronized. Of course, certain 1.0.2 capabilities might not be fully enabled until both the firmware & the LDom Manager are upgraded, but at least the system will continue to function with only one or the other upgraded.

UPDATE 4/29/2008: With the release of LDoms 1.0.2, I'm happy to report that neither of the caveats described in this entry apply when upgrading from LDoms 1.0.1 to LDoms 1.0.2. However, they do still apply if you're upgrading from LDoms 1.0 to LDoms 1.0.2 (or any subsequent release). In other words, these compatibility issues only come into play when upgrading from LDoms 1.0 to a subsequent release.

Wednesday Nov 14, 2007

Open xVM Launches

OpenxVM has been launched. Read Steve Wilson's blog entry for the details.

Thursday Nov 01, 2007

Control domain reconfiguration in LDoms 1.0.1

This note explains how control domain reconfiguration works in LDoms 1.0.1, in contrast to how it functioned in LDoms 1.0. There were severe limitations placed on control domain reconfiguration in LDoms 1.0, which have been addressed in the 1.0.1 release:
  • The control domain could only be reconfigured when running in the "factory-default" configuration. Once reconfigured, if subsequent changes were desired, one had to revert back to factory-default and re-apply the initial changes as well as any new ones.
  • The only way to instantiate a newly reconfigured control domain was by downloading the new configuration to the SP, and then power-cycling the box.

When reconfiguring the control domain under LDoms 1.0.1, the LDom Manager enters "delayed reconfiguration" mode the first time it's asked to do something that can't be immediately instantiated (i.e. just about anything other than cpu DR and adding or removing disk volumes). Once in this mode, all subsequent operations are pended in the hypervisor until the control domain reboots. This delayed reconfiguration capability actually existed in LDoms 1.0, but could not be applied to the control domain because rebooting the control domain required a full powercycle of the system to make sure the I/O subsystem was properly reset, causing the loss of any pending operations queued up in the hypervisor.

LDoms 1.0.1 introduces the ability to soft reset the I/O subsystem, allowing the control domain (or any I/O domain) to reboot while the rest of the system stays up. This in turn allows delayed reconfiguration to work on the control domain.

Utilizing delayed reconfiguration mode for control domain reconfiguration also means the reconfiguration can take place at any time, not just when running in the factory-default configuration. This allows the control domain to be reconfigured as many times as needed without having to revert to the factory-default configuration and start over each time.

To facilitate control domain reconfiguration under LDoms 1.0, the LDom Manager ran in a special "config mode" when in the factory-default configuration. In this mode, all reconfiguration requests were simply queued up within the LDom Manager, so that the new config could be downloaded to the SP when ready, and instantiated by power-cycling the box. This mode is still utilized in Ldoms 1.0.1 on UltraSPARC T1 based platforms (when booted into its factory-default configuration). This is to support non-LDoms legacy compatibility mode, since these platforms initially shipped before the advent of LDoms technology. On these systems, the first control domain reconfiguration has to be done the same way it was for 1.0.

All subsequent control domain reconfigurations (and all control domain reconfigurations on T2 based and all future LDoms-supported platforms, as they do not utilize config mode) can be accomplished via delayed reconfig operations followed by a simple reboot of the control domain.

In summary, under LDoms 1.0.1, you can now reconfigure your control domain whenever you want, as many times as you need, without having to revert to factory-default and re-apply all your previous changes, without having to save the configuration to the SP, and without having to power-cycle the box (except for the first-ever reconfiguration on a T1 based platform).

The ability to reboot the control domain without the box power-cycling (aided with some magic I'll leave to Narayan to describe) deserves a little elaboration: it means you can truly reconfigure your control domain even with active guest domains running! The guest domains stay up during the ensuing control domain reboot; VIO services are pended & automatically re-established as the control domain comes back up.

One very important note about saving your domain configuration to the SP: just because you no longer _need_ to save the new configuration to the SP before rebooting the control domain to affect a reconfiguration, doesn't mean you _shouldn't_ save it; you absolutely should! It's _strongly_ recommend to always save any new configuration you create to the SP, otherwise if the system were to lose power, it would revert to a previously saved state (or to factory-default), which is almost certainly _not_ what you want. BTW, you can safely save your configuration even if there are delayed reconfig operations pending; in this case, the configuration that gets saved is the pending one.

Saturday Oct 27, 2007

Leopard upgrade experience

I just upgraded my MacBook Pro laptop to Leopard (aka OS X 10.5). Here are my experiences so far:
  • I ran a full backup first (doesn't everyone?), then performed the default upgrade procedure.
  • On first Leopard boot, Spotlight started re-indexing both my startup drive, and my backup (external FireWire) drive, even though I had previously disabled Spotlight on the external volume. I had to disable it again. I then unmounted the external drive. Spotlight continued to index my internal drive, but starting reporting something ridiculous like -5234923 hours to go. Sure enough, next morning it had still not completed (it said 28% done by then - this is on a 120GB disk with 100GB used). I rebooted, and Spotlight finished indexing in about 10 minutes.
  • My login keychain was messed up. Apparently it didn't like that my keychain had the name of my userid; it insisted on using the keychain named "login", which was empty. Running Keychain First Aid (under the Keychain Access menu) solved the problem, migrating all my keychain data over.
  • 1passwd didn't work initially. Turns out I had an older version installed. When I upgraded to 2.5.0, it started working again. Others are apparently still having problems, see here.
  • X11 is now installed by default. It noticed I had a custom ~/.xinitrc file and warned me about it, asking me what to do (I said to leave it alone). But there are problems. There is now an in /usr/X11/. The Tiger version was in /Applications/Utilities/, and remained there after the upgrade. There also seems to be some detritus left over in /usr: in addition to the directories /usr/X11/ and /usr/X11R6/ (a symlink to /usr/X11/), there is also a /usr/X11R6 1/, which contains its own include, lib & man subdirs.
    When launching X11, I now wind up with _two_ X11 icons in my Dock. One of them blinks for a long time, then shows "Application Not Responding" when ctrl-clicking the dock icon; it can't be killed (though it goes away when I quit the original X11).
    Plus there are two other mysteries: it takes two clicks of the dock icon to switch to an X11 window (and Cmd-Tab to X11 doesn't do the right thing either). Worst of all, three button mouse emulation doesn't work correctly. In Tiger, option-click emulated middle-click and Cmd-click emulated right-click. Now, option-click emulates right-click, and Cmd-click emulates Meta-middle-click. There seems to be no way to emulate a simple middle-click. Since that's how you paste in X11, this is a real problem. This works fine with a real mouse.
  • Mail Act-On & MailTags plugins are both incompatible with Leopard. Actually, Mail Act-On can be re-enabled, look here for details.
  • The Cisco VPN client we use to login to Sun's network works fine under Leopard. This is the first time a major OS X upgrade has not also required a new VPN client. I have heard there are conflicts with the "Back to My Mac" functionality of .Mac, so I went ahead and disabled it as a precaution.
  • no longer breaks long URLs for other mail clients by adding gratuitous spaces. Hallelujah!!
  • iCal now has a preference to set an alarm on new appointments by default. No more need for third party solutions like iCalFix.
  • Quicksilver (web page currently not responding) always displays its icon in the dock, even if I tell it not to. Also, the two items it puts into the system-wide Services menu disappeared for quite some time (including through a couple reboots and rerunning of the Quicksilver setup), but they eventually returned.

    UPDATE 1: An new version of quicksilver which addresses the dock icon issue can be found here.

  • Spaces is great. I was tired of VirtueDesktop's memory leaks & crashing. I just wish it allowed more hotkey options for activating its multi-desktop view (one non-obvious tip: you can set a corner to activate Spaces, but you do it from the Exposé Pref. pane). I also wish I could name the spaces (at least for the menubar pull-down).
  • I haven't played with Time Machine; it looks to be useless for me (and perhaps most laptop users), as it requires a second drive to be attached at all times.
  • UPDATE 2: I can now with confidence state that the AirPort connection/stall problems plaguing me since upgrading to 10.4.10 are completely resolved in Leopard!! This has been a long, frustrating problem for many laptop users since 10.4.10 came out. Although I'm annoyed at Apple for how long they let this drag on, I'm glad it's over for me, and hope they release a fix for those 10.4.10 users not planning to upgrade.
    One interesting change: now when I press the AirPort menubar icon (which is what used to be needed to get the AirPort connection to start up again with this problem in 10.4.10), instead of a delay before anything happens, the menu drops down immediately, with the top line initially saying: "AirPort: Scanning..." for a second or two (in a greyed-out font), before switching to "AirPort: On".
  • UPDATE 3: Salling Clicker stopped working for me; it said Bluetooth wasn't available. There is a preliminary 3.5.1 update available which solved the problem for me.

I work on the Oracle VM Server for SPARC (nee LDoms) team.

View Eric Sharakan's profile on LinkedIn


« July 2016