Virtualization for HPC: The Heterogeneity Issue

I've been advocating for awhile now that virtualization has much to offer HPC customers (see here.) In this blog entry I'd like to focus on one specific use case, heterogeneity. It's an interesting case because while heterogeneity is either desirable or to be avoided, depending on your viewpoint, virtualization can help in either case.

The diagram above depicts a typical HPC cluster installation with each compute node running whichever distro was chosen as that site's standard OS. Homogeneity like this eases the administrative burden, but it does so at the cost of flexibility for end-users. Consider, for example, a shared compute resource like a national supercomputing center or a centralized cluster serving multiple departments within a company or other organization. Homogeneity can be a real problem for end-users whose applications only run on either other versions of the chosen cluster OS or, worse, on completely different operating systems. These users are generally not able to use these centralized facilities unless they can port their application to the appropriate OS or convinced their application provider to do so.

The situation with respect to heterogeneity for software providers, or ISVs -- independent software vendors, is quite different. These providers have been wrestling with expenses and other difficulties related to heterogeneity for years. For example, while ISVs typically develop their applications on a single platform (OS 0 above,) they must often port and support their application on several operating systems in order to address the needs of their customer base. Assuming the ISV decides correctly which operating systems should be supported to maximize revenue, it must still incur considerable expenses to continually qualify and re-qualify their application on each supported operating system version. And maintain a complex, multi-platform testing infrastructure and in-house expertise to support these efforts as well.

Imagine instead a virtualized world, as shown above. In such a world, cluster nodes run hypervisors on which pre-built and pre-configured software environments (virtual machines) are run. These virtual machines include the end-user's application and the operating system required to run that application. So far as I can see, everyone wins. Let's look at each constituency in turn:

  • End-users -- End-users have complete freedom to run any application using any operating system because all of that software is wrapped inside a virtual machine whose internal details are hidden. The VM could be supplied by an ISV, built by an open-source application's community, or created by the end-user. Because the VM is a black box from the cluster's perspective, the choice of application and operating system need no longer be restricted by cluster administrators.
  • Cluster admins -- In a virtualized world, cluster administrators are in the business of launching and managing the lifecycle of virtual machines on cluster nodes and no longer need deal with the complexities of OS upgrades, configuring software stacks, handling end-user special software requests, etc. Of course, a site might still opt to provide a set of pre-configured "standard" VMs for end-users who do not have a need for the flexibility of providing their own VMs. (If this all sounds familiar -- it should. Running a shared, virtualized HPC infrastructure would be very much like running a public cloud infrastructure like EC2. But that is a topic for another day.)
  • ISVs -- ISVs can now significantly reduce the complexity and cost of their business. Since ISV applications would be delivered wrapped within a virtual machine that also includes an operating system and other required software, ISVs would be free to select a single OS environment for developing, testing, AND deploying their application. Rather than basing their operating system choice on market share considerations, the decision could be made based on the quality of the development environment, or perhaps the stability or performance levels achievable with a particular OS, or perhaps on the ability to partner closely with an OS vendor to jointly deliver a highly-optimized, robust, and completely supported experience for end-customers.

Comments:

An interesting perspective on virtualization value in HPC.

Some scattered thoughts:
- This concept reminds me of application virtualization as well as Solaris containers.
- It's a good point that we in fact don't need homogeneity in a cluster for HPC use, but specifically homogeneity for a specific parallel application, or even more specifically for a specific parallel job. This would imply some new capabilities in the scheduler I presume. For example, the scheduler would need to be aware of how many cores and threads each compute node could support and determine how to partition inter-VM and intra-VM MPI processes. The VM would also need to be able to map the associated core affinity policies common to tuned MPI apps.
- It seems to be that this concept would greatly simplify check-point restart, as a VM is simply a file, with the associate benefits in pre-emptive scheduling and business continuity.
- This would also be a benefit for organizations with very long development cycles and the associated requirement to keep a constant tools environment for the life of the program (and beyond for root cause analysis later). The underlying cluster could change multiple times within the program lifecyle.

Posted by Ron Van Holst on January 28, 2010 at 04:58 AM EST #

Ron,

Checkpoint restart is definitely an important use-case for HPC virtualization for precisely the reason you state. (Note that CPR is covered in an earlier post I did (linked in the first line of this post) as are a few other use-cases.)

I hadn't thought of the last use-case you mentioned, but it is a good one. I will add it to my list.

Thanks for your comment.

Josh

Posted by Josh Simons on January 28, 2010 at 05:32 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 08:52 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 08:57 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 09:08 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 09:10 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 09:26 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 09:29 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 09:31 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 09:34 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 09:37 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on January 28, 2010 at 09:44 AM EST #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on April 09, 2010 at 08:48 AM EDT #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on April 15, 2010 at 04:00 AM EDT #

[Trackback] Virtualization on commodity computers is reaching market maturity, even though it’s an old concept borrowed from the days when mainframes ruled the computing landscape.  Today we have desktop virtualization, application virtualization, network vi...

Posted by Ron Van Holst's Blog on April 15, 2010 at 04:08 AM EDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Josh Simons

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today