Montag Okt 01, 2012

What's up with LDoms: Part 4 - Virtual Networking Explained

I'm back from my summer break (and some pressing business that kept me away from this), ready to continue with Oracle VM Server for SPARC ;-)

In this article, we'll have a closer look at virtual networking.  Basic connectivity as we've seen it in the first, simple example, is easy enough.  But there are numerous options for the virtual switches and virtual network ports, which we will discuss in more detail now.   In this section, we will concentrate on virtual networking - the capabilities of virtual switches and virtual network ports - only.  Other options involving hardware assignment or redundancy will be covered in separate sections later on.

There are two basic components involved in virtual networking for LDoms: Virtual switches and virtual network devices.  The virtual switch should be seen just like a real ethernet switch.  It "runs" in the service domain and moves ethernet packets back and forth.  A virtual network device is plumbed in the guest domain.  It corresponds to a physical network device in the real world.  There, you'd be plugging a cable into the network port, and plug the other end of that cable into a switch.  In the virtual world, you do the same:  You create a virtual network device for your guest and connect it to a virtual switch in a service domain.  The result works just like in the physical world, the network device sends and receives ethernet packets, and the switch does all those things ethernet switches tend to do.

If you look at the reference manual of Oracle VM Server for SPARC, there are numerous options for virtual switches and network devices.  Don't be confused, it's rather straight forward, really.  Let's start with the simple case, and work our way to some more sophisticated options later on. 

In many cases, you'll want to have several guests that communicate with the outside world on the same ethernet segment.  In the real world, you'd connect each of these systems to the same ethernet switch.  So, let's do the same thing in the virtual world:

root@sun # ldm add-vsw net-dev=nxge2 admin-vsw primary
root@sun # ldm add-vnet admin-net admin-vsw mars
root@sun # ldm add-vnet admin-net admin-vsw venus

We've just created a virtual switch called "admin-vsw" and connected it to the physical device nxge2.  In the physical world, we'd have powered up our ethernet switch and installed a cable between it and our big enterprise datacenter switch.  We then created a virtual network interface for each one of the two guest systems "mars" and "venus" and connected both to that virtual switch.  They can now communicate with each other and with any system reachable via nxge2.  If primary were running Solaris 10, communication with the guests would not be possible.  This is different with Solaris 11, please see the Admin Guide for details.  Note that I've given both the vswitch and the vnet devices some sensible names, something I always recommend.

Unless told otherwise, the LDoms Manager software will automatically assign MAC addresses to all network elements that need one.  It will also make sure that these MAC addresses are unique and reuse MAC addresses to play nice with all those friendly DHCP servers out there.  However, if we want to do this manually, we can also do that.  (One reason might be firewall rules that work on MAC addresses.)  So let's give mars a manually assigned MAC address:

root@sun # ldm set-vnet mac-addr=0:14:4f:f9:c4:13 admin-net mars

Within the guest, these virtual network devices have their own device driver.  In Solaris 10, they'd appear as "vnet0".  Solaris 11 would apply it's usual vanity naming scheme.  We can configure these interfaces just like any normal interface, give it an IP-address and configure sophisticated routing rules, just like on bare metal. 

In many cases, using Jumbo Frames helps increase throughput performance.  By default, these interfaces will run with the standard ethernet MTU of 1500 bytes.  To change this,  it is usually sufficient to set the desired MTU for the virtual switch.  This will automatically set the same MTU for all vnet devices attached to that switch.  Let's change the MTU size of our admin-vsw from the example above:

root@sun # ldm set-vsw mtu=9000 admin-vsw primary

Note that that you can set the MTU to any value between 1500 and 16000.  Of course, whatever you set needs to be supported by the physical network, too.

Another very common area of network configuration is VLAN tagging. This can be a little confusing - my advise here is to be very clear on what you want, and perhaps draw a little diagram the first few times.  As always, keeping a configuration simple will help avoid errors of all kind.  Nevertheless, VLAN tagging is very usefull to consolidate different networks onto one physical cable.  And as such, this concept needs to be carried over into the virtual world.  Enough of the introduction, here's a little diagram to help in explaining how VLANs work in LDoms:
VLANs in LDoms
Let's remember that any VLANs not explicitly tagged have the default VLAN ID of 1. In this example, we have a vswitch connected to a physical network which carries untagged traffic (VLAN ID 1) as well as VLANs 11, 22, 33 and 44.  In this example, I'll assume that the IO domain does not use the vsw as a network interface and therefore we don't have to configure VLANs on the vsw itself.  We also have two vnet devices, one for mars and one for venus.  Venus will see traffic from VLANs 33 and 44 only.  For VLAN 44, venus will need to configure a tagged interface "vnet44000".  For VLAN 33, the vswitch will untag all incoming traffic for venus, so that venus will see this as "normal" or untagged ethernet traffic.  This is very useful to simplify guest configuration and also allows venus to perform Jumpstart or AI installations over this network even if the Jumpstart or AI server is connected via VLAN 33.  Mars, on the other hand, has full access to untagged traffic from the outside world, and also to VLANs 11,22 and 33, but not 44.  On the command line, we'd do this like this:

root@sun # ldm add-vsw net-dev=nxge2 admin-vsw primary
root@sun # ldm add-vnet admin-net pvid=1 vid=11,22,33 admin-vsw mars
root@sun # ldm add-vnet admin-net pvid=33 vid=44 admin-vsw venus

Finally, I'd like to point to a neat little option that will make your live easier in all those cases where configurations tend to change over the live of a guest system.  It's the "id=<somenumber>" option available for both vswitches and vnet devices.  Normally, Solaris in the guest would enumerate network devices sequentially.  However, it has ways of remembering this initial numbering.  This is good in the physical world.  In the virtual world, whenever you unbind (aka power off and disassemble) a guest system, remove and/or add network devices and bind the system again, chances are this numbering will change.  Configuration confusion will follow suit.  To avoid this, nail down the initial numbering by assigning each vnet device it's device-id explicitly:

root@sun # ldm add-vnet admin-net id=1 admin-vsw venus

Please consult the Admin Guide for details on this, and how to decipher these network ids from Solaris running in the guest.

Thanks for reading this far.  Links for further reading are essentially only the Admin Guide and Reference Manual and can be found above.  I hope this is useful and, as always, I welcome any comments.

Montag Dez 05, 2011

Hard Partitioning!

Good news for all users of Oracle VM Server for SPARC (aka LDoms) with Oracle Software:  Since December 2, LDoms count as "Hard Partitioning".  This makes it possible to license only those cores of a server with Oracle software that you really need.  Details are available from License Management Services.

Donnerstag Okt 28, 2010

T3 twice as fast or half as expensive?

I don't usually copy other's work.  But this time, "The Register" was kind enough to do all the tedious price and performance comparisons.  So, thankful for not having to do this myself, I point to their latest article.  It discusses and shows T3 price and performance compared to the earlier T2 systems.  All that's left to ask is: Is T3 twice as fast or half as expensive?  What a nice choice!

Dienstag Sep 21, 2010

No Excuse for no Security

Ever since Sun shipped the UltraSPARC T2 CPU, there was no excuse for not using SSL security for web services.  With the new T3 chips, this is more true than ever.  Not only have the supported algorithms been modernized.  The required documentation on how to use this feature has also been updated.  Find the first two papers hers.  I expect there to be more soon.

Happy encrypting!

Dienstag Jun 01, 2010

Is my application a good fit for CMT ?

CMT-CPUs have been around for quite a while now.  That they were developed for parallel, throughput oriented loads is a well known fact.  However, finding out if a specific application is a good fit for these CPUs seems to remain a challenge, and is one of my personal FAQs.  I'll try to write down a few helpful hints that might help you answer this question yourself.

The first and most important criterion for suitability is always the service time of your application.  If this is sufficient, then the application is OK on CMT. If it is not, and the reason is actually the CPU and not some other high-latency component (like a remote database), you will need to test on other CPU architectures.

It is of course desirable that the application is multi-threaded,  and the individual threads actually perform useful work in parallel.  Only then will the application be able to make use of all the CPU resources available.  It is important to understand that (high) server utilization can never be a criterion for good application performance.  Performance always needs to be measured using application metrics like throughput (transactions per minute) or service time, or both.  Server or CPU utilization only indicate whether the application actually makes good use of the available resources.

Use threadbar to establish if an application is multi threaded and if these threads are actually doing something.  Or, if you're more the commandline type, use prstat -L.  There, the individual threads of your processes are listed, sorted by CPU usage.  What you want to see here is many threads with CPU usage higher than 0%.  What's also important is to check if any thread is limited by the single thread performance of the CPU or strand.  A first estimate of this can also be obtained from prstat.  The column "%CPU" relates to all active CPU (strands) in your system.  For example, a T5120 would show you 64 CPUs.  One strand is therefore equivalent to 1/64 or about 1.5% of the overall system.  What you want is that no one thread of your application consumes 1.5% CPU permanently.  This would be a hint that this thread is limited in performance by the single thread performance of the CPU strand.

I've put these hints and some examples into a small presentation.  Perhaps it's helpful for some of you.


Neuigkeiten, Tipps und Wissenswertes rund um SPARC, CMT, Performance und ihre Analyse sowie Erfahrungen mit Solaris auf dem Server und dem Laptop.

This is a bilingual blog (most of the time). Please select your prefered language:
The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.


« December 2016