Network performance differences within an IP Instance vs. across IP Instances
By Stw-Oracle on May 12, 2007
With IP Instances in Solaris Nevada build 57 and targeted for Solaris 10 7/07, there is the ability to configures zones with exclusive IP Instances, thus forcing all traffic leaving a zone out onto the network. This introduces additional network stack processing both on the transmit and the receive. Prompted by some customer questions regarding this, I performed a simple test to measure the difference.
On two systems, a V210 with two 1.336GHz CPUs and 8GB memory, and an x4200 with two dual-core Opteron XXXX and 8GB memory, I ran FTP transfers between zones. My switch is a Netgear GS716T Smart Switch with 1Gbps ports. The V210 has four bge interfaces and the x4200 has four e1000g interfaces.
I created four zones. Zones x1 and x2 have eXclusive IP Instances, while zones s1 and s2 have Shared IP Instances (IP is shared with the global zone). Both systems are running Solaris 10 7/07 build 06.
Relevant zonecfg info is a follows (all zones are sparse):
As a test user in each zone, I created a file using 'mkfile 1000m /tmp/file1000m'. Then I used ftp to transfer it between zones. No tuning was done whatsoever.
v210# zonecfg -z x1 info zonename: x1 zonepath: /localzones/x1 ... ip-type: exclusive net: address not specified physical: bge1 v210# zonecfg -z s1 info zonename: s1 zonepath: /localzones/s1 ... ip-type: shared net: address: 10.10.10.11/24 physical: bge3
The results are as follows.
V210: (bge) Exclusive to Exclusive x1# /usr/bin/time ftp x2 << EOF\^Jcd /tmp\^Jbin\^Jput file1000m\^JEOF real 17.0 user 0.2 sys 11.2 Exclusive to Shared x1# /usr/bin/time ftp s2 << EOF\^Jcd /tmp\^Jbin\^Jput file1000m\^JEOF real 17.3 user 0.2 sys 11.6 Shared to Shared s2# /usr/bin/time ftp s1 << EOF\^Jcd /tmp\^Jbin\^Jput file1000m\^JEOF real 6.6 user 0.1 sys 5.3 X4200: (e1000g) Exclusive to Exclusive x1# /usr/bin/time ftp x2 << EOF\^Jcd /tmp\^Jbin\^Jput file1000m\^JEOF real 9.1 user 0.0 sys 4.0 Exclusive to Shared x1# /usr/bin/time ftp s2 << EOF\^Jcd /tmp\^Jbin\^Jput file1000m\^JEOF real 9.1 user 0.0 sys 4.1 Shared to Shared s2# /usr/bin/time ftp s1 << EOF\^Jcd /tmp\^Jbin\^Jput file1000m\^JEOF real 4.0 user 0.0 sys 3.5I ran each test several times and picked a result that seemed average across the runs. Not very scientific, and a table might be nicer.
Something I noticed that surprised me was that time spent in IP and the driver is measurable on the V210 with bge, and much less so on the x4200 with e1000g.