Tuesday Nov 10, 2009

A D Script for Observing the Network

A little gem included with the Crossbow Virtual Wire Demo Tool is a DTrace script and associated Chime display written by Crossbow engineer Kais Belgaied. I've augmented the script a bit by adding an END clause to format the output, which shows the number of bits transferred over the period of time the script is run:[Read More]

Friday Oct 30, 2009

Crossbow Virtual Wire Demo Tool

At the CommunityOne West keynote last June a demo of Crossbow was given using a tool called Virtual Wire. If you haven't seen the demo before, you can watch a 10 minute replay of it here:[Read More]

Wednesday Oct 07, 2009

Netperf

Netperf is a cool little utility that I discovered while working with Network Virtualization. It's technically a network benchmarking tool, but it's fun to use to hammer your network with load and test out the bandwidth control features provided by network virtualization.

Netperf used to be available in the now decommissioned contrib repository. That's OK, it's easy enough to build.

Building Netperf for Solaris

If you want to save yourself the steps below, just save the following binaries to your /usr/bin directory: netperf and netserver. Here are the instructions from the Netperf manual.

  1. Download Netperf.  At the time of this writing the latest stable build is netperf-2.4.5.

  2. Extract the archive:
    bleonard@solaris:~/Downloads$ tar -xvf netperf-2.4.5.tar.bz2 
    netperf-2.4.5/
    netperf-2.4.5/src/
    ...
    netperf-2.4.5/doc/examples/udp_stream_script
    netperf-2.4.5/doc/netperf.info
    
    
  3. Make sure you have gcc-3 and header-math installed:
    bleonard@solaris:~$ pkg list gcc-3 header-math
    NAME (PUBLISHER)                              VERSION         STATE      UFOXI
    developer/gcc-3                               3.4.3-0.151.0.1 installed  -----
    system/library/math/header-math               0.5.11-0.151.0.1 installed  -----
  4. Run configure, overriding the default install directory of /usr/local to /usr:
    bleonard@solaris:~/Downloads/netperf-2.4.5$ ./configure --prefix=/usr
    checking build system type... i386-pc-solaris2.11
    ...
    config.status: executing depfiles commands

  5. Run make:
    bleonard@solaris:~/Downloads/netperf-2.4.5$ make
    make  all-recursive
    make[1]: Entering directory `/home/bleonard/Downloads/netperf-2.4.5'
    ..
    make[1]: Leaving directory `/home/bleonard/Downloads/netperf-2.4.5'
    
  6. The run make install:
    bleonard@solaris:~/Downloads/netperf-2.4.5$ sudo make install
    Password: 
    Making install in src
    make[1]: Entering directory `/home/bleonard/Downloads/netperf-2.4.5/src'
    ...
    make[1]: Leaving directory `/home/bleonard/Downloads/netperf-2.4.5'
    
  7. Verify the binaries were made:
    bleonard@solaris:~$ which netperf netserver
    /usr/bin/netperf
    /usr/bin/netserver
    

Using Netperf

To verify the installation, first start the benchmark server:

bleonard@solaris:~$ netserver
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC

Then run netperf over the loopback interface, which will run a TCP_STREAM test of 10 seconds:

bleonard@solaris:~$ netperf -H 127.0.0.1
TCP STREAM TEST from ::ffff:0.0.0.0 (0.0.0.0) port 0 AF_INET to ::ffff:127.0.0.1 (127.0.0.1) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

128000  49152  49152    9.99     19036.97   

Now that we know Netperf is set up correctly, we can use it to test bulk data transfer performance between 2 hosts (or 2 zones). TCP stream performance is the default test type and it's simply performed by running the command:

netperf -H <remotehost>

which will run a 10 second test between the local and remote system.

So I'm going to use Netperf to run some tests between the global zone and a local zone, which has been set up to use a virtual NIC exclusively. To create such a configuration on your own, see the blog entry Zones and Network Virtualization.

The local zone, myzone, is currently up and running:

bleonard@solaris:~$ zoneadm list -cv
  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              native   shared
   - myzone           running    /zones/myzone                  ipkg     excl 

The first thing I'm going to do is copy the netserver program over to the local zone:

bleonard@solaris:~$ sudo cp /usr/bin/netserver /zones/myzone/root/usr/bin/.

And then start the netserver:

bleonard@solaris:~$ sudo zlogin myzone /usr/bin/netserver
Password: 
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC

Now let's test the connection between the global and local zone:

bleonard@solaris:~$ netperf -H 10.0.1.25
TCP STREAM TEST from ::ffff:0.0.0.0 (0.0.0.0) port 0 AF_INET to ::ffff:10.0.1.25 (10.0.1.25) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

128000  49152  49152    10.00    1722.31  

Here we can see the throughput between my zones is 1722 Mbit/s. Now let's reduce the max bandwidth of the virtual NIC to 500 Mbit/s and try the test again:

bleonard@solaris:~$ sudo dladm set-linkprop -p maxbw=500 myzone0

bleonard@solaris:~$ dladm show-vnic
LINK         OVER         SPEED  MACADDRESS           MACADDRTYPE         VID
myzone0      e1000g0      500    2:8:20:59:0:b5       random              0

bleonard@solaris:~$ netperf -H 10.0.1.25
TCP STREAM TEST from ::ffff:0.0.0.0 (0.0.0.0) port 0 AF_INET to ::ffff:10.0.1.25 (10.0.1.25) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

128000  49152  49152    10.00     482.77   

Or at a ridiculously low 2 Mbit/s:

bleonard@solaris:~$ sudo dladm set-linkprop -p maxbw=2 myzone0

bleonard@solaris:~$ netperf -H 10.0.1.25
TCP STREAM TEST from ::ffff:0.0.0.0 (0.0.0.0) port 0 AF_INET to ::ffff:10.0.1.25 (10.0.1.25) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

128000  49152  49152    10.38       1.07  

Good stuff.

I've just scratched the surface of Netperf, but this simple introduction suits my purposes for testing network virtualization. For more fun check out the Netperf manual.

Tuesday Jul 14, 2009

Zones and Network Virtualization

If you're like me and working with zones on your laptop and/or desktop, you probably only have one network interface card to work with. Therefore, the zones I've created share the single network interface with the global zone (ip-type=shared).

Behind the scenes, Solaris creates a logical interface for the zone to use. The logical interface appears in ifconfig as your physical interface with an instance number. For example:

bleonard@solaris:~$ ifconfig -au4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
	inet 127.0.0.1 netmask ff000000 
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
	zone myzone
	inet 127.0.0.1 netmask ff000000 
e1000g0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 2
	inet 10.0.1.10 netmask ffffff00 broadcast 10.0.1.255
e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
	zone myzone
	inet 10.0.1.25 netmask ffffff00 broadcast 10.0.1.255

You can see both the loopback (loO) and physical (e1000g0) have an instance (lo0:1 and e1000g0:1) that was created for the zone myzone. These logical interfaces only exist when the zone is running. If you halt the zone, they disappear.

From inside the zone, I only see the logical interfaces:

root@myzone:~# ifconfig -au4
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 10.0.1.25 netmask ffffff00 broadcast 10.0.1.255

However, I have no control over them. For example, if I try to bring down e1000g0:1:

root@myzone:~# ifconfig e1000g0:1 inet down
ifconfig: setifflags: SIOCSLIFFLAGS: e1000g0:1: permission denied 

The global zone is responsible for managing the local zone's network configuration.

Network Virtualization

Oracle Solaris 11 introduces network virtualization technology. For example, I can create a virtual network interface card (vnic) that has all the properties of a physical nic.

bleonard@solaris:~$ sudo dladm create-vnic -l e1000g0 myzone0
bleonard@solaris:~$ dladm show-link 
LINK        CLASS    MTU    STATE    OVER
e1000g0     phys     1500   up       --
iwh0        phys     1500   down     --
vboxnet0    phys     1500   unknown  --
myzone0     vnic     1500   up       e1000g0

Now it's as if my laptop has 2 physical network interface cards. Using this "new" card, I can create a zone with an exclusive IP stack. My zone config would look something like follows:

bleonard@solaris:~$ cat myzone.config
create
set zonepath=/zones/myzone
set ip-type=exclusive	
add net
set physical=myzone0
end

Note there's no longer an IP address associated with the zone configuration. With a dedicated IP stack the zone will be able to manage its own IP.

Create the zone:

bleonard@solaris:~$ sudo zonecfg -z myzone -f myzone.config

Install the zone:

bleonard@solaris:~$ sudo zoneadm -z myzone install
A ZFS file system has been created for this zone.
   Publisher: Using solaris (https://pkg.oracle.com/solaris/support/ ).
       Image: Preparing at /zones/myzone/root.
 Credentials: Propagating Oracle_Solaris_11_Express_Support.key.pem
 Credentials: Propagating Oracle_Solaris_11_Express_Support.certificate.pem
       Cache: Using /var/pkg/download.
Sanity Check: Looking for 'entire' incorporation.
  Installing: Core System (output follows)
               Packages to install:     1
           Create boot environment:    No
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1         1/1      0.0/0.0

PHASE                                        ACTIONS
Install Phase                                  11/11 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 
               Packages to install:    45
           Create boot environment:    No
               Services to restart:     3
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                45/45 12511/12511    89.1/89.1

PHASE                                        ACTIONS
Install Phase                            17958/17958 

PHASE                                          ITEMS
Package State Update Phase                     45/45 
Image State Update Phase                         2/2 
  Installing: Additional Packages (output follows)
               Packages to install:    46
           Create boot environment:    No
               Services to restart:     2
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                46/46   4498/4498    26.5/26.5

PHASE                                        ACTIONS
Install Phase                              6143/6143 

PHASE                                          ITEMS
Package State Update Phase                     46/46 
Image State Update Phase                         2/2 

        Note: Man pages can be obtained by installing SUNWman
 Postinstall: Copying SMF seed repository ... done.
 Postinstall: Applying workarounds.
        Done: Installation completed in 486.420 seconds.

  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.

Create a configuration file for the zone. Note, here we can define the zone's IP configuration (or we could do it later):

bleonard@solaris:~$ cat sysidcfg
system_locale=C
terminal=xterms
network_interface=myzone0 {
	hostname=myzone
	ip_address=10.0.1.25
        default_route=NONE
	netmask=255.255.255.0
 	protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI

Copy the sysidcfg file to the zone:

bleonard@solaris:~$ sudo cp sysidcfg /zones/myzone/root/etc/.

Boot the zone:

bleonard@solaris:~$ sudo zoneadm -z myzone boot

Log into zone. The first login will take some time as the zone completes it's system configuration:

bleonard@solaris:~$ sudo zlogin -C myzone
[Connected to zone 'myzone' console]
100/100
Hostname: myzone
Loading smf(5) service descriptions: 3/3
 network_interface=myzone0 {
myzone0 is not a valid network interface  line 3 position 19
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: myzone0.
Note the message about myzone0 being an invalid network interface. This appears to be benign as a few lines down we see myzone0 getting configured. If you used the root_password setting from above, you can log in as root/abc123:
myzone console login: root
Password: abc123
May 31 08:30:02 myzone login: ROOT LOGIN /dev/console
Oracle Corporation      SunOS 5.11      snv_151a        April 2011
root@myzone:~#

As with shared IP, you can see the interface using ifconfig:

root@myzone:~# ifconfig -au4
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
myzone0: flags=1000863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 10.0.1.25 netmask ffffff00 broadcast 10.0.1.255
        ether 2:8:20:59:0:b5 

However, now you can also manage it. For example:

root@myzone:~# ifconfig myzone0 down
root@myzone:~# ifconfig -au4
lo0: flags=2001000849<⁞UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 

And back in the global zone, there's no more logical interfaces cluttering up the ifconfig output:

bleonard@solaris:~$ ifconfig -au4
lo0: flags=2001000849<UP,LOOPBACK,⁞RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
	inet 127.0.0.1 netmask ff000000 
e1000g0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 2
	inet 10.0.1.10 netmask ffffff00 broadcast 10.0.1.255
In addtion to this, virtual nics provide a whole bunch of control over the data passing through the network interface. For a brief introduction to that see Fun with Crossbow.

Monday Jun 29, 2009

Fun with Crossbow

The Crossbow project is probably the most exciting new feature in OpenSolaris 2009.06. It a nutshell, project Crossbow brings virtualization to the networking layer. In this quick example I'm going to create a virtual network interface card (VNIC) and dynamically alter it's maximum bandwidth as traffic is flowing over it.[Read More]
About

The Observatory is a blog for users of Oracle Solaris. Tune in here for tips, tricks and more as we explore the Solaris operating system from Oracle.

Connect with Oracle Solaris:


Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
20
21
22
23
24
25
26
27
28
29
30
   
       
Today