Monitoring packet distribution on Sun Multithreaded 10 Gigabit Ethernet

10 Gigabit Ethernet NIC relies on multiple transmit and receive DMA channels to achieve good performance. One performance problem I have encountered sometimes in the past is traffic imbalance on receive side, leading to some CPU becoming interrupt-bound. I have written a Perl script for Sun Multithreaded 10GbE that can display the packet distribution by TX and RX DMA channels and can help spot the problem easily. The script used kstat provided by nxge. Usage:

nxge-kstat-channel.pl -i <instance#> -I <interval> 

For example, to see packet distribution on nxge0,

nxge-kstat-channel.pl -i 0

The script nxge-kstat-channel.pl is shown below:

: # \*-\*-perl-\*-\*
    eval 'exec perl -S $0 "$@"'
    if $running_under_some_shell;

use Getopt::Std;
getopts("i:I:");

sub usage
{
        # command usage list
        printf("Usage: nxge-kstat-channel.pl -i <instance#> -I <interval>\\n");
        exit(-1);
}


if (!$opt_I) { $opt_I = 1;}

$cmd_tx = "kstat nxge:".$opt_i." | grep tdc_pack";
# print $cmd_tx;
$cmd_rx = "kstat nxge:$opt_i | grep rdc_pack";

@lines = `$cmd_tx`;
for ($i=0; $i < scalar(@lines); ++$i) {
        @tmp = split /[ ]+/,$lines[$i];
        chop $tmp[1];
        $tx_pkts[$i] = $tmp[1];
}

@lines = `$cmd_rx`;
for ($i=0; $i < scalar(@lines); ++$i) {
        @tmp = split /[ ]+/,$lines[$i];
        chop $tmp[1];
        $rx_pkts[$i] = $tmp[1];
}

print "Packets/sec distribution on nxge".$opt_i."\\n";
while(1) {
        @tx_lines = `$cmd_tx`;
        @rx_lines = `$cmd_rx`;

        print "TX: ";
        $nl=0;
        for ($i=0; $i < scalar(@tx_lines); ++$i) {
                @tmp = split /[ ]+/,$tx_lines[$i];
                chop $tmp[1];
                printf("Ring %2d: %-8d", $i, ($tmp[1] - $tx_pkts[$i]) / $opt_I);
                if (($i % 4) == 3) { print "\\n    "; $nl=1;}
                $tx_pkts[$i] = $tmp[1];
        }
        if (!$nl) { print "\\n"; }
        print "\\r";
        print "RX: ";
        for ($i=0; $i < scalar(@rx_lines); ++$i) {
                @tmp = split /[ ]+/,$rx_lines[$i];
                chop $tmp[1];
                printf("Ring %2d: %-8d", $i, ($tmp[1] - $rx_pkts[$i]) / $opt_I);
                if (($i % 4) == 3) { print "\\n    "; }
                $rx_pkts[$i] = $tmp[1];
        }
        print "\\n";
        # sleep 1;
        # print "\\nSleep $opt_i seconds ...\\n";
        sleep $opt_I;
} 

The script displays packet rate on all nxge interfaces if no arguments are given.

If packet distribution on receive channels are not balanced, you can tune receive load balancing policy to balance them by ndd:

ndd /dev/nxge1 class_opt_ipv4_tcp <LB policy>
0x0010:         use MAC Port (for flow key)
0x0020:         use L2DA (for flow key)
0x0040:         use VLAN (for flow key)
0x0080:         use proto (for flow key)
0x0100:         use IP src addr (for flow key)
0x0200:         use IP dest addr (for flow key)
0x0400:         use Src Port (for flow key)
0x0800:         use Dest Port (for flow key)
0xF80:          use IP 5-tuple (for flow key)
Comments:

nice script, but if one is seeing an imbalance what can one do about it?

Also, is it common to see an imbalance when you only have a single stream running? (i.e., netbackup server with only one client active)

Posted by john on February 27, 2009 at 01:59 AM PST #

Why not use the built in Sun::Solaris::KStat perm module instead of all the forking?

Posted by Jason on February 27, 2009 at 05:30 AM PST #

Post a Comment:
Comments are closed for this entry.
About

user12608924

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today