Solaris Networking | Monday, August 8, 2011

Solaris 11 Express Network Tunables

Overview


For years I, and many others, have been tuning TCP, UDP, IP, and other aspects of the Solaris network stack with ndd(1M). The ndd command is documented, however, most of the tunables were really private interface implementations, subject to change, and lacked documentation in many cases. Also, ndd does not show the default values, nor the possible values or ranges..

That is changing with Solaris 11 Express. A new command ipadm(1M) allows persistent and temporary (with the -t option) setting of key tunable values. This is a major improvement over ndd, where it is customary to create an /etc/rc2.d/S69ndd or similar script to set the parameter on every reboot. Another benefit is that ipadm shows the default value and the values that the property can be set to.

The ipadm has many features to configure the IP settings of interfaces. This blog entry focuses on how ipadm replaces ndd. Note that ipadm only supports the IP, TCP, UDP, SCTP, and ICMP protocols. Other protocols such as ipsecah and keysock still required the use of ndd.

Review of ndd


To get a list of all tunables for a specific protocol, an ndd -get operation is performed with "?" as the argument. For example, this is a way of listing all the TCP parameters.
root@Solaris11Express# ndd -get /dev/tcp \?
tcp_time_wait_interval (read and write)
tcp_conn_req_max_q (read and write)
tcp_conn_req_max_q0 (read and write)
tcp_conn_req_min (read and write)
...
tcp_dev_flow_ctl (read and write)
tcp_reass_timeout (read and write)
tcp_extra_priv_ports_add (write only)
tcp_extra_priv_ports_del (write only)
tcp_extra_priv_ports (read only)
tcp_1948_phrase (write only)
tcp_listener_limit_conf (read only)
tcp_listener_limit_conf_add (write only)
tcp_listener_limit_conf_del (write only)

To get the current value of specific parameter, list the parameter as the argument for the driver, in this case /dev/tcp.
root@Solaris11Express# ndd -get /dev/tcp tcp_conn_req_max_q
128

And to set parameter, follow it with a value.
root@Solaris11Express# ndd -set /dev/tcp tcp_conn_req_max_q 256
root@Solaris11Express# ndd -get /dev/tcp tcp_conn_req_max_q
256

And for my own benefit, I set it back to the original.
root@Solaris11Express# ndd -set /dev/tcp tcp_conn_req_max_q 128
root@Solaris11Express# ndd -get /dev/tcp tcp_conn_req_max_q
128

Using the ipadm *-prop Options


The ipadm(1M) manual page lists three sub-commands to manage TCP/IP protocol properties.
     ipadm set-prop [-t] -p prop=value[,...] protocol
ipadm reset-prop [-t] -p prop protocol
ipadm show-prop [[-c] -o field[,...]] [-p prop[,...]] [protocol]

To list all the properties for all the protocols as currently supported, I run ipadm with the show-prop sub-command.
root@Solaris11Express# ipadm show-prop
PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
ipv4 forwarding rw off -- off on,off
ipv4 ttl rw 255 -- 255 1-255
ipv6 forwarding rw off -- off on,off
ipv6 hoplimit rw 255 -- 255 1-255
ipv6 hostmodel rw weak -- weak strong,
src-priority,
weak
ipv4 hostmodel rw weak -- weak strong,
src-priority,
weak
icmp recv_maxbuf rw 8192 -- 8192 4096-65536
icmp send_maxbuf rw 8192 -- 8192 4096-65536
tcp ecn rw passive -- passive never,passive,
active
tcp extra_priv_ports rw 2049,4045 -- 2049,4045 1-65535
tcp largest_anon_port rw 65535 -- 65535 1024-65535
tcp recv_maxbuf rw 128000 -- 128000 2048-1073741824
tcp sack rw active -- active never,passive,
active
tcp send_maxbuf rw 49152 -- 49152 4096-1073741824
tcp smallest_anon_port rw 32768 -- 32768 1024-65535
tcp smallest_nonpriv_port rw 1024 -- 1024 1024-32768
udp extra_priv_ports rw 2049,4045 -- 2049,4045 1-65535
udp largest_anon_port rw 65535 -- 65535 1024-65535
udp recv_maxbuf rw 57344 -- 57344 128-1073741824
udp send_maxbuf rw 57344 -- 57344 1024-1073741824
udp smallest_anon_port rw 32768 -- 32768 1024-65535
udp smallest_nonpriv_port rw 1024 -- 1024 1024-32768
sctp extra_priv_ports rw 2049,4045 -- 2049,4045 1-65535
sctp largest_anon_port rw 65535 -- 65535 1024-65535
sctp recv_maxbuf rw 102400 -- 102400 8192-1073741824
sctp send_maxbuf rw 102400 -- 102400 8192-1073741824
sctp smallest_anon_port rw 32768 -- 32768 1024-65535
sctp smallest_nonpriv_port rw 1024 -- 1024 1024-32768

The first column lists the protocols. Of note is that there are separate IPv4 and IPv6 listings. Per the specification, there is no ttl for IPv6, as is seen by only an IPv4 property. IPv6 calls it the hoplimit, which is more indicative of how the value is actually used.

Including a protocol as an argument lists only those properties.

root@Solaris11Express# ipadm show-prop tcp
PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
tcp ecn rw passive -- passive never,passive,
active
tcp extra_priv_ports rw 2049,4045 -- 2049,4045 1-65535
tcp largest_anon_port rw 65535 -- 65535 1024-65535
tcp recv_maxbuf rw 128000 -- 128000 2048-1073741824
tcp sack rw active -- active never,passive,
active
tcp send_maxbuf rw 49152 -- 49152 4096-1073741824
tcp smallest_anon_port rw 32768 -- 32768 1024-65535
tcp smallest_nonpriv_port rw 1024 -- 1024 1024-32768

We see the current value, whether we can set it, its default value, and the possible values or range of values. Self documenting. I like it!

To get a specific property, the -p option specifies which one to list.

root@Solaris11Express# ipadm show-prop -p send_maxbuf tcp
PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
tcp send_maxbuf rw 49152 -- 49152 4096-1073741824

Now to set a property to a specific value, use the format property=value.
root@Solaris11Express# ipadm set-prop -p send_maxbuf=4096 tcp
root@Solaris11Express# ipadm show-prop -p send_maxbuf tcp
PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
tcp send_maxbuf rw 4096 4096 49152 4096-1073741824

The value of 4096 in the PERSISTENT column indicates this setting will be retained even after a reboot. To set the property only until the next reboot, use the -t option to set it temporarily.
root@Solaris11Express# ipadm set-prop -t -p send_maxbuf=4096 tcp
root@Solaris11Express# ipadm show-prop -p send_maxbuf tcp
PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
tcp send_maxbuf rw 4096 -- 49152 4096-1073741824

While it certainly possible to set the value of property back to the same one that is the default, I like the option to set it to its default. This is done with a reset. The PERSISTENT column has reverted back to its original --.
root@Solaris11Express# ipadm reset-prop -p send_maxbuf tcp
root@Solaris11Express# ipadm show-prop -p send_maxbuf tcp
PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
tcp send_maxbuf rw 49152 -- 49152 4096-1073741824

What About All Those Other ndd Configuration Parameters?


The output of the show-prop operation above is very small compared to what those who use ndd are used to for even just one of the protocols. So what about all the other ndd parameters?

There are two options:

  • Continue to use ndd
  • Use a special parameter conversion of the ndd parameter with ipadm


  • The first is business as usual. The second involves converting the protocol's ndd parameter into one that works with ipadm. The steps that have worked for me are as follows.

    • For any parameter, replace the /dev/protocol and use the protocol as the protocol argument to ipadm. So /dev/tcp becomes tcp.
    • Drop the leading protocol name from the beginning of the parameter, if there is one. So tcp_local_dack_interval becomes _local_dack_interval.
    • If there is no leading procotol name, prepend the property with an underscore (_). For example, tcp_local_dack_interval becomes _tcp_local_dack_interval.
    • For the IP protocol, if there are IPv4 and IPv6 ndd values, indicate the ipadm protocol as ipv4 and ipv6, respectively. With ndd, the lack of a 6 means IPv4.

    Examples of each are as follows.

    Dropping the leading protocol name and specifying it for the protocol argument.

    root@Solaris11Express# ndd -get /dev/tcp tcp_local_dack_interval
    50
    root@Solaris11Express# ipadm show-prop -p _local_dack_interval tcp
    PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
    tcp _local_dack_interval rw 50 -- 50 10-500

    Getting a parameter that does not start with the protocol.
    root@Solaris11Express# ndd -get /dev/ip arp_probe_interval
    1500
    root@Solaris11Express# ipadm show-prop -p _arp_probe_interval ip
    PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
    ip _arp_probe_interval rw 1500 -- 1500 10-20000

    Distinguishing between IPv4 and IPv6 parameters.
    root@Solaris11Express# ndd -get /dev/ip ip_strict_dst_multihoming
    0
    root@Solaris11Express# ndd -get /dev/ip ip6_strict_dst_multihoming
    0
    root@Solaris11Express# ipadm show-prop -p _strict_dst_multihoming ipv4
    PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
    ipv4 _strict_dst_multihoming rw 0 -- 0 0-1
    root@Solaris11Express# ipadm show-prop -p _strict_dst_multihoming ipv6
    PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
    ipv6 _strict_dst_multihoming rw 0 -- 0 0-1

    And when there is an error. All the fields have ? in them.
    root@Solaris11Express# ipadm show-prop -p _strict_dst_multihoming ip
    PROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLE
    ipadm: warning: cannot get property '_strict_dst_multihoming' for 'ip'Unknown property
    ip _strict_dst_multihoming ? ? ? ? ?

    As more properties are added to ipdam to manage there directly, it will become less necessary to do the ndd work-around.

    Join the discussion

    Comments ( 1 )
    • guest Tuesday, January 29, 2013

      Very useful article. Thanks!


    Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
     

    Visit the Oracle Blog

     

    Contact Us

    Oracle

    Integrated Cloud Applications & Platform Services