Thursday Nov 14, 2013

Performance Analysis of Oracle Traffic Director or Web Server

Performance Analysis of Oracle Trafiic Director or Oracle iPlanet Web Server using Oracle Sun Solaris Studio Performance Analyzer and DTrace scripts.

In this blog I will show how to use the Sun Studio Collector and Performance Analyzer and DTrace script to measure the performance.

1. Using Oracle Solaris Studio 12.2 Performance Analyzer with Oracle Traffic Director or Oracle iPlanet Web Server

Thanx to Basant for teaching me collector and analyzer and thanx to Julien for his help on DTrace scripts.

Install Sun Studio 12. Lets say is installed in /opt/SUNWSpro.

1.1 setting up ~/.er.rc

cat ~/.er.rc
dmetrics e.user:i.user:e!wall:i!wall:e.total:i.total:e.system:i.system:e!wait: \
i!wait:e!lock:i!lock:e!text:i!text:e!data:i!data:e!owait:i!owait:!size:!address:name
dsort e.user
scc basic:version:warn:parallel:query:loop:pipe:inline:memops:fe:cg
dcc basic:version:warn:parallel:query:loop:pipe:inline:memops:fe:cg:src
sthresh 75
dthresh 75
name long
view user
tlmode thread:root:depth=10
tldata sample:clock:hwc:heaptrace:synctrace:mpitrace:msgtrace:datarace
setpath $expts:.
tabs functions:callers-callees:source:disasm:timeline:header:
en_desc on

1.2 Collecting Data Using the collect Command

1.2.1& Run the Collector  using the collect command

# collect collect-options program program-arguments

For OTD or Web Server we edit start script bin/startserv to have a new --collect option as shown below 

Red color lines are the new lines I have added.  Copied all the lines in start option, replaced ${SERVER_BIN} (which is trafficd-wdog) by "trafficd". And added "collect $COLLECT_OPTS" before it.

In the section below replace <profiler directory> by the directory where you want the profiler to collect data.

case $COMMAND in
    --start|-start) 
        ${SERVER_BIN} -d "${SERVER_CONFIG_DIR}" \ 
            -r "${OTD_PRODUCT_HOME}" -t "${SERVER_TEMP_DIR}" \
            -u "${SERVER_USER}" ${SVC_OPT} $@
        STATUS=$?
        if [ $STATUS -ne 0 ] ; then 
             exit $STATUS
        fi
        enable_failover
        ;;
    --collect|-collect)
        PATH=$PATH:/opt/SUNWspro/bin
        COLLECT_OPTS="-t 180-300 -F all -d <profiler directory>";
        collect $COLLECT_OPTS trafficd -d "${SERVER_CONFIG_DIR}" \
            -r "${OTD_PRODUCT_HOME}" -t "${SERVER_TEMP_DIR}" \ 
            -u "${SERVER_USER}" ${SVC_OPT} $@
        STATUS=$?
        if [ $STATUS -ne 0 ] ; then
            exit $STATUS
        fi
        ;;

for web server instead of trafficd it will be webservd. 

1.2.2 start the server using --collect option

Since we replaced trafficd-wdog by trafficd, the server will start up without any watchdog process and will run in console, not in background.  

bin/startserv --collect

1.2.3 run some stress tests

1.2.4 After 5-6 minutes stop the server

This will create a directory called (e.g.) 'test.1.er' which contains the experiment.

The default name for a new experiment is test.1.er.

The Collector automatically increments n by one in the names of subsequent experiments.

1.3 Open the profile and start the Oracle Sun Studio 12.2 analyzer

Set DISPLAY env.

$ cd <profiler directory>
$ export JAVA_PATH=/opt/SUNWwbsvr/jdk 
$ /opt/SUNWspro/bin/analyzer test.1.er/

1.4 The er_print utility prints an ASCII version of the various displays supported by the Performance Analyzer.

$ /opt/SUNWspro/bin/er_print -outfile er_print1.out -functions test.1.er 

In this, functions are sorted by "Exclusive User CPU Time"

or

$ /opt/SUNWspro/bin/er_print -outfile er_print2.out \
-metrics e.user:i.user:e.%user:i.%user:e.total:i.total:e.%total:i.%total:name \
-sort i.total -functions test.1.er 

In this, functions are sorted by "Inclusive Total LWP Time"

You can look at these files and figure out which function is taking how much time.

These files look like :

Functions sorted by metric: Exclusive User CPU Time 
Excl.     Incl.     Excl.       Incl.       Excl.     Incl.      Name  
User CPU  User CPU  Total LWP   Total LWP   Sys. CPU  Sys. CPU         
   sec.      sec.         sec.        sec.     sec.      sec.     
851.666   851.666   105477.573  105477.573  106.525   106.525    <Total>
493.435   493.435      558.110     558.110   26.368    26.368    fn1
 56.840   326.308       64.015     368.568    3.042    17.352    fn2
 28.280    28.280       34.574      34.574    1.701     1.701    fn3

...


2. Using DTrace script to see how much time is spent in which function

#!/usr/sbin/dtrace -s
#pragma D option bufsize=1g
#pragma D option specsize=1g
#pragma D option aggsize=1g
#pragma D option dynvarsize=1g

pid$target:::entry
{
 self->ts[probefunc] = timestamp;
}

pid$target:::return
/self->ts[probefunc]/
{
    @time[probefunc, probemod] = sum(timestamp - self->ts[probefunc]);
    self->ts[probefunc] = 0;
}

Note I have given 1g sizes you can tune it as per your machine.

Run this D script using :

#sudo dtrace -s functime.d -p 27910 -o dtrace.log 

where 27910 is the pid of the process you are examining (in this case webservd or trafficd).

This will generate the output as shown below 

  PListFindValue                       libns-httpd40.so                                        3871
  getbucketnum                         libc.so.1                                               4995
  R_SSL_version                        libnnz11.so                                             6106
  http_format_server                   libns-httpd40.so                                        6807
  void SimpleHashTemplateBase::_insert(unsigned long,hashEntry*)  libns-httpd40.so             7059
  long atoi64(const char*)             libns-httpd40.so                                        7288

...


I wrote this wrapper script to report percentages.

To get the last column I used logic of looking for spaces after the string "lib". For LM1`ld.so.1, I temporarily added a hack.

#!/usr/bin/perl
$logfile = "dtrace.log";
$tmpfile = "temp";
open(IN,"<",$logfile) || die "Can not open $logfile: $!";
$total = 0;
$total_rounded = 0;
while (<IN>) {
    chomp($_);
    s/(.*\s*)lib([^\s]*)\s*([0-9]*)/$1lib$2=$3/g;
    my ($a, $b) = split('=');
    $total += $b;
}
print "total = $total\n";
close(IN);
open(OUT,">", $tmpfile) || die "Can not open $tmpfile: $!";
open(IN,"<",$logfile) || die "Can not open $logfile: $!";
while (<IN>) {
    chomp($_);
    s/LM1`ld.so.1/libld.so.1/g;
    s/(.*\s*)lib([^\s]*)\s*([0-9]*)/$1lib$2=$3/g;
    my ($a, $b) = split('=');
    $rounded = sprintf("%.10f", ($b*100)/$total);
    $a =~ s/libld.so.1/LM1-ld.so.1/g;
    print OUT "$b   $rounded%   $a\n";
    $total_rounded += $rounded;
}
close(IN);
close(OUT);
print "total rounded = $total_rounded\n";
`sort -n -r $tmpfile | tee $logfile.sorted`;
`rm $tmpfile`


This produced output of the following format :

20318357230394   16.4117840797%     poll                                      libc.so.1
20317702791746   16.4112554688%     _pollsys                                  libc.so.1
20313615393944   16.4079539474%     __pollsys                                 libc.so.1
2684593654698   2.1684317735%     int DaemonSession::GetConnection()          libns-httpd40.so

...

3. Using DTrace Profile Probes

 #!/usr/sbin/dtrace -s
profile-1000
/pid == $1 && arg1 != NULL/
{
    @proc[umod(arg1), ufunc(arg1),ustack()] = count(); 
}
END
{
    printa(@proc); 
}


run it as :

dtrace -x ustackframes=20 -s profiler-probes.d <pid> -o dtrace.log

It creates output in the format library name, function name, user stack and the count number of times called, newline

  libc.so.1                                           libc.so.1`mutex_lock_impl        
              libc.so.1`mutex_lock_impl+0xaf
              libc.so.1`mutex_lock+0xb
              libumem.so.1`umem_cache_alloc+0x33
              libumem.so.1`umem_alloc+0xaf
              libumem.so.1`malloc+0x2e
              libnnz11.so`sys_malloc+0x23
             1068

....



4. References

Friday Oct 11, 2013

Configuring Server Name Indication (SNI) in Oracle Traffic Director 11.1.1.6 and 11.1.1.7

What is SNI ? It is explained very well in http://en.wikipedia.org/wiki/Server_Name_Indication

If your SSL server needs certificate(s) for different domains, you can choose one of the different options :
  • Use multiple certificates using SNI feature (configure server to return different certificates for different domains) - recommended
  • Use a single certificate with SubjectAltName Extension (one hostname in CN and other hostnames in SubjectAltName extension in the certificate)
  • Use a single certificate with wild card in subject (lets say certificate with "CN=*.*.oracle.com", so it will be valid for different domains) - not preferred
  • Notes

    • Unbound Virtual Server: <virtual-server> doesn't have <http-listener> as a sub element.
    • Bound Virtual Server: <virtual-server> has a <http-listener> sub element, it is said to be bound to that http listener.
    • To figure out which Virtual server is the Default Virtual Server for a listener, look at the Virtual Server name in <default-virtual-server> of <http-listener> in server.xml.

    How to configure SNI in Oracle Traffic Director

    In this blog I will cover the following

    Enable SSL on an HTTP listener and create a certificate for it. Create two Virtual Servers both bound to an HTTP listener. One of the Virtual Server contains a certificate and the other doesn't. Send SNI and non-SNI requests to those two Virtual Servers.

    Create and add certificate for the default Virtual Server(which could be unbound or bound) and add <host> element value of <host> of our Virtual Server which doesn't have a certificate. Send a SNI request to the virtual server which doesn't have a certificate, it returns certificate from the default virtual server.

    What we will find out  is

    • If SNI host is NOT sent by the browser in SSL Handshake, then the server sends the certificate from the http listener. --------- 1
    • else (i.e. if SNI host is sent by the browser in SSLHandshake)
      • If SNI Host sent by browser doesn't match with a <host> element in any of the bound Virtual server  - goto STEP 2
      • else (i.e. If SNI host sent by browser matches with <host> element of any bound Virtual Server)
        • If that Virtual Server has certificate,  the server sends the certificate from the Virtual Server. ----------- 2
        • else (that Virtual Server DOES NOT have a certificate) - goto STEP 2

    STEP 2: get the default Virtual Server for this http listener :

      • If the default virtual Server DOES NOT have a certificate, then the server sends the cert from the http listener ------- 3
      • else (i.e. If the default virtual Server has a certificate) then the server sends the cert from this default Virtual Server ------- 4

    Exercise for readers : If Virtual Server has certificate of only one Type either ECC or RSA,  but the http listener has two types of certs one each of ECC and RSA (this should not happen in ideal case), then the server will send Virtual Server's cert has OR http listener certificate depending on the cipher requested in SSL Handshake.

     Files  Contents
    sni-abc.req
    HEAD /index.html HTTP/1.1
    Host: abc
    Connection: close
     sni-anyhost.req HEAD /index.html HTTP/1.1
    Host: anyOtherValue
    Connection: close
     sni-nocertvs.req HEAD /index.html HTTP/1.1
    Host: www.nocertvs.com
    Connection: close

    TSTCLNT="tstclnt" is NSS tool to send SSL requests to the server.

    1. Install OTD

    2. Start the Origin Server

    3. Start OTD Admin Server

    4. Create self signed cert for the http listener with subject name "www.ls.com" (for easy identification) and nickname "Server-Cert"

    $INSTANCE_HOME/bin/tadm create-selfsigned-cert --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --server-name=www.ls.com --nickname=Server-Cert --key-type=rsa

    CLI201 Command 'create-selfsigned-cert' ran successfully

    5. Enable SSL and set this self signed cert with nickname "Server-Cert" in the http listener

    $INSTANCE_HOME/bin/tadm set-ssl-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --http-listener=http-listener-1 enabled=true server-cert-nickname=Server-Cert

    CLI201 Command 'set-ssl-prop' ran successfully

    6. Create a Virtual Server VSabc with www.abc.com <host> in server.xml  and bind it to the http listener "http-listener-1"

    $INSTANCE_HOME/bin/tadm create-virtual-server --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --host-pattern=www.abc.com --http-listener-name=http-listener-1 --origin-server-pool-name=origin-server-pool-1 VSabc

    CLI201 Command 'create-virtual-server' ran successfully

    7. Create self signed cert for the Virtual Server with subject "www.abc.com" and nickname "abc"

    $INSTANCE_HOME/bin/tadm create-selfsigned-cert --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --server-name=www.abc.com --nickname=abc --key-type=rsa

    Command 'create-selfsigned-cert' ran successfully

    8. Set this certificate with nickname "abc" and subject "www.abc.com" in the Virtual Server "VSabc"

    $INSTANCE_HOME/bin/tadm set-virtual-server-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --vs=VSabc server-cert-nickname=abc

    CLI201 Command 'set-virtual-server-prop' ran successfully

    9. Create a Virtual Server VSnocertvs with "www.nocertvs.com" <host> in server.xml

    $INSTANCE_HOME/bin/tadm create-virtual-server --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --host-pattern=www.nocertvs.com --http-listener-name=http-listener-1 --origin-server-pool-name=origin-server-pool-1 VSnocertvs

    CLI201 Command 'create-virtual-server' ran successfully

    10. Set the error log level to "finest" if you wish to see log messages are logged for SNI at all levels

    $INSTANCE_HOME/bin/tadm set-log-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG log-level=finest

    CLI201 Command 'set-log-prop' ran successfully

    11. Deploy these changes

    $INSTANCE_HOME/bin/tadm deploy-config --force --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd $CONFIG

    CLI201 Command 'deploy-config' ran successfully

    12. Start the server instance

    $INSTANCE_HOME/bin/tadm start-instance --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG

    CLI204 Successfully started the server instance.

    Testing using tstclnt / Browser

    13. Just for testing add www.abc.com  and www.nocertvs.com entries in /etc/hosts.

    cat /etc/hosts | grep www.abc.com
    cat /etc/hosts | grep www.nocertvs.com

    Ideally your DNS server must resolve these hosts to the same IP address we are using in OTD http listener.

    14. Send a request via tstclnt with -a "www.abc.com"(sends this host in SSL handshake) and in request headers Host: "www.abc.com" - should get cert from the Virtual Server VSabc with subject  DN "CN=www.abc.com"

    $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 -a www.abc.com < $DEMO_DIR/sni-abc.req

    15. Send a request via tstclnt with -a "www.nocertvs.com"(sends this host in SSL handshake) and in request headers Host: "www.nocertvs.com" - should get cert from the http listener with subject DN "CN=www.ls.com" as Virtual Server VSnocertvs with <host> www.nocerts.com doesn't have any certs.

    $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 -a www.nocertvs.com < $DEMO_DIR/sni-nocertvs.req

    16. Send a NON SNI request via tstclnt i.e. WITHOUT any host in SSL Handshake - should get the cert from the http listener with subject DN "CN=www.ls.com"

    $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 < $DEMO_DIR/sni-anyhost.req

    Summary

    • If SNI host is NOT sent by the browser in SSL Handshake, then the cert is returned from http listener.
    • If SNI host is sent by the browser in SSLHandshake and it matches with <host> element in Virtual Server, cert is returned from that Virtual Server.
    • If SNI host is sent by the browser in SSLHandshake and it matches <host> element in Virtual Server which doesn't have any certificates, certificate is returned from that http listener. - This gets a bit more complicated with Default virtual servers, will discuss in the next section.

    Advanced - Default Virtual Server tests

    17. Stop the instance

    $INSTANCE_HOME/bin/tadm stop-instance --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG

    CLI205 Successfully stopped the server instance.

    18. Create self signed cert with subject "www.defaultvscert.com" for the Default Virtual Server (Virtual Server in <default-virtual-server> of http-listener in server.xml i.e. in our case it is Virtual server with vs name $CONFIG)

    $INSTANCE_HOME/bin/tadm create-selfsigned-cert --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --server-name=www.defaultvscert.com --nickname=defaultvscert --key-type=rsa

    CLI201 Command 'create-selfsigned-cert' ran successfully

    19. Set this certificate with subject "www.defaultvscert.com" in the Default Virtual Server (vs name $CONFIG)

    $INSTANCE_HOME/bin/tadm set-virtual-server-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --vs=$CONFIG server-cert-nickname=defaultvscert

    CLI201 Command 'set-virtual-server-prop' ran successfully

    20. Deploy the changes

    $INSTANCE_HOME/bin/tadm deploy-config --force --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd $CONFIG

    CLI201 Command 'deploy-config' ran successfully

    21. Start the instance

    $INSTANCE_HOME/bin/tadm start-instance --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG

    CLI204 Successfully started the server instance.

    22. Send a request via tstclnt with -a "www.nocertvs.com"(sends this host in SSL handshake) and in request headers Host: "www.nocertvs.com" - should get cert from default virtual server subject DN: CN=www.defaultvscert.com"

    $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 -a www.nocertvs.com < $DEMO_DIR/sni-nocertvs.req

    Summary

    If SNI host is sent by the browser in SSL Handshake,

    • look for every Virtual Server bound to that http listener if it has <host> element whose value matches with it,
      • if that VS has certs - return cert from this VS.
      • if that VS doesnt have any certs, then
        • get the default Virtual Server(default-virtual-server>) for this http listener(it may be bound or it may be unbound),
          • if default VS has a certificate - return cert from this default VS
          • else  - return the certificates form http listener.

    FLOW CHART OF SNI


About

Meena Vyas

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today