Thursday Nov 14, 2013

Performance Analysis of Oracle Traffic Director or Web Server

Performance Analysis of Oracle Trafiic Director or Oracle iPlanet Web Server using Oracle Sun Solaris Studio Performance Analyzer and DTrace scripts.

In this blog I will show how to use the Sun Studio Collector and Performance Analyzer and DTrace script to measure the performance.

1. Using Oracle Solaris Studio 12.2 Performance Analyzer with Oracle Traffic Director or Oracle iPlanet Web Server

Thanx to Basant for teaching me collector and analyzer and thanx to Julien for his help on DTrace scripts.

Install Sun Studio 12. Lets say is installed in /opt/SUNWSpro.

1.1 setting up ~/.er.rc

cat ~/.er.rc
dmetrics e.user:i.user:e!wall:i!!wait: \
dsort e.user
scc basic:version:warn:parallel:query:loop:pipe:inline:memops:fe:cg
dcc basic:version:warn:parallel:query:loop:pipe:inline:memops:fe:cg:src
sthresh 75
dthresh 75
name long
view user
tlmode thread:root:depth=10
tldata sample:clock:hwc:heaptrace:synctrace:mpitrace:msgtrace:datarace
setpath $expts:.
tabs functions:callers-callees:source:disasm:timeline:header:
en_desc on

1.2 Collecting Data Using the collect Command

1.2.1& Run the Collector  using the collect command

# collect collect-options program program-arguments

For OTD or Web Server we edit start script bin/startserv to have a new --collect option as shown below 

Red color lines are the new lines I have added.  Copied all the lines in start option, replaced ${SERVER_BIN} (which is trafficd-wdog) by "trafficd". And added "collect $COLLECT_OPTS" before it.

In the section below replace <profiler directory> by the directory where you want the profiler to collect data.

case $COMMAND in
        ${SERVER_BIN} -d "${SERVER_CONFIG_DIR}" \ 
            -r "${OTD_PRODUCT_HOME}" -t "${SERVER_TEMP_DIR}" \
            -u "${SERVER_USER}" ${SVC_OPT} $@
        if [ $STATUS -ne 0 ] ; then 
             exit $STATUS
        COLLECT_OPTS="-t 180-300 -F all -d <profiler directory>";
        collect $COLLECT_OPTS trafficd -d "${SERVER_CONFIG_DIR}" \
            -r "${OTD_PRODUCT_HOME}" -t "${SERVER_TEMP_DIR}" \ 
            -u "${SERVER_USER}" ${SVC_OPT} $@
        if [ $STATUS -ne 0 ] ; then
            exit $STATUS

for web server instead of trafficd it will be webservd. 

1.2.2 start the server using --collect option

Since we replaced trafficd-wdog by trafficd, the server will start up without any watchdog process and will run in console, not in background.  

bin/startserv --collect

1.2.3 run some stress tests

1.2.4 After 5-6 minutes stop the server

This will create a directory called (e.g.) '' which contains the experiment.

The default name for a new experiment is

The Collector automatically increments n by one in the names of subsequent experiments.

1.3 Open the profile and start the Oracle Sun Studio 12.2 analyzer

Set DISPLAY env.

$ cd <profiler directory>
$ export JAVA_PATH=/opt/SUNWwbsvr/jdk 
$ /opt/SUNWspro/bin/analyzer

1.4 The er_print utility prints an ASCII version of the various displays supported by the Performance Analyzer.

$ /opt/SUNWspro/bin/er_print -outfile er_print1.out -functions 

In this, functions are sorted by "Exclusive User CPU Time"


$ /opt/SUNWspro/bin/er_print -outfile er_print2.out \
-metrics \
-sort -functions 

In this, functions are sorted by "Inclusive Total LWP Time"

You can look at these files and figure out which function is taking how much time.

These files look like :

Functions sorted by metric: Exclusive User CPU Time 
Excl.     Incl.     Excl.       Incl.       Excl.     Incl.      Name  
User CPU  User CPU  Total LWP   Total LWP   Sys. CPU  Sys. CPU         
   sec.      sec.         sec.        sec.     sec.      sec.     
851.666   851.666   105477.573  105477.573  106.525   106.525    <Total>
493.435   493.435      558.110     558.110   26.368    26.368    fn1
 56.840   326.308       64.015     368.568    3.042    17.352    fn2
 28.280    28.280       34.574      34.574    1.701     1.701    fn3


2. Using DTrace script to see how much time is spent in which function

#!/usr/sbin/dtrace -s
#pragma D option bufsize=1g
#pragma D option specsize=1g
#pragma D option aggsize=1g
#pragma D option dynvarsize=1g

 self->ts[probefunc] = timestamp;

    @time[probefunc, probemod] = sum(timestamp - self->ts[probefunc]);
    self->ts[probefunc] = 0;

Note I have given 1g sizes you can tune it as per your machine.

Run this D script using :

#sudo dtrace -s functime.d -p 27910 -o dtrace.log 

where 27910 is the pid of the process you are examining (in this case webservd or trafficd).

This will generate the output as shown below 

  PListFindValue                                                     3871
  getbucketnum                                                              4995
  R_SSL_version                                                           6106
  http_format_server                                                 6807
  void SimpleHashTemplateBase::_insert(unsigned long,hashEntry*)             7059
  long atoi64(const char*)                                           7288


I wrote this wrapper script to report percentages.

To get the last column I used logic of looking for spaces after the string "lib". For LM1`, I temporarily added a hack.

$logfile = "dtrace.log";
$tmpfile = "temp";
open(IN,"<",$logfile) || die "Can not open $logfile: $!";
$total = 0;
$total_rounded = 0;
while (<IN>) {
    my ($a, $b) = split('=');
    $total += $b;
print "total = $total\n";
open(OUT,">", $tmpfile) || die "Can not open $tmpfile: $!";
open(IN,"<",$logfile) || die "Can not open $logfile: $!";
while (<IN>) {
    my ($a, $b) = split('=');
    $rounded = sprintf("%.10f", ($b*100)/$total);
    $a =~ s/;
    print OUT "$b   $rounded%   $a\n";
    $total_rounded += $rounded;
print "total rounded = $total_rounded\n";
`sort -n -r $tmpfile | tee $logfile.sorted`;
`rm $tmpfile`

This produced output of the following format :

20318357230394   16.4117840797%     poll                            
20317702791746   16.4112554688%     _pollsys                        
20313615393944   16.4079539474%     __pollsys                       
2684593654698   2.1684317735%     int DaemonSession::GetConnection()


3. Using DTrace Profile Probes

 #!/usr/sbin/dtrace -s
/pid == $1 && arg1 != NULL/
    @proc[umod(arg1), ufunc(arg1),ustack()] = count(); 

run it as :

dtrace -x ustackframes=20 -s profiler-probes.d <pid> -o dtrace.log

It creates output in the format library name, function name, user stack and the count number of times called, newline                                 `mutex_lock_impl        


4. References

Friday Oct 11, 2013

Configuring Server Name Indication (SNI) in Oracle Traffic Director and

What is SNI ? It is explained very well in

If your SSL server needs certificate(s) for different domains, you can choose one of the different options :
  • Use multiple certificates using SNI feature (configure server to return different certificates for different domains) - recommended
  • Use a single certificate with SubjectAltName Extension (one hostname in CN and other hostnames in SubjectAltName extension in the certificate)
  • Use a single certificate with wild card in subject (lets say certificate with "CN=*.*", so it will be valid for different domains) - not preferred
  • Notes

    • Unbound Virtual Server: <virtual-server> doesn't have <http-listener> as a sub element.
    • Bound Virtual Server: <virtual-server> has a <http-listener> sub element, it is said to be bound to that http listener.
    • To figure out which Virtual server is the Default Virtual Server for a listener, look at the Virtual Server name in <default-virtual-server> of <http-listener> in server.xml.

    How to configure SNI in Oracle Traffic Director

    In this blog I will cover the following

    Enable SSL on an HTTP listener and create a certificate for it. Create two Virtual Servers both bound to an HTTP listener. One of the Virtual Server contains a certificate and the other doesn't. Send SNI and non-SNI requests to those two Virtual Servers.

    Create and add certificate for the default Virtual Server(which could be unbound or bound) and add <host> element value of <host> of our Virtual Server which doesn't have a certificate. Send a SNI request to the virtual server which doesn't have a certificate, it returns certificate from the default virtual server.

    What we will find out  is

    • If SNI host is NOT sent by the browser in SSL Handshake, then the server sends the certificate from the http listener. --------- 1
    • else (i.e. if SNI host is sent by the browser in SSLHandshake)
      • If SNI Host sent by browser doesn't match with a <host> element in any of the bound Virtual server  - goto STEP 2
      • else (i.e. If SNI host sent by browser matches with <host> element of any bound Virtual Server)
        • If that Virtual Server has certificate,  the server sends the certificate from the Virtual Server. ----------- 2
        • else (that Virtual Server DOES NOT have a certificate) - goto STEP 2

    STEP 2: get the default Virtual Server for this http listener :

      • If the default virtual Server DOES NOT have a certificate, then the server sends the cert from the http listener ------- 3
      • else (i.e. If the default virtual Server has a certificate) then the server sends the cert from this default Virtual Server ------- 4

    Exercise for readers : If Virtual Server has certificate of only one Type either ECC or RSA,  but the http listener has two types of certs one each of ECC and RSA (this should not happen in ideal case), then the server will send Virtual Server's cert has OR http listener certificate depending on the cipher requested in SSL Handshake.

     Files  Contents
    HEAD /index.html HTTP/1.1
    Host: abc
    Connection: close
     sni-anyhost.req HEAD /index.html HTTP/1.1
    Host: anyOtherValue
    Connection: close
     sni-nocertvs.req HEAD /index.html HTTP/1.1
    Connection: close

    TSTCLNT="tstclnt" is NSS tool to send SSL requests to the server.

    1. Install OTD

    2. Start the Origin Server

    3. Start OTD Admin Server

    4. Create self signed cert for the http listener with subject name "" (for easy identification) and nickname "Server-Cert"

    $INSTANCE_HOME/bin/tadm create-selfsigned-cert --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --nickname=Server-Cert --key-type=rsa

    CLI201 Command 'create-selfsigned-cert' ran successfully

    5. Enable SSL and set this self signed cert with nickname "Server-Cert" in the http listener

    $INSTANCE_HOME/bin/tadm set-ssl-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --http-listener=http-listener-1 enabled=true server-cert-nickname=Server-Cert

    CLI201 Command 'set-ssl-prop' ran successfully

    6. Create a Virtual Server VSabc with <host> in server.xml  and bind it to the http listener "http-listener-1"

    $INSTANCE_HOME/bin/tadm create-virtual-server --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --http-listener-name=http-listener-1 --origin-server-pool-name=origin-server-pool-1 VSabc

    CLI201 Command 'create-virtual-server' ran successfully

    7. Create self signed cert for the Virtual Server with subject "" and nickname "abc"

    $INSTANCE_HOME/bin/tadm create-selfsigned-cert --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --nickname=abc --key-type=rsa

    Command 'create-selfsigned-cert' ran successfully

    8. Set this certificate with nickname "abc" and subject "" in the Virtual Server "VSabc"

    $INSTANCE_HOME/bin/tadm set-virtual-server-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --vs=VSabc server-cert-nickname=abc

    CLI201 Command 'set-virtual-server-prop' ran successfully

    9. Create a Virtual Server VSnocertvs with "" <host> in server.xml

    $INSTANCE_HOME/bin/tadm create-virtual-server --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --http-listener-name=http-listener-1 --origin-server-pool-name=origin-server-pool-1 VSnocertvs

    CLI201 Command 'create-virtual-server' ran successfully

    10. Set the error log level to "finest" if you wish to see log messages are logged for SNI at all levels

    $INSTANCE_HOME/bin/tadm set-log-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG log-level=finest

    CLI201 Command 'set-log-prop' ran successfully

    11. Deploy these changes

    $INSTANCE_HOME/bin/tadm deploy-config --force --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd $CONFIG

    CLI201 Command 'deploy-config' ran successfully

    12. Start the server instance

    $INSTANCE_HOME/bin/tadm start-instance --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG

    CLI204 Successfully started the server instance.

    Testing using tstclnt / Browser

    13. Just for testing add  and entries in /etc/hosts.

    cat /etc/hosts | grep
    cat /etc/hosts | grep

    Ideally your DNS server must resolve these hosts to the same IP address we are using in OTD http listener.

    14. Send a request via tstclnt with -a ""(sends this host in SSL handshake) and in request headers Host: "" - should get cert from the Virtual Server VSabc with subject  DN ""

    $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 -a < $DEMO_DIR/sni-abc.req

    15. Send a request via tstclnt with -a ""(sends this host in SSL handshake) and in request headers Host: "" - should get cert from the http listener with subject DN "" as Virtual Server VSnocertvs with <host> doesn't have any certs.

    $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 -a < $DEMO_DIR/sni-nocertvs.req

    16. Send a NON SNI request via tstclnt i.e. WITHOUT any host in SSL Handshake - should get the cert from the http listener with subject DN ""

    $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 < $DEMO_DIR/sni-anyhost.req


    • If SNI host is NOT sent by the browser in SSL Handshake, then the cert is returned from http listener.
    • If SNI host is sent by the browser in SSLHandshake and it matches with <host> element in Virtual Server, cert is returned from that Virtual Server.
    • If SNI host is sent by the browser in SSLHandshake and it matches <host> element in Virtual Server which doesn't have any certificates, certificate is returned from that http listener. - This gets a bit more complicated with Default virtual servers, will discuss in the next section.

    Advanced - Default Virtual Server tests

    17. Stop the instance

    $INSTANCE_HOME/bin/tadm stop-instance --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG

    CLI205 Successfully stopped the server instance.

    18. Create self signed cert with subject "" for the Default Virtual Server (Virtual Server in <default-virtual-server> of http-listener in server.xml i.e. in our case it is Virtual server with vs name $CONFIG)

    $INSTANCE_HOME/bin/tadm create-selfsigned-cert --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --nickname=defaultvscert --key-type=rsa

    CLI201 Command 'create-selfsigned-cert' ran successfully

    19. Set this certificate with subject "" in the Default Virtual Server (vs name $CONFIG)

    $INSTANCE_HOME/bin/tadm set-virtual-server-prop --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG --vs=$CONFIG server-cert-nickname=defaultvscert

    CLI201 Command 'set-virtual-server-prop' ran successfully

    20. Deploy the changes

    $INSTANCE_HOME/bin/tadm deploy-config --force --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd $CONFIG

    CLI201 Command 'deploy-config' ran successfully

    21. Start the instance

    $INSTANCE_HOME/bin/tadm start-instance --user=admin --port=$TD_ADMIN_PORT --password-file=$DEMO_DIR/admin.passwd --config=$CONFIG

    CLI204 Successfully started the server instance.

    22. Send a request via tstclnt with -a ""(sends this host in SSL handshake) and in request headers Host: "" - should get cert from default virtual server subject DN:"

    $TSTCLNT -c y -h $HOST -d $INSTANCE_HOME/https-$CONFIG/config -n Server-Cert -o -p $TD_PORT -2 -a < $DEMO_DIR/sni-nocertvs.req


    If SNI host is sent by the browser in SSL Handshake,

    • look for every Virtual Server bound to that http listener if it has <host> element whose value matches with it,
      • if that VS has certs - return cert from this VS.
      • if that VS doesnt have any certs, then
        • get the default Virtual Server(default-virtual-server>) for this http listener(it may be bound or it may be unbound),
          • if default VS has a certificate - return cert from this default VS
          • else  - return the certificates form http listener.


Thursday Nov 08, 2012

Configuring Oracle iPlanet WebServer / Oracle Traffic Director to use crypto accelerators on T4-1 servers

Configuring Oracle iPlanet Web Server / Oracle Traffic Director to use crypto accelerators on T4-1 servers

Jyri had written a technical article on Configuring Solaris Cryptographic Framework and Sun Java System Web Server 7 on Systems With UltraSPARC T1 Processors. I tried to find out what has changed since then in T4.

I have used a T4-1 SPARC system with Solaris 10. Results slightly vary for Solaris 11.  For Solaris 11, the T4 optimization was implemented in while it was in for Solaris 10.

Overview of T4 processors is here in this blog.

Many thanx to Chi-Chang Lin and Julien for their help.

1. Install Oracle iPlanet Web Server / Oracle Traffic Director.  Go to instance/config directory.

 # cd /opt/oracle/webserver7/https-hostname.fqdn/config

2. List default PKCS#11 Modules

# ../../bin/modutil -dbdir . -list
Listing of PKCS #11 Modules
1. NSS Internal PKCS #11 Module
slots: 2 slots attached
status: loaded

slot: NSS Internal Cryptographic Services
token: NSS Generic Crypto Services

slot: NSS User Private Key and Certificate Services
token: NSS Certificate DB

2. Root Certs
library name:
slots: 1 slot attached
status: loaded

slot: NSS Builtin Objects
token: Builtin Object Token

3. Initialize the soft token data store in the $HOME/.sunw/pkcs11_softtoken/ directory

# pktool setpin keystore=pkcs11
Enter token passphrase: olderpassword
Create new passphrase: password
Re-enter new passphrase: password
Passphrase changed.

4. Offload crypto operations to Solaris Crypto Framework on T4

$ ../../bin/modutil -dbdir . -nocertdb -add SCF -libfile /usr/lib/ -mechanisms RSA:AES:SHA1:MD5

Module "SCF" added to database.

Note that

  • -nocertdb means modutil won't try to open the NSS softoken key database. It doesn't even have to be present.

  • PKCS#11 library used is /usr/lib/ If the server is running in 64 bit mode, we have to use /usr/lib/64/

  • Unlike T1 and T2, in T4 we do not have to disable mechanisms in softtoken provider using cryptoadm.

5. List again to check that a new module SCF is added

# ../../bin/modutil -dbdir . -list
Listing of PKCS #11 Modules
1. NSS Internal PKCS #11 Module
slots: 2 slots attached
status: loaded

slot: NSS Internal Cryptographic Services
token: NSS Generic Crypto Services

slot: NSS User Private Key and Certificate Services
token: NSS Certificate DB

2. SCF
library name: /usr/lib/
slots: 2 slots attached
status: loaded

slot: Sun Metaslot
token: Sun Metaslot

slot: n2rng/0 SUNW_N2_Random_Number_Generator
token: n2rng/0 SUNW_N2_RNG
3. Root Certs
library name:
slots: 1 slot attached
status: loaded
slot: NSS Builtin Objects
token: Builtin Object Token

6.  Create certificate in “Sun Metaslot” :

I have used certutil, but you must use Admin Server CLI / GUI

# ../../bin/certutil -S -x -n "Server-Cert" -t "CT,CT,CT" -s "CN=*.fqdn" -d . -h "Sun Metaslot"
Enter Password or Pin for "Sun Metaslot": password

7. Verify that the certificate is created properly in “Sun Metslaot”

# ../../bin/certutil -L -d . -h "Sun Metaslot"
Certificate Nickname Trust Attributes
Enter Password or Pin for "Sun Metaslot": password
Sun Metaslot:Server-Cert CTu,Cu,Cu

8. Associate this newly created certificate to http listener using Admin CLI/GUI. After that server.xml should have

<http-listener> ...
        <server-cert-nickname>Sun Metaslot:Server-Cert</server-cert-nicknamer>

Note the prefix "Sun Metaslot"

9. Disable PKCS#11 bypass

To use the accelerated AES algorithm, turn off PKCS#11 bypass, and configure modutil to have the AES mechanism go to the Metaslot.

After you disable PKCS#11 bypasss using Admin GUI/CLI,  check that server.xml should have

<server> ....

With PKCS#11 bypass enabled, Oracle iPlanet Web Server will only use the RSA capability of the T4, provided certificate and key are stored in the T4 slot (Metaslot). Actually, the RSA op is never bypassed in NSS, it's always done with PKCS#11 calls. So the bypass settings won't affect the behavior of the probes for RSA at all. The only thing that matters if where the RSA key and certificate live, ie. which PKCS#11 token, and thus which PKCS#11 module gets called to do the work. If your certificate/key are in the NSS certificate/key db, you will see libsoftokn3/libfreebl libraries doing the RSA work. If they are in the Sun Metaslot, it should be the Solaris code.

10. Start the server instance

# ../bin/startserv
Oracle iPlanet Web Server 7.0.16 B09/14/2012 03:33
Please enter the PIN for the "Sun Metaslot" token: password
info: HTTP3072: http-listener-1: https://hostname.fqdn:80 ready to accept requests
info: CORE3274: successful server startup

11. Figure out which process to run this DTrace script on

# ps -eaf | grep webservd | grep -v dog
18224 18223 0 13:17:25 ? 0:07 webservd -d /opt/oracle/webserver7/https-hostname.fqdn/config -r /opt/
18225 18224 0 13:17:25 ? 0:00 webservd -d /opt/oracle/webserver7/https-hostname.fqdn/config -r /opt/

(For Oracle Traffic Director look for process named "trafficd")

We see that the child process id is “18225

12. Clients for testing :

You can use any browser. I used NSS tool tstclnt for testing

$cat > req.txt
GET /index.html HTTP/1.0

For checking both RSA and AES, I used cipher “:0035” which is TLS_RSA_WITH_AES_256_CBC_SHA

$./tstclnt -h hostname -p 80 -d . -T -f -o -v -c “:0035” < req.txt

13. How do I make sure that crypto accelerator is being used

13.1 Create DTrace script

The following D script should be able to uncover whether T4-specific crypto routine are being called or not. It also displays stats per second.

# cat > t4crypto.d
#!/usr/sbin/dtrace -s

    @ops[probemod, probefunc] = count();


Invoke with './t4crypto.d -p <pid> '

13.2 EXPECTED PROBES FOR Solaris 10 :

If offloading to T4 HW are correctly set up, the expected DTrace output would have these probes and libraries





soft_decrypt_rsa_pkcs_decode, soft_encrypt_rsa_pkcs_encode soft_rsa_crypt_init_common soft_rsa_decrypt, soft_rsa_encrypt soft_rsa_decrypt_common, soft_rsa_encrypt_common


yf_aes_instructions_present yf_aes_expand256, yf_aes256_cbc_decrypt, yf_aes256_cbc_encrypt, yf_aes256_load_keys_for_decrypt, yf_aes256_load_keys_for_encrypt,

Note that these are for 256, same for 128, 192...

these are for cbc, same for ecb, ctr, cfb128...


yf_des_expand, yf_des_instructions_present yf_des_encrypt


yf_md5_multiblock, yf_md5_instruction_present


yf_sha1_instruction_present, yf_sha1_multibloc


# ./t4crypto.d -p 18225   soft_decrypt_rsa_pkcs_decode    1   soft_rsa_crypt_init_common      1   soft_rsa_decrypt                1   big_mp_mul_yf                   2   mpm_yf_mpmul                    2   mpmul_arr_yf                    2   rijndael_key_setup_enc_yf       2   soft_rsa_decrypt_common         2   yf_aes_expand256                2   yf_aes256_cbc_decrypt           3   yf_aes256_load_keys_for_decrypt 3   big_mont_mul_yf                 6   mm_yf_montmul                   6   yf_des_instructions_present     6   yf_aes256_cbc_encrypt           8   yf_aes256_load_keys_for_encrypt 8   yf_mpmul_present                8   yf_aes_instructions_present    13   yf_des_encrypt                 18                yf_md5_multiblock              41                yf_md5_instruction_present     72                yf_sha1_instruction_present    82                yf_sha1_multiblock             82

This indicates that both RSA and AES ops are done in Solaris Crypto Framework.


# ./t4crypto.d -p 18225   soft_decrypt_rsa_pkcs_decode 1   soft_rsa_crypt_init_common   1   soft_rsa_decrypt             1   soft_rsa_decrypt_common      1   big_mp_mul_yf                2   mpm_yf_mpmul                 2   mpmul_arr_yf                 2   big_mont_mul_yf              6   mm_yf_montmul                6   yf_mpmul_present             8

For this cipher, when I enable PKCS#11 bypass, Only RSA probes are being hit AES probes are not being hit.

13.5 ustack() for RSA operations / probefunc == "soft_rsa_decrypt" /

Shows that is calling C_* functions of which is calling functions of for both cases with and without bypass.

When PKCS#11 bypass is disabled (allow-bypass is 0)`soft_rsa_decrypt`soft_rsa_decrypt_common+0x94`soft_unwrapkey+0x258`C_UnwrapKey+0x1ec`meta_unwrap_key+0x17c`meta_UnwrapKey+0xc4`C_UnwrapKey+0xfc`pk11_AnyUnwrapKey+0x6b8`PK11_PubUnwrapSymKey+0x8c`ssl3_HandleRSAClientKeyExchange+0x1a0`ssl3_HandleClientKeyExchange+0x154`ssl3_HandleHandshakeMessage+0x440`ssl3_HandleHandshake+0x11c`ssl3_HandleRecord+0x5e8`ssl3_GatherCompleteHandshake+0x5c`ssl_GatherRecord1stHandshake+0x30`ssl_Do1stHandshake+0xec`ssl_SecureRecv+0x1c8`ssl_Recv+0x9c`__1cNDaemonSessionDrun6M_v_+0x2dc

When PKCS#11 bypass is enabled (allow-bypass is 1)`soft_rsa_decrypt`soft_rsa_decrypt_common+0x94`C_Decrypt+0x164`meta_do_operation+0x27c`meta_Decrypt+0x4c`C_Decrypt+0xcc`PK11_PrivDecryptPKCS1+0x1ac`ssl3_HandleRSAClientKeyExchange+0xe4`ssl3_HandleClientKeyExchange+0x154`ssl3_HandleHandshakeMessage+0x440`ssl3_HandleHandshake+0x11c`ssl3_HandleRecord+0x5e8`ssl3_GatherCompleteHandshake+0x5c`ssl_GatherRecord1stHandshake+0x30`ssl_Do1stHandshake+0xec`ssl_SecureRecv+0x1c8`ssl_Recv+0x9c`__1cNDaemonSessionDrun6M_v_+0x2dc`ThreadMain+0x1c`_pt_root+0xe8

13.6 ustack() FOR AES operations / probefunc == "yf_aes256_cbc_encrypt" /

When PKCS#11 bypass is disabled (allow-bypass is 0)`yf_aes256_cbc_encrypt`aes_block_process_contiguous_whole_blocks+0xb4`aes_crypt_contiguous_blocks+0x1cc`soft_aes_encrypt_common+0x22c`C_EncryptUpdate+0x10c`meta_do_operation+0x1fc`meta_EncryptUpdate+0x4c`C_EncryptUpdate+0xcc`PK11_CipherOp+0x1a0`ssl3_CompressMACEncryptRecord+0x264`ssl3_SendRecord+0x300`ssl3_FlushHandshake+0x54`ssl3_SendFinished+0x1fc`ssl3_HandleFinished+0x314`ssl3_HandleHandshakeMessage+0x4ac`ssl3_HandleHandshake+0x11c`ssl3_HandleRecord+0x5e8`ssl3_GatherCompleteHandshake+0x5c`ssl_GatherRecord1stHandshake+0x30`ssl_Do1stHandshake+0xec

Shows that is calling C_* functions of which is calling functions of

However when PKCS#11 bypass is disabled (allow-bypass is 1) this stack isn't getting called.


# ./t4crypto.d -p 18225 -l
55720 pid18225 yf_md5_instruction_present entry
55721 pid18225 yf_sha256_instruction_present entry
55722 pid18225 yf_sha512_instruction_present entry
55723 pid18225 yf_sha1_instruction_present entry
55724 pid18225 yf_sha256 entry
55725 pid18225 yf_sha256_multiblock entry
55726 pid18225 yf_sha512 entry
55727 pid18225 yf_sha512_multiblock entry
55728 pid18225 yf_sha1 entry
55729 pid18225 yf_sha1_multiblock entry
55730 pid18225 yf_md5 entry
55731 pid18225 yf_md5_multiblock entry
55732 pid18225 yf_aes_instructions_present entry
55733 pid18225 rijndael_key_setup_enc_yf entry
55734 pid18225 yf_aes_expand128 entry
55735 pid18225 yf_aes_encrypt128 entry
55736 pid18225 yf_aes_decrypt128 entry
55737 pid18225 yf_aes_expand192 entry
55738 pid18225 yf_aes_encrypt192 entry
55739 pid18225 yf_aes_decrypt192 entry
55740 pid18225 yf_aes_expand256 entry
55741 pid18225 yf_aes_encrypt256 entry
55742 pid18225 yf_aes_decrypt256 entry
55743 pid18225 yf_aes128_load_keys_for_encrypt entry
55744 pid18225 yf_aes192_load_keys_for_encrypt entry
55745 pid18225 yf_aes256_load_keys_for_encrypt entry
55746 pid18225 yf_aes128_ecb_encrypt entry
55747 pid18225 yf_aes192_ecb_encrypt entry
55748 pid18225 yf_aes256_ecb_encrypt entry
55749 pid18225 yf_aes128_cbc_encrypt entry
55750 pid18225 yf_aes192_cbc_encrypt entry
55751 pid18225 yf_aes256_cbc_encrypt entry
55752 pid18225 yf_aes128_ctr_crypt entry
55753 pid18225 yf_aes192_ctr_crypt entry
55754 pid18225 yf_aes256_ctr_crypt entry
55755 pid18225 yf_aes128_cfb128_encrypt entry
55756 pid18225 yf_aes192_cfb128_encrypt entry
55757 pid18225 yf_aes256_cfb128_encrypt entry
55758 pid18225 yf_aes128_load_keys_for_decrypt entry
55759 pid18225 yf_aes192_load_keys_for_decrypt entry
55760 pid18225 yf_aes256_load_keys_for_decrypt entry
55761 pid18225 yf_aes128_ecb_decrypt entry
55762 pid18225 yf_aes192_ecb_decrypt entry
55763 pid18225 yf_aes256_ecb_decrypt entry
55764 pid18225 yf_aes128_cbc_decrypt entry
55765 pid18225 yf_aes192_cbc_decrypt entry
55766 pid18225 yf_aes256_cbc_decrypt entry
55767 pid18225 yf_aes128_cfb128_decrypt entry
55768 pid18225 yf_aes192_cfb128_decrypt entry
55769 pid18225 yf_aes256_cfb128_decrypt entry
55771 pid18225 yf_des_instructions_present entry
55772 pid18225 yf_des_expand entry
55773 pid18225 yf_des_encrypt entry
55774 pid18225 yf_mpmul_present entry
55775 pid18225 yf_montmul_present entry
55776 pid18225 mm_yf_montmul entry
55777 pid18225 mm_yf_montsqr entry
55778 pid18225 mm_yf_restore_func entry
55779 pid18225 mm_yf_ret_from_mont_func entry
55780 pid18225 mm_yf_execute_slp entry
55781 pid18225 big_modexp_ncp_yf entry
55782 pid18225 big_mont_mul_yf entry
55783 pid18225 mpmul_arr_yf entry
55784 pid18225 big_mp_mul_yf entry
55785 pid18225 mpm_yf_mpmul entry
55786 pid18225 nsapi_rsa_set_priv_fn entry
55795 pid18225 prepare_rsa_priv_key_export_for_asn1 entry
55796 pid18225 sunw_dst_rsaref_init entry
55797 pid18225 NSS_Get_SEC_UniversalStringTemplate entry
55813 pid18225 prepare_low_rsa_priv_key_for_asn1 entry
55814 pid18225 rsa_FormatOneBlock entry
55815 pid18225 rsa_FormatBlock entry
55816 pid18225 lg_prepare_low_rsa_priv_key_for_asn1 entry
55817 pid18225 rsa_build_from_primes entry
55818 pid18225 rsa_is_prime entry
55819 pid18225 rsa_get_primes_from_exponents entry
55820 pid18225 rsa_PrivateKeyOpNoCRT entry
55821 pid18225 rsa_PrivateKeyOpCRTNoCheck entry
55822 pid18225 rsa_PrivateKeyOpCRTCheckedPubKey entry
55823 pid18225 key_gen_rsa_by_value entry
55824 pid18225 get_rsa_private_key entry
55825 pid18225 get_rsa_public_key entry
55826 pid18225 soft_rsa_encrypt entry
55827 pid18225 soft_rsa_decrypt entry
55828 pid18225 soft_rsa_crypt_init_common entry
55829 pid18225 soft_rsa_encrypt_common entry
55830 pid18225 soft_rsa_decrypt_common entry
55831 pid18225 soft_rsa_sign_verify_init_common entry
55832 pid18225 soft_rsa_sign_common entry
55833 pid18225 soft_rsa_verify_common entry
55834 pid18225 generate_rsa_key entry
55835 pid18225 soft_rsa_genkey_pair entry
55836 pid18225 get_rsa_sha1_prefix entry
55837 pid18225 soft_rsa_digest_sign_common entry
55838 pid18225 soft_rsa_digest_verify_common entry
55839 pid18225 soft_rsa_verify_recover entry
55840 pid18225 rsa_pri_to_asn1 entry
55841 pid18225 asn1_to_rsa_pri entry
55842 pid18225 soft_encrypt_rsa_pkcs_encode entry
55843 pid18225 soft_decrypt_rsa_pkcs_decode entry
55844 pid18225 soft_sign_rsa_pkcs_encode entry
55845 pid18225 soft_verify_rsa_pkcs_decode entry
55770 profile tick-1sec

Tuesday Nov 06, 2012

Interesting articles and blogs on SPARC T4

Interesting articles and blogs on SPARC T4 processor

I have consolidated all the interesting information I could get on SPARC T4 processor and its hardware cryptographic capabilities.  Hope its useful.

1. Advantages of SPARC T4 processor

Most important points in this T4 announcement are :

"The SPARC T4 processor was designed from the ground up for high speed security and has a cryptographic stream processing unit (SPU) integrated directly into each processor core. These accelerators support 16 industry standard security ciphers and enable high speed encryption at rates 3 to 5 times that of competing processors. By integrating encryption capabilities directly inside the instruction pipeline, the SPARC T4 processor eliminates the performance and cost barriers typically associated with secure computing and makes it possible to deliver high security levels without impacting the user experience."

Data Sheet has more details on these :

"New on-chip Encryption Instruction Accelerators with direct non-privileged support for 16 industry-standard cryptographic algorithms plus random number generation in each of the eight cores: AES, Camellia, CRC32c, DES, 3DES, DH, DSA, ECC, Kasumi, MD5, RSA, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512"

I ran "isainfo -v" command on Solaris 11 Sparc T4-1 system. It shows the new instructions as expected :

$ isainfo -v
64-bit sparcv9 applications
crc32c cbcond pause mont mpmul sha512 sha256 sha1 md5 camellia kasumi des aes ima hpc vis3 fmaf asi_blk_init vis2 vis popc
32-bit sparc applications
  crc32c cbcond pause mont mpmul sha512 sha256 sha1 md5 camellia kasumi des aes ima hpc vis3 fmaf asi_blk_init vis2 vis popc v8plus div32 mul32

2. Dan Anderson's Blog have some interesting points about how these can be used :

"New T4 crypto instructions include: aes_kexpand0, aes_kexpand1, aes_kexpand2, 
aes_eround01, aes_eround23, aes_eround01_l, aes_eround_23_l, aes_dround01, aes_dround23, aes_dround01_l, aes_dround_23_l.

Having SPARC T4 hardware crypto instructions is all well and good, but how do we access it ?

The software is available with Solaris 11 and is used automatically if you are running Solaris a SPARC T4.  It is used internally in the kernel through kernel crypto modules. It is available in user space through the PKCS#11 library."

3. Dan Anderson's Blog on Where's the Crypto Libraries?

Although this was written in 2009 but still is very useful

"Here's a brief tour of the major crypto libraries shown in the digraph:
  • The libpkcs11 library contains the PKCS#11 API (C_\*() functions, such as C_Initialize()).
  • That in turn calls library pkcs11_softtoken or pkcs11_kernel, for userland or kernel crypto providers. The latter is used mostly for hardware-assisted cryptography (such as n2cp for Niagara2 SPARC processors), as that is performed more efficiently in kernel space with the "kCF" module (Kernel Crypto Framework). Additionally, for Solaris 10, strong crypto algorithms were split off in separate libraries, pkcs11_softtoken_extra
  • libcryptoutil contains low-level utility functions to help implement cryptography.
  • libsoftcrypto (OpenSolaris and Solaris Nevada only) implements several symmetric-key crypto algorithms in software, such as AES, RC4, and DES3, and the bignum library (used for RSA).
  • libmd implements MD5, SHA, and SHA2 message digest algorithms"

4. Dan Anderson's Blog on  How to tell if SPARC T4 crypto is being used?

5. Difference in T3 and T4

Diagram in this blog is good and self explanatory.

Jeff's blog also highlights the differences 

"The T4 servers have improved crypto acceleration, described at It is "just built in" so administrators no longer have to assign crypto accelerator units to domains - it "just happens". Every physical or virtual CPU on a SPARC-T4 has full access to hardware based crypto acceleration at all times. .... For completeness sake, it's worth noting that the T4 adds more crypto algorithms, and accelerates Camellia, CRC32c, and more SHA-x."

6. About performance counters

In this blog, performance counters are explained :

  • "Note that unlike T3 and before, T4 crypto doesn't require kernel modules like ncp or n2cp, there is no visibility of crypto hardware with kstats or cryptoadm.
  • T4 does provide hardware counters for crypto operations. You can see these using cpustat: cpustat -c pic0=Instr_FGU_crypto 5
  • You can check the general crypto support of the hardware and OS with the command "isainfo -v".
  • Since T4 crypto's implementation now allows direct userland access, there are no "crypto units" visible to cryptoadm.  "

For more details refer Martin's blog as well.

7. How to turn off  SPARC T4 or Intel AES-NI crypto acceleration

 I found this interesting blog from Darren about how to turn off  SPARC T4 or Intel AES-NI crypto acceleration.

"One of the new Solaris 11 features of the linker/loader is the ability to have a single ELF object that has multiple different implementations of the same functions that are selected at runtime based on the capabilities of the machine. The alternate to this is having the application coded to call getisax(2) system call and make the choice itself. We use this functionality of the linker/loader when we build the userland libraries for the Solaris Cryptographic Framework (specifically and

The Solaris linker/loader allows control of a lot of its functionality via environment variables, we can use that to control the version of the cryptographic functions we run. To do this we simply export the LD_HWCAP environment variable with values that tell to not select the HWCAP section matching certain features even if isainfo says they are present. This will work for consumers of the Solaris Cryptographic Framework that use the Solaris PKCS#11 libraries or use interfaces directly. For SPARC T4 : export LD_HWCAP="-aes -des -md5 -sha256 -sha512 -mont -mpul" .. For Intel systems with AES-NI support: export LD_HWCAP="-aes""

Note that LD_HWCAP is explained in

"LD_HWCAP, LD_HWCAP_32, and LD_HWCAP_64 -  Identifies an alternative hardware capabilities value... A “-” prefix results in the capabilities that follow being removed from the alternative capabilities."

8. Whitepaper on High Performance Security For Oracle Database and Fusion Middleware Applications using SPARC T4

This whitepaper on "High Performance Security For Oracle Database and Fusion Middleware Applications using SPARC T4 explains more details. It has DTrace scripts which may come in handy :

"To ensure the hardware-assisted cryptographic acceleration is configured to use and working with the security scenarios, it is recommended to use the following Solaris DTrace script. "

#!/usr/sbin/dtrace -s




    @ops[probemod, probefunc] = count();







Note that I have slightly modified the D Script to have *rsa* and to make it work for both Solaris 10 and 11  as per recommendations from Chi-Chang Lin.

For Solaris 11, the T4 optimization is implemented in while it is in for Solaris 10. So just add these two probes for Solaris 10 :

9. References

Monday Jul 02, 2012

How to export ECC key and Cert from NSS DB and import into JKS keystore and Oracle Wallet

How to export ECC key and Cert from NSS DB and import into JKS keystore and Oracle Wallet

In this blog I will write about how to extract a cert and key from NSS Db and import it to a JKS Keystore and then import that JKS Keystore into Oracle Wallet.

1. Set Java Home

I pointed it to JRE 1.6.0_22

$ export JAVA_HOME=/usr/java/jre1.6.0_22/

2. Create a self signed ECC cert in NSS DB

I created NSS DB with self signed ECC certificate. If you already have NSS Db with ECC cert (and key) skip this step.

$export NSS_DIR=/export/home/nss/

$$NSS_DIR/certutil -N -d .

$$NSS_DIR/certutil -S -x -s "CN=test,C=US" -t "C,C,C" -n ecc-cert -k ec -q nistp192 -d .

3. Export ECC cert and key using pk12util

Use NSS tool pk12util to export this cert and key into a p12 file 

    $$NSS_DIR/pk12util -o ecc-cert.p12 -n ecc-cert -d . -W password

4. Use keytool to create JKS keystore and import this p12 file

4.1 Import p12 file created above into a JKS keystore

$JAVA_HOME/bin/keytool -importkeystore -srckeystore ecc-cert.p12 -srcstoretype PKCS12 -deststoretype JKS -destkeystore ecc.jks -srcstorepass password -deststorepass password -srcalias ecc-cert -destalias ecc-cert -srckeypass password -destkeypass password -v

But if an error as shown is encountered,

keytool error: Get Key failed: EC KeyFactory not available Get Key failed: EC KeyFactory not available
       at Source)
        at Source)
        at Source)
        at Source)
        at Source)
        at Source)
        at Source)
        at Source)
        at Source)
Caused by: EC KeyFactory not available
        at<init>(Unknown Source)
        at Source)
        ... 9 more

4.2 Create a new PKCS11 provider

If you didn't get an error as shown above skip this step.

Since we already have NSS libraries built with ECC, we can create a new PKCS11 provider

Create ${java.home}/jre/lib/security/nss.cfg as follows:

name = NSS
    nssLibraryDirectory = ${nsslibdir}
    nssDbMode = noDb
    attributes = compatibility

where nsslibdir should contain NSS libs with ECC support.

Add the following line to ${java.home}/jre/lib/security/ : ${java.home}/lib/security/nss.cfg

Note that those who are using Oracle iPlanet Web Server or Oracle Traffic Director, NSS libs built with ECC are in <ws_install_dir>/lib or <otd_install_dir>/lib.

4.3. Now keytool should work

Now you can try the same keytool command and see that it succeeds :

$JAVA_HOME/bin/keytool -importkeystore -srckeystore ecc-cert.p12 -srcstoretype PKCS12 -deststoretype JKS -destkeystore ecc.jks -srcstorepass password -deststorepass password -srcalias ecc-cert -destalias ecc-cert -srckeypass password -destkeypass password -v

[Storing ecc.jks]

5. Convert JKS keystore into an Oracle Wallet

You can export this cert and key from JKS keystore and import it into an Oracle Wallet if you need using orapki tool as shown below.

Make sure that orapki you use supports ECC.

Also for ECC you MUST use "-jsafe" option.

$ orapki wallet create -pwd password  -wallet .  -jsafe

$ orapki wallet jks_to_pkcs12 -wallet . -pwd password -keystore ecc.jks -jkspwd password -jsafe


$orapki wallet display -wallet . -pwd welcome1  -jsafe
Oracle PKI Tool : Version
Copyright (c) 2004, 2012, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=test,C=US
Trusted Certificates:
Subject:        OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
Subject:        OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        CN=test,C=US

As you can see our ECC cert in the wallet.

You can follow the same steps for RSA certs as well.

6. References


Meena Vyas


« August 2016