Configuring Oracle iPlanet WebServer / Oracle Traffic Director to use crypto accelerators on T4-1 servers

Configuring Oracle iPlanet Web Server / Oracle Traffic Director to use crypto accelerators on T4-1 servers

Jyri had written a technical article on Configuring Solaris Cryptographic Framework and Sun Java System Web Server 7 on Systems With UltraSPARC T1 Processors. I tried to find out what has changed since then in T4.

I have used a T4-1 SPARC system with Solaris 10. Results slightly vary for Solaris 11.  For Solaris 11, the T4 optimization was implemented in while it was in for Solaris 10.

Overview of T4 processors is here in this blog.

Many thanx to Chi-Chang Lin and Julien for their help.

1. Install Oracle iPlanet Web Server / Oracle Traffic Director.  Go to instance/config directory.

 # cd /opt/oracle/webserver7/https-hostname.fqdn/config

2. List default PKCS#11 Modules

# ../../bin/modutil -dbdir . -list
Listing of PKCS #11 Modules
1. NSS Internal PKCS #11 Module
slots: 2 slots attached
status: loaded

slot: NSS Internal Cryptographic Services
token: NSS Generic Crypto Services

slot: NSS User Private Key and Certificate Services
token: NSS Certificate DB

2. Root Certs
library name:
slots: 1 slot attached
status: loaded

slot: NSS Builtin Objects
token: Builtin Object Token

3. Initialize the soft token data store in the $HOME/.sunw/pkcs11_softtoken/ directory

# pktool setpin keystore=pkcs11
Enter token passphrase: olderpassword
Create new passphrase: password
Re-enter new passphrase: password
Passphrase changed.

4. Offload crypto operations to Solaris Crypto Framework on T4

$ ../../bin/modutil -dbdir . -nocertdb -add SCF -libfile /usr/lib/ -mechanisms RSA:AES:SHA1:MD5

Module "SCF" added to database.

Note that

  • -nocertdb means modutil won't try to open the NSS softoken key database. It doesn't even have to be present.

  • PKCS#11 library used is /usr/lib/ If the server is running in 64 bit mode, we have to use /usr/lib/64/

  • Unlike T1 and T2, in T4 we do not have to disable mechanisms in softtoken provider using cryptoadm.

5. List again to check that a new module SCF is added

# ../../bin/modutil -dbdir . -list
Listing of PKCS #11 Modules
1. NSS Internal PKCS #11 Module
slots: 2 slots attached
status: loaded

slot: NSS Internal Cryptographic Services
token: NSS Generic Crypto Services

slot: NSS User Private Key and Certificate Services
token: NSS Certificate DB

2. SCF
library name: /usr/lib/
slots: 2 slots attached
status: loaded

slot: Sun Metaslot
token: Sun Metaslot

slot: n2rng/0 SUNW_N2_Random_Number_Generator
token: n2rng/0 SUNW_N2_RNG
3. Root Certs
library name:
slots: 1 slot attached
status: loaded
slot: NSS Builtin Objects
token: Builtin Object Token

6.  Create certificate in “Sun Metaslot” :

I have used certutil, but you must use Admin Server CLI / GUI

# ../../bin/certutil -S -x -n "Server-Cert" -t "CT,CT,CT" -s "CN=*.fqdn" -d . -h "Sun Metaslot"
Enter Password or Pin for "Sun Metaslot": password

7. Verify that the certificate is created properly in “Sun Metslaot”

# ../../bin/certutil -L -d . -h "Sun Metaslot"
Certificate Nickname Trust Attributes
Enter Password or Pin for "Sun Metaslot": password
Sun Metaslot:Server-Cert CTu,Cu,Cu

8. Associate this newly created certificate to http listener using Admin CLI/GUI. After that server.xml should have

<http-listener> ...
        <server-cert-nickname>Sun Metaslot:Server-Cert</server-cert-nicknamer>

Note the prefix "Sun Metaslot"

9. Disable PKCS#11 bypass

To use the accelerated AES algorithm, turn off PKCS#11 bypass, and configure modutil to have the AES mechanism go to the Metaslot.

After you disable PKCS#11 bypasss using Admin GUI/CLI,  check that server.xml should have

<server> ....

With PKCS#11 bypass enabled, Oracle iPlanet Web Server will only use the RSA capability of the T4, provided certificate and key are stored in the T4 slot (Metaslot). Actually, the RSA op is never bypassed in NSS, it's always done with PKCS#11 calls. So the bypass settings won't affect the behavior of the probes for RSA at all. The only thing that matters if where the RSA key and certificate live, ie. which PKCS#11 token, and thus which PKCS#11 module gets called to do the work. If your certificate/key are in the NSS certificate/key db, you will see libsoftokn3/libfreebl libraries doing the RSA work. If they are in the Sun Metaslot, it should be the Solaris code.

10. Start the server instance

# ../bin/startserv
Oracle iPlanet Web Server 7.0.16 B09/14/2012 03:33
Please enter the PIN for the "Sun Metaslot" token: password
info: HTTP3072: http-listener-1: https://hostname.fqdn:80 ready to accept requests
info: CORE3274: successful server startup

11. Figure out which process to run this DTrace script on

# ps -eaf | grep webservd | grep -v dog
18224 18223 0 13:17:25 ? 0:07 webservd -d /opt/oracle/webserver7/https-hostname.fqdn/config -r /opt/
18225 18224 0 13:17:25 ? 0:00 webservd -d /opt/oracle/webserver7/https-hostname.fqdn/config -r /opt/

(For Oracle Traffic Director look for process named "trafficd")

We see that the child process id is “18225

12. Clients for testing :

You can use any browser. I used NSS tool tstclnt for testing

$cat > req.txt
GET /index.html HTTP/1.0

For checking both RSA and AES, I used cipher “:0035” which is TLS_RSA_WITH_AES_256_CBC_SHA

$./tstclnt -h hostname -p 80 -d . -T -f -o -v -c “:0035” < req.txt

13. How do I make sure that crypto accelerator is being used

13.1 Create DTrace script

The following D script should be able to uncover whether T4-specific crypto routine are being called or not. It also displays stats per second.

# cat > t4crypto.d
#!/usr/sbin/dtrace -s

    @ops[probemod, probefunc] = count();


Invoke with './t4crypto.d -p <pid> '

13.2 EXPECTED PROBES FOR Solaris 10 :

If offloading to T4 HW are correctly set up, the expected DTrace output would have these probes and libraries





soft_decrypt_rsa_pkcs_decode, soft_encrypt_rsa_pkcs_encode soft_rsa_crypt_init_common soft_rsa_decrypt, soft_rsa_encrypt soft_rsa_decrypt_common, soft_rsa_encrypt_common


yf_aes_instructions_present yf_aes_expand256, yf_aes256_cbc_decrypt, yf_aes256_cbc_encrypt, yf_aes256_load_keys_for_decrypt, yf_aes256_load_keys_for_encrypt,

Note that these are for 256, same for 128, 192...

these are for cbc, same for ecb, ctr, cfb128...


yf_des_expand, yf_des_instructions_present yf_des_encrypt


yf_md5_multiblock, yf_md5_instruction_present


yf_sha1_instruction_present, yf_sha1_multibloc


# ./t4crypto.d -p 18225   soft_decrypt_rsa_pkcs_decode    1   soft_rsa_crypt_init_common      1   soft_rsa_decrypt                1   big_mp_mul_yf                   2   mpm_yf_mpmul                    2   mpmul_arr_yf                    2   rijndael_key_setup_enc_yf       2   soft_rsa_decrypt_common         2   yf_aes_expand256                2   yf_aes256_cbc_decrypt           3   yf_aes256_load_keys_for_decrypt 3   big_mont_mul_yf                 6   mm_yf_montmul                   6   yf_des_instructions_present     6   yf_aes256_cbc_encrypt           8   yf_aes256_load_keys_for_encrypt 8   yf_mpmul_present                8   yf_aes_instructions_present    13   yf_des_encrypt                 18                yf_md5_multiblock              41                yf_md5_instruction_present     72                yf_sha1_instruction_present    82                yf_sha1_multiblock             82

This indicates that both RSA and AES ops are done in Solaris Crypto Framework.


# ./t4crypto.d -p 18225   soft_decrypt_rsa_pkcs_decode 1   soft_rsa_crypt_init_common   1   soft_rsa_decrypt             1   soft_rsa_decrypt_common      1   big_mp_mul_yf                2   mpm_yf_mpmul                 2   mpmul_arr_yf                 2   big_mont_mul_yf              6   mm_yf_montmul                6   yf_mpmul_present             8

For this cipher, when I enable PKCS#11 bypass, Only RSA probes are being hit AES probes are not being hit.

13.5 ustack() for RSA operations / probefunc == "soft_rsa_decrypt" /

Shows that is calling C_* functions of which is calling functions of for both cases with and without bypass.

When PKCS#11 bypass is disabled (allow-bypass is 0)`soft_rsa_decrypt`soft_rsa_decrypt_common+0x94`soft_unwrapkey+0x258`C_UnwrapKey+0x1ec`meta_unwrap_key+0x17c`meta_UnwrapKey+0xc4`C_UnwrapKey+0xfc`pk11_AnyUnwrapKey+0x6b8`PK11_PubUnwrapSymKey+0x8c`ssl3_HandleRSAClientKeyExchange+0x1a0`ssl3_HandleClientKeyExchange+0x154`ssl3_HandleHandshakeMessage+0x440`ssl3_HandleHandshake+0x11c`ssl3_HandleRecord+0x5e8`ssl3_GatherCompleteHandshake+0x5c`ssl_GatherRecord1stHandshake+0x30`ssl_Do1stHandshake+0xec`ssl_SecureRecv+0x1c8`ssl_Recv+0x9c`__1cNDaemonSessionDrun6M_v_+0x2dc

When PKCS#11 bypass is enabled (allow-bypass is 1)`soft_rsa_decrypt`soft_rsa_decrypt_common+0x94`C_Decrypt+0x164`meta_do_operation+0x27c`meta_Decrypt+0x4c`C_Decrypt+0xcc`PK11_PrivDecryptPKCS1+0x1ac`ssl3_HandleRSAClientKeyExchange+0xe4`ssl3_HandleClientKeyExchange+0x154`ssl3_HandleHandshakeMessage+0x440`ssl3_HandleHandshake+0x11c`ssl3_HandleRecord+0x5e8`ssl3_GatherCompleteHandshake+0x5c`ssl_GatherRecord1stHandshake+0x30`ssl_Do1stHandshake+0xec`ssl_SecureRecv+0x1c8`ssl_Recv+0x9c`__1cNDaemonSessionDrun6M_v_+0x2dc`ThreadMain+0x1c`_pt_root+0xe8

13.6 ustack() FOR AES operations / probefunc == "yf_aes256_cbc_encrypt" /

When PKCS#11 bypass is disabled (allow-bypass is 0)`yf_aes256_cbc_encrypt`aes_block_process_contiguous_whole_blocks+0xb4`aes_crypt_contiguous_blocks+0x1cc`soft_aes_encrypt_common+0x22c`C_EncryptUpdate+0x10c`meta_do_operation+0x1fc`meta_EncryptUpdate+0x4c`C_EncryptUpdate+0xcc`PK11_CipherOp+0x1a0`ssl3_CompressMACEncryptRecord+0x264`ssl3_SendRecord+0x300`ssl3_FlushHandshake+0x54`ssl3_SendFinished+0x1fc`ssl3_HandleFinished+0x314`ssl3_HandleHandshakeMessage+0x4ac`ssl3_HandleHandshake+0x11c`ssl3_HandleRecord+0x5e8`ssl3_GatherCompleteHandshake+0x5c`ssl_GatherRecord1stHandshake+0x30`ssl_Do1stHandshake+0xec

Shows that is calling C_* functions of which is calling functions of

However when PKCS#11 bypass is disabled (allow-bypass is 1) this stack isn't getting called.


# ./t4crypto.d -p 18225 -l
55720 pid18225 yf_md5_instruction_present entry
55721 pid18225 yf_sha256_instruction_present entry
55722 pid18225 yf_sha512_instruction_present entry
55723 pid18225 yf_sha1_instruction_present entry
55724 pid18225 yf_sha256 entry
55725 pid18225 yf_sha256_multiblock entry
55726 pid18225 yf_sha512 entry
55727 pid18225 yf_sha512_multiblock entry
55728 pid18225 yf_sha1 entry
55729 pid18225 yf_sha1_multiblock entry
55730 pid18225 yf_md5 entry
55731 pid18225 yf_md5_multiblock entry
55732 pid18225 yf_aes_instructions_present entry
55733 pid18225 rijndael_key_setup_enc_yf entry
55734 pid18225 yf_aes_expand128 entry
55735 pid18225 yf_aes_encrypt128 entry
55736 pid18225 yf_aes_decrypt128 entry
55737 pid18225 yf_aes_expand192 entry
55738 pid18225 yf_aes_encrypt192 entry
55739 pid18225 yf_aes_decrypt192 entry
55740 pid18225 yf_aes_expand256 entry
55741 pid18225 yf_aes_encrypt256 entry
55742 pid18225 yf_aes_decrypt256 entry
55743 pid18225 yf_aes128_load_keys_for_encrypt entry
55744 pid18225 yf_aes192_load_keys_for_encrypt entry
55745 pid18225 yf_aes256_load_keys_for_encrypt entry
55746 pid18225 yf_aes128_ecb_encrypt entry
55747 pid18225 yf_aes192_ecb_encrypt entry
55748 pid18225 yf_aes256_ecb_encrypt entry
55749 pid18225 yf_aes128_cbc_encrypt entry
55750 pid18225 yf_aes192_cbc_encrypt entry
55751 pid18225 yf_aes256_cbc_encrypt entry
55752 pid18225 yf_aes128_ctr_crypt entry
55753 pid18225 yf_aes192_ctr_crypt entry
55754 pid18225 yf_aes256_ctr_crypt entry
55755 pid18225 yf_aes128_cfb128_encrypt entry
55756 pid18225 yf_aes192_cfb128_encrypt entry
55757 pid18225 yf_aes256_cfb128_encrypt entry
55758 pid18225 yf_aes128_load_keys_for_decrypt entry
55759 pid18225 yf_aes192_load_keys_for_decrypt entry
55760 pid18225 yf_aes256_load_keys_for_decrypt entry
55761 pid18225 yf_aes128_ecb_decrypt entry
55762 pid18225 yf_aes192_ecb_decrypt entry
55763 pid18225 yf_aes256_ecb_decrypt entry
55764 pid18225 yf_aes128_cbc_decrypt entry
55765 pid18225 yf_aes192_cbc_decrypt entry
55766 pid18225 yf_aes256_cbc_decrypt entry
55767 pid18225 yf_aes128_cfb128_decrypt entry
55768 pid18225 yf_aes192_cfb128_decrypt entry
55769 pid18225 yf_aes256_cfb128_decrypt entry
55771 pid18225 yf_des_instructions_present entry
55772 pid18225 yf_des_expand entry
55773 pid18225 yf_des_encrypt entry
55774 pid18225 yf_mpmul_present entry
55775 pid18225 yf_montmul_present entry
55776 pid18225 mm_yf_montmul entry
55777 pid18225 mm_yf_montsqr entry
55778 pid18225 mm_yf_restore_func entry
55779 pid18225 mm_yf_ret_from_mont_func entry
55780 pid18225 mm_yf_execute_slp entry
55781 pid18225 big_modexp_ncp_yf entry
55782 pid18225 big_mont_mul_yf entry
55783 pid18225 mpmul_arr_yf entry
55784 pid18225 big_mp_mul_yf entry
55785 pid18225 mpm_yf_mpmul entry
55786 pid18225 nsapi_rsa_set_priv_fn entry
55795 pid18225 prepare_rsa_priv_key_export_for_asn1 entry
55796 pid18225 sunw_dst_rsaref_init entry
55797 pid18225 NSS_Get_SEC_UniversalStringTemplate entry
55813 pid18225 prepare_low_rsa_priv_key_for_asn1 entry
55814 pid18225 rsa_FormatOneBlock entry
55815 pid18225 rsa_FormatBlock entry
55816 pid18225 lg_prepare_low_rsa_priv_key_for_asn1 entry
55817 pid18225 rsa_build_from_primes entry
55818 pid18225 rsa_is_prime entry
55819 pid18225 rsa_get_primes_from_exponents entry
55820 pid18225 rsa_PrivateKeyOpNoCRT entry
55821 pid18225 rsa_PrivateKeyOpCRTNoCheck entry
55822 pid18225 rsa_PrivateKeyOpCRTCheckedPubKey entry
55823 pid18225 key_gen_rsa_by_value entry
55824 pid18225 get_rsa_private_key entry
55825 pid18225 get_rsa_public_key entry
55826 pid18225 soft_rsa_encrypt entry
55827 pid18225 soft_rsa_decrypt entry
55828 pid18225 soft_rsa_crypt_init_common entry
55829 pid18225 soft_rsa_encrypt_common entry
55830 pid18225 soft_rsa_decrypt_common entry
55831 pid18225 soft_rsa_sign_verify_init_common entry
55832 pid18225 soft_rsa_sign_common entry
55833 pid18225 soft_rsa_verify_common entry
55834 pid18225 generate_rsa_key entry
55835 pid18225 soft_rsa_genkey_pair entry
55836 pid18225 get_rsa_sha1_prefix entry
55837 pid18225 soft_rsa_digest_sign_common entry
55838 pid18225 soft_rsa_digest_verify_common entry
55839 pid18225 soft_rsa_verify_recover entry
55840 pid18225 rsa_pri_to_asn1 entry
55841 pid18225 asn1_to_rsa_pri entry
55842 pid18225 soft_encrypt_rsa_pkcs_encode entry
55843 pid18225 soft_decrypt_rsa_pkcs_decode entry
55844 pid18225 soft_sign_rsa_pkcs_encode entry
55845 pid18225 soft_verify_rsa_pkcs_decode entry
55770 profile tick-1sec


Hello Meena,

Thank You for the blogpost, it is awesome!

But :),

i have installed a fresh Solaris 10 (u10) T4-1 server, patched with EIS-2012.08, and deployed an iPlantet 7.0 u15. Configured for instance using https, the certificate is a Verisign signed 2048 bit key (SSL3-RSA).

When i start the instance i become the following message:

[20/Nov/2012:08:38:33] info ( 6356): HTTP3072: http-listener-1: ready to accept requests
[20/Nov/2012:08:38:33] info ( 6356): CORE3274: successful server startup
[20/Nov/2012:08:38:33] catastrophe ( 6356): CORE3260: Server crash detected (signal SIGBUS)
[20/Nov/2012:08:38:33] info ( 6356): CORE3262: Crash occurred in function C_GenerateRandom from module /usr/lib/security/sparcv9/pkc
[20/Nov/2012:08:38:34] info ( 6119): CORE3050: Spawning replacement for worker process 6356

After i configure the "Allow PKCS11 Bypass" back to TRUE, the server can operate well....

Where i make the mistake? Or where is the problem? (What is not neccessarily a great problem, just for the blog :) )



Posted by Ferike on November 29, 2012 at 05:55 PM IST #

With "allow Pkcs11 Bypass true", only RSA operations will by done by h/w crypto. It won't send AES operations to h/w crypto. Please raise a bug/escalation with Oracle. this needs to be fixed.

Posted by Meena on November 30, 2012 at 02:14 PM IST #

You can write to us in webtier forum

Posted by Meena on November 30, 2012 at 02:44 PM IST #

It IS MY failure. The origin server, where i the configuration has tested is a T2000 and not a T4-1.... Therefore is the failure.

best regards,


Posted by guest on November 30, 2012 at 07:10 PM IST #

For configuring WebServer on T1, refer my blog

Still server should not crash. You can file an escalation if you want.

Posted by Meena on November 30, 2012 at 08:26 PM IST #

Failures in C_GenerateRandom() can occur if you have Metaslot disabled. To see if it's disabled, type "cryptoadm list metaslot". To enable it, type "cryptoadm enable metaslot". Also, make sure you are up to the current patch level for S10. If these don't fix it, I suggest contacting your Oracle service representative.

Posted by Dan on November 30, 2012 at 11:27 PM IST #


Are you folks sure that you tested Oracle Traffic Director on Sparc T4 servers. Last I heard, Oracle Traffic Director is not available on Solaris Sparc.

Any pointers for Oracle Traffic Director availability on SPARC.

Posted by guest on February 22, 2013 at 02:53 AM IST #

OTD will be available on SSC very shortly.

Posted by Sriram Natarajan on July 31, 2013 at 12:56 PM IST #


This was a very informative blog post. I'd like to also ask if you could dedicate a post to show how to upgrade the NSS library on an Oracle iPlanet web server installation without having to upgrade the server binaries.

Lastly, I noticed that you are using Oracle iPlanet web server version 7.0.16, yet the oracle download page only shows version 7.0.15 available for download. May I know how and where you got the latest version from?


Posted by guest on August 16, 2013 at 09:36 AM IST #

We do not recommend you to upgrade NSS libraries yourself.

Release notes for Web server 7.0.17 is available
I am trying to figure out.

Posted by Meena on August 16, 2013 at 09:50 AM IST #

iPlanet Web Server is currently available within Oracle Support. You should be able to login to Oracle Support and search for the following patch to download our latest - 7.0.17 service pack release.

Patch 145843-10: Oracle iPlanet Web Server 7.0 Update 17

this service pack , like any other iPlanet Web Server updates, can be updated on top of an existing installation.

Posted by Sriram Natarajan on August 17, 2013 at 04:48 AM IST #

Post a Comment:
  • HTML Syntax: NOT allowed

Meena Vyas


« April 2014