Thursday Nov 08, 2012

Configuring Oracle iPlanet WebServer / Oracle Traffic Director to use crypto accelerators on T4-1 servers

Configuring Oracle iPlanet Web Server / Oracle Traffic Director to use crypto accelerators on T4-1 servers

Jyri had written a technical article on Configuring Solaris Cryptographic Framework and Sun Java System Web Server 7 on Systems With UltraSPARC T1 Processors. I tried to find out what has changed since then in T4.

I have used a T4-1 SPARC system with Solaris 10. Results slightly vary for Solaris 11.  For Solaris 11, the T4 optimization was implemented in while it was in for Solaris 10.

Overview of T4 processors is here in this blog.

Many thanx to Chi-Chang Lin and Julien for their help.

1. Install Oracle iPlanet Web Server / Oracle Traffic Director.  Go to instance/config directory.

 # cd /opt/oracle/webserver7/https-hostname.fqdn/config

2. List default PKCS#11 Modules

# ../../bin/modutil -dbdir . -list
Listing of PKCS #11 Modules
1. NSS Internal PKCS #11 Module
slots: 2 slots attached
status: loaded

slot: NSS Internal Cryptographic Services
token: NSS Generic Crypto Services

slot: NSS User Private Key and Certificate Services
token: NSS Certificate DB

2. Root Certs
library name:
slots: 1 slot attached
status: loaded

slot: NSS Builtin Objects
token: Builtin Object Token

3. Initialize the soft token data store in the $HOME/.sunw/pkcs11_softtoken/ directory

# pktool setpin keystore=pkcs11
Enter token passphrase: olderpassword
Create new passphrase: password
Re-enter new passphrase: password
Passphrase changed.

4. Offload crypto operations to Solaris Crypto Framework on T4

$ ../../bin/modutil -dbdir . -nocertdb -add SCF -libfile /usr/lib/ -mechanisms RSA:AES:SHA1:MD5

Module "SCF" added to database.

Note that

  • -nocertdb means modutil won't try to open the NSS softoken key database. It doesn't even have to be present.

  • PKCS#11 library used is /usr/lib/ If the server is running in 64 bit mode, we have to use /usr/lib/64/

  • Unlike T1 and T2, in T4 we do not have to disable mechanisms in softtoken provider using cryptoadm.

5. List again to check that a new module SCF is added

# ../../bin/modutil -dbdir . -list
Listing of PKCS #11 Modules
1. NSS Internal PKCS #11 Module
slots: 2 slots attached
status: loaded

slot: NSS Internal Cryptographic Services
token: NSS Generic Crypto Services

slot: NSS User Private Key and Certificate Services
token: NSS Certificate DB

2. SCF
library name: /usr/lib/
slots: 2 slots attached
status: loaded

slot: Sun Metaslot
token: Sun Metaslot

slot: n2rng/0 SUNW_N2_Random_Number_Generator
token: n2rng/0 SUNW_N2_RNG
3. Root Certs
library name:
slots: 1 slot attached
status: loaded
slot: NSS Builtin Objects
token: Builtin Object Token

6.  Create certificate in “Sun Metaslot” :

I have used certutil, but you must use Admin Server CLI / GUI

# ../../bin/certutil -S -x -n "Server-Cert" -t "CT,CT,CT" -s "CN=*.fqdn" -d . -h "Sun Metaslot"
Enter Password or Pin for "Sun Metaslot": password

7. Verify that the certificate is created properly in “Sun Metslaot”

# ../../bin/certutil -L -d . -h "Sun Metaslot"
Certificate Nickname Trust Attributes
Enter Password or Pin for "Sun Metaslot": password
Sun Metaslot:Server-Cert CTu,Cu,Cu

8. Associate this newly created certificate to http listener using Admin CLI/GUI. After that server.xml should have

<http-listener> ...
        <server-cert-nickname>Sun Metaslot:Server-Cert</server-cert-nicknamer>

Note the prefix "Sun Metaslot"

9. Disable PKCS#11 bypass

To use the accelerated AES algorithm, turn off PKCS#11 bypass, and configure modutil to have the AES mechanism go to the Metaslot.

After you disable PKCS#11 bypasss using Admin GUI/CLI,  check that server.xml should have

<server> ....

With PKCS#11 bypass enabled, Oracle iPlanet Web Server will only use the RSA capability of the T4, provided certificate and key are stored in the T4 slot (Metaslot). Actually, the RSA op is never bypassed in NSS, it's always done with PKCS#11 calls. So the bypass settings won't affect the behavior of the probes for RSA at all. The only thing that matters if where the RSA key and certificate live, ie. which PKCS#11 token, and thus which PKCS#11 module gets called to do the work. If your certificate/key are in the NSS certificate/key db, you will see libsoftokn3/libfreebl libraries doing the RSA work. If they are in the Sun Metaslot, it should be the Solaris code.

10. Start the server instance

# ../bin/startserv
Oracle iPlanet Web Server 7.0.16 B09/14/2012 03:33
Please enter the PIN for the "Sun Metaslot" token: password
info: HTTP3072: http-listener-1: https://hostname.fqdn:80 ready to accept requests
info: CORE3274: successful server startup

11. Figure out which process to run this DTrace script on

# ps -eaf | grep webservd | grep -v dog
18224 18223 0 13:17:25 ? 0:07 webservd -d /opt/oracle/webserver7/https-hostname.fqdn/config -r /opt/
18225 18224 0 13:17:25 ? 0:00 webservd -d /opt/oracle/webserver7/https-hostname.fqdn/config -r /opt/

(For Oracle Traffic Director look for process named "trafficd")

We see that the child process id is “18225

12. Clients for testing :

You can use any browser. I used NSS tool tstclnt for testing

$cat > req.txt
GET /index.html HTTP/1.0

For checking both RSA and AES, I used cipher “:0035” which is TLS_RSA_WITH_AES_256_CBC_SHA

$./tstclnt -h hostname -p 80 -d . -T -f -o -v -c “:0035” < req.txt

13. How do I make sure that crypto accelerator is being used

13.1 Create DTrace script

The following D script should be able to uncover whether T4-specific crypto routine are being called or not. It also displays stats per second.

# cat > t4crypto.d
#!/usr/sbin/dtrace -s

    @ops[probemod, probefunc] = count();


Invoke with './t4crypto.d -p <pid> '

13.2 EXPECTED PROBES FOR Solaris 10 :

If offloading to T4 HW are correctly set up, the expected DTrace output would have these probes and libraries





soft_decrypt_rsa_pkcs_decode, soft_encrypt_rsa_pkcs_encode soft_rsa_crypt_init_common soft_rsa_decrypt, soft_rsa_encrypt soft_rsa_decrypt_common, soft_rsa_encrypt_common


yf_aes_instructions_present yf_aes_expand256, yf_aes256_cbc_decrypt, yf_aes256_cbc_encrypt, yf_aes256_load_keys_for_decrypt, yf_aes256_load_keys_for_encrypt,

Note that these are for 256, same for 128, 192...

these are for cbc, same for ecb, ctr, cfb128...


yf_des_expand, yf_des_instructions_present yf_des_encrypt


yf_md5_multiblock, yf_md5_instruction_present


yf_sha1_instruction_present, yf_sha1_multibloc


# ./t4crypto.d -p 18225   soft_decrypt_rsa_pkcs_decode    1   soft_rsa_crypt_init_common      1   soft_rsa_decrypt                1   big_mp_mul_yf                   2   mpm_yf_mpmul                    2   mpmul_arr_yf                    2   rijndael_key_setup_enc_yf       2   soft_rsa_decrypt_common         2   yf_aes_expand256                2   yf_aes256_cbc_decrypt           3   yf_aes256_load_keys_for_decrypt 3   big_mont_mul_yf                 6   mm_yf_montmul                   6   yf_des_instructions_present     6   yf_aes256_cbc_encrypt           8   yf_aes256_load_keys_for_encrypt 8   yf_mpmul_present                8   yf_aes_instructions_present    13   yf_des_encrypt                 18                yf_md5_multiblock              41                yf_md5_instruction_present     72                yf_sha1_instruction_present    82                yf_sha1_multiblock             82

This indicates that both RSA and AES ops are done in Solaris Crypto Framework.


# ./t4crypto.d -p 18225   soft_decrypt_rsa_pkcs_decode 1   soft_rsa_crypt_init_common   1   soft_rsa_decrypt             1   soft_rsa_decrypt_common      1   big_mp_mul_yf                2   mpm_yf_mpmul                 2   mpmul_arr_yf                 2   big_mont_mul_yf              6   mm_yf_montmul                6   yf_mpmul_present             8

For this cipher, when I enable PKCS#11 bypass, Only RSA probes are being hit AES probes are not being hit.

13.5 ustack() for RSA operations / probefunc == "soft_rsa_decrypt" /

Shows that is calling C_* functions of which is calling functions of for both cases with and without bypass.

When PKCS#11 bypass is disabled (allow-bypass is 0)`soft_rsa_decrypt`soft_rsa_decrypt_common+0x94`soft_unwrapkey+0x258`C_UnwrapKey+0x1ec`meta_unwrap_key+0x17c`meta_UnwrapKey+0xc4`C_UnwrapKey+0xfc`pk11_AnyUnwrapKey+0x6b8`PK11_PubUnwrapSymKey+0x8c`ssl3_HandleRSAClientKeyExchange+0x1a0`ssl3_HandleClientKeyExchange+0x154`ssl3_HandleHandshakeMessage+0x440`ssl3_HandleHandshake+0x11c`ssl3_HandleRecord+0x5e8`ssl3_GatherCompleteHandshake+0x5c`ssl_GatherRecord1stHandshake+0x30`ssl_Do1stHandshake+0xec`ssl_SecureRecv+0x1c8`ssl_Recv+0x9c`__1cNDaemonSessionDrun6M_v_+0x2dc

When PKCS#11 bypass is enabled (allow-bypass is 1)`soft_rsa_decrypt`soft_rsa_decrypt_common+0x94`C_Decrypt+0x164`meta_do_operation+0x27c`meta_Decrypt+0x4c`C_Decrypt+0xcc`PK11_PrivDecryptPKCS1+0x1ac`ssl3_HandleRSAClientKeyExchange+0xe4`ssl3_HandleClientKeyExchange+0x154`ssl3_HandleHandshakeMessage+0x440`ssl3_HandleHandshake+0x11c`ssl3_HandleRecord+0x5e8`ssl3_GatherCompleteHandshake+0x5c`ssl_GatherRecord1stHandshake+0x30`ssl_Do1stHandshake+0xec`ssl_SecureRecv+0x1c8`ssl_Recv+0x9c`__1cNDaemonSessionDrun6M_v_+0x2dc`ThreadMain+0x1c`_pt_root+0xe8

13.6 ustack() FOR AES operations / probefunc == "yf_aes256_cbc_encrypt" /

When PKCS#11 bypass is disabled (allow-bypass is 0)`yf_aes256_cbc_encrypt`aes_block_process_contiguous_whole_blocks+0xb4`aes_crypt_contiguous_blocks+0x1cc`soft_aes_encrypt_common+0x22c`C_EncryptUpdate+0x10c`meta_do_operation+0x1fc`meta_EncryptUpdate+0x4c`C_EncryptUpdate+0xcc`PK11_CipherOp+0x1a0`ssl3_CompressMACEncryptRecord+0x264`ssl3_SendRecord+0x300`ssl3_FlushHandshake+0x54`ssl3_SendFinished+0x1fc`ssl3_HandleFinished+0x314`ssl3_HandleHandshakeMessage+0x4ac`ssl3_HandleHandshake+0x11c`ssl3_HandleRecord+0x5e8`ssl3_GatherCompleteHandshake+0x5c`ssl_GatherRecord1stHandshake+0x30`ssl_Do1stHandshake+0xec

Shows that is calling C_* functions of which is calling functions of

However when PKCS#11 bypass is disabled (allow-bypass is 1) this stack isn't getting called.


# ./t4crypto.d -p 18225 -l
55720 pid18225 yf_md5_instruction_present entry
55721 pid18225 yf_sha256_instruction_present entry
55722 pid18225 yf_sha512_instruction_present entry
55723 pid18225 yf_sha1_instruction_present entry
55724 pid18225 yf_sha256 entry
55725 pid18225 yf_sha256_multiblock entry
55726 pid18225 yf_sha512 entry
55727 pid18225 yf_sha512_multiblock entry
55728 pid18225 yf_sha1 entry
55729 pid18225 yf_sha1_multiblock entry
55730 pid18225 yf_md5 entry
55731 pid18225 yf_md5_multiblock entry
55732 pid18225 yf_aes_instructions_present entry
55733 pid18225 rijndael_key_setup_enc_yf entry
55734 pid18225 yf_aes_expand128 entry
55735 pid18225 yf_aes_encrypt128 entry
55736 pid18225 yf_aes_decrypt128 entry
55737 pid18225 yf_aes_expand192 entry
55738 pid18225 yf_aes_encrypt192 entry
55739 pid18225 yf_aes_decrypt192 entry
55740 pid18225 yf_aes_expand256 entry
55741 pid18225 yf_aes_encrypt256 entry
55742 pid18225 yf_aes_decrypt256 entry
55743 pid18225 yf_aes128_load_keys_for_encrypt entry
55744 pid18225 yf_aes192_load_keys_for_encrypt entry
55745 pid18225 yf_aes256_load_keys_for_encrypt entry
55746 pid18225 yf_aes128_ecb_encrypt entry
55747 pid18225 yf_aes192_ecb_encrypt entry
55748 pid18225 yf_aes256_ecb_encrypt entry
55749 pid18225 yf_aes128_cbc_encrypt entry
55750 pid18225 yf_aes192_cbc_encrypt entry
55751 pid18225 yf_aes256_cbc_encrypt entry
55752 pid18225 yf_aes128_ctr_crypt entry
55753 pid18225 yf_aes192_ctr_crypt entry
55754 pid18225 yf_aes256_ctr_crypt entry
55755 pid18225 yf_aes128_cfb128_encrypt entry
55756 pid18225 yf_aes192_cfb128_encrypt entry
55757 pid18225 yf_aes256_cfb128_encrypt entry
55758 pid18225 yf_aes128_load_keys_for_decrypt entry
55759 pid18225 yf_aes192_load_keys_for_decrypt entry
55760 pid18225 yf_aes256_load_keys_for_decrypt entry
55761 pid18225 yf_aes128_ecb_decrypt entry
55762 pid18225 yf_aes192_ecb_decrypt entry
55763 pid18225 yf_aes256_ecb_decrypt entry
55764 pid18225 yf_aes128_cbc_decrypt entry
55765 pid18225 yf_aes192_cbc_decrypt entry
55766 pid18225 yf_aes256_cbc_decrypt entry
55767 pid18225 yf_aes128_cfb128_decrypt entry
55768 pid18225 yf_aes192_cfb128_decrypt entry
55769 pid18225 yf_aes256_cfb128_decrypt entry
55771 pid18225 yf_des_instructions_present entry
55772 pid18225 yf_des_expand entry
55773 pid18225 yf_des_encrypt entry
55774 pid18225 yf_mpmul_present entry
55775 pid18225 yf_montmul_present entry
55776 pid18225 mm_yf_montmul entry
55777 pid18225 mm_yf_montsqr entry
55778 pid18225 mm_yf_restore_func entry
55779 pid18225 mm_yf_ret_from_mont_func entry
55780 pid18225 mm_yf_execute_slp entry
55781 pid18225 big_modexp_ncp_yf entry
55782 pid18225 big_mont_mul_yf entry
55783 pid18225 mpmul_arr_yf entry
55784 pid18225 big_mp_mul_yf entry
55785 pid18225 mpm_yf_mpmul entry
55786 pid18225 nsapi_rsa_set_priv_fn entry
55795 pid18225 prepare_rsa_priv_key_export_for_asn1 entry
55796 pid18225 sunw_dst_rsaref_init entry
55797 pid18225 NSS_Get_SEC_UniversalStringTemplate entry
55813 pid18225 prepare_low_rsa_priv_key_for_asn1 entry
55814 pid18225 rsa_FormatOneBlock entry
55815 pid18225 rsa_FormatBlock entry
55816 pid18225 lg_prepare_low_rsa_priv_key_for_asn1 entry
55817 pid18225 rsa_build_from_primes entry
55818 pid18225 rsa_is_prime entry
55819 pid18225 rsa_get_primes_from_exponents entry
55820 pid18225 rsa_PrivateKeyOpNoCRT entry
55821 pid18225 rsa_PrivateKeyOpCRTNoCheck entry
55822 pid18225 rsa_PrivateKeyOpCRTCheckedPubKey entry
55823 pid18225 key_gen_rsa_by_value entry
55824 pid18225 get_rsa_private_key entry
55825 pid18225 get_rsa_public_key entry
55826 pid18225 soft_rsa_encrypt entry
55827 pid18225 soft_rsa_decrypt entry
55828 pid18225 soft_rsa_crypt_init_common entry
55829 pid18225 soft_rsa_encrypt_common entry
55830 pid18225 soft_rsa_decrypt_common entry
55831 pid18225 soft_rsa_sign_verify_init_common entry
55832 pid18225 soft_rsa_sign_common entry
55833 pid18225 soft_rsa_verify_common entry
55834 pid18225 generate_rsa_key entry
55835 pid18225 soft_rsa_genkey_pair entry
55836 pid18225 get_rsa_sha1_prefix entry
55837 pid18225 soft_rsa_digest_sign_common entry
55838 pid18225 soft_rsa_digest_verify_common entry
55839 pid18225 soft_rsa_verify_recover entry
55840 pid18225 rsa_pri_to_asn1 entry
55841 pid18225 asn1_to_rsa_pri entry
55842 pid18225 soft_encrypt_rsa_pkcs_encode entry
55843 pid18225 soft_decrypt_rsa_pkcs_decode entry
55844 pid18225 soft_sign_rsa_pkcs_encode entry
55845 pid18225 soft_verify_rsa_pkcs_decode entry
55770 profile tick-1sec

Tuesday Nov 06, 2012

Interesting articles and blogs on SPARC T4

Interesting articles and blogs on SPARC T4 processor

I have consolidated all the interesting information I could get on SPARC T4 processor and its hardware cryptographic capabilities.  Hope its useful.

1. Advantages of SPARC T4 processor

Most important points in this T4 announcement are :

"The SPARC T4 processor was designed from the ground up for high speed security and has a cryptographic stream processing unit (SPU) integrated directly into each processor core. These accelerators support 16 industry standard security ciphers and enable high speed encryption at rates 3 to 5 times that of competing processors. By integrating encryption capabilities directly inside the instruction pipeline, the SPARC T4 processor eliminates the performance and cost barriers typically associated with secure computing and makes it possible to deliver high security levels without impacting the user experience."

Data Sheet has more details on these :

"New on-chip Encryption Instruction Accelerators with direct non-privileged support for 16 industry-standard cryptographic algorithms plus random number generation in each of the eight cores: AES, Camellia, CRC32c, DES, 3DES, DH, DSA, ECC, Kasumi, MD5, RSA, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512"

I ran "isainfo -v" command on Solaris 11 Sparc T4-1 system. It shows the new instructions as expected :

$ isainfo -v
64-bit sparcv9 applications
crc32c cbcond pause mont mpmul sha512 sha256 sha1 md5 camellia kasumi des aes ima hpc vis3 fmaf asi_blk_init vis2 vis popc
32-bit sparc applications
  crc32c cbcond pause mont mpmul sha512 sha256 sha1 md5 camellia kasumi des aes ima hpc vis3 fmaf asi_blk_init vis2 vis popc v8plus div32 mul32

2. Dan Anderson's Blog have some interesting points about how these can be used :

"New T4 crypto instructions include: aes_kexpand0, aes_kexpand1, aes_kexpand2, 
aes_eround01, aes_eround23, aes_eround01_l, aes_eround_23_l, aes_dround01, aes_dround23, aes_dround01_l, aes_dround_23_l.

Having SPARC T4 hardware crypto instructions is all well and good, but how do we access it ?

The software is available with Solaris 11 and is used automatically if you are running Solaris a SPARC T4.  It is used internally in the kernel through kernel crypto modules. It is available in user space through the PKCS#11 library."

3. Dan Anderson's Blog on Where's the Crypto Libraries?

Although this was written in 2009 but still is very useful

"Here's a brief tour of the major crypto libraries shown in the digraph:
  • The libpkcs11 library contains the PKCS#11 API (C_\*() functions, such as C_Initialize()).
  • That in turn calls library pkcs11_softtoken or pkcs11_kernel, for userland or kernel crypto providers. The latter is used mostly for hardware-assisted cryptography (such as n2cp for Niagara2 SPARC processors), as that is performed more efficiently in kernel space with the "kCF" module (Kernel Crypto Framework). Additionally, for Solaris 10, strong crypto algorithms were split off in separate libraries, pkcs11_softtoken_extra
  • libcryptoutil contains low-level utility functions to help implement cryptography.
  • libsoftcrypto (OpenSolaris and Solaris Nevada only) implements several symmetric-key crypto algorithms in software, such as AES, RC4, and DES3, and the bignum library (used for RSA).
  • libmd implements MD5, SHA, and SHA2 message digest algorithms"

4. Dan Anderson's Blog on  How to tell if SPARC T4 crypto is being used?

5. Difference in T3 and T4

Diagram in this blog is good and self explanatory.

Jeff's blog also highlights the differences 

"The T4 servers have improved crypto acceleration, described at It is "just built in" so administrators no longer have to assign crypto accelerator units to domains - it "just happens". Every physical or virtual CPU on a SPARC-T4 has full access to hardware based crypto acceleration at all times. .... For completeness sake, it's worth noting that the T4 adds more crypto algorithms, and accelerates Camellia, CRC32c, and more SHA-x."

6. About performance counters

In this blog, performance counters are explained :

  • "Note that unlike T3 and before, T4 crypto doesn't require kernel modules like ncp or n2cp, there is no visibility of crypto hardware with kstats or cryptoadm.
  • T4 does provide hardware counters for crypto operations. You can see these using cpustat: cpustat -c pic0=Instr_FGU_crypto 5
  • You can check the general crypto support of the hardware and OS with the command "isainfo -v".
  • Since T4 crypto's implementation now allows direct userland access, there are no "crypto units" visible to cryptoadm.  "

For more details refer Martin's blog as well.

7. How to turn off  SPARC T4 or Intel AES-NI crypto acceleration

 I found this interesting blog from Darren about how to turn off  SPARC T4 or Intel AES-NI crypto acceleration.

"One of the new Solaris 11 features of the linker/loader is the ability to have a single ELF object that has multiple different implementations of the same functions that are selected at runtime based on the capabilities of the machine. The alternate to this is having the application coded to call getisax(2) system call and make the choice itself. We use this functionality of the linker/loader when we build the userland libraries for the Solaris Cryptographic Framework (specifically and

The Solaris linker/loader allows control of a lot of its functionality via environment variables, we can use that to control the version of the cryptographic functions we run. To do this we simply export the LD_HWCAP environment variable with values that tell to not select the HWCAP section matching certain features even if isainfo says they are present. This will work for consumers of the Solaris Cryptographic Framework that use the Solaris PKCS#11 libraries or use interfaces directly. For SPARC T4 : export LD_HWCAP="-aes -des -md5 -sha256 -sha512 -mont -mpul" .. For Intel systems with AES-NI support: export LD_HWCAP="-aes""

Note that LD_HWCAP is explained in

"LD_HWCAP, LD_HWCAP_32, and LD_HWCAP_64 -  Identifies an alternative hardware capabilities value... A “-” prefix results in the capabilities that follow being removed from the alternative capabilities."

8. Whitepaper on High Performance Security For Oracle Database and Fusion Middleware Applications using SPARC T4

This whitepaper on "High Performance Security For Oracle Database and Fusion Middleware Applications using SPARC T4 explains more details. It has DTrace scripts which may come in handy :

"To ensure the hardware-assisted cryptographic acceleration is configured to use and working with the security scenarios, it is recommended to use the following Solaris DTrace script. "

#!/usr/sbin/dtrace -s




    @ops[probemod, probefunc] = count();







Note that I have slightly modified the D Script to have *rsa* and to make it work for both Solaris 10 and 11  as per recommendations from Chi-Chang Lin.

For Solaris 11, the T4 optimization is implemented in while it is in for Solaris 10. So just add these two probes for Solaris 10 :

9. References


Meena Vyas


« April 2014