HOWTO Turn off SPARC T4 or Intel AES-NI crypto acceleration.

Since we released hardware crypto acceleration for SPARC T4 and Intel AES-NI support we have had a common question come up: 'How do I test without the hardware crypto acceleration?'.

Initially this came up just for development use so developers can do unit testing on a machine that has hardware offload but still cover the code paths for a machine that doesn't (our integration and release testing would run on all supported types of hardware anyway).  I've also seen it asked in a customer context too so that we can show that there is a performance gain from the hardware crypto acceleration, (not just the fact that SPARC T4 much faster performing processor than T3) and measure what it is for their application.

With SPARC T2/T3 we could easily disable the hardware crypto offload by running 'cryptoadm disable provider=n2cp/0'.  We can't do that with SPARC T4 or with Intel AES-NI because in both of those classes of processor the encryption doesn't require a device driver instead it is unprivileged user land callable instructions.

Turns out there is away to do this by using features of the Solaris runtime loader (ld.so.1). First I need to expose a little bit of implementation detail about how the Solaris Cryptographic Framework is implemented in Solaris 11.  One of the new Solaris 11 features of the linker/loader is the ability to have a single ELF object that has multiple different implementations of the same functions that are selected at runtime based on the capabilities of the machine.  The alternate to this is having the application coded to call getisax() and make the choice itself.  We use this functionality of the linker/loader when we build the userland libraries for the Solaris Cryptographic Framework (specifically libmd.so, and the unfortunately misnamed due to historical reasons libsoftcrypto.so)

The Solaris linker/loader allows control of a lot of its functionality via environment variables, we can use that to control the version of the cryptographic functions we run.  To do this we simply export the LD_HWCAP environment variable with values that tell ld.so.1 to not select the HWCAP section matching certain features even if isainfo says they are present. 

For SPARC T4 that would be:

export LD_HWCAP="-aes -des -md5 -sha256 -sha512 -mont -mpmul" 

and for Intel systems with AES-NI support:

export LD_HWCAP="-aes"

This will work for consumers of the Solaris Cryptographic Framework that use the Solaris PKCS#11 libraries or use libmd.so interfaces directly.  It also works for the Oracle DB and Java JCE.  However does not work for the default enabled OpenSSL "t4" or "aes-ni" engines (unfortunately) because they do explicit calls to getisax() themselves rather than using multiple ELF cap sections.

However we can still use OpenSSL to demonstrate this by explicitly selecting "pkcs11" engine  using only a single process and thread. 

$ openssl speed -engine pkcs11 -evp aes-128-cbc
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      54170.81k   187416.00k   489725.70k   805445.63k  1018880.00k

$ LD_HWCAP="-aes" openssl speed -engine pkcs11 -evp aes-128-cbc
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      29376.37k    58328.13k    79031.55k    86738.26k    89191.77k

We can clearly see the difference this makes in the case where AES offload to the SPARC T4 was disabled. The "t4" engine is faster than the pkcs11 one because there is less overhead (again on a SPARC T4-1 using only a single process/thread - using -multi you will get even bigger numbers).

$ openssl speed -evp aes-128-cbc
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      85526.61k    89298.84k    91970.30k    92662.78k    92842.67k

Yet another cool feature of the Solaris linker/loader, thanks Rod and Ali.

Note these above openssl speed output is not intended to show the actual performance of any particular benchmark just that there is a significant improvement from using hardware acceleration on SPARC T4. For cryptographic performance benchmarks see the http://blogs.oracle.com/BestPerf/ postings.

Comments:

Hello,
thanks for sharing that. However, it seems Solaris 11 does not utilize AES-NI for ZFS encryption at all (Intel Xeon E3-12xx and numerous people at hardforum.com, etc.)
Is it broken?

Posted by KHM on December 15, 2011 at 05:53 AM GMT #

Hi,

Thanks for the article, was quite helpful.

I was wondering whether the same test might work on Solaris 10 or only valid on Solaris 11. Minutes ago, I did the same testing on Solaris 10 and got around the same values:

# /usr/sfw/bin/openssl speed -engine pkcs11 -evp aes-128-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 20588.77k 78096.38k 256764.93k 592054.95k 997711.87k

# export LD_HWCAP="-aes -des -md5 -sha256 -sha512 -mont -mpul"
# /usr/sfw/bin/openssl speed -engine pkcs11 -evp aes-128-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 20524.21k 78137.98k 258737.92k 600083.11k 999112.70k

# /usr/sfw/bin/openssl engine
(pkcs11) PKCS #11 engine support

# /usr/sfw/bin/openssl version
OpenSSL 0.9.7d 17 Mar 2004

This arises because I have custom-compiled apache version presumably to use PKCS11 (Compiled with CFLAGS='-DSSL_ENGINE' ./configure -enable-ssl --with-ssl=/usr/sfw) and enabled within the apache configuration file (SSLCryptoDevice pkcs11), which I would have to get a real sense of how the SSL Offload is improving the application performance.

I've been looking for some information in the matter, but every blog post from oracle uses either Solaris 11 or T1/T2/T3 hardware.

Thanks for your time reading this in advance.
JM

Posted by guest on June 12, 2012 at 03:19 PM BST #

The example
export LD_HWCAP="-aes -des -md5 -sha256 -sha512 -mont -mpul"
should be
export LD_HWCAP="-aes -des -md5 -sha256 -sha512 -mont -mpmul"
(according to isainfo)

Posted by Ian on July 09, 2012 at 07:39 PM BST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

DarrenMoffat

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today