PKCS#11 engine patch update for OpenSSL 0.9.8i

I've updated the PKCS#11 patch to the latest OpenSSL 0.9.8i version. The patch includes RFEs and fixes we integrated into OpenSolaris since the last patch release.

Note that from now on we use some POSIX thread functions in the engine code. There is no problem on Solaris but the build may fail on some Linux distros, and possibly other systems. The problem is that OpenSSL always tries to build a threaded library unless "no-threads" is used when configuring but sometimes doesn't succeed. See the README file, section FAQs, about how to solve the problem. I don't have any Linux box around with the PKCS#11 support so I just built the library on Gentoo (I had to manually tweak the Configure script as described in the FAQs) but run no tests. I also built on FreeSBD 5.3. Hopefully nothing has changed and it will work. The patch to download is pkcs11_engine-0.9.8i.patch.2008-12-02.gz.


FreeBSD 5.3? Huh you should really think about updating it to 7.0 or 7.1 (out soon)

anyway, good work, keep it up! :)

Posted by Anon Ymous on December 03, 2008 at 12:31 AM CET #

Jan, i tried this patch out and it seems to hang when accessing an SSL CGI script "if SSLCryptoDevice is not specified".

Of course, with this patch, one would usually use:

SSLCryptoDevice pkcs11

Without that line, when accessing the same script in non-SSL mode, it works. When accessing a static content via SSL, it works. When accessing a CGI via SSL, it hangs. Have you experienced anything like that?

OpenSSL was compiled using gcc with the following Configure option:

./Configure solaris-sparcv9-gcc --pk11-libname=/usr/lib/ shared threads -Wl,-R/usr/local/lib,-R/usr/local/ssl/lib && dmake -j 8

Apache is using MPM worker. The process affected contain the following:

----------------- lwp# 33 / thread# 33 --------------------
fef45898 lwp_park (0, 0, 0)
feec1e00 _prefork_handler (10d8, fef73580, 1000, 1a75b8, 1ba268, 0) + 38
fef37e90 fork (1, 0, fea92240, fee58000, fe887800, fef73580) + bc
ff0ddbcc apr_proc_create (1ba520, 1a6cf0, 1a75a8, 1a75b8, 1ba268, 1a5968) + 74
fea92240 run_cgi_child (fc477b94, fc477b90, fc477b8c, 1a6cf0, 1a75a8, 1a59a8) + 1b0
fea9299c cgi_handler (1a59a8, 5d800, 1a59a8, 0, c3fc0, 5) + 230
0003b724 ap_run_handler (1a59a8, 1e, 4, 1a59a8, 19a1b0, 19bed0) + 3c
0003bba0 ap_invoke_handler (1a59a8, 5d800, 1a59a8, 78800, 78860, 176f10) + b8
00048348 ap_process_request (1a59a8, 1e, 4, 1a59a8, 19a1b0, 19bed0) + 160
000455ac ap_process_http_connection (19a1b8, 0, 0, 78800, 78860, 176f10) + 10c
00041dd8 ap_run_process_connection (19a1b8, 199f08, 199f08, 1e, 19a1b0, 19bed0) + 3c
0004ce84 worker_thread (14a560, 0, 0, 78800, 78800, 78) + 20c
ff0dead0 dummy_worker (14a560, fc47c000, 0, 0, ff0deac4, 1) + c
fef457f8 _lwp_start (0, 0, 0, 0, 0, 0)

Let me know if you would like further information. Thanks.

Posted by Lawrence Ong on December 07, 2008 at 11:42 PM CET #

we haven't seen that yet. Please use, that's a better place to discuss it. Thx.

Posted by Jan on December 08, 2008 at 07:31 AM CET #

I tried applying this patch and then compiling as follows:

export PATH

LD_OPTIONS="-R/opt/app/openssl/lib -L/opt/app/openssl/lib"

./Configure \\
--prefix=/opt/app/openssl \\
--openssldir=/opt/app/openssl \\
--pk11-libname=/usr/lib/ \\
shared \\


The compile is using gcc from /usr/sfw

However when testing, it doesn't work:

openssl s_server -engine pkcs11 -cert /path/to/my/server.crt -www

openssl s_client -ssl3 -connect

15920:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1053:SSL alert number 40
15920:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:530:

If I omit "-engine pkcs11" from the "openssl s_server" command, everything works fine.

Posted by deckrider on January 02, 2009 at 10:34 AM CET #

One small addition ... interestingly, when the client and server BOTH use '-engine pkcs11' then things work, but if one uses '-engine pkcs11' and the other does not, then there is the failure I reported above.

Posted by deckrider on January 02, 2009 at 10:41 AM CET #

you seem to be using Solaris, any reason you don't want to use the shipped OpenSSL version which already offer the PKCS#11 engine support?

from your PATH and from what you have pasted I'd say that you run the shipped OpenSSL version since /usr/sfw/bin is the first directory in your PATH. You don't mention the Solaris version so if you use S10 be sure that your system is fully patched - whether you use the shipped version or the built one. If you use OpenSolaris you probably don't need this patch at all unless you need something specific that is not in OpenSSL 0.9.8a.

Posted by Jan on January 02, 2009 at 01:05 PM CET #

/usr/sfw/bin/openssl claims to be:

"OpenSSL 0.9.7d 17 Mar 2004 (+ security patches to 2006-09-29)"

So I wanted to ensure we had a more recently updated version, thus the desire to compile our own.

It seems that both our version and the above bundled version have the same problem.

So maybe we need some patch like this:

We have this:

uname -a
SunOS foo 5.10 Generic_127111-09 sun4v sparc SUNW,T5240

cat /etc/release
Solaris 10 8/07 s10s_u4wos_12b SPARC
Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 16 August 2007

Posted by deckrider on January 02, 2009 at 01:38 PM CET #

Deckrider, that's not the CR. See README from the patch. Basically, you should have all the fixes mentioned there. The first group of the fixes were already released as a patch, the 2nd one is in progress. If you still see the problem with the shipped OpenSSL after you apply all relevant existing patches, file an escalation if you have a support contract; you might get those fixes earlier. Otherwise, just wait until the other patches are released.

CRs from the README should give you enough info about what you are looking for. If you have problem with the patch on OpenSolaris, please send the problem description to

thanks, J.

Posted by Jan on January 05, 2009 at 09:40 AM CET #

I would like to add the PKCS11 .0.9.8 patch to the OpenSSL-fips-1.0.2 source I am getting the errors as Hunk failed.
is there any PKCS11 patch for OpenSSL-fips-1.1.2?

Posted by satish gummadelli on August 13, 2009 at 12:04 AM CEST #

to Satish: no and it won't be. Please do not forget that if you modify the OpenSSL FIPS source code you break the FIPS certification of it. There is more about it in the docs on OpenSSL FIPS certification and use.

Posted by Jan on August 13, 2009 at 04:32 AM CEST #

Hi Jan,

This is the only thread I can find about this. We've just ordered a couple of T5120s, and I'm trying to figure out how to compile openssl to use the SCF. Currently the docs say we have to use the /usr/sfw openssl. Is it even possible to compile your own SCF-enabled openssl? If so, is it worth it? The included openssl is only 0.9.7d I think.



Posted by Ryan on January 07, 2010 at 12:50 PM CET #

Ryan, you do not need to do anything, OpenSSL shipped with Solaris 10 (and OpenSolaris) has the engine integrated. Yes, it's 0.9.7d but with security vulnerabilities fixed (you must watch out for security patches, of course).

Also, you may consider to install OpenSolaris, Sun even offers support for it (see

For more info on OpenSSL in S10, please see this blog entry:

Posted by Jan on January 07, 2010 at 01:54 PM CET #

Post a Comment:
  • HTML Syntax: NOT allowed

Jan Pechanec


« April 2014