Friday Mar 02, 2012

Using tshark to debug SSL connections

Jyri had explained in his blog how to use ssldump to debug SSL connections. We can also use tshark. On my Linux server, tshark is installed in /usr/sbin/tshark.

Support team guys need these steps for finding out what is happening. First try to reproduce the problem in a test environment with self-signed certificate and follow the steps given in this blog.

I started Oracle iPlanet Web Server 7.0 instance on IP lets say 11.111.111.111 and port 15000.

Exporting Private Key from NSS DB

In NSS Database, I have a Server Certificate named "Server-Cert" as shown below.

$ cd <WS_install-root>/https-<instance>/config

$ ../../bin/certutil -L -d .

Certificate Nickname                         Trust Attributes
                                             SSL,S/MIME,JAR/XPI
Server-Cert                                     u,u,u

First use pk12util to extract server certificate and its key into a file "server.keycert".

$ ../../bin/pk12util -o server.keycert -n "Server-Cert" -d .

Enter Password or Pin for "NSS Certificate DB": nssdbpassword


Enter password for PKCS12 file: pkcs12password
Re-enter password: pkcs12password


pk12util: PKCS12 EXPORT SUCCESSFUL


then I use openssl to get just the RSA private key

$ openssl pkcs12 -nodes -in server.keycert -out key.pem -nocerts -nodes
Enter Import Password: pkcs12password
MAC verified OK 

$ rm server.keycert

If you look at the file, its contents are like :

$ cat key.pem
Bag Attributes
    friendlyName: Server-Cert
    localKeyID: ...
Key Attributes: <No Attributes>
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

Edit the file key.pem manually and remove the first 4 lines.

Now the file starts with line "-----BEGIN RSA PRIVATE KEY-----" and end with "-----END RSA PRIVATE KEY-----"

Note that we should be very careful with this key as its not so safe to leave it unprotected.

You can protect it by another password if you like.

I would prefer if wireshark can take NSS DB or Oracle Wallets as input directly.

Running tshark

Now as root run tshark

$ /usr/sbin/tshark -o "ssl.desegment_ssl_records: TRUE" \

-o "ssl.desegment_ssl_application_data: TRUE" \

-o "ssl.keys_list:11.111.111.111,15000,http,key.pem" \

-o "ssl.debug_file:ssldebug.log" \

-f "tcp port 15000" \

-R "ssl" \

-V -x 2>&1 | tee tshark.log

when I had not given IP address in ssl.key_list, it wasn't associating key to some of my connections.

Note that 
I used capture filter "tcp port 15000" and display filter "ssl". 
I used -V to show more verbose output and 
I also used -x to get both hex and ASCII dumps. 

You can try your own options.

Now send a request through a browser to https://11.111.111.1111:15000/index.html, close the browser and after a while, press control c on the window where tshark is running and kill it.

Delete the private key file key.pem.

ssldebug.log should have a message that says key was loaded successfully

$ grep -i "private key" ssldebug.log

Private key imported: KeyID ...
ssl_init private key file key.pem successfully loaded

Note that ssldebug.log MUST NOT contain any error messages about key not being used etc.

Now look at tshark.log, look for "Secure Socket Layer" sections one such section is shown below :

Secure Socket Layer
  SSL Record Layer: Handshake Protocol: Client Hello
      Content Type: Handshake (22)
      Version: TLS 1.0 (0x0301)
      Length: 168
      Handshake Protocol: Client Hello
          Handshake Type: Client Hello (1)
          Length: 164
          Version: TLS 1.0 (0x0301)
          Random
              gmt_unix_time: Mar  2, 2012 00:01:26.000000000
              random_bytes: .......

          Session ID Length: 0
          Cipher Suites Length: 72
          Cipher Suites (36 suites)
            Cipher Suite: Unknown (0x00ff)
            Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
            Cipher Suite: TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0088)
...


In the end you can see SSL data being decrypted :

Decrypted SSL data (1 bytes):

0000  48                                                H

 Decrypted SSL data (225 bytes):

0000  54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d 0a   TTP/1.1 200 OK..
...
0040  46 72 69 2c 20 30 32 20 4d 61 72 20 32 30 31 32   Fri, 02 Mar 2012
0050  20 30 39 3a 31 32 3a 32 38 20 47 4d 54 0d 0a 4c    09:12:28 GMT..L
0060  61 73 74 2d 6d 6f 64 69 66 69 65 64 3a 20 57 65   ast-modified: We
0070  64 2c 20 32 39 20 46 65 62 20 32 30 31 32 20 31   d, 29 Feb 2012 1
0080  31 3a 33 38 3a 31 39 20 47 4d 54 0d 0a 43 6f 6e   1:38:19 GMT..Con
0090  74 65 6e 74 2d 6c 65 6e 67 74 68 3a 20 31 39 0d   tent-length: 19.
00a0  0a 45 74 61 67 3a 20 22 31 33 2d 34 66 34 65 30   .Etag: "13-4f4e0
00b0  65 32 62 22 0d 0a 41 63 63 65 70 74 2d 72 61 6e   e2b"..Accept-ran
00c0  67 65 73 3a 20 62 79 74 65 73 0d 0a 0d 0a 54 68   ges: bytes....Th
00d0  69 73 20 69 73 20 69 6e 64 65 78 2e 68 74 6d 6c   is is index.html
00e0  0a                                                .

This log shows different stages of SSL

$grep "Handshake Protocol" tshark.log

    Handshake Protocol: Client Hello
    Handshake Protocol: Server Hello
    Handshake Protocol: Certificate
    Handshake Protocol: Server Hello Done
    Handshake Protocol: Client Key Exchange
    Handshake Protocol: Finished
    Handshake Protocol: Finished

Exporting Private Key from Wallet

If your product uses Oracle wallet instead of NSS DB, to extract the key and certificate from the Wallet you can use openssl command as shown below

$openssl pkcs12 -in ewallet.p12 -passin pass:walletpassword -out ewallet.txt -nodes

MAC verified OK

If you look at this file it has  "-----BEGIN RSA PRIVATE KEY-----" and "-----END RSA PRIVATE KEY-----".

$cat ewallet.txt


Bag Attributes
    localKeyID: ...
Key Attributes: <No Attributes>
-----BEGIN RSA PRIVATE KEY-----

...
-----END RSA PRIVATE KEY-----


Bag Attributes
    localKeyID: ...
subject=/C=US/CN=*.oracle.com
issuer=/C=US/CN=root
-----BEGIN CERTIFICATE-----

...

-----END CERTIFICATE-----

...

Edit this file and copy only the lines starting with " -----BEGIN RSA PRIVATE KEY-----" and ending with " -----END RSA PRIVATE KEY-----" into a new file key.pem. Rest of the steps remain the same.

I wanted to check if we are getting "close notify" in a connection, I saw in the presentation

http://www.powershow.com/view/29ec1-OWNkM/SSL_Troubleshooting_with_Wireshark_and_Tshark_flash_ppt_presentation

useful commands to get a particular field in tshark:

$tshark -G fields | fgrep "ssl." and hence used
$tshark -R "ssl.alert_message"

References

  1. http://wiki.wireshark.org/SSL
  2. https://forums.oracle.com/forums/thread.jspa?threadID=830575
  3. http://www.powershow.com/view/29ec1-OWNkM/SSL_Troubleshooting_with_Wireshark_and_Tshark_flash_ppt_presentation

Wednesday Sep 21, 2011

More about PKCS11 Bypass in Oracle iPlanet Web Server 7.0

More about PKCS11 Bypass in Oracle iPlanet Web Server 7.0

Jyri's blog explains the concepts about PKCS11 Bypass in Oracle iPlanet Web server 7.0.

    By default in Oracle iPlanet Web Server 7.0, PKCS11Bypass is enabled. To know if PKCS11 Bypass is actually enabled or disabled in your server instance, run the server instance in <log-level>fine</log-level> and check the error log for lines containing the words "PKCS11 bypass". 

When PKCS11 Bypass is enabled When PKCS11 Bypass is disabled
server.xml

<pkcs11>
   <allow-bypass>true</allow-bypass>
</pkcs11>

<pkcs11>
   <allow-bypass>false</allow-bypass>
</pkcs11>

Error log

fine: PKCS#11 bypass is enabled

fine: PKCS#11 bypass is disabled

    Even though PKCS11 Bypass is enabled in server.xml, it is possible that that check of "SSL_CanBypass" fails in that case PKCS11 Bypass is not enabled. So its essential to check error log contents.

    Lets use DTrace scripts to see what's going on at function call level when PKCS11 Bypass is enabled or disabled. Lets analyse the scenario where AES cipher suite is negotiated in SSL Handshake. We know that  "AES_Encrypt" will be called in that case. So we write a script to print stack when "AES_Encrypt" function is called.

#!/usr/sbin/dtrace -s
#pragma D option quiet

pid$1::AES_Encrypt*:entry
{
    printf("thread %d:  stack is : \n", tid);
    ustack();
}

Note that if in SSL Handshake, others cipher suites were negotiated,  for example if RC4 is negotiated, "RC4_Encrypt" function will be called instead of "AES_Encrypt" and so on... Ideally we need a script with the full list of freebl algorithms but for our simple testing this will serve the purpose.

We run this D script and pass the the highest webservd pid as the first argument. Now send a (HTTPS) request via browser to the Web Server instance, here is the stack we get in both the cases (when PKCS11 Bypass is enabled and disabled).


User stack when PKCS11 Bypass is enabled

User stack when PKCS11 Bypass is disabled*

#./ssl.d 22437
thread 17: stack is :

libsoftokn3.so`AES_Encrypt




libssl3.so`ssl3_CompressMACEncryptRecord+0x5a8
libssl3.so`ssl3_SendRecord+0x38c
libssl3.so`ssl3_FlushHandshake+0x1cc
libssl3.so`ssl3_SendFinished+0x448
libssl3.so`ssl3_HandleFinished+0x5a4
libssl3.so`ssl3_HandleHandshakeMessage+0x8d0
libssl3.so`ssl3_HandleHandshake+0x2d8
libssl3.so`ssl3_HandleRecord+0xb60
libssl3.so`ssl3_GatherCompleteHandshake+0x110
libssl3.so`ssl_GatherRecord1stHandshake+0xd0
libssl3.so`ssl_Do1stHandshake+0x308
libssl3.so`ssl_SecureRecv+0x230
libssl3.so`ssl_Recv+0x124
libnspr4.so`PR_Recv+0x48
libns-httpd40.so`int DaemonSession::GetConnection()+0x470
libns-httpd40.so`void DaemonSession::run()+0xdc
libnsprwrap.so`void Thread::run_()+0x28

#./ssl.d 22469
thread 17: stack is :

libsoftokn3.so`AES_Encrypt

libsoftokn3.so`NSC_EncryptUpdate+0x490
libnss3.so`PK11_CipherOp+0x28c


libssl3.so`ssl3_CompressMACEncryptRecord+0x5a8
libssl3.so`ssl3_SendRecord+0x38c
libssl3.so`ssl3_FlushHandshake+0x1cc
libssl3.so`ssl3_SendFinished+0x448
libssl3.so`ssl3_HandleFinished+0x5a4
libssl3.so`ssl3_HandleHandshakeMessage+0x8d0
libssl3.so`ssl3_HandleHandshake+0x2d8
libssl3.so`ssl3_HandleRecord+0xb60
libssl3.so`ssl3_GatherCompleteHandshake+0x110
libssl3.so`ssl_GatherRecord1stHandshake+0xd0
libssl3.so`ssl_Do1stHandshake+0x308
libssl3.so`ssl_SecureRecv+0x230
libssl3.so`ssl_Recv+0x124
libnspr4.so`PR_Recv+0x48
libns-httpd40.so`int DaemonSession::GetConnection()+0x470
libns-httpd40.so`void DaemonSession::run()+0xdc
libnsprwrap.so`void Thread::run_()+0x28


*Note that we see calls to NSS softoken starting with NSC_.  If we use libpkcs11.so, the names of the symbols to look might be different, i.e. C_Encrypt, C_Decrypt.

From the above results we find that 

  • When PKCS11 Bypass is enabled,  function "ssl3_CompressMACEncryptRecord" (and others) in libssl3.so directly call "AES_Encrypt" in the same library (i.e. libssl3.so).
  • When PKCS11 Bypass is disabled,  it calls "PK11_CipherOP" function in libnss3.so which then calls "NSC_EncryptUpdate" function in libsofttoken3.so which in turn calls "AES_Encrypt" function in libsofttoken3.so. 

So by enabling PKCS11 Bypass we are eliminating two layers of function calls and that's why its faster. 

References

http://blogs.oracle.com/jyrivirkki/entry/pkcs_11_and_ssl_performance

Which ciphers are enabled in Oracle iPlanet Web Server 7.0 instance? and how do I find information about ciphers that are actually used at run time ?

Which ciphers are enabled in Oracle iPlanet Web Server 7.0 instance? and how do I find information about ciphers that are actually used at run time ?

A lot of people ask me how do I know which ciphers are enabled in Oracle iPlanet Web Server 7.0.

 The list of ciphers and whether they are enabled or disabled is given in the table http://download.oracle.com/docs/cd/E19146-01/821-0794/gcfbv/index.html . This may slightly vary from update release to another.

 The best way to know this is to change <log-level> in server.xml from "info" to "finest" and start the server instance. You will see these log messages at server startup which will tell  you which cipher was enabled or disabled.

>...
fine: Initializing "NSS Generic Crypto Services" PKCS #11 token
fine: Initializing "internal" PKCS #11 token
....
fine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: RC4, mac: MD5, key bits: 128): SSL_RSA_WITH_RC4_128_MD5
fine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: RC4, mac: SHA1, key bits: 128): SSL_RSA_WITH_RC4_128_SHA
fine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: 3DES, mac: SHA1, key bits: 112): SSL_RSA_WITH_3DES_EDE_CBC_SHA
...
fine: disabling cipher (cert: RSA, auth: RSA, kea: ECDHE, enc: 3DES, mac: SHA1, key bits: 112): TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
fine: disabling cipher (cert: RSA, auth: RSA, kea: ECDHE, enc: AES, mac: SHA1, key bits: 256): TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
fine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: AES, mac: SHA1, key bits: 128): TLS_RSA_WITH_AES_128_CBC_SHA
fine: enabling cipher (cert: RSA, auth: RSA, kea: RSA, enc: AES, mac: SHA1, key bits:  256): TLS_RSA_WITH_AES_256_CBC_SHA
fine: SSLv3/TLS is enabled and 18 SSLv3/TLS ciphers are enabled
fine: 0 export ciphers enabled
fine: PKCS#11 bypass is enabled
fine: 1 RSA certificate(s) present, 6 suitable cipher(s) enabled
fine: 0 ECC certificate(s) present, 12 suitable cipher(s) enabled

Or if you are familir with Admin CLI you can use the following CLI

wadm>list-ciphers --config=<config> --http-listener=<listener> --verbose --all

 For more information refer :

http://docs.oracle.com/cd/E19146-01/821-0792/list-ciphers-1/index.html


Here is my blog about how to use Dtrace to collect information about ciphers used in the connection :

http://blogs.oracle.com/meena/entry/dtarce_script_to_collect_information

You can also modify server.xml to print cipher in access log :

<access-log>
    <file>../logs/access</file>
   <format>%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%
%Ses->client.cipher%</format>
</access-log>

Access log will have an new cipher entry in the last column of each row. For example it may show "AES-256","RC4" etc.

Friday Aug 19, 2011

HTTPS Oracle iPlanet Web Server 7.0 Reverse Proxy Server and HTTP Origin Server

 HTTPS Oracle iPlanet Web Server 7.0 Reverse Proxy Server and HTTP Origin Server

Origin Server <--- HTTP ---> Reverse Proxy Server <--- HTTPS ---> client/Browser

There are various SSL and non SSL configurations we can have for Reverse Proxy and Origin Servers

  1. Origin Server <--- HTTP ---> Reverse Proxy Server <--- HTTP ---> client/Browser
  2. Origin Server <-- HTTP ---> Reverse Proxy Server <-- HTTPS --> client/Browser i.e. Reverse proxy as SSL termination End point
  3. Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTP --> client/Browser
  4. Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTPS --> client/Browser

In this blog I will try out SCENARIO 2 - reverse proxy server as SSL termination end point.

For this I have setup two Oracle iPlanet Web Server 7.0 update 11 instances. One acting as a reverse proxy (instance name rps on port 8080) and the other origin server(instance name origs on port 4444).


1. Enable SSL on Reverse Proxy Server

1.1 Install Server Certificate in Reverse Proxy Server instance

I created self signed server certificate in reverse proxy server. Use Admin Server CLI to create a self-signed certificate (recommened)

$./bin/wadm --user=admin Please enter admin-user-password> Connected to localhost:8989 Oracle iPlanet Web Server 7.0.11 B03/11/2011 08:38 wadm>

wadm> list-configs rps wadm>create-selfsigned-cert --config=rps

--server-name=www.rps.com --nickname=Server-Cert-RP

wadm>deploy-config rps

Or you can use certutil followed by pull-config .

$cd <reverse-proxy-install-dir>/https-rps/config
$../../bin/certutil -N -d . (if DBs do not exist already)
$../../bin/certutil -S -d . -n Server-Cert-RP 
          -s "CN=www.rps.com" -x -t "CT,CT,CT"

Verify it with certutil that the certificate got installed :

$../../bin/certutil -L -d <reverse-proxy-install-dir>/https-rps/config
Certificate Nickname                       Trust Attributes
                                           SSL,S/MIME,JAR/XPI
Server-Cert-RP                             u,u,u

1.2 Enable SSL in http-listener. Set the server certificate nickname (if its different from "Server-Cert"). 

Use "set-ssl-prop" Admin Server CLI to enable SSL for this listener, set the server certificate nickname and then run deploy config CLI.

wadm> list-http-listeners --config=rps http-listener-1 wadm> set-ssl-prop --config=rps --http-listener=http-listener-1

server-cert-nickname=Server-Cert-RP enabled=true

wadm>deploy-config rps

server.xml should get modified to look like

<http-listener>
    <name>http-listener-1</name>
    <port>8080</port>
    <server-name>www.rps.com</server-name>
    <default-virtual-server-name>rps</default-virtual-server-name>
   <ssl>
   <server-cert-nickname>Server-Cert-RP</server-cert-nickname>
   </ssl>
</http-listener>

1.3 Run create-reverse-proxy CLI from Administration server

wadm> list-virtual-servers --config=rps
rps

wadm>create-reverse-proxy --config=rps --vs=rps --uri-prefix=/ 
    --server=http://www.origs.com:4444

wadm>deploy-config rps

rps.obj.conf gets modified as shown below :

<Object name="default">
AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true"
NameTrans fn="map" from="/" name="reverse-proxy-/" to="http:/"
...
</Object>
<Object ppath="http:*">
Service fn="proxy-retrieve" method="*"
</Object>
<Object name="reverse-proxy-/">
Route fn="set-origin-server" server="http://www.origs.com:4444"
</Object>

In rps.obj.conf I have configured reverse proxy in such a way that all requests are redirected to origin server.

In real world situation you can if you want redirect only certain requests depending on your requirements.

1.4 OPTIONAL : Change access log format in Reverse Proxy Server

Modify access log format in Reverse Proxy Server to contain %Ses->client.cipher% and %Ses->client.ssl-id%
wadm> get-access-log-prop --config=rps
enabled=true
file=../logs/access
format=%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%]
       "%Req->reqpb.clf-request%"
       %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%
mode=text

wadm> enable-access-log --file="../logs/access" --config="rps"
        --format="%Ses->client.ip% -
        %Req->vars.auth-user% \[%SYSDATE%\]
       \\"%Req->reqpb.clf-request%\\" %Req->srvhdrs.clf-status%
        %Req->srvhdrs.content-length% 
        %Ses->client.cipher% %Ses->client.ssl-id%

wadm>deploy-config rps

Or manually add in server.xml the log format
<access-log>
    <file>../logs/access</file>
    <format>
    %Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] 
    "%Req->reqpb.clf-request%"
    %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% 
    %Ses->client.cipher% %Ses->client.ssl-id%
   </format>
 </access-log>

Note the contents in between <format></format> should be in one line

I have put it in 2 lines so that its easy to see.

1.5  (OPTIONAL) Create a file test.html file in origin server :

$cat ../docs/test.html
This is test.html

1.6 Send a request through browser to Reverse Proxy Server on URI /test.html.

Start the origin server and reverse proxy server instances. Send a request via the browser to reverse proxy server on https://www.rps.com:8080/test.html. Origin should send the content to reverse proxy serverw hcih should send it to the browser.

We can check the entries in both the access logs of reverse proxy server and origin server to see what's happening.

Note that browser may give a warning that this reverse proxy server is not issued by a valid CA. That's ok because its a self signed certificate. If you install a certificate from trusted CAs this message will not come up.

When you check access log entries of reverse porxy server :
$cat access
format=%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% %Ses->client.cipher% %Ses->client.ssl-id%
xxx.xxx.xxx.xx1 - - [19/Aug/2011:14:02:51 +0530] "GET /test.html HTTP/1.1" 200 18 AES-256 Ux0aq03pRHCNZaDxLX1mrBKzmM7ac4YUAspbTX5s8pw=

Its printing the ciphers used and the SSL session id.

When you check access log entries of origin server :

$cat .access
format=%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%
xxx.xxx.xxx.xx2  - - [19/Aug/2011:13:48:20 +0530] "GET /test.html HTTP/1.1" 200 18


References

Thursday May 26, 2011

What's new in NSS 3.12.* : Chrome's and SSL False Start

What's new in NSS 3.12.* - Chrome and SSL False Start

As you are aware that Chrome uses NSS for SSL/TLS. A friend of mine forwarded this blog link to me which was quite interesting to read.

"SSL False Start is a client-side only change to reduce one round-trip from the SSL handshake. ...

  We implemented SSL False Start in Chrome 9, and the results are stunning, yielding a significant decrease in overall SSL connection setup times. SSL False Start reduces the latency of a SSL handshake by 30% ..."

Recent related news on this  http://www.engadget.com/2012/04/12/google-puts-false-start-ssl-experiment-down-nobody-notices/

References :

About

Meena Vyas

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today