Tuesday Feb 05, 2008

Using HTTP compression to speed up content delivery in Sun Java System Web Server 7.0 update 2

Using HTTP compression to speed up content delivery in Sun Java System Web Server 7.0 update 2


If you are looking for faster web page downloads, you can use HTTP compression feature which compresses your content to speed it up. It is especially beneficial for low bandwidth connections. This also reduces the number of bytes transferred and improves the overall server performance.

How it Works

Browsers that support compressed content send an Accept-encoding header with value gzip, deflate. Our Web Server sees the header and chooses to provide compressed content, and sends Content-encoding: gzip response header. The browser on seeing this header tries to decompress the content and renders it.

Now the question is how to enable HTTP compression feature can be configured in obj.conf. Let me explain that in detail in this blog.

HTTP Compression of Static Files

    For dynamic compression of static files in  Sun Java System Web Server 7.0 Update 2, we need to use compress-file SAF with find-compressed SAF.

    The compress-file function creates the compressed file in the subdirectory specified if the file size is in between min-size and max-size the first time a request is sent on the URI. If Web Server needs to compress that static file for every static request then it may not really improve performance. It will reduce the network traffic but will cause more CPU usage. So content compression will only happen once or until the uncompressed original file is modified. If check-age is set to true, it recreates the compressed file if the compressed version is not as recent as the non-compressed version. The find-compressed function checks if a compressed version of the requested file is available. If the following conditions are met, find-compressed changes the path to point to the compressed file:
  • A compressed version is available.
  • The compressed version is as recent as the non-compressed version.
  • The client supports compression.
  • If the HTTP method is GET or HEAD.

Example

    This is an example of how server compresses html files and places them in .compressed directory (relative to the original file location). Modify the default object of obj.conf as shown below

<Object name="default">
NameTrans fn="assign-name" from="\*.html" name="find-compressed"
...
Service method=(GET|HEAD|POST) type=\*~magnus-internal/\* fn=compress-file subdir=".compressed-files"
Service method=(GET|HEAD|POST) type=\*~magnus-internal/\* fn=send-file
...
</Object>

<Object name="find-compressed">
PathCheck fn="find-compressed"
</Object>

    However, if you plan to put precompressed content in the location where the original file is present yourself, you can just use find-compressed SAF . A compressed version of a file must have the same file name as the non-compressed version but with a .gz suffix. For example, the compressed version of a file named /httpd/docs/index.html would be named /httpd/docs/index.html.gz. To compress files, you can use the freely available gzip program.
<Object name="default">
NameTrans fn="assign-name" from="\*.html" name="find-compressed"
...
</Object>
<Object name="find-compressed">
PathCheck fn="find-compressed"

</Object>

HTTP Compression of Dynamic Content

    The http-compression filter compresses outgoing  dynamic content such as the output from SHTML pages, CGI programs, or pages created with JavaServer PagesTM (JSPTM) technology that need to be interpreted by the server.

Example

Output fn="insert-filter" type="text/\*" filter="http-compression" vary="on" compression-level="9"

In this example, type="text/\*" restricts compression to documents that have a MIME type of text/\* (for example, text/ascii, text/css, text/html, and so on).

Suppose you want to compress only JSPs. Modify the default object of obj.conf as shown below
<Object name="default">
NameTrans fn="assign-name" from="\*.jsp" name="compress-on-demand"
...
</Object>

<Object name="compress-on-demand">
Output fn="insert-filter" filter="http-compression"
</Object>

Parameters

Here is a list of parameters you can use if you want to tune these functions

Parameters for compress-file

  • subdir
    • (Optional) This should be a directory name only relative to the directory in which the original non-compressed file is located.
    •  To overwrite a pre-compressed compressed file lying in docroot  set the subdir to ".".
    • The default value  is "." will overwrite any precompressed gz files if any.
  • check-age            (Optional) The values can be true or false.        The default value is true.
  • vary                   (Optional) The values can be true or false.        The default value is true.
  • compression-level (Optional) The values can be 1 to 9.                  The default value is 6.
  • min-size             (Optional) The values can be 0 to INT_MAX.           The default value is 256.
  • max-size         (Optional) The values can be min-size to INT_MAX. The default value is 1048576.

Parameters for find-compressed

  • check-age
    • (Optional)  Specifies whether to check if the compressed version is older than the non-compressed version. 
    • The values can be yes or no. By default, the value is set to yes.
    • If set to yes, the compressed version will not be selected if it is older than the non-compressed version.
    •  If set to no, the compressed version is always selected, even if it is older than the non-compressed version.
  • vary 
    • (Optional) Specifies whether to insert a Vary: Accept-Encoding header.
    • The values can be yes or no. By default, the value is set to yes.
    • If set to yes, a Vary: Accept-Encoding header is always inserted when a compressed version of a file is selected.
    •  If set to no, a Vary: Accept-Encoding header is never inserted. 

Parameters for http-compression

  • vary 
    • Controls whether the filter inserts a Vary: Accept-encoding header.
    • If vary is absent, the default value is yes. yes tells the filter to insert a Vary: Accept-encoding header when it compresses content. 
    • no tells the filter to never insert a Vary: Accept-encoding header.
  • fragment-size
    • Size in bytes of the memory fragment used by the compression library to control how much to compress at a time.
    • The default value is 8096.
  • compression-level
    • Controls the compression level used by the compression library.
    • Valid values are from 1 to 9. A value of 1 results in the best speed.
    • A value of 9 results in the best compression.
    • The default value is 6.
  • window-size 
    • Controls an internal parameter of the compression library.
    • Valid values are from 9 to 15.
    • Higher values result in better compression at the expense of memory usage.
    • The default value is 15.
  • memory-level 
    • Controls how much memory is used by the compression library.
    • Valid values are from 1 to 9.
    • A value of 1 uses the minimum amount of memory but is slow.
    • A value of 9 uses the maximum amount of memory for optimal speed.
    • The default value is 8.

Using wadm (Administration CLI) to configure these HTTP compression features

You can use the following wadm commands to configure these.
For compression of static content i.e. find-compressed and compress-file functions use
wadm> enable-precompressed-content
Usage: enable-precompressed-content --help|-?
  or   enable-precompressed-content [--echo] [--no-prompt] [--verbose] [--uri-pattern=pattern] [--no-vary-header] [--no-age-check] [--compress-file] [--sub-dir=sub-directory] [--compression-level=1-9] [--min-size=size-in-bytes] [--max-size=size-in-bytes] --config=config-name --vs=vs-name
CLI014 vs is a required option.
wadm>
For compression of dynamic content i.e. http-compression function use
wadm> enable-on-demand-compression
Usage: enable-on-demand-compression --help|-?
  or   enable-on-demand-compression [--echo] [--no-prompt] [--verbose] [--uri-pattern=pattern] [--no-vary-header] [--fragment-size=size] [--compression-level=1-9] --config=config-name --vs=vs-name
CLI014 vs is a required option.

You can also try experimenting with other related CLIs
  • get-precompressed-content-prop
  • get-on-demand-compression-prop
  • disable-precompressed-content
  • disable-on-demand-compression

How do I test if my configuration is working or not

    You can use telnet to submit HTTP requests, impersonating a web browser. The stuff in bold is the stuff I typed.  Here's an example on a Solaris machine testserver, where I have a Web Server instance running on port 2600:
$ telnet testserver 2600
Trying 1.2.3.4...
Connected to testserver.
Escape character is '\^]'.
GET / HTTP/1.0
[press enter twice]

HTTP/1.1 200 OK
Server: Sun-Java-System-Web-Server/7.0
Date: Wed, 06 Feb 2008 12:09:45 GMT
Content-type: text/html
Last-modified: Fri, 25 Jan 2008 00:12:38 GMT
Content-length: 355
Connection: close

<html>...</html>
Connection to testserver closed by foreign host.
$
I'll try that same HTTP request again, but this time indicate that I want a compressed response:
$telnet testserver 2600
Trying 1.2.3.4...
Connected to testserver.
Escape character is '\^]'.
GET / HTTP/1.0
[press enter]
Accept-encoding: gzip [press enter twice]


HTTP/1.1 200 OK
Server: Sun-Java-System-Web-Server/7.0
Date: Wed, 06 Feb 2008 12:09:45 GMT
Content-type: text/html

Last-modified: Fri, 25 Jan 2008 00:12:38 GMT
Content-encoding: gzip
Vary: accept-encoding
Connection: close

MPËNÄ0...ó²õã\\å
Connection to testserver closed by foreign host.
$

When I ask for a compressed response, the server responds with a Content-encoding: gzip header and returns compressed response instead of the html.


\*\*Note that compress-file SAF is not available in Web Server 6.1, in that case you have to put pre compressed content into document root for find-compressed to work.

References

  1. Web Server software forum questions
  2. Compressing Web Content with mod_gzip and mod_deflate
  3. Best Practices for Speeding Up Your Web Site
  4. HTTP Compression
  5. Web Server 7.0 update 2 documentation about find-compressed function
  6. Web Server 7.0 update 2 documentation about http-compression function
  7. my previous blog Dynamic Compression Of Static Files

Wednesday Sep 19, 2007

Using built-in hardware crypto accelerators of SPARC Enterprise T5120 server (powered by UltraSPPARC T2 i.e. Niagara2 processor) in Oracle iPlanet Web Server 7.0


Using built-in hardware crypto accelerators of SPARC Enterprise T5120 server (powered by UltraSPARC T2 i.e. Nigara2 processor) in Oracle iPlanet Web Server 7.0 update 9


In my previous blog I talked about SCF framework and Sun Java System Web Server 7.0 in general. This time I tried to make use of built-in hardware crypto accelerators of SPARC Enterprise T5120 server (powered by Ultra SPARC-T2 i.e. Niagara 2 processor) in Oracle iPlanet Web Server 7.0 update 9.

T5120 server has intergrated onboard cryptographic acceleration supporting 10 embedded security industry-standard ciphers including DES, 3DES, AES, RC4, SHA1, SHA256, MD5, RSA to 2048 key, ECC, and CRC32.

Here is what I did :

Step 1 : Go to Webserver installation directory and start admin server

# ./admin-server/bin/startserv

Step 2.1 : Go to Web Server 7.0 instance's config directory and perform these manual steps

# cd https-foo.com/config/


2.2) First move the existing database to another directory

# mv /.sunw ./sunw.old

Setpin

# pktool setpin
Create new passphrase: type-password-here
Re-enter new passphrase: type-password-here
Passphrase changed.


2.3) List the current PKCS#11modules

# ../../bin/modutil -list -dbdir .

Listing of PKCS #11 Modules
-----------------------------------------------------------
  1. NSS Internal PKCS #11 Module
         slots: 2 slots attached
        status: loaded

         slot: NSS Internal Cryptographic Services
        token: NSS Generic Crypto Services

         slot: NSS User Private Key and Certificate Services
        token: NSS Certificate DB

  2. Root Certs
        library name: libnssckbi.so
         slots: 1 slot attached
        status: loaded

         slot: NSS Builtin Objects
        token: Builtin Object Token
-----------------------------------------------------------

2.4) Add SCF module

# ../../bin/modutil  -dbdir . -add "Solaris Crypto Framework" -libfile /usr/lib/libpkcs11.so -mechanisms RSA

...

Module "Solaris Crypto Framework" added to database.


2.5) Enable SCF module

# ../../bin/modutil -enable "Solaris Crypto Framework" -dbdir .
...

Slot "Sun Metaslot" enabled.
Slot "n2cp/0 Crypto Accel Bulk 1.0" enabled.
Slot "ncp/0 Crypto Accel Asym 1.0" enabled.
Slot "n2rng/0 SUNW_N2_Random_Number_Generator" enabled.



2.6) List modules to make sure add and enable stuff above succeeded.

# ../../bin/modutil  -list -dbdir .

Listing of PKCS #11 Modules
-----------------------------------------------------------
  1. NSS Internal PKCS #11 Module
         slots: 2 slots attached
        status: loaded

         slot: NSS Internal Cryptographic Services
        token: NSS Generic Crypto Services

         slot: NSS User Private Key and Certificate Services
        token: NSS Certificate DB


2. Solaris Crypto Framework
        library name: /usr/lib/libpkcs11.so
         slots: 4 slots attached
        status: loaded

         slot: Sun Metaslot
        token: Sun Metaslot

         slot: n2cp/0 Crypto Accel Bulk 1.0
        token: n2cp/0 Crypto Accel Bulk 1.0

         slot: ncp/0 Crypto Accel Asym 1.0
        token: ncp/0 Crypto Accel Asym 1.0

         slot: n2rng/0 SUNW_N2_Random_Number_Generator
        token: n2rng/0 SUNW_N2_RNG

  3. Root Certs
        library name: libnssckbi.so
         slots: 1 slot attached
        status: loaded

         slot: NSS Builtin Objects
        token: Builtin Object Token
-----------------------------------------------------------

2.7) List cryptoadm providers and their mechanisms:

# cryptoadm list -p

User-level providers:
=====================
/usr/lib/security/$ISA/pkcs11_kernel.so: all mechanisms are enabled. random is enabled.
/usr/lib/security/$ISA/pkcs11_softtoken_extra.so: all mechanisms are enabled. random is enabled.

Kernel software providers:
==========================
des: all mechanisms are enabled.
aes256: all mechanisms are enabled.
arcfour2048: all mechanisms are enabled.
blowfish448: all mechanisms are enabled.
sha1: all mechanisms are enabled.
sha2: all mechanisms are enabled.
md5: all mechanisms are enabled.
rsa: all mechanisms are enabled.
swrand: random is enabled.

Kernel hardware providers:
==========================
n2cp/0: all mechanisms are enabled.
ncp/0: all mechanisms are enabled.
n2rng/0: all mechanisms are enabled. random is enabled.


2.9) Disable the following mechanisms in User Level Providers

cryptoadm disable  provider=/usr/lib/security/\\$ISA/pkcs11_softtoken_extra.so \\  
             mechanism=CKM_SSL3_PRE_MASTER_KEY_GEN,CKM_SSL3_MASTER_KEY_DERIVE,CKM_SSL3_KEY_AND_MAC_DERIVE,\\
                       CKM_SSL3_MASTER_KEY_DERIVE_DH,CKM_SSL3_MD5_MAC,CKM_SSL3_SHA1_MAC

2.10) List to make sure these were disabled

# cryptoadm list -p
User-level providers:
=====================
/usr/lib/security/$ISA/pkcs11_kernel.so: all mechanisms are enabled. random is enabled.
/usr/lib/security/$ISA/pkcs11_softtoken_extra.so: all mechanisms are enabled, except CKM_SSL3_SHA1_MAC,CKM_SSL3_MD5_MAC,CKM_SSL3_MASTER_KEY_DERIVE_DH,CKM_SSL3_KEY_AND_MAC_DERIVE,
CKM_SSL3_MASTER_KEY_DERIVE,CKM_SSL3_PRE_MASTER_KEY_GEN. random is enabled.

Kernel software providers:
==========================
des: all mechanisms are enabled.
aes256: all mechanisms are enabled.
arcfour2048: all mechanisms are enabled.
blowfish448: all mechanisms are enabled.
sha1: all mechanisms are enabled.
sha2: all mechanisms are enabled.
md5: all mechanisms are enabled.
rsa: all mechanisms are enabled.
swrand: random is enabled.

Kernel hardware providers:
==========================
n2cp/0: all mechanisms are enabled.
ncp/0: all mechanisms are enabled.
n2rng/0: all mechanisms are enabled. random is enabled.
2.11) a) Now run the following Web Server CLI commands

# ../../bin/wadm --user=admin
Please enter admin-user-password> type-admin-server-password-here
wadm> list-configs
foo
wadm> pull-config --config=foo foo
CLI201 Command 'pull-config' ran successfully
wadm> list-tokens --config=foo
internal
Sun Metaslot
wadm>  create-selfsigned-cert --config=foo --server-name=foo --nickname=Server-Cert --token="Sun Metaslot"
Please enter token-pin> type-password-here
CLI201 Command 'create-selfsigned-cert' ran successfully
wadm> list-http-listeners  --config=foo
http-listener-1
wadm> set-ssl-prop --config=foo --http-listener=http-listener-1 server-cert-nickname="Sun Metaslot:Server-Cert" enabled=true
CLI201 Command 'set-ssl-prop' ran successfully
wadm>  deploy-config foo
CLI201 Command 'deploy-config' ran successfully
wadm>

\*\*

2.11) b) If you are using older version of Web Server and you do not have Admin CLI, you can use the following command to create the self signed certificate
# ../../bin/certutil -S -x -n "Server-Cert" -t "u,u,u" -s "CN=foo" -d . -h "Sun Metaslot"
Enter Password or Pin for "Sun Metaslot": type-password-here
...
Continue typing until the progress meter is full:

|\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|

Finished. Press enter to continue:

     Generating key. This may take a few moments...
     #

2.11) c) OR you can create a self-signed certificate in NSS DB and export it and import it into Sun Metaslot :
# ../../bin/certutil -S -x -n "Server-Cert" -t "u,u,u" -s "CN=foo" -d .
Enter Password or Pin for "Sun Metaslot": type-password-here
...
Continue typing until the progress meter is full:

|\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*|


Finished. Press enter to continue:


Generating key. This may take a few moments...
#

# ../../bin/pk12util  -o key-cert-data.pk12 -n "Server-Cert" -d .
Enter password for PKCS12 file: type-a-new-password-here-or-press-enter
Re-enter password: type-a-new-password-here-or-press-enter
pk12util: PKCS12 EXPORT SUCCESSFUL

# ../../bin/pk12util -i key-cert-data.pk12 -d . -h "Sun Metaslot"
Enter Password or Pin for "Sun Metaslot": type-password-here
Enter password for PKCS12 file: type-the-new-password-here
pk12util: PKCS12 IMPORT SUCCESSFUL

2.12) Now manually double check if the certificate exists
# ../../bin/certutil -L -d . -h "Sun Metaslot"
Certificate Nickname                Trust Attributes
                                   SSL,S/MIME,JAR/XPI
Enter Password or Pin for "Sun Metaslot": type-password-here
Sun Metaslot:Server-Cert                      u,u,u

Check that server.xml contains server-cert-nickname element
 <http-listener>
    <name>http-listener-1</name>
    <port>80</port>
    <server-name>foo</server-name>
    <default-virtual-server-name>foo</default-virtual-server-name>
    <ssl>
      <server-cert-nickname>Sun Metaslot:Server-Cert</server-cert-nickname>
    </ssl>
  </http-listener>

2.13) Start the server
#../bin/startserv
Oracle iPlanet Web Server 7.0.9 B07/04/2010 02:15
Please enter the PIN for the "Sun Metaslot" token: type-password-here
info: CORE5076: Using [Java HotSpot(TM) Server VM, Version 1.6.0_20] from [Sun Microsystems Inc.]
info: HTTP3072: http-listener-1: https://foo:80 ready to accept requests
info: CORE3274: successful server startup

2.14) Check the statistics using kstat

# kstat -n ncp0 | grep rsa

        rsagenerate                     6
        rsaprivate                      18
        rsapublic                       15

Send a request through the browser (with a cipher containing RSA) to this server https://foo:80/test.html, it should show test.html
I used tstclnt (a client bundled with NSS) to send request to the server using cipher c i.e. SSL3 RSA WITH RC4 128 MD5.

# tstclnt -h accel -p 81 -d . -n Server-Cert  -T -f -o -v  -c c < req.txt
tstclnt: connecting to accel:81 (address=10.133.169.154)
...

      tstclnt: SSL version 3.0 using 128-bit RC4 with 128-bit MD5 MAC
      tstclnt: Server Auth: 1024-bit RSA, Key Exchange: 1024-bit RSA
      ...
      HTTP/1.1 200 OK
      Server: Oracle-iPlanet-Web-Server/7.0
      Content-type: text/html
      Last-modified: Mon, 10 Jan 2011 09:45:03 GMT
      Etag: "12-4d2ad51f"
      Accept-ranges: bytes
      Content-length: 18
      Date: Mon, 10 Jan 2011 09:45:23 GMT
      Connection: close

      This is test.html
      ...

Now run kstat again,  if it shows an increase that means we are ok.

kstat -n ncp0 | grep -i rsa 
    rsagenerate                     6 
    rsaprivate                      18

    rsapublic                       16


Note : If

1. Web server does not present the Intermediate CA certificates installed as Server Certificate Chain to the browser and that causes the certificate validation by the browser to fail.
or
2. Client authentication fails with the following error message in the errors log .  Root CA cert has been installed to the certificate database.

failure (16670): HTTP3068: Error receiving request from 123.45.67.897(SEC_ERROR_UNKNOWN_ISSUER: Peer's certificate is signed by an unknown issuer)

These two issues are caused by the /.sunw directory not being accessible by the web server running user "webservd". That directory has permissions 0700 and is owned by root. Web Server starts up as root and then changes (using setuid) to user "webservd".  Solution to this is
1) Have the web server running as root
2) Open up the permission on /.sunw so that it is readable by the web server running user
3) Set  the environment variable SOFTTOKEN_DIR to point to some directory that is owned by webservd before the web server is started. The SCF will then access the files in $SOFTTOKEN_DIR/pkcs11_softoken/ during execution.

References


\*\*If you get an error in list-configs command like : "CLI104 Unable to communicate with the administration server: No such file or directory" then you are probably seeing bug "6606384 SCF consumers crash after mechanisms are disabled using cryptoadm when using libumem". Upgrade to Solaris 10 update 9 or apply patches. core dump should look like

# mdb ../../admin-server/config/core.18103
>::stack
libc.so.1`_lwp_kill+8(6, 0, 20f04, ff34ba3c, ff36a000, ff36abdc)
libumem.so.1`umem_do_abort+0x1c(44, e4ced4e8, 6, 20e40, ff356ad8, 0)
libumem.so.1`umem_err_recoverable+0x7c(ff357b54, a, 20d38, 0, ff36d0e8, ff357b5f)
libumem.so.1`process_free+0x114(12d3ee8, 1, 0, 3e3a1000, 1ec08, ff)
libpkcs11.so.1`pkcs11_slottable_delete+0x158(12d3ee8, a11628, a11628, f97c6bb0, 1, 1)
libpkcs11.so.1`pkcs11_fini+0x4c(f97c6b8c, 1, f97ae1c8, f97c6000, 17aac, f97c6b84)
libc.so.1`_postfork_child_handler+0x30(1d18, fd543800, 1c00, 4, fba5ca00, fd543800)
libc.so.1`fork+0x144(0, 2, 0, 3c, fd543800, fba5ca00)
libns-httpd40.so`CHILDEXEC_ERR ChildExec::_startListener()+0x19c(...)
libns-httpd40.so`CHILDEXEC_ERR ChildExec::PerformListenerOp(ListenerOp,int&)+0x14(...)
libns-httpd40.so`CHILDEXEC_ERR ChildExec::StartListener(int&)+0x48(...)
libns-httpd40.so`CHILDEXEC_ERR ChildExec::initialize(int&)+0xa8(...)
libns-httpd40.so`PRStatus cgistub_child_exec_init()+0x2d0(...)
libns-httpd40.so`PRStatus cgistub_init()+0x58(...)
...

Monday Jul 23, 2007

Configuring reverse proxy in Sun Java System Web Server 7.0 when origin server has SSL enabled

Configuring reverse proxy in Oracle iPlanet Web Server 7.0 when origin server has SSL enabled.

Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTP -->  client/Browser

There are various SSL and non SSL configurations we can have for Reverse Proxy and Origin Servers

  1. Origin Server <--- HTTP ---> Reverse Proxy Server <--- HTTP ---> client/Browser
  2. Origin Server <-- HTTP ---> Reverse Proxy Server <-- HTTPS --> client/Browser i.e. Reverse proxy SSL termination End point
  3. Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTP --> client/Browser
  4. Origin Server <-- HTTPS ---> Reverse Proxy Server <-- HTTPS --> client/Browser
In my last blog I explained configuration of a simple non SSL reverse proxy(i.e. scenario 1). In this blog I have tried to set up a SCENARIO 3 shown above, where a non SSL Oracle iPlanet Web Server 7.0 tries to connect to origin server which is SSL enabled

Creating SSL enabled origin server

If you already have an SSL enabled origin server you can skip this.

For ease of use I have used Oracle iPlanet Web Server 7.0 as origin server also.
Start the administration server, and go to wadm
>./wadm --user=admin
Please enter admin-user-password> \*\*\*\*

Create a self signed certificate

wadm>create-selfsigned-cert --config=test --server-name=www.test.com --nickname=OriginServerServer-Cert

Optional : Create a HTTP listener (or use existing one)

wadm> create-http-listener --listener-port=8888 --config=test --server-name=www.test.com --default-virtual-server-name=www.test.com mylistener

Enable SSL for this listener and set the server certificate nickname

wadm> set-ssl-prop --config=test --http-listener=mylistenerserver-cert-nickname=OriginServerServerCert enabled=true

Deploy the changes

wadm> deploy-config test

Start this origin server instance.

Settings in Web Server 7.0 instance

Lets say we want to forward all requests to /xyz to the origin server.

Go the Web Server instance config directory and modify the obj.conf as given below

<Object name="default">
AuthTrans fn="match-browser"browser="\*MSIE\*" ssl-unclean-shutdown="true"
NameTrans fn="ntrans-j2ee" name="j2ee"
NameTrans fn="pfx2dir" from="/mc-icons" dir="/export2/mv/lib/icons" name="es-internal"
NameTrans fn="map" from="/xyz" name="reverse-proxy-/" to="/xyz"
PathCheck fn="uri-clean"
...
</Object>
<Object ppath="\*">
Service fn="proxy-retrieve" method="\*"
</Object>
<Object name="reverse-proxy-/">
Route fn="set-origin-server" server="https://test.sun.com:8888"
</Object>


\*\*Note that I have given manual steps here. In my last blog I have given Administration CLI steps.

Lets say for this instance server.xml has <port>8080</port>.
Make sure that the origin server is up and running.
Start the server and access http://test.sun.com:8080/xyz/ should show you xyz directory in the docroot of the origin server.

Troubleshooting

In case we get a Gateway Timeout error and in error logs we see some error like

[23/Jul/2007:16:44:11] failure (27927): for host .... trying to GET ...., service-http reports: HTTP7758: error sending request (SEC_ERROR_UNTRUSTED_ISSUER: Client certificate is signed by an untrusted issuer.)


We get this error because the origin server's certificate was not issued by a trusted CA. It means we need to export CA certificate of the origin server instance and import it into Web Server instance. In this case, Origin Server certificate (nickname OriginServerServerCert) is the CA for itself so we import that certificate.

This step is not required if the Origin Server's certificate chain ends at a root CA Certificate which is a trusted CA and is present in built-in root CA certificate DB.

Export the origin server's CA certificate

Go to <server-instance>/config  directory of the origin server, and list certificates and then use pk12util to export the certificate.

$../../bin/pk12util -o /tmp/exported.crt -n OriginServerServerCert -d .

Import the origin server CA certificate in server instance config directory

Initialize NSS Database

To import certificate in server instance config directory you have to first initialize the NSS database.

Note that if this Web Server instance is SSL enabled you can skip this NSS database intialization part.

$../../bin/certutil -N -d .

Import the certificate

Lets say the file /tmp/exported.crt contained the CA cert of the origin server, import that to NSS database.

$ ../../bin/pk12util -i /tmp/exported.crt -d .

Confirm by listing certs using certutil

$../../bin/certutil -L -d .
OriginServerServerCert          u,u,

Modify trust flags if required (if its a self signed cert)

You can see that the certificates imported doesn't contain 'CT' trust flags.

$../../bin/certutil -M -n OriginServerServer-Cert -t 'CTu,CTu,CTu' -d .

Now u can confirm

$../../bin/certutil -L -d .
OriginServerServerCert       CTu,CTu,CTu

Restart the server instance and things should work fine now.


Thursday Mar 15, 2007

Troubleshooting memory leaks and memory corruptions in Sun Java System Web Server 7.0

Troubleshooting memory leaks and memory corruptions in Sun Java System Web Server 7.0

In this blog I am trying to show how to use libumem and watchmalloc to find more information about memory leaks and memory corruptions in Sun Java System Web Server 7.0.

Using libumem to find memory leaks and memory corruptions

1) Disable pools
Add this line in magnus.conf
Init fn="pool-init" disable="true"
Note this is not a supported public interface.
2) Set environment variable UMEM_DEBUG
$export UMEM_DEBUG="default"
Refer man pages for umem_debug (3MALLOC) for more details.
3)  Start the Web Server instance
$./bin/startserv
4) Note down the pid of the webservd process
$ps -eaf |grep webservd
You will see two webservd processes, note down the highest pid.
5) Dump the initial core
$gcore -o core.pid.start pid
6) Run tests , send some requests which cause memory corruption or leaks.
7) Dump core again
gcore -o core.pid.end pid
8) Compare memory allocated between these two should show the leaks.
$mdb core.pid.start  
>::findleaks -d ! cat > findleaks.start.log
( Or you can also run $echo ::findleaks -d |mdb core.pid.start > findleaks.start.log )

$mdb core.pid.end
>::findleaks -d ! cat > findleaks.end.log

compare these two logs manually to see if there are leaks.
If these logs are unreadable you can use Sun studio's c++filt or in mdb give this command as shown below
> $G
C++ symbol demangling enabled
>

9) For memory corruptions, try umem_status, bufctl_audir and umem_verify commands in the last core:
$mdb core.pid.end
>::umem_status
...
If it shows a corrupted buffer try looking at its contents
(For example >26a3640/40X)
or
>>bufferaddress::bufctl_audit
(For example >26a4618::bufctl_audit)

umem_status command, also shows if a buffer is accessed after its already freed. For example, once it showed me
> ::umem_status
Status: ready and active
Concurrency: 2
Logs: (inactive)
Message buffer:
umem allocator: buffer modified after being freed
modification occurred at offset 0x8 (0xdeadbeefdeadbeef replaced by 0xde20be20de20be20)
...

You can also use umem_verify command to see if one of the umem caches has a corrupted buffer.

Using watchmalloc for memory corruptions

1) Disable pools
Add the following line in magnus.conf
Init fn="pool-init" disable="true"
2) Add LD_PRELOAD watchmalloc with MALLOC_DEBUG=WATCH in start script , comment out libumem and libCld portions

$diff -u startserv startserv.watchmalloc
--- startserv     Wed Dec  7 14:03:40 2005
+++ startserv.watchmalloc     Thu Mar 15 13:19:06 2007
@@ -9,6 +9,9 @@
 SERVER_BIN_DIR="/space/wsDec7/iplanet/ias/server/work/B1/SunOS5.8_DBG.OBJ/bin"
 SERVER_LIB_DIR="/space/wsDec7/iplanet/ias/server/work/B1/SunOS5.8_DBG.OBJ/lib"
 SERVER_BIN=webservd-wdog
+LIBWMC=/usr/lib/watchmalloc.so.1
+LD_PRELOAD_32="${LIBWMC} ${LD_PRELOAD}"; export LD_PRELOAD_32
+MALLOC_DEBUG="WATCH"; export MALLOC_DEBUG

 # Add path to server binaries to PATH
 PATH="${SERVER_BIN_DIR}:${SERVER_LIB_DIR}:/bin:${PATH}"; export PATH
@@ -47,26 +50,6 @@
 # Add instance-specific information to SHLIB_PATH for HP-UX
 SHLIB_PATH="${SERVER_LIB_PATH}:${SERVER_JVM_LIBPATH}:${SHLIB_PATH}"; export SHLIB_PATH

-# Preload libumem to improve performance on Solaris 10
-LIBUMEM_32=/usr/lib/libumem.so
-if [ -f "${LIBUMEM_32}" ] ; then
-    if [ `uname -r | sed s/\\\\\\.//` -ge 510 ] ; then
-        LD_PRELOAD_32="${LIBUMEM_32} ${LD_PRELOAD_32}"; export LD_PRELOAD_32
-    fi
-fi
-LIBUMEM_64=/usr/lib/64/libumem.so
-if [ -f "${LIBUMEM_64}" ] ; then
-    if [ `uname -r | sed s/\\\\\\.//` -ge 510 ] ; then
-        LD_PRELOAD_64="${LIBUMEM_64} ${LD_PRELOAD_64}"; export LD_PRELOAD_64
-    fi
-fi
-
-# Preload libCld to resolve -compat=4/-compat=5 C++ ABI issues on Solaris
-LIBCLD="${SERVER_LIB_DIR}/libCld.so"
-if [ -f "${LIBCLD}" ] ; then
-    LD_PRELOAD_32="${LIBCLD} ${LD_PRELOAD_32}"; export LD_PRELOAD_32
-fi
-
 if [ $# -eq 0 ] ; then
     COMMAND=--start;
 elif [ "$1" = "-i" ] ; then
3) Comment out compat4/compat5 NSAPI Init plugins in magnus.conf if they exist (most probably not)
4) Start the Web Server instance
$./bin/startserv
This makes the server extremely slow.
5) Run tests, send requests to the server,
The server may crash at proper places rather than random ones.


In general if you want to see what's happening when you send a request you can try using truss : truss -o truss.log -u "\*" -d -fael -rall -vall -wall -p <pid>

More information about this is in http://docs.sun.com/app/docs/doc/816-5165/truss-1?l=ja&a=view

Note that -u option will show the user-level function call tracing.

Links:
http://sunsolve.sun.com/search/document.do?assetkey=1-9-70641
http://developers.sun.com/solaris/articles/libumem_library.html

Wednesday Mar 14, 2007

Troubleshooting Web Server crashes : enabling core dumps

Troubleshooting Web Server crashes : enabling core dumps

I get a lot of questions regarding what to do when Web Server instance crashes as shown by error logs and you do not find the core file.

The core file will be dumped in the Web Server instance's config directory. For example, if the Web Server is installed in /opt/SUNWwbsvr directory, core file will be /opt/SUNWwbsvr/https-hostname/config/core.

If you are using SSL, core dumps will be disabled by default. You can force them on by setting the SSL_DUMP environment variable before starting Web Server:
$ SSL_DUMP=1
$ export SSL_DUMP
$ ./start

Check coreadm(1M) and ulimit(1) to see if Solaris will allow processes to dump core. Use the pstack(1M) command to obtain a stack trace. You can use the same coreadm (core file administration) command to set appropriate values.
$coreadm
  global core file pattern:
  global core file content: default
  init core file pattern: core.%p
  initcore file content: default
  global core dumps: disabled
  per-process core dumps: enabled
  global setid core dumps: disabled
  per-process setid core dumps: disabled
  global core dump logging: disabled


Note that I have set core file pattern to core.%p instead of the usual core . If a process with pid lets say 1000 dumps core, it will generate a core file with name core.1000 to avoid overriding in case the server dumps multiple core files. But this is not necessary.

If your Operating System is Linux, make sure that you set ulimit to unlimited before starting the server.
$ulimit -c unlimited or

$ulimit -S -c unlimited or
or edit /etc/security/limits.conf and followed by editing /etc/profile

To get core dumps on Windows, first make drwatson as a default debugger:
C:\\WINDOWS\\system32>drwtsn32.exe -i

To change various settings of this drwatson,
C:\\WINDOWS\\system32>drwtsn32.exe
This opens a window where you can specify where to dump the core.

To set back the default debugger to MSVC, change registry key HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\AeDebug to "C:\\Program Files\\Microsoft Visual Studio\\Common\\MSDev98\\Bin\\msdev.exe" -p %ld -e %ld
Set Auto to 0.
Set UserDebuggerHotKey to 0.

Some more suggested reading

  1. http://publib.boulder.ibm.com/httpserv/ihsdiag/coredumps.html
  2. http://developers.sun.com/solaris/articles/DebugLibraries/DebugLibraries_content.html  - The article shows how to use elfdump -p and dbx to check corefiles.



About

Meena Vyas

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today