Wednesday Apr 29, 2009

UnsatisfiedLinkError: no jss4 in java.library.path

As support engineer, considerable amount of my time goes in trying to replicate customer problems in the labs within Sun. When setting up things in the lab, sometimes I run into other issues that I have to overcome before I can possibly reproduce the actual problem reported by the customer.

Here is one such error I ran into when  I installed  Access Manager 7.1 from JES 5 on Appserver 8.2.

Installation of the compoents went through fine but when the web container was restarted,  it was throwing the error:  java.lang.UnsatisfiedLinkError: no jss4 in java.library.path

To overcome this, I had to add the following 3 entries to AMConfig.properties and then restart the container.

com.iplanet.security.SecureRandomFactoryImpl=com.iplanet.am.util.SecureRandomFactoryImpl
com.iplanet.security.SSLSocketFactoryImpl=netscape.ldap.factory.JSSESocketFactory
com.iplanet.security.encryptor=com.iplanet.services.util.JCEEncryption

Snippet of the exception in the application server log

[#|2009-04-13T16:50:59.482+1000|SEVERE|sun-appserver-pe8.2|javax.enterprise.system.container.web|_ThreadID=10;|WebModule[/amserver]Servlet /amserver threw load() exception javax.servlet.ServletException: no jss4 in java.library.path
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:300)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:165)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:118)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1093)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:931)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4183)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4535)
at com.sun.enterprise.web.WebModule.start(WebModule.java:241)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1086)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:847)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1086)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:483)
at org.apache.catalina.startup.Embedded.start(Embedded.java:894)
at com.sun.enterprise.web.WebContainer.start(WebContainer.java:741)at com.sun.enterprise.web.PEWebContainer.startInstance(PEWebContainer.java:515)
at com.sun.enterprise.web.PEWebContainerLifecycle.onStartup(PEWebContainerLifecycle.java:54)
at com.sun.enterprise.server.ApplicationServer.onStartup(ApplicationServer.java:300)
at com.sun.enterprise.server.PEMain.run(PEMain.java:292)
at com.sun.enterprise.server.PEMain.main(PEMain.java:218)
----- Root Cause -----
java.lang.UnsatisfiedLinkError: no jss4 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1682)
at java.lang.Runtime.loadLibrary0(Runtime.java:822)
at java.lang.System.loadLibrary(System.java:993)
at org.mozilla.jss.CryptoManager.loadNativeLibraries(CryptoManager.java:1339)
at org.mozilla.jss.CryptoManager.initialize(CryptoManager.java:827)
at com.iplanet.services.util.JSSEncryption.<clinit>(JSSEncryption.java:250)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at com.iplanet.services.util.Crypt.createInstance(Crypt.java:133)
at com.iplanet.services.util.Crypt.<clinit>(Crypt.java:103)
at com.iplanet.services.ldap.LDAPUser.getPasswd(LDAPUser.java:117)
at com.iplanet.services.ldap.ServerInstance.getPasswd(ServerInstance.java:128)
at com.sun.identity.security.ServerInstanceAction.run(ServerInstanceAction.java:92)
at java.security.AccessController.doPrivileged(Native Method)
at com.iplanet.am.util.AdminUtils.<clinit>(AdminUtils.java:82)
at com.sun.identity.sm.SMSEntry.<clinit>(SMSEntry.java:169)
at com.sun.identity.sm.ServiceManager.<clinit>(ServiceManager.java:74)
at com.sun.identity.authentication.service.AuthD.<init>(AuthD.java:247)
at com.sun.identity.authentication.service.AuthD.getAuth(AuthD.java:507)
at com.sun.identity.authentication.UI.LoginLogoutMapping.initializeAuth(LoginLogoutMapping.java:89)
at com.sun.identity.authentication.UI.LoginLogoutMapping.init(LoginLogoutMapping.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:249)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:282)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:165)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:118)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1093)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:931)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4183)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4535)
at com.sun.enterprise.web.WebModule.start(WebModule.java:241)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1086)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:847)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:847)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1086)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:483)
at org.apache.catalina.startup.Embedded.start(Embedded.java:894)
at com.sun.enterprise.web.WebContainer.start(WebContainer.java:741)
at com.sun.enterprise.web.PEWebContainer.startInstance(PEWebContainer.java:515)
at com.sun.enterprise.web.PEWebContainerLifecycle.onStartup(PEWebContainerLifecycle.java:54)
at com.sun.enterprise.server.ApplicationServer.onStartup(ApplicationServer.java:300)
at com.sun.enterprise.server.PEMain.run(PEMain.java:292)
at com.sun.enterprise.server.PEMain.main(PEMain.java:218)

 

Monday Feb 25, 2008

Access Manager Session State Diagram

Here is the session state diagram  I found in the access manager code, session.java.

  \* The following is the state diagram for a session:
\*
\* |
\* |
\* |
\* V
\* ---------------invalid
\* | |
\* | |creation (authentication OK)
\* | |
\* |max login time | max idle time
\* |destroy V ------------------->
\* | valid inactive --
\* | | <------------------ |
\* | | reactivate |
\* | | |
\* | | logout | destroy
\* | | destroy | max session time
\* | | max session time |
\* | V |
\* --------------> destroy <---------------------





 

LARES - Liberty-like AuthNResponse

LARES stands for  Liberty-like AuthNResponse.   This is the base64 encoded value of SAML assertion sent by the access manager to the agent when CDSSO is enabled.

The Access manager sends this as a hidden value in it's response message as shown below:

    <HTML>
    <BODY Onload="document.Response.submit()">
    <FORM NAME="Response" METHOD="POST" ACTION="http://hostname.singapore.sun.com:80/test.asp?sunwMethod=GET">
    <INPUT TYPE="HIDDEN" NAME="LARES" VALUE="PGxpYjpBdXRoblJlc3BvbnNlIHhtbG5zOmxpYj0iaHR0cDovL3Byb2plY3RsaWJlcnR5Lm9yZy9zY2hlbWFzL2NvcmUvMjAwMi8xMiIgeG1sbnM6c2FtbD0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6MS4wOmFzc2VydGlvbiIgeG1sbnM6c2FtbHA9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjEuMDpwcm90b2NvbCIgeG1sbnM6ZHM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvMDkveG1sZHNpZyMiIHhtbG5zOnhzaT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTY2hlbWEtaW5zdGFuY2UiIFJlc3BvbnNlSUQ9InM1MjU5NjZlNmUzODhlNDQ4Y2I4OGFiYjVkNzNjZDBmNmM5ZDg4YmFkIiAgSW5SZXNwb25zZVRvPSIyNzE0NSIgIE1ham9yVmVyc2lvbj0iMSIgIE1pbm9yVmVyc2lvbj0iMCIgIElzc3VlSW5zdGFudD0iMjAwOC0wMi0yNVQwMTozODoyNloiPjxzYW1scDpTdGF0dXM%2BCjxzYW1scDpTdGF0dXNDb2RlIFZhbHVlPSJzYW1scDpTdWNjZXNzIj4KPC9zYW1scDpTdGF0dXNDb2RlPgo8L3NhbWxwOlN0YXR1cz4KPHNhbWw6QXNzZXJ0aW9uICB4bWxuczpzYW1sPSJ1cm46b2FzaXM6bmFtZXM6dGM6U0FNTDoxLjA6YXNzZXJ0aW9uIiB4bWxuczp4c2k9Imh0dHA6Ly93d3cudzMub3JnLzIwMDEvWE1MU2NoZW1hLWluc3RhbmNlIiAgeG1sbnM6bGliPSJodHRwOi8vcHJvamVjdGxpYmVydHkub3JnL3NjaGVtYXMvY29yZS8yMDAyLzEyIiAgaWQ9InMyOTBkODYyN2ZiYmQ4OGU4OGIyMGZjMjE2NmNjODhjODI3MDE2OWZkMDEiIE1ham9yVmVyc2lvbj0iMSIgTWlub3JWZXJzaW9uPSIwIiBBc3NlcnRpb25JRD0iczI5MGQ4NjI3ZmJiZDg4ZTg4YjIwZmMyMTY2Y2M4OGM4MjcwMTY5ZmQwMSIgSXNzdWVyPSJodHRwOi8vdjEwMC1jLmF1cy5zdW4uY29tOjgwODAvYW1zZXJ2ZXIvY2Rjc2VydmxldCIgSXNzdWVJbnN0YW50PSIyMDA4LTAyLTI1VDAxOjM4OjI2WiIgSW5SZXNwb25zZVRvPSIyNzE0NSIgeHNpOnR5cGU9ImxpYjpBc3NlcnRpb25UeXBlIj4KPHNhbWw6Q29uZGl0aW9ucyAgTm90QmVmb3JlPSIyMDA4LTAyLTI1VDAxOjM4OjI2WiIgTm90T25PckFmdGVyPSIyMDA4LTAyLTI1VDAxOjM5OjI2WiIgPgo8c2FtbDpBdWRpZW5jZVJlc3RyaWN0aW9uQ29uZGl0aW9uPgo8c2FtbDpBdWRpZW5jZT5odHRwOi8vdjR1LTIyMHJhLXNpbjA2LnNpbmdhcG9yZS5zdW4uY29tOjgwL2FtYWdlbnQ8L3NhbWw6QXVkaWVuY2U%2BCjwvc2FtbDpBdWRpZW5jZVJlc3RyaWN0aW9uQ29uZGl0aW9uPgo8L3NhbWw6Q29uZGl0aW9ucz4KPHNhbWw6QXV0aGVudGljYXRpb25TdGF0ZW1lbnQgIEF1dGhlbnRpY2F0aW9uTWV0aG9kPSJMREFQIiBBdXRoZW50aWNhdGlvbkluc3RhbnQ9IjIwMDgtMDItMjVUMDE6Mzg6MjRaIiBSZWF1dGhlbnRpY2F0ZU9uT3JBZnRlcj0iMjAwOC0wMi0yNVQwMTozOToyNloiIHhzaTp0eXBlPSJsaWI6QXV0aGVudGljYXRpb25TdGF0ZW1lbnRUeXBlIj48c2FtbDpTdWJqZWN0ICAgeHNpOnR5cGU9ImxpYjpTdWJqZWN0VHlwZSI%2BPHNhbWw6TmFtZUlkZW50aWZpZXIgTmFtZVF1YWxpZmllcj0iaHR0cDovL3YxMDAtYy5hdXMuc3VuLmNvbTo4MDgwL2Ftc2VydmVyL2NkY3NlcnZsZXQiPkFRSUM1d00yTFk0U2Zjd2tDTmFQJTJCemVMbWpwTVBmUUZ0czkxNzlBcGZrOHM5VlklM0QlNDBBQUpUU1FBQ01ERSUzRCUyMzwvc2FtbDpOYW1lSWRlbnRpZmllcj4KPHNhbWw6U3ViamVjdENvbmZpcm1hdGlvbj4KPHNhbWw6Q29uZmlybWF0aW9uTWV0aG9kPnVybjpvYXNpczpuYW1lczp0YzpTQU1MOjEuMDpjbTpiZWFyZXI8L3NhbWw6Q29uZmlybWF0aW9uTWV0aG9kPgo8L3NhbWw6U3ViamVjdENvbmZpcm1hdGlvbj4KPGxpYjpJRFBQcm92aWRlZE5hbWVJZGVudGlmaWVyICBOYW1lUXVhbGlmaWVyPSJodHRwOi8vdjEwMC1jLmF1cy5zdW4uY29tOjgwODAvYW1zZXJ2ZXIvY2Rjc2VydmxldCIgPkFRSUM1d00yTFk0U2Zjd2tDTmFQJTJCemVMbWpwTVBmUUZ0czkxNzlBcGZrOHM5VlklM0QlNDBBQUpUU1FBQ01ERSUzRCUyMzwvbGliOklEUFByb3ZpZGVkTmFtZUlkZW50aWZpZXI%2BCjwvc2FtbDpTdWJqZWN0PjxzYW1sOlN1YmplY3RMb2NhbGl0eSAgSVBBZGRyZXNzPSIxMC4xNS4zLjE0NiIgRE5TQWRkcmVzcz0idjEwMC1jLmF1cy5zdW4uY29tIiAvPjxsaWI6QXV0aG5Db250ZXh0PjxsaWI6QXV0aG5Db250ZXh0Q2xhc3NSZWY%2BaHR0cDovL3d3dy5wcm9qZWN0bGliZXJ0eS5vcmcvc2NoZW1hcy9hdXRoY3R4L2NsYXNzZXMvUGFzc3dvcmQ8L2xpYjpBdXRobkNvbnRleHRDbGFzc1JlZj48bGliOkF1dGhuQ29udGV4dFN0YXRlbWVudFJlZj5odHRwOi8vd3d3LnByb2plY3RsaWJlcnR5Lm9yZy9zY2hlbWFzL2F1dGhjdHgvY2xhc3Nlcy9QYXNzd29yZDwvbGliOkF1dGhuQ29udGV4dFN0YXRlbWVudFJlZj48L2xpYjpBdXRobkNvbnRleHQ%2BPC9zYW1sOkF1dGhlbnRpY2F0aW9uU3RhdGVtZW50Pjwvc2FtbDpBc3NlcnRpb24%2BCjxsaWI6UHJvdmlkZXJJRD5odHRwOi8vdjEwMC1jLmF1cy5zdW4uY29tOjgwODAvYW1zZXJ2ZXIvY2Rjc2VydmxldDwvbGliOlByb3ZpZGVySUQ%2BPC9saWI6QXV0aG5SZXNwb25zZT4K
    </FORM>
    </BODY>
</HTML>

 

 

 

If you stick the value in the LARES param to a base64 decoder, it should decode to the actual SAML assertion sent by the Access Manager.
 

<lib:AuthnResponse xmlns:lib="http://projectliberty.org/schemas/core/2002/12" xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion" xmlns:samlp="urn:oasis:names:tc:SAML:1.0:protocol" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ResponseID="s525966e6e388e448cb88abb5d73cd0f6c9d88bad"  InResponseTo="27145"  MajorVersion="1"  MinorVersion="0"  IssueInstant="2008-02-25T01:38:26Z"><samlp:Status?
</saml:Conditions>
<saml:AuthenticationStatement AuthenticationMethod="LDAP" AuthenticationInstant="2008-02-25T01:38:24Z" ReauthenticateOnOrAfter="2008-02-25T01:39:26Z" xsi:type="lib:AuthenticationStatementType"><saml:Subject xsi:type="lib:SubjectType"?
</saml:Subject><saml:SubjectLocality IPAddress="10.15.3.146" DNSAddress="v100-c.aus.sun.com" /><lib:AuthnContext><lib:AuthnContextClassRef?�</saml:AuthenticationStatement></saml:Assertion?
�</lib:AuthnResponse>


 

 Here is the snapshot of the actual packet as seen in ethereal. Choose View image and then zoom in to view the image correctly.


 


 



Monday Nov 19, 2007

libumem to detect modify-after-free corruptions.

Here is another example of how libumem can help us detect memory corruption..

The problem reported by the customer was that the Sun Web Server 6.1 protected by Sun's Access Manager Policy Agent  running in production was crashing.  This problem was not showing up in their staging environment, it was only the servers in production that were crashing.

Customer sent in several core files for us to analyze.  The stack trace in the first  core file that I looked at showed that we were aborting in libmtmalloc.so.1. 

=>[1] libc.so.1:__lwp_kill(0x0, 0x6, 0x0, 0xfe0bc000, 0x5, 0xaf295c8), at 0xfe0a0218
[2] libc.so.1:raise(0x6, 0x0, 0xe1efba50, 0xffffffff, 0x74ab0, 0xaf02900), at 0xfe050c80
[3] libc.so.1:abort(0x0, 0x0, 0x88, 0xffffffff, 0x74ab0, 0xaf02870), at 0xfe06e98
[4] libmtmalloc.so.1:free(0xaf28060, 0xfffb, 0xaf02800, 0xaf02828, 0x74ab0, 0xaf02828),

 I vaguely remembered that there was a bug in mtmalloc that returned an already freed pointer. I searched our bug database and found the bug . As I read through the synopsis of the bug  it became clear that we were NOT running into this bug. The bug was about : libmtmalloc's realloc() returning an already freed pointer. I don't think, we were doing a realloc here.  I ruled out that bug and just to be sure, I checked the mtmalloc patch level on the system and found that they were running the latest mtmalloc patch.

I looked at the other core files sent by the customer, and this  time, the crash was somewhere else. As I analyzed the core files and pstack output I noticed that the crashes were random in nature. The randomness of the crash indicated  memory corruption.  I initially suspected that this could be double-free type of  an error.

 fe0a0218 _lwp_kill (6, 0, e1ffba50, ffffffff, 74620, 2969700) + 18
fe036e98 addsev   (0, 0, 88, ffffffff, 74620, 2969670)
ff390bcc free     (3e0d188, fffb, 2969600, 2969628, 74620, 2969628) + 1e0
fd726f08 void\*__Crun::vector_del(void\*,unsigned,void(\*)(void\*)) (3e0d188, 0, 0, ffffffff, 0,
fd70e934 std::vector<bool>::iterator&std::vector<bool>::iterator::operator++()

I requested the customer to use libumem , the customer obliged (BIG Thank you) and sent us three corefiles generated with libumem enabled.

I opened the corefile in mdb, ran ::umem_verify and to my surprise, the integrity of all the caches came up as clean. Where do I go from here ?

Okay, I ran ::umem_status command and that printed the exact nature of corruption, including the stacktrace of the thread that last freed this buffer and the offset at which someone wrote to this buffer after it was being freed.

This information was sufficient for Sun Access Manager engineering team to come up with a fix and release  hotpatch 2.2-01 against Policy agent.  

How good is that ! I told you, libumem is so powerful yet so simple and easy to use.

mdb>::umem_status
Status: ready and active
Concurrency: 8
Logs: transaction=256k content=256k fail=256k (inactive)
Message buffer:
umem allocator: buffer modified after being freed modification occurred at offset 0x20 (0xdeadbeefdeadbeef replaced by 0xdeadbeeedeadbeef)
buffer=a7a0708 bufctl=a813f68 cache: umem_alloc_96
previous transaction on buffer a7a0708:
thread=28 time=T-284.970780700 slab=a5dede0 cache: umem_alloc_96
libumem.so.1'umem_cache_free+0x4c
libumem.so.1'process_free+0x006c
libames6.so'PR_DestroyLock
libames6.so'~Node
libames6.so'~RefCntPtr
libames6.so'~Tree
libames6.so'~PolicyEntry
libames6.so'~RefCntPtr
libames6.so'cleanup+0x0030
libames6.so'operator+0x01fc
libames6.so'spin+0x023c
libnspr4.so'_pt_root+0x00d4
libthread.so.1'_lwp_start
umem: heap corruption detected


About

Hema

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today