11i Post cloning issues: APPSLOCALLOGIN.JSP gives JAVA.LANG.NOCLASSDEFFOUNDERROR - how is it related to JTF Distributed caching.. (Experiments from the Field..Based on True Stories)

Preface

This situation was faced after cloning: in spite of repeated apache bouncing, we could not make the self service login page to work, in a multi-tier(load balanced) application architecture for 11.5.10 instance. A generic error like java.lang.NoClassDefFoundError was coming up.

Many a times, it can be resolved by making sure that your CLASSPATH or AF_CLASSPATH has all the right .zip/.jar files/$JAVA_TOP etc in $IAS_ORACLE_HOME/Apache/Jserv/etc or adovars.env, BUT at times even those checks did not work.

Debugging and trying all sorts of things first..

Well, we did not come to the understanding of the actual problem till much later. Initially, we tried many things like recompiling JSP cache in $COMMON_TOP/_pages.

sandbox:web_dev> perl ojspCompile.pl --compile --fast
identifying apache_top.../ORACLE/dev/9iAS
identifying apache_config_top.../ORACLE/dev/9iAS
identifying java_home.../usr/java/j2sdk1.4.2_07
identifying jsp_dir.../ORACLE/apps/dev/common/html
identifying pages_dir.../ORACLE/apps/dev/common
identifying classpath...file:///ORACLE/dev/9iAS/Apache/Jserv/etc/jserv.properties
auto-logging to /ORACLE/apps/dev/common/_pages/ojsp_error.log
starting...(compiling delta)
using 8i internal ojsp ver: 1.1.3.5.2
including compatibility flag -whiteSpaceBetweenScriptlet
synchronizing dependency file:
enumerating jsps...14016
parsing jsp...14016
writing deplist...14016
initializing compilation:
eliminating children...10724 (-3292)
searching uncompiled...10692
translating and compiling:
searching untranslated...10692
translating jsps...10692/10692 [failed: 204] in 3m58s
compiling jsps...10488/10488 [failed: 75] in 5m44s
Finished!

Now, when we tried accessing the AppsLocalLogin.jsp page, the following contents could be seen in the _pages directory and the error was still very much present:

sandbox:web_dev> pwd
/ORACLE/apps/dev/common/_pages/_oa__html

sandbox:web_dev> ls -l *Apps*
-rw-r--r--    1 applmgr oaa         11356 Nov 14 09:23 _AppsChangePassword.class
-rw-r--r--    1 applmgr oaa         11494 Nov 14 09:23 _AppsChangePassword$__jsp_StaticText.class
-rw-r--r--    1 applmgr oaa          2037 Nov 14 09:23 _AppsCustomizedSSOLogin.class
-rw-r--r--    1 applmgr oaa           852 Nov 14 09:23 _AppsCustomizedSSOLogin$__jsp_StaticText.class
-rw-r--r--    1 applmgr oaa          4626 Nov 14 09:23 _AppsLocalLogout.class
-rw-r--r--    1 applmgr oaa           888 Nov 14 09:23 _AppsLocalLogout$__jsp_StaticText.class
-rw-r--r--    1 applmgr oaa         25964 Nov 14 09:28 _qotSZzpAppsLink.class
-rw-r--r--    1 applmgr oaa          3714 Nov 14 09:28 _qotSZzpAppsLink$__jsp_StaticText.class

Since the .class files were very much present, it meant that the .jsp files had been converted into runtime .class files.

Snippets from relevant log files

mod_jerv.log

[12/11/2007 14:33:53:558] (ERROR) ajp12: Servlet Error: OracleJSP: java.io.FileNotFoundException
[12/11/2007 14:34:57:972] (ERROR) ajp12: Servlet Error: OracleJSP: java.io.FileNotFoundException
[12/11/2007 14:39:43:301] (ERROR) ajp12: Servlet Error: OracleJSP: oracle.jsp.provider.JspCompileException: <H3>Errors compiling:/ORACLE/apps/dev/common/_pages/_oa__html//_ahlBrowserTitleIncl.java</H3><TABLE BORDER=1 WIDTH=100%><TR><TH ALIGN=CENTER>Line #</TH><TH ALIGN=CENTER>Error</TH></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>63</TD><TD>[jsp src:line #:26]<br> cannot resolve symbol symbol : class FrameworkException location: class _oa__html._ahlBrowserTitleIncl         catch(FrameworkException fe)               </TD></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>68</TD><TD>[jsp src:line #:31]<br> cannot resolve symbol symbol : variable csiPageContext location: class _oa__html._ahlBrowserTitleIncl         csiPageContext.setAppId(AhlcmAppConst.APP_ID);         </TD></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>69</TD><TD>[jsp src:line #:32]<br> cannot resolve symbol symbol : variable csiPageContext location: class _oa__html._ahlBrowserTitleIncl         Hashtable browserTitlePrmpts = CsifutRegion.getPrompts(csiPageContext, "AHL_BROWSER_TITLES");                                                                </TD></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>69</TD><TD>[jsp src:line #:32]<br> cannot resolve symbol symbol : variable CsifutRegion location: class _oa__html._ahlBrowserTitleIncl         Hashtable browserTitlePrmpts = CsifutRegion.getPrompts(csiPageContext, "AHL_BROWSER_TITLES");                                        </TD></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>73</TD><TD>[jsp src:line #:36]<br> cannot resolve symbol symbol : variable AHL_BROWSER_TITLE_PRMT location: class _oa__html._ahlBrowserTitleIncl           browserTitle = (String)browserTitlePrmpts.get(AHL_BROWSER_TITLE_PRMT);                                                         </TD></TR></TABLE>
[12/11/2007 14:39:43:301] (ERROR) an error returned handling request via protocol "ajpv12"

OACoreGroup.0.stderr

Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initialization failed. The Distributed Caching System failed to initialize on port: 12362. The list of hosts in the distributed caching system is: sandbox1 sandbox2 . The port 12362 should be free on each host running the JVMs. The default port 12362 can be overridden using -Doracle.apps.jtf.cache.IASCacheProvidercacheProvider.port=<port number>
        at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:220)
        at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1444)
        at oracle.apps.jtf.cache.CacheManager.initCache(CacheManager.java:752)
        at oracle.apps.jtf.cache.CacheManager.<clinit>(CacheManager.java:378)
        at oracle.apps.fnd.cache.Cache.setCacheFullName(Cache.java:228)
        at oracle.apps.fnd.cache.Cache.initCache(Cache.java:114)
        at oracle.apps.fnd.cache.Cache.<init>(Cache.java:89)
        at oracle.apps.fnd.cache.AppsCache.<init>(AppsCache.java:86)
        at oracle.apps.fnd.cache.AolCaches.getCache(AolCaches.java:155)
        at oracle.apps.fnd.profiles.Profiles.<clinit>(Profiles.java:241)
        at oracle.apps.fnd.profiles.ExtendedProfileStore.init(ExtendedProfileStore.java:498)
        at oracle.apps.fnd.profiles.ExtendedProfileStore.<init>(ExtendedProfileStore.java:119)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:274)
        at oracle.apps.fnd.common.AppsContext.instantiateProfileStore(AppsContext.java:4312)
        at oracle.apps.fnd.common.AppsContext.makeProfileStore(AppsContext.java:1171)
        at oracle.apps.fnd.common.Context.setProfileStore(Context.java:970)
        at oracle.apps.fnd.common.Context.setProfileStore(Context.java:952)
        at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:958)
        at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:918)
        at oracle.apps.fnd.common.AppsContext.<init>(AppsContext.java:752)
        at oracle.apps.mwa.container.ApplicationsObjectLibrary.AOLInit(ApplicationsObjectLibrary.java:172)
        at oracle.apps.mwa.container.ApplicationsObjectLibrary.<init>(ApplicationsObjectLibrary.java:76)
        at oracle.apps.mwa.container.MWALib.setObjectLibrary(MWALib.java:333)
        at oracle.apps.mwa.wap.engine.WapServlet.init(WapServlet.java:158)
        at org.apache.jserv.JServServletManager.load_init(JServServletManager.java:755)
        at org.apache.jserv.JServServletManager.loadServlet(JServServletManager.java:659)
        at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager.java:789)
        at org.apache.jserv.JServServletManager.init(JServServletManager.java:447)
        at org.apache.jserv.JServ.start(JServ.java:625)
        at org.apache.jserv.JServ.main(JServ.java:234)
.....
.....
java.lang.NoClassDefFoundError
        at org.apache.jserv.JServServletManager.load_init(JServServletManager.java:765)
        at org.apache.jserv.JServServletManager.loadServlet(JServServletManager.java:659)
        at org.apache.jserv.JServConnection.processRequest(JServConnection.java:394)
        at org.apache.jserv.JServConnection.run(JServConnection.java:294)
        at java.lang.Thread.run(Thread.java:534)
.....
.....
java.lang.NoClassDefFoundError
        at oracle.apps.fnd.framework.webui.OAPageContextImpl.getMACValidationLevel(OAPageContextImpl.java)
        at oracle.apps.fnd.framework.webui.OAPageBean.isMacEnabled(OAPageBean.java:7254)
        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:667)
        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:508)
        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:429)
        at _oa__html._OA._jspService(_OA.java:85)
        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)
        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)
        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)
        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)
        at oracle.jsp.JspServlet.service(JspServlet.java:156)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)
        at org.apache.jserv.JServConnection.processRequest(JServConnection.java:456)
        at org.apache.jserv.JServConnection.run(JServConnection.java:294)
        at java.lang.Thread.run(Thread.java:534)

javacache.log

sandbox:web_dev> pwd
/ORACLE/apps/dev/common/rgf/dev_sandbox1

....
....
[2007-11-14 Wed 11:59:31.099 AM EST] -001 [Grp] Transport initialization failed.
java.lang.IllegalArgumentException: Port value out of range: 144957
        at java.net.ServerSocket.<init>(ServerSocket.java:180)
        at oracle.ias.cache.group.Transport.createServerSocket(Unknown Source)
        at oracle.ias.cache.group.Transport.ssinit(Unknown Source)
        at oracle.ias.cache.group.Transport.init(Unknown Source)
        at oracle.ias.cache.group.GroupCommunication.init(Unknown Source)
        at oracle.ias.cache.Net.init(Unknown Source)
        at oracle.ias.cache.CacheInternal.init(Unknown Source)
        at oracle.ias.cache.Cache.init(Unknown Source)
        at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:199)
        at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1444)
        at oracle.apps.jtf.cache.CacheManager.initCache(CacheManager.java:752)
        at oracle.apps.jtf.cache.CacheManager.<clinit>(CacheManager.java:378)
        at oracle.apps.fnd.cache.Cache.setCacheFullName(Cache.java:228)
        at oracle.apps.fnd.cache.Cache.initCache(Cache.java:114)
        at oracle.apps.fnd.cache.Cache.<init>(Cache.java:89)
        at oracle.apps.fnd.cache.AppsCache.<init>(AppsCache.java:86)
        at oracle.apps.fnd.cache.AolCaches.getCache(AolCaches.java:155)
        at oracle.apps.fnd.profiles.Profiles.<clinit>(Profiles.java:241)
        at oracle.apps.fnd.profiles.ExtendedProfileStore.init(ExtendedProfileStore.java:498)
        at oracle.apps.fnd.profiles.ExtendedProfileStore.<init>(ExtendedProfileStore.java:119)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:274)
        at oracle.apps.fnd.common.AppsContext.instantiateProfileStore(AppsContext.java:4312)
        at oracle.apps.fnd.common.AppsContext.makeProfileStore(AppsContext.java:1171)
        at oracle.apps.fnd.common.Context.setProfileStore(Context.java:970)
        at oracle.apps.fnd.common.Context.setProfileStore(Context.java:952)
        at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:958)
....
....

So what is Distributed JTF cache?

Thats when distributed JTF cache feature comes in, which seems to be enabled by default in 11.5.10 (not sure as of which ATG patchset though). Please my previous article What is the Java cache feature: Framework, Features, Administration and How does it tie up with Workflow Business Event System (BES)

Essentially, it is just one kind of Java cache component for the JTF product and has a direct bearing on the compilation of the AppsLocalLogin.jsp page. If some FND profile options were changed from a non-java interface like Forms or sqlplus, AND the Workflow Java Deferred Agent Listener GSM component is NOT running, then there is a high possibility that the JTF cache has been invalidated.

Working around it...

Thankfully, this is when we came across other bugs that talked about disabling the distributed JTF java cache by modifying the following parameters in the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties and bouncing Apache:

###############################################
# By default the value of DLONG_RUNNING_JVM is true
###############################################
wrapper.bin.parameters=-DLONG_RUNNING_JVM=false

###############################################
# You may have to add this parameter
###############################################
wrapper.bin.parameters=-DCACHEMODE=LOCAL

And this is when it all the features of Java caching started to make sense. Again, for more information and background, I would highly advise you to read the following two articles:

What is the Java cache feature: Framework, Features, Administration and How does it tie up with Workflow Business Event System (BES)

Java Caching for Oracle Applications 11i: Part 1

Conclusion

Java caching can be a pain in the neck, if we do not understand how it works. This can really cause grief in post cloning situations, since FND profile options have to changed to suit to the new environement. In the above case, the issue could also have been resolved by making sure the pending events of Workflow Java Deferred Agent Listener GSM component were processed.

The catch was to understand the Java caching framework and how it was connected with the Workflow Business Event System (BES) events.

Comments (1)

Periyasamy:

Nice diagnostic path... Is there any metalink which helps in increasing the no.of JVM's in a 11.5.10.2 environment?

Posted by Periyasamy | January 16, 2008 12:50 AM

Posted on January 16, 2008 00:50

Experiments from the Field..Based on True Stories