Oracle E-Business Suite Technology

Ok.

So I have created a bayesian network to diagnose java caching problems and it is working remarkly well (the ROC / AUC are looking good for it).

Lets exchange notes if you are interested.

Sincerely,

-Mohsin

Mike/Steve, Thanks for this valuable information. Hope to see some tips/insiders in SSO registration information in Apps (Things hidden behind SSOSDK) and mystery of txkrun.pl
As always this blog is my inspiration.

Atul Kumar

Very interesting notes!

We actually hit both "issues" with responsibilities and logon issue in our test environments. Usually the NoClassDefFoundError comes after cloning but since Apache bounce resolves it and it doesn't reappear after that we haven't been looking deeper into it.

Hi Mike/Steve,

We occasionally hit NoClassDefFound errors with the environments. To fix this, we've found that if we remove the port values in the context file (s_fnd_cache_port_range) and leave it null - allowing the system to dynamically select ports, the problem vanishes away. (We run the application with a default of 1 core JVM - ie OACoreGroup of 1).

Would you have a clue on this ?

Thanks,
Rakesh.

Hello Mohsin, yes please. Will discuss with you offline

Atul, thanks for your thoughts, SSO registration/integration is certainly an area that needs to be unpicked a little I think

Simon and Rakesh, useful information so will make note and look into further

Excellent notes. I hit almost all the problems before. Even I had resolved the problems, but I did not have the whole picture. These notes definitely help me understand the problems.

We had NoClassDefFoundError for the first login to almost every new cloned environment we created. What I do is to remove $COMMON_TOP/_pages/*, bounce the Apache, the problem will go away for goods. The interesting thing is that even before I start up the Apache the very first time, I had cleaned $COMMON_TOP/_pages/* , I still have the problem. I have to remove $COMMON_TOP/_pages/_oa__html again and bounce Apache, now the problem goes away for goods.

Hello Mike!

I seems that java caching problems remain in oAS 10g. We have missing responsibilities issues with OEBS Release 12 and similar problems with OCS 10.1.2 both based on oAS 10.1.2. I've read your metalink note 455194.1 "Diagnosing database invalidation issues with Java Cache for eBusiness Suite" but could not realise what should I do to avoid java caching problems in OEBS Release 12.
Regards,
Viacheslav Leichinsky

Hello Tianhua,

So far as I know, cloning does not clear out the JTF_PREFAB_WSH_POES_B table, so if this has entries relating to your source system, this could be causing temporary confusion for Java Cache. Next time you do a clone, try truncating the JTF_PREFAB_WSH_POES_B table (with all eBiz middle tier services shutdown) The JTF_PREFAB_WSH_POES_B table is repopulated when the services startup so there is no problem doing this on a shutdown system.

Let me know if this works and we can get that fixed

Hello Viacheslav,

Release 11i and Release 12 both use pretty much the same underlying code with respect to Java Caching, so the diagnostic techniques will be very similar in both cases.

I would recommend you raise a SR with Oracle Support to have your specific case investigated. To give the investigations a head start, upload the data/script output from parts 1 through 4 of the "Detailed diagnostic steps" in Note 455194.1, as most of this information is the same for R12. The main differences are the config files name/locations and the specific patch numbers that are being listed.

Thanks Mike, I will give it a shot and let you know.

I just ran into the NoClassDefFoundError for the first login issue this morning. I have cleaned up _pages and bounced Apache. It seems to work now.

Thanks,
Sriram

Hi Mike,
I have tried "truncating JTF_PREFAB_WSH_POES_B" in two clone envs, and it works, Thanks for the tip.

T Wu

Hi Mike,

I have encountered one problem lately and I suspect it might be related with Java Cache.
Basically, one a user update a profile, we do not see the changes even after boucing the Apache. However, it would eventually show in the apps after another period of time. If I bounced all applications, it would show immediately.
Unfortunately this only happens only in production, I could not play with it to decide which apps I should bounce at the same time. We are at 11.5.10.2 (ATG.RUP5) and 10.2.0.2 (DB).

I just want to know what your thought is?

Thanks,

T Wu

Hello Tianhua,

Your issue doesn't seem to directly relate to java cache and sounds more likely an issue with the processing of the records through the Workflow Java Deferred Agent Listener. Probably best plan is to go through Note 455194.1 and then raise a SR with Oracle Support

Hope this helps

Mike

Thanks again, Mike! I will look into it.

T Wu

I am trying to understand the steps by which membership is attained in a distributed cache and the process by which
a coordinator for JOC is appointed.

Is there any document out there that explains this ?

Thanks

Ravi

Hello Ravi,

Ebiz utilizes the underlying technology delivered by the AS10g team, so at the level of deteail you are looking into you can refer to the standard documentation set for the Java Object Cache

This is described in the "Oracle Containers for J2EE Services Guide 10g Release 3 (10.1.3) " at http://download-west.oracle.com/docs/cd/B25221_03/web.1013/b14427/joc.htm

Hope this helps

Mike

Hi Mike
I've only recently come across your articles. We have had an ongoing problem with our HTML forms since implementing 11i CU2. We are using HTML Quoting with the side bar navigation links. The problem is that we have missing links from that side bar navigation menu that we can not pinpoint as to why they disappear. The only thing that we have found to correc this is an Apache bounce. If the Apache bounce is not performed correctly, the html links do not reappear. This has been happening for the past 4 years. We have had numerous SRs and OWCs but to date, there has been no correction the problem. Is there any assistance that you can provide?

Thanks
Linda

Hello Linda,

I am not a Quoting expert, and believe it uses it's own product specific mechanism to use the underlying Java technology (from memory) so may not be the best person to help

Can you email me a SR number (or two) to take a look at and will see how best to approach the problem......

regards

Mike

Hi Mike,

One of our customer has around 50 JVM's running , enabling Distributed Caching is causing performance issue since each JVM need to communicate with other 50 JVM's .
Do we have any option in 11.5.10 to group JVM's let us say 5 groups with each group of abount 10 JVM's . If we can do it then each JVM need to communicate with only 10 JVM's .

Regards
Santhosh

Hello Santhosh,

You can only have one group, but 50 JVMs isn't an unusually large number of JVMs for a PROD instance so not sure why you would see an appreciable performance hit.

Probably best to raise a Service Request with ATG Support so we can take a look at the specifics of your case

regards

Mike

Hi Mike,

We have 3 OACore JVMs per test instance - all on the same server, with only one appstier per instance - they all have their databases on a separate server with no firewall in between - so we have s_fnd_cache_port_range set to nothing.

Does setting s_fnd_cache_port_range mean that this port is allocated dynamically ?

We seem to get intermittent, but frequent problems with some of the test environments taking the ports which are defined for other environments in the xml file.

This issue Never occurs in production (Dedicated server - not sharing with any other envs)

When I run a netstat -anp | grep - the output shows that one of the other environments is listening on the port in question.

Do you think having s_fnd_cache_port_range to nothing maybe causing this ?

Thanks

Oli

Hello Oli,

Yes, if "s_fnd_cache_port_range" is not set then the port number is random.

From your description of the issue, then setting "s_fnd_cache_port_range" to a specific port range for each test environment would likely resolve that problem

Hope this helps

Mike

Thanks very much Mike - we will try to implement this to see if it helps.

On the Server we only have Oracle Apps running.

cat /proc/sys/net/ipv4/ip_local_port_range gives
1024 65000

Is there a recommended/standard port number to use for the port range ?

I saw some articles giving examples of (portpool + 3000-portpool+3000+Max_num_JVMs)

Thanks very much for you help

Oli

s_fnd_cache_port_range is discussed in Note 287176.1 "Oracle E-Business Suite 11i Configuration in a DMZ" but we do not suggest any specific port range to use, so you can come up with your own conventions.

Hope this helps

Mike

Hi Mike Thanks for info:

In section

5.9: Enable Distributed Oracle Java Object Cache Functionality

How do you do this:Identify the number of java processes spawned by the concurrent manager tier. For eg: if there are 3 JVMs spawned by the ICM, take the number as 3 .

Thanks

Oli

5.9: Enable Distributed Oracle Java Object Cache Functionality

Identifiy the number of java processes spawned by the concurrent manager tier.?For eg: if there are 3 JVMs spawned by the ICM, take the number as 3 .
Add this to the number of oacore JVMs . In the example given above, the total number JVMs become 6 . So, six ports need to be opened in the firewall.

Do know how to identifiy the number of java processes spawned by the concurrent manager tier.?

Thanks

Oli

Hello Oli,

I was sure we had documented how to calculate this number, but had to dig hard to find it !

The technique is mentioned in Note 380490.1 "Oracle E-Business Suite Release 12 Configuration in a DMZ" which suggests:

You can use the 'pstree' command to check the number of java processes spawned by the concurrent manager parent process. For eg: pstree -p 26258 where 26258 is the process ID of the FNDSM process

Hope this helps

Mike

Thanks very much for this Mike - one last thing...

pstree output loads of java processes - I piped it through wc -l and got 222 processes. I was expecting it to be a smaller number - any advice ?

Thanks - heres a small sample of the output

$ pstree -p 1103
FNDSM(1103)ââ¬âFNDCRM(1365)
ââFNDLIBR(1372)
ââFNDLIBR(1373)
ââFNDLIBR(1374)
ââFNDLIBR(1375)
ââFNDLIBR(1376)
ââFNDLIBR(1384)
ââFNDLIBR(1388)
ââFNDLIBR(1389)
ââFNDLIBR(1391)
ââFNDLIBR(1392)
ââFNDLIBR(1393)
ââFNDLIBR(1394)
ââFNDSCH(1369)
ââPOXCON(1242)
ââPOXCON(1243)
ââPOXCON(1244)
ââPOXCON(1245)
ââPOXCON(1246)
ââPOXCON(1247)
ââRCVOLTM(1239)
ââRCVOLTM(1240)
ââRCVOLTM(1241)
ââjava(1248)ââ¬â{java}(1249)
â ââ{java}(1250)
â ââ{java}(1251)
â ââ{java}(1252)
â ââ{java}(1253)
â ââ{java}(1254)
â ââ{java}(1255)
â ââ{java}(1256)

Hello Oli,

By the looks, the output is showing each thread as a sub-process to the main java process. So PID 1248 is the "real" process and 1249->1256 are threads which are shown as children

In other words, you should be able to just look at the "java" processes at the top of the tree and ignore the rest

Hope this helps

Mike

Hi Mike, Thankyou this helps a lot.

I identified the FNDSM - process was spawning 27 real java processes - so I added the 3 for the OACORE JVMs and updated the context file for a port range of 30501-30530 - ran autoconfig - checked the profile options were updated. Then I restarted all the apps services

It seems the java processes are now using the ports I defined - but not exclusivley -Netsat shows connections between java processes through the ports I defined to other random ports. I increased the port range from 30 upto 250 (30750) to see if this allowed enough ports for the processes to just use the ones in the range but it gave the following:

>>
>> Edited to remove this section
>>

Is there something Im missing

Thanks very much

Oli

Hello Oli,

Sorry you are having these problems with your setup. Seems like you are at the stage where you need to raise a Service Request (SR) for Oracle Support to take a more detailed look into your specific configuration. Please go ahead and raise a SR but by all means let me know the SR number.

As an aside, you will notice I took out site specific data from your last update, as didn't think it prudent to publish

regards

Mike

Thanks Mike - SR number 7756431.994

Regards

Oli

Hi Mike,

After the changes the issues went away for nearly a month - but once has cropped up again - SR 7756431.994 any ideas?

Thanks

Oli