How Can We Improve EBS Techstack Support?

Welcome to Lake Wobegon, where all the women are strong, all the men are good-looking, and all the children are above average.

~ Garrison Keillor

The New Year is a good time for reflection.  In that spirit, I'm going to try something different here.  One of our regular readers recently made a pointed remark about his clients' general desires to avoid reaching out to Oracle Support unless absolutely necessary.  This isn't the first time I've heard this sentiment but it crystallized my thoughts on something I've been meaning to discuss here for a while.

"See One, Do One, Teach One"

Cover of Complications-Atul-Gawande.jpg

One of my favorite medical writers is Atul Gawande, an accomplished surgeon and frequent contributor to the New York Times and New Yorker.  In his superb first book, Complications: A Surgeon's Notes on an Imperfect Science, Gawande makes this critical observation:

Every day, surgeons are faced with uncertainties.  Information is inadequate; the science is ambiguous; one's knowledge and abilities are never perfect.  Even with the simplest operation, it cannot be taken for granted that the patient will come through better off -- or even alive.

... In surgery, as in anything else, skill and confidence are learned through experience -- haltingly and humiliatingly... we need to practice to get good at what we do.

... This is the uncomfortable truth about teaching.  By traditional ethics and public insistence (not to mention court rulings), a patient's right to the best care possible must trump the objective of training novices.  We want perfection without practice.  Yet everyone is harmed if no one is trained for the future.

The Impossibility of Perfection Without Practice

Technical support isn't surgery, of course.  But there are many parallels in the complexities of the diagnostic process and procedures for addressing a particular situation. 

Although I'm part of our Development group, I get frequent opportunities to work with our Support team in a number of different capacities.  Like any large organization, I've found that Support has staff with varying degrees of technical depth and experience.  There are seasoned Support staff from whom I learn something with each conversation, and there are novices whom I occasionally coach or correct.

Like the proverbial box of chocolates, you'll never know who you're going to get when you log a Service Request (SR) against Metalink.  You might get a seasoned veteran or you might get someone who's still learning the ropes for your particular problem.  Support engineers usually search our internal knowledge base and discuss tricky SRs with the grizzled vets before making recommendations to you.  Regardless, the odds are that some of your SRs will be handled by engineers in the early stages of their careers.  This is inevitable.  If you understand Gawande's reasoning, it's also desirable and necessary.

What Can We Do Better?

Understanding that, I'd like to invite you to suggest ways in which we can improve how we provide support for E-Business Suite technology stack issues.  Remember that I'm in the Applications Technology Integration (Development) team, not Support, and can't make any commitments on Support's behalf.  I can promise you that I'll make sure that your feedback gets heard by the right people responsible for supporting our EBS technology stack.

Here are some tips for keeping this conversation productive:

  • Keep things focussed on EBS technology stack support, not product support (e.g. for Financials or Human Resources).
  • Please keep it constructive.  We can't do much with "Support sucks."
  • Avoid ad hominem attacks and unreasonable flames.  These will be moderated.
  • Be specific, drawing upon your actual experiences.
  • Provide Service Request numbers if you feel they're illustrative (or email them to me directly if the details are confidential).
Comments:

Great post, I agree. But I do not avoid creating SR's, I log them as needed with the hopes that I will either learn something in the process or I will have identified a bug that will help Oracle and other customers. Folks on the receiving end of an SR will keep me happy by either 1) working hard to resolve the problem or 2) having authoritative answers to my questions. So basically "don't ignore me and if you don't know, help me" (and you) find out the answer.

Cheers!

Posted by Stephen Manning on January 06, 2009 at 10:36 AM PST #

I'm going to think about this and will probably end up with a few hopefully constructive and insightful comments. Off the top of my head these things come to mind.

1) When CPU patches are released, some of the patches for the platforms I deal with (linux, solaris) have not been available when the CPU is released, or some will be password protected, requiring me to open SRs to get patches Oracle has already recommended. These aren't huge, but they are annoying and do slow down adoption.

2) On the password protecting of certain patches - why is it that way in the first place? We all have to log in to metalink (or the ftp site with our metalink id) to get the patch, and I know these records are kept of what we downloaded, because I occasionally get emails about patches being "reloaded". If we must keep password protecting of certain patches, can there be some sort of alert / flag to the analyst to tell them when recommending a patch to a user if it's password protected?

3) Back in the day of the paper TRMs and to a lesser degree with the eTRMs, if I wanted / needed to, I could trace back all the connections and get a better idea of how things fit together. With tech stack this isn't as cut and dried, but finding one stop documentation sources for say, what each autoconfig xml parameter does and where it gets propagated would be great. These days you can put in most init.ora parameters into metalink and get a useful hit explaining it's values, usage, effect. Entering in an xml tag for autoconfig into metalink you'll get bugs that are related, but an explanation? Not as much.

4) I realize I've mentioned this one before, but bring back the automated required init.ora parameter checking script for apps DBs, updated for 11g. It might be back, I have to look.

5) It's been awhile since an ATG RUP came out for 11i Apps (RUP 6) but with each RUP there has been confusion as to what apps security patches are included in that ATG RUP. When I have a client that's behind on apps patches and we apply , say, RUP 6 which (thank you) states CPUs thru XX/XX have been included, that's great, but if I am anal-retentive and do my own checking (select * from ad_bugs where bug_number = 'XXXXXXX') after applying it to a development instance and a patch wasn't applied, umm - that's worrying and SR time.

6) All (ok, most) patches have pre-reqs and they're listed in the readmes generally in a standard format of
XXXXXXX - description of XXXXXXXXX
and occasionally the less helpful
11i.FND.H - description of FND.H

Two thoughts here,
a) can you all include a simple script or even section of the readme that does the select * from ad_bugs where bug_number = 'XXXXXXX' for the pre-reqs - nothing fancy, just plain easy sql that I can cut and paste (very useful especially with multiple patches)?
b) Stop saying things like the 11i.FND.H example above - I admit it, I just don't remember the patch # for FND.H and if every instance I manage has it. When I see these things, I have to then go out to my latest output of patchsets.sh for the instance and see what patchset it's on.

Ok, speaking of patches, my patching for the evening is done, so I'll stop this probably too long already comment.

Thanks Stephen for this blog. I really really find it useful and am always turning other DBAs on to it. It's a great resource.

Posted by Jay Weinshenker on January 06, 2009 at 11:04 AM PST #

Hello Steven,

Just wonder if Oracle Support strategic management considered to introduced some type of Metalink users technical ranking and use it in SR routing process for avoiding nasty situation then Experienced technical expert from customer side hits junior support analyst?

I believe many of us has been in the situation then you carefully describe a problem and diagnostics steps you went through based on your 1x years experience working with particular product, describe your experiences and awards in first few lines of SR but in response getting annoying request to complete hundreds of basic steps you did already. After that unreasonable RDA requests comes etc.

I agree that in inexperienced customer case those steps might lead to find a solution quickly but for another 10% of experienced Metalink users those don’t add any value and people start trying to avoid creating SR-s as I do now.

If we assume that Experience makes Support expert better, than if customers seniors avoids communicating with Support it doesn’t allow Oracle Support to develop more expert level professionals reducing overall competence. It looks like the current situation leads Oracle Support to degradation in terms of total quality and in-house experience.

Customers’ experts ranking might be the solution to allow customers with certain level of knowledge to communicated with corresponding Support representatives.

Best regards,
Jurijs

Posted by Jurijs on January 06, 2009 at 03:05 PM PST #

Thanks for the great comments so far, I will first respond to a couple of the technical issues Jay has raised (in no particular order)....

- Note 174605.1 "bde_chk_cbo.sql - Reports Database Initialization Parameters related to an Apps 12 or 11i instance" allows you to automate the checking of RDBMS parameters, although Note 216205.1 and Note 396009.1 should still be considered the single source of truth....

- The Autoconfig parameters are self documented. You can see the descriptions if you are changing them using the "proper" way (via OAM)... or can just take a peek at $AD_TOP/admin/template/adctxinf.tmp if you prefer manually hacking files

- selecting from AD_BUGS is not an authoritative way to confirm if a patch is applied or not. Using this mechanism can potentially give you both false positive and false negative results. Using OAM's "Applied patches" functionality is the recommended way of course.

- OAM's "Recommend/Analyze Patches" would probably help with patch pre-req checking and/or you could enable Pre-req checking by adpatch using "options=prereq" on the command line

Jurijs also raises some interesting points.

- Support have to be careful about how much experience we assume a customer has and how many assumptions are being made about the issue that is being reported - so the default situation is to "start from scratch" with a SR. Oracle Configuration Manager (http://blogs.oracle.com/stevenChan/2006/06/automated_configuration_collec.html) is specifically designed to give both you and Oracle Support a head start on all issues you are facing, but in addition to this, if you have already reviewed Metalink notes and done your own diagnostics then you certainly should be mentioning this on the SR and uploading the relevant output for our review. Chris Warticki's Blog (http://blogs.oracle.com/Support/) is interesting reading if you are keen to get the best out of Oracle Support. Amongst other things, his team also provide web based training (http://www.oracle.com/support/seminars.html) on things like Support best practises, using Metalink and so on

- I would be more than happy to be able to rank customers who are logging calls with Support, but unfortunately this is a can of worms that probably shouldn't be opened. (for example, what if you don't agree with my assessment of your skillsand even what do we actually do with the ranking once we have it....) Even a simplistic approach of using OCP certifications as a guide is fraught with problems, as firstly what ones would Apps DBA's need (http://blogs.oracle.com/stevenChan/2007/07/four_essential_oracle_ocp_cert.html) and then what would we do with customers who don't have DBA's with these OCP's

Overall, Oracle are always aiming at reducing the number of SRs that need to be raised, by improving the product, the documentation, the diagnostics provided, self repair and the Metalink Notes, and also by empowering the customer to solve their own issues wherever possible by using the tools like OAM and providing good research tools like Metalink. When SRs do need to be raised, customers should expect a professional and constant approach for all SRs no matter who is dealing with it. Hopefully we are going in the right direction with this objective.

Posted by Mike Shaw on January 06, 2009 at 06:54 PM PST #

Hi Steve, is there a goal to get EBS12 to the point that it only requires one AS version be installed, instead of two (one for Forms, and one for Java)? That might simplify things?

Posted by Keith on January 07, 2009 at 04:26 AM PST #

Hi, Stephen,

Thanks for your comments. That's the kind of attitude that is appreciated by all of our Support staff. They might not claim to know all of the answers, but will strive to find the right person to provide them.

Regards,
Steven

Posted by Steven Chan on January 08, 2009 at 04:35 AM PST #

Hi, Jurijs,

Interesting idea. I'd hate to get into the business of ranking the expertise of Metalink users, though; that's a slippery slope.

However, I think that it becomes clear very quickly to both the Support engineer and the SR filer as to who has relatively more expertise. It's always fair for customers to point out that a given Support engineer might be barking up the wrong tree, and to request that person to consult with a more senior engineer. Speaking with a Duty Manager can result in more experienced staff being pulled into a given SR, too.

If we jointly appreciate that Oracle Support, Development, and customers all have something to teach each other, that's a good start. We all benefit from technically-intensive dialogue, regardless of whether the "source of truth" is Oracle or our customers.

Don't give up on filing SRs! I agree that it can be painful sometimes, but it sounds like you understand the broader value of that.

Regards,
Steven

Posted by Steven Chan on January 08, 2009 at 04:43 AM PST #

Hi, Keith,

I hear you. From a maintenance standpoint, having multiple ORACLE_HOMEs is a pain. From an architectural standpoint, there are strong benefits.

For example, having separate ORACLE_HOMEs allows us to upgrade one without affecting the other one. If you're happy with your Forms version and don't want to destabilize it with an upgrade, fine; you can just upgrade the OC4J components in the other ORACLE_HOME to speed up Java-related content (or improve stability).

So, to answer your question directly: I'm afraid not. The fine-grained nature of maintaining different ORACLE_HOMEs for different techstack components offers structural benefits that we believe outweigh their costs, so this is our architectural strategy for the conceivable future.

Regards,
Steven

Posted by Steven Chan on January 08, 2009 at 04:54 AM PST #

Thanks, everyone, for the comments so far.

To respond to Jay's question about password-protecting certain patches:

I'll simplify our release process a bit for the purposes of this discussion. Individual product teams have the option of selecting the distribution method for their patches. Patches can be flagged for internal Oracle/Development use only, for general availability to everyone, and for a specific subset of users.

Our general practice is that emergency patches whose impacts aren't well-understood (but need to be released quickly due to reported P1s) may be considered sketchy. These patches should only be applied after an experienced Support engineer has confirmed that the customer is really hitting the problem resolved by that particular patch, and that the customer understands the risk, and that the risk of applying a sketchy patch is outweighed by the benefits.

Unlike the majority of the readers of this blog, who are generally in the upper-quartile of Apps DBA skills and experience, there are DBAs who will blindly apply whatever patches they can get their hands on. Password-protecting certain patches give us a fighting chance of preventing certain well-intentioned but misinformed users from damaging their environments inadvertently.

That's the theory, anyway.

Regards,
Steven

Posted by Steven Chan on January 09, 2009 at 04:38 AM PST #

Following up on Mike's response about AutoConfig parameters being self-documented: this is true, but I know that some of this "self-documentation" may leave a bit to be desired.

I can't provide a lot of specifics at this point, but our AutoConfig team has heard your repeated questions along these lines. They're working on a utility that can potentially provide more useful information about AutoConfig parms. Watch this blog for sneak previews (if I can persuade Ivo Dujmovic to free up some time to write a blog article about this).

Regards,
Steven

Posted by Steven Chan on January 09, 2009 at 04:41 AM PST #

Steven,

From a philosophical standpoint, I don't open SRs with the presumption that they will be resolved, but I also don't hesitate in opening them while I work the issue on my own.
We pay for support, so I use it, in hopes the analyst can find something I can't see or research on my own.

While this has the potential to be a vast topic, I thought I'd comment on a specific area which I find incredibly frustrating. We lovingly call it "busy work" from the analysts; when you're asked to run a script to determine what what rollup of ATG, etc, has been applied. Don't get me wrong, I understand that this is information which is required. What I find frustrating is why this is still required when I open an SR using the new "My Oracle Support" interface. I'm not the biggest fan of the flash-heavy new interface, but I see the value of using Oracle Configuration Manager and the new interface to provide this information up front to the analyst. Despite this, it seems on nearly every SR I'm asked to provide information which should (or could in the future) already be known.

Posted by Kevin Kempf on January 13, 2009 at 10:13 PM PST #

Hi, Kevin,

Thanks for your insight. I think that Support, too, would consider themselves to be partners in searching for answers (rather than seeing themselves as the sole source of those answers).

Your point about OCM's configuration information is really intriguing. I was under the impression that Support engineers could draw upon the database as part of the SR investigation process. That said, there's a lot of sensitivity around who can access a customer's potentially privileged and proprietary information, so my assumption may be naive.

I've escalated that point to our Support management chain. It does seem a little silly to collect that info and then not use it for SRs.

Regards,
Steven

Posted by Steven Chan on January 14, 2009 at 05:39 AM PST #

Hi, Kevin,

Our Support team confirms that all engineers should have access to OCM configuration data. OCM isn't universally used by all customers, though, and the catch is that it may be sometimes difficult to tell from an SR whether such data exists.

In response to your comment, Support would like to do some refresher training for staff as needed. Would you be able to provide some of your SRs as samples of those times where you were asked to perform "busy work" that should have been obviated by your OCM data?

Feel free to email the SR numbers to me if that's sensitive info.

Regards,
Steven

Posted by Steven Chan on January 15, 2009 at 02:24 AM PST #

I would like to add a few of my thoughts. I worked extensively with EBS Teckstack support since we started integrating EBS 11i with AS10g (Portal/SSO/OID) in 2006. I found both documentation and support analysts in the area improved significantly. I would say that Oracle support is the best among the vendor support that I have worked with. Especially, Edwin Wiggins’ support team in Orlando, FL. Without their support, including Steven Chan’s support, we wouldn’t be where we are today.

A few constructive comments with an eye towards improvement: The true value of working with support most of the time is to find the best analyst in the field. This makes all the difference under critical circumstances. The question of improvement is ultimately tied to the organizations capacity to increase its number of star analysts. Strong organizations recognize strong performers and create incentives.

My second thought is to build more advanced environments inside support so the support analysts can reproduce issues that customers report. In some cases, I had to transfer SRs worldwide in order to find an analyst who has the access to more an advanced configuration/environment.

My last thought is how to make engaging development a bit easier. After a bug is acknowledged, if development works a bit closer with customers, that may lead the bug being fixed more quickly. We have had a bug logged for about a year in the EBS Teckstack. Development couldn’t figure out the causes. We offered an environment that they could dial into, turn debugging on, make changes, but no developers responded. It seems both development and support staff is a bit overloaded sometimes, which may affect the quality of work.

Hope this helps.
Jennifer.

Posted by Jennifer Chen on February 11, 2009 at 11:51 AM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
4
5
6
7
8
9
10
11
12
13
14
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today