Part Two: Actual User Feedback

Back to Summary
Part One: Gathering Feedback from Recent Usability Research and Studies

In Part One I covered how we gathered feedback from customers for the new My Oracle Support. In this part, we will discuss the detailed results of those studies by looking at two studies we have done in the last year. One done last summer, which was a large study of twelve customers, covering basic navigation and usability. The second a detailed study of the design of the health check features found in production for former Metalink3 customers and coming soon for Oracle customers (November of 2009).

In the first study, participants spent about an hour doing basic tasks mostly focused on SR creation in My Oracle Support. The problems and issues were documented by the usability tester and then reviewed and triaged by the team. Twelve customers participated. Each session was video recorded with their permission. The videos are used to both clarify issues and when filing a bug, a link to the video can be provided to the designer or developer to exactly see the issue.

Results
12 Customers
88 Actionable Issues, by category:

Create SR (42)
General Issues (5)
Delete, Saving, Breadcrumbs, Templates (4)
Wizard: General Information Step (11)
Wizard: Knowledge Step (5)
Wizard : Upload (4)
Wizard: Problem Details (7)
Wizard: Product and Problem (2)
Wizard: Review (4)

 

 


View SR (15)
Draft SRs (5)
Knowledge (8)
General Usability, Dashboard, Filtering, PowerView, Customize, Timezones, etc.. (15)
Bug Region (3)

 

For each of these issues, we cataloged how many of the customers experienced the problem during the session, and then rated the issues to help prioritize them. This was done in a spreadsheet and includes details of when the problem occurred, for who, and screen shots. Later bugs could be filed and pointing the developer to a link to the video to see for themselves the issue.

Let's look at some examples of the issues found, that are fixed for the current or upcoming release.

 










































Issue

Problem

Design Solution / Notes

MUST not allow you to leave SR without saving...

Customer could lose work by navigating away without clicking Save

Adding confirmation when user attempts to navigate away without saving

(Working towards more "Auto - Save" functionality, so you never can lose work and don't need to remember to "Save", but this is still a work in progress)

No way to delete draft SRs

Allow the user more control to remove old draft's without going into the wizard

Adding the Remove icon to the table

"By System" text in Create SR flow is confusing

Many customers do not know what we define as a System. They might confuse our Configuration Manager collected "System" with just the name of their "system".

Redesigned AutoFill Region to reference "System/Configuration" - disabled when they do not have configurations

Remember last 10 choices in Product Selector

Difficult to find "your" product in the 1000's Oracle sells. So keeping track of ones I have used in the past help me find my content next time.

Added Recently Used items to many drop lists and selectors for the 3.2.1 release for MetaLink3 customers (coming soon for Oracle customers). Product List, List of Platforms and Languages, etc... to all have a recently used list.

Opening SR WAY TO SLOW!!!

Large SRs were having trouble rendering in the History region.

Multiple fixes were attempted, and some are still in the works to bring back the "bubble" view, to make it quicker to scan for what you say and what Oracle says. For now even large SRs should load fast in the History region.

Double Scroll bars annoying

Double scroll bars make it very difficult to view content. You have to play with each bar back and forth. Really annoying.

Make sure that there are not two sets of scroll bars for regions which grow. Turn off the scroll policy of one of the items to allow it to grow to its natural size up to the full size of the window. Then let the scroll bar appear, as needed. Fixed in SR region, Attach SR

Can't find the "Send" button

Submitting an SR requires them Send it to Oracle.

Button should be moved to bottom right position of the wizard and highlight the button.

 

So from this study about 30% of the bugs are currently fixed in production releases. I highlight this study because this has a low fix rate. This is something we should work to fix in upcoming releases. While in the next study done for the Health Check area, 70 of the 72 issues were resolved prior to the first release of the Health Check user interface for the MetaLink3 users in August of 2009. Those results were interesting for a few reasons.

The Health Check region for My Oracle Support was effectively designed starting with customer feedback. While the SR form has been around forever, Health Checks are reasonably new and don't have a lot of baggage associated with their design. So we went to about 6 customers without any concept of what we should build and asked them how they want health checks to work and what features would be useful to them. This eventually created a roadmap of features. We then took the key features and design a series of design ideas and floated those by customers. Here are some examples of the evolution of those screens over time.

First this is the screen where we just mapped content up into three regions and called it a day. If you want to see details for a check, you have to click on the finding and go to another page. In this example, you can click on the bars and it will change the health checks shown below. This was a hack we did to help moving between Critical, Warning and Informational. Prior to that you had to return to the home page to change between Critical, Warning and Informational. Ouch!

 

clip_image002.jpg

 

We heard a few key things which made it into the design showed below. This is the most recent release of Health Checks, but still not the end-state. Notice the ability to Suppress a check. This was one of the key communications. Customers wanted to suppress based on a variety of options. We currently allow for the three shown (the issue, an entire target, and an entire check). Some customers also want other ways of suppressing, but more options yields more complexity. So we started with the "big three".

Notice also the ability to see the details of the check without navigating. This improves performance. You can also group by findings and multiple select rows and then suppress. Also notice it is clearer how to navigate to the different classes of checks. And a short cut to view any suppressed items. We also know we had issues with what to call the "Suppression" function (Hide?, Archive?, Put in Trash?), and did our best but also notice how we extended the Suppress menu with a Help link to "Learn about Suppression". So if you don't get it right away, you might explore and learn from the Help system.

 

clip_image002_0000.jpg

 

But we know we were not done. In other studies concerning Patch Recommendations we were asked to; bring the advances from the Patch Recommendation UI into Health Recommendations, and to further extend the features of Health Checks to support a broader array of checks, systems to check and features. Thus the future should see us handle larger numbers of checks for our biggest customers, view the check in more detail, provide feedback to Oracle and the Oracle community about their use of this check (or issues with the check) and other similar features we have now released for Patch Recommendations (sorry can't give you a timeline for this).

Overall, we took in 72 customer issues. Each resulted in a design, assigned to developers and we fixed 70 of the issues prior to shipping the first release of this in Spring of 2009 for the customers on support.oracle.com (Nov 2009 for metalink.oracle.com customers). Try it out for yourself and see what you think.

And in the meantime, take a look at an "idea" I am working on based on recent customer feedback. As with anything this is just a working concept. No guarantee it would ever be released... and I will let the picture do the talking... Want to get involved? Let us know via the Blog.

 

clip_image002_0001.jpg

 

So, this is a look into some of the feedback we have received for usability issues. We get more than we can discuss here. It hurts to hear complaints, but it worse to know we have issues. So do provide us feedback, this makes it easier to focus attention on fixing issues which you really want addressed. And one tip... Don't just tell us, "My Oracle Support is slow!", tell us what is slow and provide details, "When I load My Oracle Support via VPN from my office in China it takes 45 seconds before my dashboard comes up, and that is too slow!". Speaking of "slow", performance has been on our mind for a while. In Part Three I will cover some specific areas of special interest to customers, including Performance.

Next: Part Three: Special Areas of Feedback: Content, Quality, and Performance

Comments:

This is a really interesting series of blog postings. It's great that you guys are listening, and I appreciated the feedback and resolution to the issues I commented about here. You need to somehow bridge the gap between "you" and "us", because a "non technical SR" is hardly the most instinctive terminology for leaving feedback, and too often the response is not a considered one. I'm more than willing to contribute my time to improving site, because I have a vested interest in it being better - but there's no way I'm going to waste my time fighting with someone on the end of an SR trying to get them to understand that I'm not an idiot that needs handholding, that what I'm saying actually needs to go beyond level 1 support. I guess you need some way of recognising, or maybe just a separate communications route, for people wanting to seriously feedback and who aren't just saying they've lost their password or where's the power-on button :)

Posted by rnm1978 on October 25, 2009 at 10:17 PM PDT #

From a design perspective I agree. Using the negative of a word should be reservered for when the negative has the clear meaning of "not". A non-technical SR means what? I think there was some attempt to cover non-feedback (;->) in this space (like Media Requests), which are not technical SRs and well... not feedback. Feedback gets more confusing in the new release being called "Contact Us" (but does not contain the typical "contact us" options, really on containing the same feedback form. I will see if I can help resolve this in a future release. If you mean the "response" to your non-technical SR from support? I think maybe what you are seeing is the "level 1" response. BUT, your feedback is getting all the way back to Product Managers, Development Managers and design. Even if the response is to "simple" for your comment or question. I am going to ping someone else to chime on this, I think he might have a good idea or two on the subject.

Posted by richard.miller on October 26, 2009 at 01:35 AM PDT #

I know it can be painful, but those non-technical SRs help us - they feed the formal process in place to make sure all feedback gets logged and a bug or enhancement request created if need be. All CRs created as part of this process are reviewed by the My Oracle Support team (product mgt, development, business process) and prioritized for upcoming fixes. In addition, we do bi-weekly to monthly rollups of all the feedback and look for common themes or issues that are occurring more frequently. We use these feedback themes to highlight areas where you, our customers, are telling us we need to work harder, and to prioritize issues for fixing or enhancement. OK, so far, but how do you know your voice was heard? We have published release notes on what is fixed in the latest release of My Oracle Support as well as currently known issues. The release notes can be found in article 943285.1 on http://metalink.oracle.com, preparing you for what you should see post migration. They are also posted on http://support.oracle.com in note 870864.5. We will continue to update the release notes going forward. There is also the opportunity to engage with Product Management and Development on the My Oracle Support Communities. The "Using My Oracle Support" community is designed just for this. Customer feedback is critical to our success. We do process and use it, but we need to be better about engaging in discussions and closing the loop.

Posted by Chris Welnak on October 27, 2009 at 02:19 AM PDT #

Richard, I'm a little confused .. is the "suppress" feature available on MOS already, or is it planned for the next release ? This is EXACTLY what I've been looking for ! The prototype screen looks great as well -- get the details of the issue right there, without having to drill down through multiple screens

Posted by Edward on October 28, 2009 at 05:29 AM PDT #

Sorry about the confusion. Suppress is available in production right now if you use support.oracle.com It will be available shortly with the Oracle migration planned for Nov for metalink.oracle.com users.

Posted by richard.miller on October 28, 2009 at 06:02 AM PDT #

Umm... no wonder... I've been using ML on MOS platform all this time and had no clue ... Now I see all the functionality when I go tu MOS on suport.oracle.com. There are a few technical issues I'm having with that site, but I think I'll log an SR rather then bothering you. But I'll ask this: are all the SRs my group's created in MOS version of ML going to be accessible though support.oracle.com? I tried to look for them both by name and number ( not found either way ). Thank you !

Posted by Edward on October 29, 2009 at 03:50 AM PDT #

Please do file SR, we want to fix issues. SR's will be available post migration. The numbers will change.... I read this... Service Requests that existed before the migration already have a unique tracking number. During the migration, these SRs will receive a new SR number with a prefix of 2 (i.e. 2-xxxxxxx). For your reference, the original SR number will still be listed on the Service Request in the ‘Legacy SR’ field. SRs created in My Oracle Support after the migration will have a prefix of 3 (i.e. 3-xxxxxxx) For Oracle On Demand customers, Change Management SRs will be migrated to Requests for Change (RFCs) and will retain the same number. New RFCs created in My Oracle Support will have the format 3-xxxxxx. from Article 841061.1 from metalink.oracle.com (search for article ID by typing the ID into the top right search field AND choosing to search by Article ID from the drop down in front of the field (so you go right to the article)

Posted by richard.miller on October 29, 2009 at 05:26 AM PDT #

6 customers? 12 customers? What percentage of your customers is that? If we rolled out a system for say 500 customers and only consulted with say 1 of them, I wouldn't be surprised that the new system is a complete and utter failure. Try talking to a larger number of customers, not partners, not companies with a vested interest in oracle who need to "tow the line" or else lose lucrative discounts or "free" consulting. Take lets say, 1% of your # of active CSI or people who have used the site in the last 60 days and actively communicate with those customers (you have their email address at least), not the management, but the actual people doing the work, the people who's household, kids braces and general nutrional intake rely on their ability to make the management happy by keeping things running and not looking like a fool at a morning meeting when they repeatedly have to say "I don't know I can't log into oracle support". You would have the best support site on the planet for the next few generations because it would be fast, dedicated to searching and getting issues in and resolved in minutes instead of weeks or months. I have asked this question in the past, post real numbers of the percentage of customers using the configuration manager, and I mean using it, actually getting some information out of it. I have been an Oracle DBA since 1997 and love the product, but thankfully I am capable enough to not have to open many SR's because it is a painful experience dealing with oracle support, in the past it has been the person on the other end of the SR but at least the site worked not too bad. Now it is the person on the other end of the SR AND the site. It is a sad sad day when Microsoft support beats oracle support hands down.

Posted by Jeremy on November 11, 2009 at 12:49 PM PST #

There is a lot of truth in what you say. But we have been doing dozens of one-on-one usabiilty sessions with customers - with the studies we mentioned. Every one of them is a real user, who typically uses MOS every day. I don't talk with parnets, or companies with vested interest, in fact we look for those who like to complain. And we try to encorage folks to not hold back. Having said all of that, there is no point in testing for bugs with customers. if we know we have bugs we should fix them. Right now quality and reliability are the biggest issues. It is not reasonable to even have a discussion around usability when the darn thing is not working well. Totally understand. In the meantime, from a design perspective, I have to think that the product is supposed to work. I can't design thinking it is not going to work. Knowledge performance is key, sure enough it is the larged use of My Oracle Support. But not the only thing folks do. The configuration manager has a variety of customer productivity advantages, which are discussed (even with customer input) on the collector tab. But you seem like the kind of person who wants direct value right now for you. How about this. Do you ever "diff" the config of a DB when it is acting up? If you had the collector installed, you could compare your config from today to say Monday with one click. Thas should be worth something (and recall it is provided as part of your support contract). How about this. Getting priority to your SR (when you do file them) so that it gets triaged and worked before folks who don't have the collector associated with the SR. That should be be worth something (and recall it is provided as part of your support contract). There are other capabilites you might like. Check out the training videos, or try installing the collector on one of your Databases. Just one! AND THEN, we should get together for an hour and hear your complaints directly. Because we REALLY do want to make the best support site possible. I don't want to be compared to Microsoft, I want to be compared to Apple. ;->

Posted by richard.miller on November 12, 2009 at 03:57 AM PST #

Just tried to register an account.."AMAZINGLY ENOUGH, NO WAY TO REGISTER BECAUSE "THE COUNTRY UNITED STATES", WAS LEFT OFF THE REGISTRATION SELECTION LIST", now I have truly seen it all... Was not found at the top of the list or alphabetically...way to go o great Oracle........

Posted by Steve on May 08, 2010 at 10:32 AM PDT #

Funny, it is listed as "USA". Not what I would have expected either. And of course, it is not MOS, but the SSO system which does this. Will try to find a way to fix that... both to use the correct name for our (my) country. And 2) to put it at the top of the list not because we want to be USA centric, but because most of our customers are from the USA and with one little change it reduces selection time quite a bit. Thanks

Posted by richard.miller on July 01, 2010 at 05:13 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
5
6
8
9
10
11
12
13
14
15
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today