Supporting our Customers

The last Month was not the best on in my life.  The 24th of February my mother in law died at the age of 73 caused by cancer.   Because we are a close family and the speed of this tragic incident (one week) it had a huge impact on my creativity, you start to focus on important things and you learn to relativize quickly.

One important thing I learned is that the service in the hospital is on very high level and a good place to learn where WE can improve.   I would like to share some observations, there is a learning in everything, even in the sad moments in life...

Communication is key.  Starting with the top. The specialists where very good in (frequently) explaining what happened, the approach they take and the showing options (will she survive?) of the treatment.   This approach gives a lot of 'piece of mind' to the relatives. Now, 3 weeks later we do not have the feeling 'if we would have done this then....' .
The medial people on the floor (nurses).  They where great in executing the approach suggested by the specialists and where key player in the process by regular checks, measurements (blood/heartbeat/temperature/... etc) and feedback to all parties.  They also took ownership and showed initiatives to proactively inform family and medical staff but they where also professional enough to keep distance and point to the specialists for more detailed information.
Finally the overall 'atmosphere' in the hospital.  Although you 'know' this is a routine, the staff was able to give the patient and their family the feeling that they really care about you. From the moment we entered the hospital till we left it for last time .

Oke, now the IT world.   We seem to promote as much self service as possible.  When you have a real problem that is not tackled in the online documentation, please log a service request and use the reference number in the rest of the conversation.  When you have a real problem, call use, and we might even decide to escalate. You might even get an actionplan (when you are lucky) .  However realize that we have more customers with even more severe problems so be patient.... 

I agree this is a bit exaggerated but it makes my point clear... WE can do much better when we are willing to raise the bar. IT community, lets take better care of our (critical) customers.  Make sure there are specialists who can provide and communicate a clear action plan. Ensure that the support organization follows the action plan with the customer. Perform regular checks and measurment to track progress. Stay in contact with the stakeholders, even in times when they are less responsive because they have other things on their minds. Last but not least, create the atmosphere that YOU CARE !! even if this is escalation #10 of the week.....

Hans

Last week one of our customers faced a complex escalation where a large (2+ TB) database did not successfully completed the migration. I will not go into the details but the problem was that the data-header blocks where not consistent with the data in the blocks - in layman's terms "Houston we have a problem".
As a result - no access and.. the key business application down...   in Support terms a REAL SEVERITY 1...  in real life an emergency call Monday morning at 6.45 AM.

Our usual reflex is to assign a crisis manager who starts to engage with all parties involved (customer/partner-implementer/support/development). Starting point is to triage the situation, assign ownership and manage the process which in fact means communicate-communicate-communicate and let the techies do their work.

During the day we had a lot if discussions with the customer about all aspects of this project.   What is your plan?, how is the contingency  setup ?. Is there a fallback ? and did you see this problem during testing ?.   All interesting areas to discuss in this blog but lets focus on the testing

A review of the steps revealed that they spend a lot of time in this area. A team of specialists spend many hours in setting-up and preparing the test, covering all areas of the migration.  The team performed the planned data migration using an identical data set on an identical operating system using the cookbook designed for the migration weekend.

So... the 1M dollar question was - "Why did they not found out about the data-corruption in the test".  I can tell you there where a lot of speculations about going on during the day.

During the day we got the answer, simple and a great learning point; "We tested the migration but did not run the application accessing the data. This was considered as a separate project and was planned in a later stage....by a different team"

So, project leaders, please keep this learning in mind when planning migration.  Don't let a limited scope of the project limit your thinking about testing.
 

Hans


..and how about the problem...
-Repeating the migration would take 7 days
-Fallback would delay the project 3 months, resulting in huge costs and loss in confidence.
-Using DUL would be an option
-The Oracle team decided to manually detect the wrong blocks and patch the manually .. 12.00PM the same day the 2+ TB Database was up and running and the customer went into production as planned (after running the dataccess tests)