Disaster Averted (aka Attack of the POD people!)
By user12601629 on Nov 06, 2006
Have you ever had one of those jobs that you knew was really important, but no one else seemed to know it? Or, have you ever had one of those jobs where no one seemed to notice all the great work you've been doing until something goes terribly wrong?
Inside the Connected Systems Network there is a quiet, little piece of infrastructure called Patch Operations and Delivery (POD for short). POD is the key part of the pipeline that internal engineers at Sun use to push out patches and updates to products like Solaris and the Java Enterprise System. It isn't directly visible to customers, but it's the back-end of important services like Update Connection and SunSolve. If POD were to suddenly go away people would notice!
that's just what happened recently. POD had a whopper of a melt
down. In one of those classic IT mishaps, a sleeply little
UltraSPARC-II system in a non-production lab had crept into the transaction flow for the production
database. It had been running fine for years, but then there was failure in the storage array attached to the
Without going into all the details of what transpired (there were repair scripts written, data pulled from mirrors across the planet, and people working through the night and over the weekend) I'm happy to report that POD is back to full strength. There were some serious heroics involved in this incredibly delicate fix. Here's a quick list of people that deserve a real thank you.
- Jan Birkelund
- Slim Heilpern
- Darcy O'Connor
- Simon Ip
- Philippe Nave
- Derk Norton
- Mike Tanaka
- Janet Bacon
- Don Gritzmacher
- Darl Kuhn
In honor of you brave warriors of software, I give you Wired Magazine's List of the Coolest Movie Weapons. These may come in handy next time you have to fight a problem this big!
Thanks Again Gang!