Diversity in your connections...
By relling on Jun 07, 2004
Cingular is building a cell phone tower on the ranch. This view from the front porch of the construction reminds me that diversity in your connections is, and will always be, very important for maintaining a highly available system. The pole on the right next to the backhoe's bucket supplies all of the power and telephone to my house. If the bucket were to move about a foot to the right, it could knock out the power and phone service to my house. Given the nature of this sort of failure, recovery could take many hours, and ruin my day.
I use diversity to overcome this single point of failure to my house. For power, I've got a generator and generally use laptop computers with batteries. In a pinch, I could connect some solar power panels and charge controllers to run off-the-power-grid. For telephone service, I have a cellular phone (not Cingular, since they don't have service in my area, yet) with fairly good coverage on the ranch. For internet, I have a wireless connection to Thunder Mountain Wireless, a local wireless service provider run by my friend Terry.
As you build your highly available service, you should also consider these things. A backhoe many miles away could cut your power and telecom service. If you do not have diverse supplies for these crucial services, then your system is at risk. In large, metropolitan areas it may be very difficult to achieve diverse supplies for power and telecom services. But when you need it, you need it fast. Plan ahead to avoid this pitfall.
It is worth noting that this cell site is also susceptible to this same fault. The backhoe is digging the trench to supply power and telephone to the cell site. Generally, the site has batteries and perhaps a generator, but the connection to the telecom network is via the wired phone system (T1). If we fast-forward a few years and dig again, then we could put the site out of commission.
A few years ago I posted a similar experience I experienced in Central Florida where a developer was putting in a subdivision and the backhoe cut the power. In this specific datacenter, everything was functional except the automatic transfer switch (ATS) which was supposed to change the data center power over to the batteries and generators. The ATS had not been tested in several years and did indeed fail. Remember, an ATS is a single point of failure. My colleagues in Europe asked me what a "backhoe" was. I think this picture, and its direct application to my ranch, should drive the point home.