Cingular is building a cell phone tower
on the ranch. This view from the front
porch of the construction reminds me that diversity in your connections is, and
will always be, very important for maintaining a highly available system. The
pole on the right next to the backhoe's bucket supplies all of the power and
telephone to my house. If the bucket were to move about a foot to the right, it
could knock out the power and phone service to my house. Given the nature of this
sort of failure, recovery could take many hours, and ruin my day.
I use diversity to overcome this single point of failure to my house. For power, I've
got a generator and generally use laptop computers with batteries. In a pinch, I could
connect some solar power panels and charge controllers to run off-the-power-grid.
For telephone service, I have a cellular phone (not Cingular, since they don't have
service in my area, yet) with fairly good coverage on the ranch. For internet, I have
a wireless connection to Thunder Mountain Wireless, a local wireless service provider run by my friend Terry.
As you build your highly available service, you should also consider these things.
A backhoe many miles away could cut your power and telecom service. If you do not
have diverse supplies for these crucial services, then your system is at risk. In
large, metropolitan areas it may be very difficult to achieve diverse supplies for
power and telecom services. But when you need it, you need it fast. Plan ahead
to avoid this pitfall.
It is worth noting that this cell site is also susceptible to this same fault.
The backhoe is digging the trench to supply power and telephone to the cell site.
Generally, the site has batteries and perhaps a generator, but the connection
to the telecom network is via the wired phone system (T1). If we fast-forward a
few years and dig again, then we could put the site out of commission.
A few years ago I posted a similar experience I experienced in Central Florida
where a developer was putting in a subdivision and the backhoe cut the power. In
this specific datacenter, everything was functional except the automatic transfer
switch (ATS) which was supposed to change the data center power over to the
batteries and generators. The ATS had not been tested in several years and did
indeed fail. Remember, an ATS is a single point of failure. My colleagues in
Europe asked me what a "backhoe" was. I think this picture, and its direct
application to my ranch, should drive the point home.