Busy day today
By was on May 05, 2008
It's been a busy day today, OpenSolaris was announced and the downloads started coming in fast and furious. It overwhelmed our bandwidth locally, so we shifted our traffic to one of our CDN providers. I'm happy that we've built into most of our systems the ability to shift traffic dynamically. It allows me to balance budget vs. performance in near real-time. Ok, it's still not where I want it (where I can just tweak some knobs and have things magically work), but it's really dang close.
Rama asked me after we fixed the problem "How did you know it was bandwidth?", to which I replied - "Experience". Ok, it was experience and paranoia - I watched the server dashboard light up (in a bad way) and I was thinking "gee, I wonder if OpenSolaris is going to get hit hard this AM..." Checked the network bandwidth graphs and sure enough, we were seeing some ugly flat spots, way up at 961mb/s. Yep, nearly a gigabit/second was being driven through the infrastructure.
Good news is the firewalls, load balancers, et. al. were doing fine; just couldn't go any faster. So Joe shifted some traffic to our CDNs and presto, we're back in operating territory.