Busy day today

It's been a busy day today, OpenSolaris was announced and the downloads started coming in fast and furious. It overwhelmed our bandwidth locally, so we shifted our traffic to one of our CDN providers. I'm happy that we've built into most of our systems the ability to shift traffic dynamically. It allows me to balance budget vs. performance in near real-time. Ok, it's still not where I want it (where I can just tweak some knobs and have things magically work), but it's really dang close. 

Rama asked me after we fixed the problem "How did you know it was bandwidth?", to which I replied - "Experience". Ok, it was experience and paranoia - I watched the server dashboard light up (in a bad way) and I was thinking "gee, I wonder if OpenSolaris is going to get hit hard this AM..." Checked the network bandwidth graphs and sure enough, we were seeing some ugly flat spots, way up at 961mb/s. Yep, nearly a gigabit/second was being driven through the infrastructure.

Good news is the firewalls, load balancers, et. al. were doing fine; just couldn't go any faster. So Joe shifted some traffic to our CDNs and presto, we're back in operating territory.



 

Comments:

almost posted a similar entry some time ago (netbeans6 downloads or something), but Ops wanted me to sanitize the graph so much as to render it uninteresting. guess the fear was someone could exploit this information. i decided to pick my battles and left it unposted...

Posted by rama on May 05, 2008 at 08:12 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

I run the engineering group responsible for Sun.com and the high volume websites at Sun.

Will Snow
Sr. Engineering Director

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today