I have a psychic connection to my webservers...

I swear I do. I woke up at 4am and couldn't get back to sleep. I'm thinking it's the wind noise making something rattle around outside or some animal. So I get up, get some water, wander around the house, and try to go back to bed. 4:30AM, pager goes off, followed by cell phone. Wife punts me out of bed and I trundle off to answer the phone. www.sun.com is seeing "variable" performance.

That's special. Something you always want to hear, "variable" on the most visible site at Sun. Yippee. I spend the next few minutes logging in, confirming what I've been told, starting IM to talk to the on-call folks in the UK. Yep, it's variable. Seems that with varying traffic levels, we're dropping connections from certain locations in the world.

We've just swapped one of our switches last week, and I'm thinking it might be the new switch or the other one is having issues. I'm a hands-on kinda guy, and so I poke and prod a bit, checking connectivity from several locations (I run a few personal websites). I finally call my lead network engineer, and he switches over to the backup switch. Everything is resolved, for now. Of course, the predawn light hits my eyes and now I'm up for good. Another early start to my day.

The real kicker here is that we've built out a new network infrastructure that has significantly more capacity and performance. We started moving machines already, but as the site is live, you have to move things in a very studied fashion. Slowly and carefully and with lots of testing. The old network is running some network gear that is old and starting to feel it's age. I'm now \*very\* motivated to move stuff to the new network faster...

Comments:

Post a Comment:
Comments are closed for this entry.
About

I run the engineering group responsible for Sun.com and the high volume websites at Sun.

Will Snow
Sr. Engineering Director

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today