Tuesday's Site Outage

[webmaster]

What a way to start the new year! At 8:12 am today the data center that hosts my server had a major failure that caused 6 hours and 40 minutes of downtime. Sorry for the inconvenience.

For all you techies out there here is the most recent status report on the failure:

The WALTHAM, MA facility is experiencing a complete outage. We have updated information confirming that both the primary and secondary redundant fiber paths went down. The primary fiber uplink went down earlier this weekend, and before engineers could restore it to full operation, the secondary fiber link went down this morning at about 8:12 AM EST. Savvis does have engineers working on the issue with the highest priority. This is affecting ALL customers of Savvis in their Boston 1 and Boston 2 datacenters.

The site does still seem to be a little slow but that should clear as the data center is brought back to full capacity.

Thanks for your patience.

[Skip]

....for keeping us posted.

I noticed a few of my other common sites were experiencing difficulties also...the biggie being the Manchester Union Leader site....now you have me curious as to if they are utilizing the same site!

Thanks again and Happy New Year,

Skip

EDIT: Wow, just checked in at the Union Leader and at the top of the fold had the same explanation as yours! Wow, you really do run with the big fish now!!!

[webmaster]

Originally Posted by Skip

Wow, just checked in at the Union Leader and at the top of the fold had the same explanation as yours! Wow, you really do run with the big fish now!!!

Savvis is a huge data center and this outage must have affected thousands of customers including banks, government and media sites, like the Union Leader.

While the outage was a minor inconvenience for us, the big fish must have been going nuts. Fortunately a simultaneous failure of redundant systems is very rare and unusual.

[RI Swamp Yankee]

I was a bit concerned when I saw even the DNS servers were down and could not be reached. Glad it is back up. Seems to be at full speed.

[ghfromaltonbay]

I sent an email last night to DRH, Pepper, IG and Island Sib. This morning when I logged on I had an email message telling me that the delivery was delayed to DRH and Pepper. Both have cable modems from Metrocast. Then I couldn't access the Union Leader or this website. I did manage to pull up Foster's Daily Democrat which must go thru a different server in New England. Now as the Pink Panther would say..."the mystery is solv-ed."

[Steve]

Originally Posted by webmaster

Fortunately a simultaneous failure of redundant systems is very rare and unusual.

You know Don, I wonder about this though. The issues had to do with routers and circuits today. Seems to me that they weren't very redundant. I'll be curious to know more, but it doesn't seem like everything failed over like it should, which makes me question their policies and procedures.

[webmaster]

Originally Posted by Steve

You know Don, I wonder about this though. The issues had to do with routers and circuits today. Seems to me that they weren't very redundant. I'll be curious to know more, but it doesn't seem like everything failed over like it should, which makes me question their policies and procedures.

They certainly could have used triple redundancy in this case but you make a good point Steve. If I was running a critical business server at SAVVIS I think I'd be asking some tough questions and considering my alternatives. For the travel & tourism sites that run on my server the downtime was more of an inconvenience than a crisis and I wouldn't move the server unless this becomes a regular event.

I'm sure that in response to concerned (irate?) customers they will soon have a plan to make sure this can never happen again.

[RI Swamp Yankee]

It seems the problem wasn't directly with any servers or routers at the SAVIS location, it was the fiber links from them to the world. They need to take a hard look at their transport providers.

[Steve]

I believe they did have internal router issues as well. Not 100% sure though if it was related to failover, or a perfect storm of events.

[Blue Thunder]

I have heard and read about several fiber optic cable failures here in Eastern Mass that started last Friday. The one in Ipswich MA has been worked on 24/7 since last Friday. Verizon is in the process of replacing several thousand feet of Fiber Optic cable. I read about some other towns with similar issues over the weekend. Is there any way they could be related, ie; over loading of existing circuits when they transfer the circuits off to another line while they replace the failed one??? Seems a little coincidental to me.

Blue Thunder

[webmaster]

Lycos was down too: http://computerworld.com/action/arti...intsrc=hm_list

[Waterbaby]

another site i frequent, nascar.com, also had the same problem..... it was down from about 8 this morning until about 2-something this afternoon........ almost sounds like what we all expected for Y2K! delayed reaction, possibly?

[Weirs guy]

Oh, the site was down? Thank god, I thought I was still drunk from New Years!!