Tuesday's Site Outage

Steve · 01-02-2007, 05:41 PM

Quote:

Originally Posted by webmaster

Fortunately a simultaneous failure of redundant systems is very rare and unusual.

You know Don, I wonder about this though. The issues had to do with routers and circuits today. Seems to me that they weren't very redundant. I'll be curious to know more, but it doesn't seem like everything failed over like it should, which makes me question their policies and procedures.

webmaster · 01-02-2007, 10:39 PM

Quote:

Originally Posted by Steve

You know Don, I wonder about this though. The issues had to do with routers and circuits today. Seems to me that they weren't very redundant. I'll be curious to know more, but it doesn't seem like everything failed over like it should, which makes me question their policies and procedures.

They certainly could have used triple redundancy in this case but you make a good point Steve. If I was running a critical business server at SAVVIS I think I'd be asking some tough questions and considering my alternatives. For the travel & tourism sites that run on my server the downtime was more of an inconvenience than a crisis and I wouldn't move the server unless this becomes a regular event.

I'm sure that in response to concerned (irate?) customers they will soon have a plan to make sure this can never happen again.

RI Swamp Yankee · 01-02-2007, 11:43 PM

It seems the problem wasn't directly with any servers or routers at the SAVIS location, it was the fiber links from them to the world. They need to take a hard look at their transport providers.

Steve · 01-03-2007, 01:08 AM

I believe they did have internal router issues as well. Not 100% sure though if it was related to failover, or a perfect storm of events.

Blue Thunder · 01-03-2007, 07:55 AM

I have heard and read about several fiber optic cable failures here in Eastern Mass that started last Friday. The one in Ipswich MA has been worked on 24/7 since last Friday. Verizon is in the process of replacing several thousand feet of Fiber Optic cable. I read about some other towns with similar issues over the weekend. Is there any way they could be related, ie; over loading of existing circuits when they transfer the circuits off to another line while they replace the failed one??? Seems a little coincidental to me.

Blue Thunder

webmaster · 01-03-2007, 01:21 PM

Lycos was down too: http://computerworld.com/action/arti...intsrc=hm_list

01-02-2007, 11:43 PM	#3
RI Swamp Yankee Senior Member Join Date: Jul 2002 Location: North Kingstown RI Posts: 688 Thanks: 143 Thanked 83 Times in 55 Posts	It seems the problem wasn't directly with any servers or routers at the SAVIS location, it was the fiber links from them to the world. They need to take a hard look at their transport providers. __________________ Gene ~ aka "another RI Swamp Yankee"

01-03-2007, 07:55 AM	#5
Blue Thunder Senior Member Join Date: Apr 2004 Location: Eastern MA & Frye Island/Sebago Lake, Maine Posts: 955 Thanks: 256 Thanked 351 Times in 158 Posts	Could this be a larger problem? I have heard and read about several fiber optic cable failures here in Eastern Mass that started last Friday. The one in Ipswich MA has been worked on 24/7 since last Friday. Verizon is in the process of replacing several thousand feet of Fiber Optic cable. I read about some other towns with similar issues over the weekend. Is there any way they could be related, ie; over loading of existing circuits when they transfer the circuits off to another line while they replace the failed one??? Seems a little coincidental to me. Blue Thunder __________________ " Live for today because yesterday is gone and tomorrow may never come"

01-03-2007, 01:21 PM	#6
webmaster Moderator Join Date: Jun 2002 Posts: 2,476 Blog Entries: 1 Thanks: 463 Thanked 3,985 Times in 854 Posts	the magnitude of yesterday's outage Lycos was down too: http://computerworld.com/action/arti...intsrc=hm_list

01-03-2007, 01:08 AM	#4
Steve Senior Member Join Date: Apr 2004 Location: Southern NH Posts: 109 Thanks: 0 Thanked 0 Times in 0 Posts	I believe they did have internal router issues as well. Not 100% sure though if it was related to failover, or a perfect storm of events.