Shaw downtown on fire?

[frizzlefry]

Originally posted by Xtrema


Just because its highlighted doesn't mean it will be acted on.

You will NEVER have a 100% redundant. The cost raises exponentially for every 9 you put behind 99%

Yup, but you can at least choose a better datacenter. Q9 ain't cheap but they have clean agent fire suppression, pre-emptive air monitoring for smoke and you are not allowed to bring any combustable material inside (like cardboard). Plus no office workers sharing the space. I would think a bank would have chosen a better location.

[Mibz]

I was under the impression that sprinklers/fire suppression didn't go off in the actual data center, but that the generator/AC unit explosion caused sprinklers to go off on the 7th floor which then forced the power off in the data center.

This was a freak accident (provided whatever exploded was maintained properly) and could have happened just as easily at Q9 IMO.

EDIT: I also imagine a lot of stuff in the Shaw DC (and other shitty and/or old DCs) was in there before Q9 even came to Calgary.

[frizzlefry]

Originally posted by Mibz
I was under the impression that sprinklers/fire suppression didn't go off in the actual data center, but that the generator/AC unit explosion caused sprinklers to go off on the 7th floor which then forced the power off in the data center.

This was a freak accident (provided whatever exploded was maintained properly) and could have happened just as easily at Q9 IMO.

EDIT: I also imagine a lot of stuff in the Shaw DC (and other shitty and/or old DCs) was in there before Q9 even came to Calgary.

Could not happen at Q9, no water to take out power. There is always a chance that whatever causes an electrical failure will bring fire with it. So lets put in sprinklers! Bare minimum they could have used a dry-pipe water system but, because people work there, they may not have been able to. Plus, having mission critical servers in a place with a 13th floor is just asking for it

[Mibz]

I should have been clearer and said that "A freak accident causing a similar outage" could have happened at Q9, not this specific scenario.

LOL @ 13th floor :P

[rage2]

Originally posted by frizzlefry
Could not happen at Q9

I'm sure the customers at Shaw had the exact same thoughts before their datacenter blew up on Wednesday haha.

Fact is, any datacenter, throw any name behind it, can and will go down unexpectedly. Difference is, Q9 had some great marketing back in the day and everyone thinks they're the best. I did a tour of their facilities, it's nice, but it's no different than any other top tier DC.

[ipeefreely]

Originally posted by dexlargo
^Yeah, our mainframe application is up and working as of yesterday afternoon, but the motor vehicle registry is still down. Hopefully it'll be up today, but I have only heard that we should get an update this morning.

Got this notice around noon:

- Current best estimate is Monday, July 16 for the resumption of Registry services and GOA application across multiple ministries.

- Disaster recovery to another facility is in progress.

[Xtrema]

Originally posted by frizzlefry


Yup, but you can at least choose a better datacenter. Q9 ain't cheap but they have clean agent fire suppression, pre-emptive air monitoring for smoke and you are not allowed to bring any combustable material inside (like cardboard). Plus no office workers sharing the space. I would think a bank would have chosen a better location.

Trust me. You can't account for everything and hindsight is 20/20.

And management type loves SLAs as a cheap way to cover their asses. "Well the <insert vendor> said it's covered"

And unless the infrastructure is designed to be highly available over multiple sites, most DRP will take 3 to 4 days to complete after at least a day or 2 after incident to assess the need to go DRP. Going DR isn't fun because you probably need another outage to bring data back to the primary site.

[revelations]

Originally posted by LOLzilla
This is what organizations get for placing all their eggs in one basket. It is no fault of Shaw's that organizations using their services did not have a redundant site. AHS? ATB? This should have/would have been high lighted in their risk assessments.

Yeah, I heard about ATB yesterday. Thats a BIG fuckup.

[rage2]

Originally posted by revelations
Yeah, I heard about ATB yesterday. Thats a BIG fuckup.

The US government and banks that we deal with, they won't even talk to us unless we have multiple sites in different geographical locations for disaster recovery. And that's for managing their non bank related data.

[dexlargo]

Originally posted by ipeefreely


Got this notice around noon:

- Current best estimate is Monday, July 16 for the resumption of Registry services and GOA application across multiple ministries.

- Disaster recovery to another facility is in progress.

The motor vehicles application just came up - not sure if registry agents are being allowed to connect yet.

[U4RIA]

Originally posted by dexlargo
The motor vehicles application just came up - not sure if registry agents are being allowed to connect yet.

I called three different registry offices an hour ago, all three told me they anticipate to be back up and running on Monday.

So... now there's a chance it could be today?

[dexlargo]

Originally posted by U4RIA


I called three different registry offices an hour ago, all three told me they anticipate to be back up and running on Monday.

So... now there's a chance it could be today?

I don't know. All I can tell you is the part of that system that I can see is up and running. I don't have the same access that registry agents have and don't know if they're being allowed to access it yet. If they're being told Monday, that's probably when they'll get access.

[frizzlefry]

Originally posted by Xtrema


Trust me. You can't account for everything and hindsight is 20/20.

And management type loves SLAs as a cheap way to cover their asses. &quot;Well the &lt;insert vendor&gt; said it's covered&quot;

And unless the infrastructure is designed to be highly available over multiple sites, most DRP will take 3 to 4 days to complete after at least a day or 2 after incident to assess the need to go DRP. Going DR isn't fun because you probably need another outage to bring data back to the primary site.

You sure can't account for everything. And this shaw incident sure has me rethinking our DR. But I would never call a facility that has a water based fire suppression system a proper datacenter. The very thing there to protect your systems from fire will take them down. That would be a deal breaker right of the bat for me if I were choosing a colocation facility.

[LOLzilla]

Originally posted by Xtrema


Just because its highlighted doesn't mean it will be acted on.

You will NEVER have a 100% redundant. The cost raises exponentially for every 9 you put behind 99%

I completely understand this but when it comes to critical business functions I cannot see there being a really solid excuse. The days of 'it costs too much money' seems negligent to me. How much money did the affected businesses lose due to this?

[ipeefreely]

Originally posted by dexlargo
The motor vehicles application just came up - not sure if registry agents are being allowed to connect yet.

Our HR application (employee time sheet, job info etc...) was available at around 2pm today.

They said you could enter your time sheet. So they must be making progress on all the applications (and are confident that new data can be entered)...

[Mibz]

I thought we went over this, there weren't water-based sprinklers in the actual data center, only on the same floor. But them being triggered also caused the power to be cut. [citation needed] The design flaw here isn't servers getting wet, it's that a small fire in one corner of the floor takes out the power in the DC.

That said, there's the possibility that the exploded generator was supposed to supply power to the DC in the event of a 7th floor fire and it was just a REALLY unfortunate circumstance.

We don't know these things, we have no idea. The fact that I'm even here speculating despite a clear lack of details upsets me.

Originally posted by LOLzilla
I completely understand this but when it comes to critical business functions I cannot see there being a really solid excuse. The days of 'it costs too much money' seems negligent to me. How much money did the affected businesses lose due to this?

As with everything else it comes down to risk. Sure, the damage of this failure was huge, but the likelihood was negligible. You try convincing an executive to double your budget for an astronomically low chance.

As was mentioned earlier, every 9 you add on to that uptime exponentially increases cost and to say "Redundancy at any cost" is more negligent than assessing the risk for that specific service for that specific company and budgeting accordingly.

[rage2]

Originally posted by frizzlefry
You sure can't account for everything. And this shaw incident sure has me rethinking our DR. But I would never call a facility that has a water based fire suppression system a proper datacenter. The very thing there to protect your systems from fire will take them down. That would be a deal breaker right of the bat for me if I were choosing a colocation facility.

From what I've been told, the datacenter isn't on sprinklers. The water was from firefighting efforts and sprinklers elsewhere above the datacenter, which worked its way down to the 7th floor.

[dragonone]

Originally posted by rage2

The US government and banks that we deal with, they won't even talk to us unless we have multiple sites in different geographical locations for disaster recovery. And that's for managing their non bank related data.

For businesses, should they have the same requirements if they expect to be up and running 24/7? I don't have a full grasp of how their backup works but for a city to be that dependent upon Shaw's services, it seems odd that the contingency plan is not more clear for the media or its customers.

[Xtrema]

Originally posted by LOLzilla
How much money did the affected businesses lose due to this?

#1 As we have beat it to death, you can't blame Shaw if customer is too cheap to have proper design. For example, some businesses buy desktop towers rebadged servers and say the 4hr response from HP is good enough. And when shit hit the fan, they are out for 4hrs and parts may be here next day. They are fucked. Same mindset will affect DRP design.

#2 Who makes the call on what outage cost is? Some business may not be much. Some may be regulated and have huge government fines. Some like ATB and AHS just say fuck it we will take a chance on civil lawsuits which may be cheaper than proper DRP. DRP budget is always a guesstimate and perceived risk. ie. How much does Shaw value a tarnish reputation (justified or not) over this incident to put in more redundant circuits?


Just like us buying car insurance. If it wasn't the law, none of us will spend on car insurance.

[rage2]

Here's an article with some interesting details.

http://www.calgaryherald.com/busines...459/story.html

IBM Canada, the province’s contractor whose Shaw Court data centre remained powerless most of Thursday, had to fly the analogue tapes that stored all Alberta’s vehicle and property registration data to a backup facility in Markham, Ont., and carefully feed it all through new servers.

Some systems had “mirror” backups and can be up by Friday, but the registries and other systems will take up to 72 hours to restore.

“For other systems that run on tapes, that are older systems, those take longer to get back online and we know that,” Bhullar told reporters by teleconference.

I think IBM dropped the ball on this one in relations to the AB government apps and services. It's outsourced to them, and if they have to fly tapes to Ontario to get it working, well, that's a terrible DRP.