Shaw downtown on fire?

[dannie]

Just the NOC

[Cos]

.

[Disoblige]

This sucks! No internet or cable as I live downtown. Never thought I'd be so happy I have Telus LTE on my phone.

[eblend]

My old empoloyer ERCB (quit last friday) is right across the shaw building and after many years trying to retire the mainframe moved their mainframe services to that shaw building, so speaking with some of my old co-workers, their mainframe is down. I bet this would be an all hands on board situation in many IT orgs affected, so I am sure I would have been pulled in to this as well. My new employer, while still gov doesn't use the mainframe so we are all peachy Guess i took a good time to leave. Surprised their site is down though, would assume they would have some offsite provider at least be able to host an outage page

[codetrap]

Originally posted by dannie
Keep in mind, Registry services and Alberta Health Care is IBM based too.... All registries are unable to process at this time....

That's interesting, because the registries network homes into servers that are located in the BP building here in Calgary, and to my knowledge, those were unaffected.

[Oz-]

Signal all the Business Continuity specialists.

[cam_wmh]

[narou]

Hopefully they get something up for ahs. I got work to do.

[Cos]

.

[hedge]

I can't believe some of these places don't have a DR plan in place, we'd be back up in about 5 minutes.

[G-ZUS]

Originally posted by narou
Hopefully they get something up for ahs. I got work to do.

[TurboMedic]

Originally posted by 1-Bar
Fun times!! Computers in amboolances are functional. Just email and info pushes to electronic paperwork is down.

Yah, that was helpful today........

[rage2]

Originally posted by hedge
I can't believe some of these places don't have a DR plan in place, we'd be back up in about 5 minutes.

I think the biggest issue is that most companies, city/provincial government is just relying on the Shaw datacenter SLA to make sure services are up, which is the biggest mistake. SLA's are useless, in extended outages such as this, you get refunded some of your fees, which is tiny compared to the $ lost for your business/government.

It's not up to the datacenter/colocation to provide datacenter redundancy. Datacenters go down, even Amazon EC2 out east went down for hours last weekend from intense storms which took out Instagram and Netflix.

The servers/software architecture that's being hosted has to be designed to work in a failover site scenario. Most software isn't designed for this and is impossible to make it failover to an offsite datacenter reliably without dataloss or corruption. If the city was expecting 100% uptime even with a datacenter loss, then that's an oversight right there.

Fact is, as I mentioned earlier, datacenter loss is rare, and when you present a cost/risk/benefit analysis for an offsite mirrored backup to management, it makes very little sense and will rarely get approval at most companies. A SAN is expensive. SAN replication over WAN is retarded. And that's just 1 small piece of the puzzle in a multi-site environment.

With that being said, some of the critical services at the government level should have been designed and hosted in a multi-site environment from the start. I wouldn't fully blame Shaw, I would blame the people responsible for the systems that are down right now.

[hampstor]

Originally posted by rage2

I deal with a lot of DR stuff at work, and a datacenter like Shaw's would have tons of redundancy built in at every level, except for physical location. Meaning if they host a website for you, don't expect them to have a second datacenter with your website mirrored and ready to go.

Datacenter explosions like this is pretty rare. Having a backup datacenter for redundancy is *very* expensive for most companies to even consider, so a worst case scenario like this usually isn't covered by most companies. It's a very high cost/low risk scenario.

What I'm surprised at though, is that the City of Calgary, and Alberta Health Services does not have redundant datacenters that they can kick in if the primary DC goes down. If anything, critical services like that should be housed in multiple locations. Same goes for some of Shaw's critical infrastructure, ie home phones, and even their website. The city's 911 service isn't hosted with Shaw, I'm curious now if they have redundant backup datacenter in case their primary blew up.

As it's an IBM colo site, there would be a fair number of IBM mainframes there. AHS and the City of Calgary were big IBM mainframe users...

Over the last year, I discovered many organizations that still operate mainframes do not have a DR site for the mainframes because they can't afford to. I would imagine that any systems that still rely on the mainframes are actually what is down right now. Would make sense since the provinces' registry system is very old.

I'm just guessing of course... I was recently working closely with a team on an IBM mainframe rehost project and this is the first thing that came to my mind when I saw the list of customers down and this being an IBM site.

[rage2]

Originally posted by hampstor
As it's an IBM colo site, there would be a fair number of IBM mainframes there. AHS and the City of Calgary were big IBM mainframe users...

Over the last year, I discovered many organizations that still operate mainframes do not have a DR site for the mainframes because they can't afford to. I would imagine that any systems that still rely on the mainframes are actually what is down right now. Would make sense since the provinces' registry system is very old.

Not only is it expensive, it's sometimes impossible from an architectural point of view with these legacy systems.

Lucky for us, VMWare has made things a lot easier to do multi-site redundancy for software that wasn't designed for it. Taking the software architectural costs out of the equation is a huge money saver.

[dannie]

Originally posted by hampstor


As it's an IBM colo site, there would be a fair number of IBM mainframes there. AHS and the City of Calgary were big IBM mainframe users...

Over the last year, I discovered many organizations that still operate mainframes do not have a DR site for the mainframes because they can't afford to. I would imagine that any systems that still rely on the mainframes are actually what is down right now. Would make sense since the provinces' registry system is very old.

I'm just guessing of course... I was recently working closely with a team on an IBM mainframe rehost project and this is the first thing that came to my mind when I saw the list of customers down and this being an IBM site.

You got it. Registries use the IBM mainframe at that site. What surprised us was that registries went down a solid 45 minutes before AHS did. I don't know enough about the systems to go into further detail or why one system took longer to fail. But as of right now, it's still down and there is no current ETA for a fix.

[1-Bar]

Originally posted by TurboMedic


Yah, that was helpful today........

Haha you were on day shift i presume?!?

[M.alex]

1.40am ... I could connect back to my work VPN. Yay

[sputnik]

Originally posted by ercchry
with all the water-free fire suppression systems out there for server rooms... they had sprinklers?!

Sprinkers are actually the choice of monitored data centers.

With fire supression like Inergen everyone has to evacuate at even the smallest puff of smoke that sets off the system. With sprinklers you can fight the fire with an extinguisher and the only the sprinker above the fire gets triggered.

Go to almost any high end data center (Q9, Shaw, Telus etc) and you will see sprinklers. Most will only have Inergen for if the fire is too big to handle and those systems are only manually engaged.

[Zhariak]

Originally posted by rage2

Not only is it expensive, it's sometimes impossible from an architectural point of view with these legacy systems.

Lucky for us, VMWare has made things a lot easier to do multi-site redundancy for software that wasn't designed for it. Taking the software architectural costs out of the equation is a huge money saver.

Vmware HA and DRS for the win yooooo!!!!

Fire? What fire???