Shaw downtown on fire?

[Alterac]

Originally posted by sputnik

Just make sure you have diverse paths for those fibre connections or a backhoe could take you out in a matter of seconds.

Q9 Guarantees 100% Uptime on their internet to your Rack. (Lol 100%. Oh Well, its in Writing)

If we were bringing in redundant loops to our downtown office, we would have to get enmax to trench in physical diversity to the building.. so it was cheaper to relocate to Q9.


With multiple Nexus 7000's, and vBlocks (Cisco Networking, Cisco Blade Servers, VMWare, EMC San, all in 1, under 1 support vendor), with WAN SAN Replication (LOL Money Money Money Money!), we can have our critical stuff back up in minutes.

[Mibz]

Originally posted by Alterac
With multiple Nexus 7000's, and vBlocks (Cisco Networking, Cisco Blade Servers, VMWare, EMC San, all in 1, under 1 support vendor), with WAN SAN Replication (LOL Money Money Money Money!), we can have our critical stuff back up in minutes.

Are you hiring?

[r3ccOs]

Originally posted by Alterac



Q9 Guarantees 100% Uptime on their internet to your Rack. (Lol 100%. Oh Well, its in Writing)

If we were bringing in redundant loops to our downtown office, we would have to get enmax to trench in physical diversity to the building.. so it was cheaper to relocate to Q9.


With multiple Nexus 7000's, and vBlocks (Cisco Networking, Cisco Blade Servers, VMWare, EMC San, all in 1, under 1 support vendor), with WAN SAN Replication (LOL Money Money Money Money!), we can have our critical stuff back up in minutes.

talking to my former collegues who are now at IBM, BP, Suncor and ATB, they are awaiting a formal report to announce what the condition of the DC is, and IBM flew in a few BCP/DR specialist and post disaster analysts from Markham to evaluate what is possibly salvagable...

there were some expensives pieces of kit in there, from some z9 mainframes, IBM P5/P6 595 AIX chassis hosting LPARs for various critical services for their customers (SAP app servers/websphere/ECM/DB2 and its instance) to appliances ranging from F5 loadbalancers, wan scalers, VPN devices, Checkpoint/Cisco RSWs & FWs

on a few DR projects I was on in the past, we always took IBM's recommendation of running HADR on DB2 and utalizing GPFS for our DR sites to have an active file mount and parterning with HDS for TC for their SAN replication on fiber channel (always via dark fiber using MCdata fabric directors and Brocades FC controllers)

however hadr didn't quite work so well for a couple databases, and we've had to restore from tape and are rebuilding from transaction logs :P

even in a perfect world, with various test condtions ran against a highly resilient environment, there are always unpredictable variables you can never account for.

[Skyline_Addict]

Yup. All business critical systems are still down at ATB Financial.

[LollerBrader]

Originally posted by r3ccOs

even in a perfect world, with various test condtions ran against a highly resilient environment, there are always unpredictable variables you can never account for.

Well said.

Too many propellerheads have come out to monday morning quarterback with all of their little technofixes that "could have prevented this"- thus Demonstrating an ignorance of real world situations in which there are so many competing requirements and imperfect circumstances.

[rage2]

Originally posted by LollerBrader
Too many propellerheads have come out to monday morning quarterback with all of their little technofixes that "could have prevented this"- thus Demonstrating an ignorance of real world situations in which there are so many competing requirements and imperfect circumstances.

I was more annoyed at the huge # of people (mostly on twitter) jumping on the bandwagon and criticizing shaw for not having a working DRP. It's not up to a datacenter host to ensure multisite redundancy for all the servers and apps hosted there. Shifting the blame to Shaw is retarded when they've essentially lost an entire datacenter.

[nzwasp]

My coworker used to work for IBM and said that some of the AHS sites had racks made out of plywood because they were so cheap with their IT budget. It seems far fetched but given the state of our healthcare I wouldn't put it past them.

[cam_wmh]

Originally posted by rage2

I was more annoyed at the huge # of people (mostly on twitter) jumping on the bandwagon and criticizing shaw for not having a working DRP. It's not up to a datacenter host to ensure multisite redundancy for all the servers and apps hosted there. Shifting the blame to Shaw is retarded when they've essentially lost an entire datacenter.

Exactly.

How about our societal value to Public Health Records, and it's data priority as compared to the Private sector? [/ZenOps]

[JfuckinC]

YOu know i bet they could have avoided all this encoded data loss if they just had a backlink hooked up to the DNS Server matrix.. Simple solution to monitor the incoming and outcoming power and hit a DERP switch to trigger the circuit board malfunction relays and start the fiberlink regeneration.. How did no one think of this? now all the capacitors and LED's are probably fried on the micro chips.

Idiots at shaw

[speedog]

Originally posted by rage2

I was more annoyed at the huge # of people (mostly on twitter) jumping on the bandwagon and criticizing shaw for not having a working DRP. It's not up to a datacenter host to ensure multisite redundancy for all the servers and apps hosted there. Shifting the blame to Shaw is retarded when they've essentially lost an entire datacenter.

Too true - too many arm-chair critics on Twitter and elsewhere (even here?). I've been out of the industry for almost 5 years now and can't even begin to offer up an informed opinion anymore because - A) things change so fast and B) the old noggin just can't remember a lot of that shit - just too, too long ago.

[sputnik]

Originally posted by rage2

I was more annoyed at the huge # of people (mostly on twitter) jumping on the bandwagon and criticizing shaw for not having a working DRP. It's not up to a datacenter host to ensure multisite redundancy for all the servers and apps hosted there. Shifting the blame to Shaw is retarded when they've essentially lost an entire datacenter.

So many companies look at IT completely wrong.

The directors spend (almost begrudgingly) what they do to get up and running and then as long as they have a couple of IT guys on the payroll they think they are done.

If anything goes wrong they just blame the IT guys and complain that they weren't doing something right, or whatever excuse they can come up with.

However these are the same execs that run half-baked solutions on undersized or poor quality hardware with ridiculously tight budgets and when the IT guys insist of certain hardware replacements or upgrades their requests fall on deaf ears.

Hopefully this failure at Shaw just drives companies to get off their wallets and reconsider their IT infrastructure budgets.

[95EG6P]

Originally posted by Skyline_Addict
Yup. All business critical systems are still down at ATB Financial.

down since 1 30pm yesterday

[Spoons]

Originally posted by 95EG6P


down since 1 30pm yesterday

Most of the Shaw employee's got sent home with pay....

I wish I worked at Shaw

[Chester]

Originally posted by Spoons


Most of the Shaw employee's got sent home with pay....

I wish I worked at Shaw

Yup. I dont work at shaw court but i work in the engineering office, network went down yesterday at 1:30, dont have to go to work till monday. Doesnt really matter though as ive been attending corporate binge drinking events all week lol

[95EG6P]

Fuck I work at atb wish I got to go early fuck I had to be on the phones says our systems are down for 8 hours

[dexlargo]

I'm now told that our mainframe backup is running and will be coming online this afternoon. The delay now is just that they have verify data integrity, run the overnight batch processes that were missed then run some quick tests before going fully live to production. Not sure if this will be the same for all government mainframe services. Mine is not the AHS service or the registry systems that are the most noticeable to the public. I have no ideas what's happening with those.

[8Ball]

AHS post

We’ve been advised we could begin to restart systems by 2 p.m. today. This time is subject to change. The process is expected to take several hours.

2pm, followed by 'several hours'

Love the ambiguity

[BigDL]

Originally posted by 8Ball
AHS post




2pm, followed by 'several hours'

Love the ambiguity

When it comes to big systems like that, who knows exactly how long it will take let alone if shit happens during start up.

[narou]

Originally posted by BigDL


When it comes to big systems like that, who knows exactly how long it will take let alone if shit happens during start up.

We have a lot running out of Shaw. I doubt it is as easy as flipping a switch to get everything running properly.

[8Ball]

Originally posted by BigDL


When it comes to big systems like that, who knows exactly how long it will take let alone if shit happens during start up.

Originally posted by narou


We have a lot running out of Shaw. I doubt it is as easy as flipping a switch to get everything running properly.

Did you guys give the update to ahs too?

2pm and then