beyond.ca forum outage - June 19-20, 2012

[Tik-Tok]

DoS attacks.

[C_Dave45]

Haven't been able to get in most of the day yesterday, and for the past couple of hours.
Is it just me?

[lasimmon]

Anonymous attacked.

[revelations]

srs ? dos?

[rage2]

Yea, we must've pissed off some script kiddies. DDOS attack which crippled our ISP.

Unfortunately, we've moved our hosting offsite about 2 years ago, and with no control ourselves to the networking layer, it took a while to get things under control.

Really sorry it took 2 days to get everything back up and running properly. Thanks for your patience tho! Things *should* be stable now!

[flipstah]

Oh, phew; thought it was just me.

I was afraid my work blocked Beyond. :rofl:

[D'z Nutz]

It's all these fuckers' fault!
http://forums.beyond.ca/st/355595/cyber-attacks-at-your-workplace/

[kvg]

I'm glad people won't have to waste their day working today:clap:

[jampack]

It's funny how I somehow felt my day (yesterday) wasn't THAT complete without looking at beyond... :rofl:

[flipstah]

Originally posted by D'z Nutz
It's all these fuckers' fault!
http://forums.beyond.ca/st/355595/cyber-attacks-at-your-workplace/

Thread pops up; Beyond goes down days after.

:whipped:

[JRSC00LUDE]

Thank god, I was starting to panic!

[bulaian]

I thought I got blocked or something, so I tried tablet, phone, library computer..... I was going through Beyond withdrawls

[Twin_Cam_Turbo]

I was at home sick yesterday, pretty painful day with no beyond!

[Hallowed_point]

Originally posted by jampack
It's funny how I somehow felt my day (yesterday) wasn't THAT complete without looking at beyond... :rofl:

x2 :(

[baygirl]

I am stuck in the middle of butt-fuck nowhere, no cell service and no beyond..it was horrible:(

[kvg]

beyonds fb page has never seen that much action:rofl:

[rage2]

Again, my apologies it went down for so long.

On the bright side, we're still getting hit right now, and we're OK. So things are looking good for now. We're eating up around 100mbit of traffic.

[Sugarphreak]

Oh thank god it is back up... time had actually stopped at work, I've been here for what felt like 100 hours already today.

[spikerS]

so who did we piss off this time? Do we have any idea who it is, or could it be something else hosted on the same off site server?

[baygirl]

Originally posted by rage2
Again, my apologies it went down for so long.

On the bright side, we're still getting hit right now
Does this mean it might happen again:eek:

And why do you keep apologizing, it's not like it's your fault lol

[rage2]

Originally posted by baygirl
And why do you keep apologizing, it's not like it's your fault lol
It's my job to keep the site up, regardless of what's causing an outage, so if the site is down, it's my fault. Accountability I guess? haha

[bourge73]

God that was brutal, I actually had to work.....and look at 780 Tuners.....The humanity!

[jibber]

Thankfully, beyond is back up and running. No apology needed, but a thank you is in order... thanks to rage for the help in getting it all sorted out!

[littledan]

damn! I totally thought my work blocked beyond as well! haha. also would like to know who beyond pissed off... haters gonna hate!

[flipstah]

Originally posted by littledan
damn! I totally thought my work blocked beyond as well! haha. also would like to know who beyond pissed off... haters gonna hate!

I bet it's Revscene.

Flame suit, engage.

[bourge73]

I bet the number of posts today are going to be through the roof! Ha Ha

[mr2mike]

Originally posted by flipstah
Oh, phew; thought it was just me.

I was afraid my work blocked Beyond. :rofl:

Same. haha.

[nykz]

I never realized how many times I click on my beyond tab a day until yesterday. Just wasn't the same without beyond, glad everything's okay now though :thumbsup:

[littledan]

in other news... trev006 revenues have plummeted and he will be forced to post videos in the comments section of calgarysun

[AndyL]

Originally posted by rage2

It's my job to keep the site up, regardless of what's causing an outage, so if the site is down, it's my fault. Accountability I guess? haha

No no... this is not correct, if theres one thing we know from beyond - nobody should be accountable for their actions. Blame Kenny :)

I still think you must have tried to post another spam review, and made the spambot detectors explode :rofl: they don't know what to do with the real thing :D

[schocker]

Originally posted by Tik-Tok
DoS attacks.
http://farm5.staticflickr.com/4119/4876696414_d3f62d18be_z.jpg

[C_Dave45]

Originally posted by jibber
Thankfully, beyond is back up and running. No apology needed, but a thank you is in order... thanks to rage for the help in getting it all sorted out!

Dammit....I thought I was actually going to get things done for a change. :rofl:

[kenny]

Originally posted by AndyL


No no... this is not correct, if theres one thing we know from beyond - nobody should be accountable for their actions. Blame Kenny :)


Hey now... let's keep me outta this :rofl:

[Tik-Tok]

Originally posted by schocker
Are you a wizard

I found a sweet Wizards Cap in D3, that lets me post answers before OP's questions...

[toyboy88]

Originally posted by Sugarphreak
Oh thank god it is back up... time had actually stopped at work, I've been here for what felt like 100 hours already today.

Touche! :poosie:

[FraserB]

I seriously thought work blocked me lol.

[dr_jared88]

How about you provide us the attackers source address and we all reverse attack them? :dunno:

[-relk-]

Originally posted by dr_jared88
How about you provide us the attackers source address and we all reverse attack them? :dunno:

:werd:

[BokCh0y]

I just wanted to see my thread asking for muffler shop recommendations. glad beyond is back though!!!

[rage2]

It's a small botnet of around ~3-4k hosts. Each host is just infected with a bot trojan, and had no idea they're being used to attack other sites.

I have all the IPs, but it's useless to attack them.

[D'z Nutz]

Originally posted by dr_jared88
How about you provide us the attackers source address and we all reverse attack them? :dunno:

mK0q7d3pLH0

[95EG6P]

[i[\I was afraid my work blocked Beyond. :rofl: [/B]

:werd:

[Mibz]

Originally posted by D'z Nutz
mK0q7d3pLH0 Fuck.Yes

[TomcoPDR]

Originally posted by baygirl
I am stuck in the middle of butt-fuck nowhere, no cell service and no beyond..it was horrible:(

Aren't you close to Dildo?

[TomcoPDR]

Dildo, Newfoundland

[spikerS]

Originally posted by TomcoPDR
Dildo, Newfoundland

fairly close, round about 45 mins, depending on how you drive...

[baygirl]

Originally posted by TomcoPDR


Aren't you close to Dildo?



Originally posted by TomcoPDR
Dildo, Newfoundland

:rofl: :rofl: :rofl: :rofl: :rofl: :rofl:

[jibber]

Originally posted by TomcoPDR



Originally posted by baygirl
I am stuck in the middle of butt-fuck nowhere, no cell service and no beyond..it was horrible


Aren't you close to Dildo?

:rofl: See, this why I missed beyond when it was down... nowhere else can you find a single post with "butt-fuck" and "Dildo" and "...it was horrible" all conveniently together. And posted with a straight face!

[blitz]

Originally posted by dr_jared88
How about you provide us the attackers source address and we all reverse attack them? :dunno:

What, send them a bunch of emails with horrible grammar and caps lock turned on?

[dr_jared88]

Originally posted by blitz


What, send them a bunch of emails with horrible grammar and caps lock turned on?

If only that would work. This forum is packed full of experts in that area!

[max_boost]

About time it's back up! :nut:

[spacerz]

^+1.

"www.beyond.ca" is still showing an error message for me, but www.forums.beyond.ca still works.

Edit: both are working now.

[googe]

If you send the list of IPs to their ISPs, the ISPs will usually pull the plug on them until they confirm they've cleaned the bot. That is assuming none are businesses with SLAs.

[Cos]

.

[rage2]

Originally posted by googe
If you send the list of IPs to their ISPs, the ISPs will usually pull the plug on them until they confirm they've cleaned the bot. That is assuming none are businesses with SLAs.
haha ya already done that. It takes a while though, 4000 IPs from everywhere, have to dig up who's responsible for each IP, find their abuse@ info, then ship it out.

I have a guy doing it already.

[Heru-Farukon]

Originally posted by Cos
Facebook exploded with all the yippy comments from beyonders. Me and a few others all thought we got banned.

:rofl:

[mr2mike]

Originally posted by BokCh0y
I just wanted to see my thread asking for muffler shop recommendations.

I just wanted to see everyone recommend him Minute Muffler on 32nd Ave as a 1 stop shop for everything.

[LOLzilla]

Seems like a perfect time to host beyond on AWS.

[rage2]

Originally posted by LOLzilla
Seems like a perfect time to host beyond on AWS.
We use EC2 at work, while it can scale to handle the onslaught, we'll be paying for the bandwidth of the attack. So not an ideal solution.

Right now, we're not paying anything extra for attack bandwidth. Only legit requests are getting charged.

The attack has slowed down since last night. Right now we're seeing 50mbit bursts every few mins, and roughly 200k connections over ~2800 bots. At the peak when we deployed the firewall changes, we were seeing 100mbit, 500k connections, and ~4000 bots. We have a guy going through every IP and reporting 'em, so at a minimum someone will eventually lose out on 4000 bots.

Here's a google map of where the current points of attack are coming from. Fun stuff.

http://i1213.photobucket.com/albums/cc475/rage2amg/ddos.jpg

[CompletelyNumb]

Script kiddy knew a few tricks it looks like.

[LOLzilla]

Originally posted by rage2

We use EC2 at work, while it can scale to handle the onslaught, we'll be paying for the bandwidth of the attack. So not an ideal solution.


It matters what your objective is. If you wanted 100% availability it would be a valid option. I guess the only risk associated with beyond.ca being down is angry neckbeards.

I suspect you have done this already but it would appear that %50~ of the traffic is coming from outside of North America. Using iptables (http://ipinfodb.com/ip_country_block_iptables.php) or similar you could country block temporarily. Some of my clients keep a 'oshit' firewall ACL that will be only be used in time of DDoS attacks.

[dj_rice]

Thank god Beyond's back. I had to resort to browsing through 780Tuners at work.

[flipstah]

Originally posted by dj_rice
Thank god Beyond's back. I had to resort to browsing through 780Tuners at work.

That's as bad as wanting to go to Arby's.

[toastgremlin]

Originally posted by flipstah


That's as bad as wanting to go to Arby's. If Arby's gave you herpes, sure.

[canadian_hustla]

thank god beyond is back. I thought it has something to do with my ISP blocking it or something because every other site worked LOL :nut:

and yes, as others have said last night was super boring didn't really know what to do with myself :banghead:

[Disoblige]

You know what Beyond members are like when the forum goes down for a day or so and people think they got banned :rofl:

[FraserB]

Productivity in the core peaked for about a day it seems.

[Cos]

.

[Disoblige]

Something funky's happening again..

[kenny]

Originally posted by Disoblige
Something funky's happening again..

*Jedi Wave* Everything is fine.

[kvg]

It gives me extra time to watch porn:devil:

[rage2]

Originally posted by kvg
It gives me extra time to watch porn:devil:
5 min outage there, all the time you need! :rofl:

[JfuckinC]

man i was going crazy on thurs trying to get on...

do you think somone you banned is lashing back? :rofl:

[toyboy88]

Originally posted by Disoblige
Something funky's happening again..

+1

The forums.beyond.ca page works/loads OK now, but something not right keeps happening still (ever since the outage)...

I go to login (which works fine), and then 'get new posts' (which also loads fine onto "page 1 of X")...but then when I either: A) open thread link into new tab or B) click to go to "page 2 of x", it basically says "no thread specified" and/or logs me out for some reason. And then I have to keep re-logging in!

Anybody else having this problem? I've tried on both my Mac and PC, both seem to have this issue...? :(

[AwkwardAzn]

Originally posted by toyboy88


+1

The forums.beyond.ca page works/loads OK now, but something not right keeps happening still (ever since the outage)...

I go to login (which works fine), and then 'get new posts' (which also loads fine onto "page 1 of X")...but then when I either: A) open thread link into new tab or B) click to go to "page 2 of x", it basically says "no thread specified" and/or logs me out for some reason. And then I have to keep re-logging in!

Anybody else having this problem? I've tried on both my Mac and PC, both seem to have this issue...? :(

ya this happens to me have to log in multiple times!! :banghead:

[kvg]

Originally posted by rage2

5 min outage there, all the time you need! :rofl:

And still had time to write an email:thumbsup:

[spikerS]

I dont know if it is just me or not, but happening on 3 diff comps with 3 diff internet connections, and beyond is dragging ass on all 3...

[speedog]

Ditto

[pf0sh0]

Same with me, takes a couple tries to get things going.

Tried a couple different browsers and same result

[rage2]

I got alerted to a new DDOS attack starting at 7am this morning. Our network guys have been tweaking the firewalls to clean things up since then (kudos to them). It's been slightly slower here and there whenever they're deploying changes, but mostly it's been fast from my cell phone.

I'm on the road, so I haven't been actively working on it. I trust that our network guys will have a handle on it shortly.

We were down for about 5 mins this morning when the attack first started. Our firewalls are eating up 130mbps right now. At the peak, we were getting hit with around 200mbps. Nothing our connection can't handle at this point. :)

[spikerS]

Man, someone or someones are really trying to take Beyond down.

Rage, is there other sites hosted on the server beyond is on that might be targeted, or is it beyond itself?

[rage2]

It's targetting us for sure haha. Sucks that DDOS is so easy to do nowadays. Anyone can pay someone $200 to take a site down for a week. Lucky we have some good network guys on staff to take care of things.

I have a case open with the RCMP and FBI, so maybe they'll be able to catch them.

[spikerS]

with the FBI? wow, probably showing my ignorance here, but how hard was that to accomplish? I am surprised they would be interested in something like this.

[speedog]

Could be as easy as this - http://www.ic3.gov/default.aspx

[rage2]

Cyber crime has no borders. It's pretty easy to get in touch with the FBI. I've been dealing with them since the mid 90's for various stuff.

[Go4Long]

Originally posted by rage2

I have a guy doing it already.

Beyond.ca...we have people.

[ExtraSlow]

Originally posted by Go4Long


Beyond.ca...we have people.
OT, but my wife swears that for anything someone could need, beyond has a guy that does that.

Mattress, Check.
Mercedes, Check
Realtor, check
Insurance, Check,
cop, check,
Engineers, check
Rigs, check
Garage doors, check
Tow truck/day trader/long haul trucker, check
Restaurant, check

there's way more than that too, but you get my drift.

[jdmakkord]

For some reason I cannot access Beyond via my iPhone any more. Does this have something to do with the outage?

[sxtasy]

the forums have been slow for me as well

[Twin_Cam_Turbo]

Really slow right now! Trying my best to get my fix!

[Cody D]

I keep thinking that all the people sitting hitting refresh trying to get beyond to work probably isn't helping the problem, haha.

[rage2]

For you guys experiencing problems, are you browsing on pc or phone/tablet? 3G or wifi? I've been on 3G all day with no issues, but I'm definately getting reports of some users having probs from some locations.

I'll pass whatever info to our network guys. Thanks.

[CompletelyNumb]

It's somewhat slow on my pc, telus optik downtown. Nothing unbearable though.

[ipeefreely]

Originally posted by rage2
For you guys experiencing problems, are you browsing on pc or phone/tablet? 3G or wifi? I've been on 3G all day with no issues, but I'm definately getting reports of some users having probs from some locations.

I'll pass whatever info to our network guys. Thanks.


Slow on PC... had to try three times to get to this tread! :nut:

[Hallowed_point]

It could be my shitty laptop , but I've noticed that I have to refresh and go back quite a bit now to view posts. And I'm getting the "Waiting for forums.beyond.ca..." message :confused:

[msommers]

Telus 3G works per usual. Just tried on my computer and it's pretty slow; other webpages load at normal speeds. Must be a product of your firewall on PC I guess

[Offroad]

Its Neshi and city hall hackers :poosie:

[88CRX]

Wifi on iPhone barely loads. 3G on iPhone isnt really any better.

Computer nothing as well.

[rage2]

Is it better now? The guys made a change a few mins ago.

[88CRX]

Seems like its all good now.