Computer nerds! What's the deal with overwhelmed servers?

[ExtraSlow]

So, y'all know lots more than me about this. But how's this work?
How do servers get "overwhelmed" during busy times? Like when the reservealbertaparks website was horrific yesterday for camping reservations. Apparently they made 8000 reservations in one day, and that's like 10x more than the normal opening day volume.

But, like, what actually does that mean? What can be done? Does having cloud servers help? What are best practices for site like this that have wildly variable demand.


ELI5, Coles, FAQ, lmfao and whatever else.

Wut?

[adam c]

ELI5

It's like too many people walking up to you and trying to have a conversation with you, you can only process so much at once, however once you have that conversation that person leaves and you can focus on the next person

[rage2]

This quote is hidden because you are ignoring this member. Show Quote

So, y'all know lots more than me about this. But how's this work?
How do servers get "overwhelmed" during busy times? Like when the reservealbertaparks website was horrific yesterday for camping reservations. Apparently they made 8000 reservations in one day, and that's like 10x more than the normal opening day volume.

But, like, what actually does that mean? What can be done? Does having cloud servers help? What are best practices for site like this that have wildly variable demand.


ELI5, Coles, FAQ, lmfao and whatever else.

Wut?

I dunno if I can ELI5 but I'll try to break it down so its understandable. I'm waiting for my food delivery, so I got some time to kill haha. First, let's get the terminology right. The camping website is a service, which runs on server(s). adam c's analogy precisely ELI5 what happened.

Now, how does cloud servers help? First, let's understand why building your own server doesn't work when a service gets overwhelmed with way too much usage. When building your own servers, you have to build for the peak, and you have to really guess what your peak looks like, and build for that. Using adam c's analogy, if you expect 4000 people to talk to you, you ensure you have 1000 people on standby to handle the conversations in a timely manner. 8000 people show up wanting to talk, and it's a massive queue and things move slowly with only 1000 people handling it when you need 2000. Back to servers, building/ordering servers takes lead time, so it's all about properly estimating your peak. This model is also a waste of money, because 99% of the time you're wasting away these expensive servers that can handle 4000 people and only serving 20 when there is no peak load. Plus not being able to handle 8000 when you've got your estimates wrong. At that point, you're just sitting around hoping 4000 people give up and fuck off and things will start moving again. That's basically what happened with the camping site.

Onto cloud servers, how does that work? Cloud services let you fire up servers whenever you want, it's like being able to bring 1000 people to handle the unexpected load in a matter of minutes, and you just pay for what you use. In a well managed setup, the systems detect additional servers needed, and fires it up automatically to handle the conversations. That's how beyond works today. We had a spike of 3x the traffic for about 2 hours today, and it automatically fired up 2 more servers without anyone noticing it. Once that extra traffic subsided, it killed off the unnecessary servers. This is automatic scaling, where the system detects and scales up and down (bring in kill off servers) based on demand, and I only pay for what's being used.

That said, it's not foolproof and there is still a level of estimation that needs to be done. In cases like camping, ticket sales, shoe sales that start at a specific time, your system won't be able to properly guess that you're going from needing 10 people to 2000 in a matter of seconds. So what we would do is pre-scale, and in adam c's analogy again, forcibly bring in 2000 people right off the bat before the shit goes nuts. So similar to the build your server model, except I can give these extra people the boot after I'm done with them. An example of this is our work cloud servers that handle time tracking, we pre-scale on Monday morning at 6:45am EST because at 7am EST hundreds of thousands of east coast workers are punching in for the day, and we need to process that instantaneous load that we've predicted but the system can't react fast enough to. The bonus in this flexibility is that we can scale down to as little as 2 servers at night, so that the system is still available but we're paying 1/10th of the cost.

I guess if I had to ELI5 the cloud model, it'd be like being the shift manager at a 24/7 supermarket, where you have the ability to have anyone on call within minutes, and kick people off to go away when you don't need them, without worrying about a union so you can get away with paying the bare minimum of wages to handle your business.

[pheoxs]

Rage did a better answer but if I had to try and explain:

A server can only pass so much data so quickly. Because sites take data to load (particularly images) there is a limit on how many requests a server can handle at once.

But websites aren't sent in entire packages, instead you send packets of data so a normal browser goes hey give me a bit, servers gives it, hey give me some more, server gives more, etc until it's done. When there's way too many people asking the server starts missing requests. Then it gets behind and can't catch up. It's sending one little packet out here and there to people but it's not enough for anyone to render the full page.

Places like Ticketmaster fix this issue by having a queueing server where it takes people arriving at the page and then directs them to a secondary server where they place their order. You can have a hundred secondary server with the queue server splitting the load up so no one's overwhelmed. Think airport security where they direct you to a line but pause people so none of the lines block each other

[rage2]

Here's a snapshot of beyond on 4/20.



Purple line is a representation of traffic on the left Axis, green line is the # of servers on the right axis. A search engine kicked off loading the site at a high rate adding a bunch of requests. System detects that it's starting to be under load and just added a server. Once the search engine loads slowed down, it killed one back off.

This is all automatic. I had no idea this happened until right now haha. I helped another forum in Montreal move to the cloud and they went from daily problems to zero.

[benyl]

ELI5: think of each camping request like a drop of water. The garden hose is only so big and can only handle a certain gpm. Once it is overwhelmed, you can’t push anymore water through it. Only way is to add more hoses (cloud servers).

So, if you were trying to put out a fire in your back yard and one hose isn’t enough, then you need to turn on more hoses.

[rage2]

I love this shit. Wrote all that while waiting for food and then eating my Pho haha. I don't get to touch any of this stuff anymore in my new role, so this is very enjoyable.

[ExtraSlow]

This quote is hidden because you are ignoring this member. Show Quote

I love this shit. Wrote all that while waiting for food and then eating my Pho haha. I don't get to touch any of this stuff anymore in my new role, so this is very enjoyable.

So, that camping reservation service should have an initial "queueing" server and that server can be the bottleneck while the rest runs more smoothly? They know they have a small number of busy days per year, and then a low normal base load.
So Coles is that gubbermints done gone had a bad setup.

[rage2]

This quote is hidden because you are ignoring this member. Show Quote

So, that camping reservation service should have an initial "queueing" server and that server can be the bottleneck while the rest runs more smoothly? They know they have a small number of busy days per year, and then a low normal base load.
So Coles is that gubbermints done gone had a bad setup.

For what they have right now, yes. Queuing would have solved their problems if they are handcuffed by the hardware they have. I’d argue it’s a lot tougher to develop the software for a queuing service than to move the whole thing into a cloud scalable service. To give you an idea of how much effort it took to move beyond from our physical servers in a datacenter to AWS, I completed the entire proof of concept move in 8 hours at a bar in Banff on a weekend.

And to clarify, the queueing service can’t be bottlenecked either. There’s a lot of design considerations to make it work. It has to be designed so that it communicates with the user as little as possible, while accounting for idiots that mass refresh thinking they’ll get a better place in line. Overload the queuing server and you’ll have a bunch of people pissed off at losing their place in line.

Of course important businesses (banks, government) will argue they can’t have sensitive data stored outside their control so they can’t store it in the cloud. That’s fair, but camping data isn’t sensitive. You can still host your own identity services in house, keeping sensitive information out of the hands of your cloud provider, while running non sensitive camping information in the cloud that leverages the identity service eliminating all the risks.

Even if they stored everything in house, an ideal datacenter model should be something like a massive VMware cluster with all services running within that, so in essence a private cloud that allows them to spin up servers as needed, and thereby sharing all that physical hardware across government because not everything needs peak capacity on the same day.

There are so many ways to solve the problem, but clearly the camping service chose a 20 year old solution to the problem.

[CompletelyNumb]

I'm mostly impressed that you got vb4 working on AWS in a day. The underlying PHP is not built to handle high load.

[thetransporter]

i m sure cloud is just another name for a server

" Does having cloud servers help? "

[LilDrunkenSmurf]

This quote is hidden because you are ignoring this member. Show Quote

Here's a snapshot of beyond on 4/20.



Purple line is a representation of traffic on the left Axis, green line is the # of servers on the right axis. A search engine kicked off loading the site at a high rate adding a bunch of requests. System detects that it's starting to be under load and just added a server. Once the search engine loads slowed down, it killed one back off.

This is all automatic. I had no idea this happened until right now haha. I helped another forum in Montreal move to the cloud and they went from daily problems to zero.

Which service? I'm running a forum on AWS, but haven't set up auto scaling as I'm not sure how to deploy the server config.

[rage2]

This quote is hidden because you are ignoring this member. Show Quote

Which service? I'm running a forum on AWS, but haven't set up auto scaling as I'm not sure how to deploy the server config.

EC2 handles load balancing, auto scale, and the virtual front end servers. EFS for the shared file services where the forums and attachments are stored (you don’t want any persistent data on the VMs because they get created and destroyed automatically). RDS for the database backend. Cloud watch to monitor and alert on anything.

If you want a visual rundown of how it’s all setup, I’d be happy to host a zoom one night to show you under the hood of how it all ties in together.

[LilDrunkenSmurf]

This quote is hidden because you are ignoring this member. Show Quote

EC2 handles load balancing, auto scale, and the virtual front end servers. EFS for the shared file services where the forums and attachments are stored (you don’t want any persistent data on the VMs because they get created and destroyed automatically). RDS for the database backend. Cloud watch to monitor and alert on anything.

If you want a visual rundown of how it’s all setup, I’d be happy to host a zoom one night to show you under the hood of how it all ties in together.

I got the basis of it. The problem is I'm using a 3rd party AMI that has the software pre-installed, so it hosts the DB locally on the EC2 instance. I've setup cron jobs to do a DB export, and backup to S3 nightly, and that's fine for my purposes, for all of 11 users, but I'm looking to future proof it.

I could be less lazy, move the DB to RDS, but I never considered using EFS for the attachments.

Thanks for the tips. I'm using SMF vs vBulletin, but I'm sure the same basis would apply. I might still poke you and ask for some help.

I also have the entire thing setup via Terraform, because I can.

[rage2]

This quote is hidden because you are ignoring this member. Show Quote

I got the basis of it. The problem is I'm using a 3rd party AMI that has the software pre-installed, so it hosts the DB locally on the EC2 instance. I've setup cron jobs to do a DB export, and backup to S3 nightly, and that's fine for my purposes, for all of 11 users, but I'm looking to future proof it.

I could be less lazy, move the DB to RDS, but I never considered using EFS for the attachments.

Thanks for the tips. I'm using SMF vs vBulletin, but I'm sure the same basis would apply. I might still poke you and ask for some help.

I also have the entire thing setup via Terraform, because I can.

I built my own AMIs. Didn’t use terraform. Was trying to keep it as simple as possible. Ya using a 3rd party forum out of the box AMI with DB running there doesn’t make sense from a scale perspective. You want to segregate each function so they can scale independently of one another. Also some of those 3rd party AMIs charge hourly on top of EC2 costs.

You still have my number? Feel free to msg me. As long as you’re on an iPhone lol.

[Xtrema]

queuing has fixed cost. Bursting in cloud will have increased cost.

It's like having 1 cashier at check out vs 2+

[rage2]

This quote is hidden because you are ignoring this member. Show Quote

queuing has fixed cost. Bursting in cloud will have increased cost.

It's like having 1 cashier at check out vs 2+

The fact that you can minimize cloud spend during off peak hours easily offset any burst cloud costs.

[ExtraSlow]

As a follow-up to my original post, can someone go tell the folks at ALberta Parks to fix thier damned website so this isn't such an issue next time? I hear they could be releasing the second 50% of sites later this summer . . . .

[rage2]

https://www.cbc.ca/news/canada/calga...lure-1.5576905

Single server, panic, reboot!

They’re looking to rebuild it but won’t be ready till next year? Oh boy.

[ExtraSlow]

This quote is hidden because you are ignoring this member. Show Quote

They’re looking to rebuild it but won’t be ready till next year? Oh boy.

Just selecting a vendor is probably an eight month process for the federales.