software-engineer-blog logoSoftware Engineer Blog

Module 9 · Delivery and operations

Unit 33 of 49

Unit 33 · Module 9 · Delivery and operations

Load balancing and autoscaling

More traffic than one box can take, and the new box always arrives late.

Unit 33 of the free 49-unit computer-science course, in delivery and operations. 4 topics to watch or read, 3 interview questions answered in full and a short self-check.

Watch or read

4 topics make up this unit. Take each one whichever way suits you, then answer the questions below.

Load balancers from scratch

A load balancer distributes incoming requests across multiple servers to prevent any single server from becoming a bottleneck. Learn how round-robin routing, health checks, and redundancy keep sites online during traffic spikes.

ReelRead

Horizontal vs vertical scaling

Your app is slowing down under load — do you scale up or scale out? Vertical scaling means moving to a bigger machine: same app, same code, more CPU/RAM/disk. Horizontal scaling means adding more machines behind a load balancer. Here's how they really differ on cost, ceiling, fault tolerance, and the coordination tax — with a clear rule for which to reach for, plus how the same trade-off shows up when you serve LLMs.

ReelRead

Autoscaling

Autoscaling is a reaction, not a prediction. A metric is sampled every ~60 seconds, has to hold over a threshold for two samples, and only then does a machine launch — and launched is not serving. Boot, image pull, app warmup and health checks add another ~90 seconds before the load balancer sends it a single request. Here's the full loop, why it is structurally late by exactly one boot cycle, and what actually absorbs a spike.

ReelRead

Stateless vs stateful

Stateless servers forget you after each request, scaling infinitely but requiring every call to carry full context. Stateful servers remember you, enabling natural continuity but pinning you to one node.

ReelRead

Interview questions this unit unlocks

Asked out loud, answered out loud. Read the answer, then say it in your own words.

What does a load balancer actually do, beyond spreading requests?

Three jobs, and only the first is the obvious one. It distributes — round robin, least connections, or a hash for stickiness. It health checks, which is the part that turns a crashed instance into a non-event rather than one in every N requests failing. And it terminates TLS and normalises the connection, so your application servers speak plain HTTP on a warm keep-alive rather than doing a handshake per client.

Layer 4 balances connections and is fast and blind; layer 7 reads the request and can route by path or header, at the cost of parsing it.

Horizontal or vertical scaling — how do you choose?

Vertical is a bigger machine: no code changes, no distributed-systems tax, and it is bounded by the largest instance and leaves you a single point of failure. Horizontal is more machines: effectively unbounded and redundant, but it requires the application to be stateless, which is a design decision you have to have made earlier. In practice you scale up until it is embarrassing, and out once the ceiling or the availability requirement forces it.

Why do new instances always arrive two minutes too late?

Because autoscaling is a feedback loop with lag. The metric is averaged over a window, the alarm needs consecutive breaches, the instance has to boot, warm its caches and pass a health check — and the spike that triggered all this arrived in seconds. So you scale on a leading signal like queue depth rather than a trailing one like CPU, keep warm capacity for the step you know is coming, and set the scale-down cooldown longer than the scale-up so you do not oscillate.

For a known event — a launch, a campaign — scheduled scaling beats reactive scaling every time. The best autoscaler is the one that already scaled.

Self-check — 3 questions

Answer alone, at 2am, with no interviewer in the room.

Part of Everything You Need to Know About Computer Science.