Load Balancer Explained From Scratch — How One Site Survives a Traffic Flood

Prefer to watch? ▶ Load Balancer Explained From Scratch — How One S in 90 seconds ✈ Telegram

When a single server maxes out, most engineers reach for a bigger machine. That ceiling arrives fast, costs climb steeply, and you're still one server crash away from total downtime. A load balancer solves this by sitting in front of multiple identical servers and distributing traffic across them.

Mental model: A load balancer is a switchboard operator at the front desk—it takes every incoming call and hands it to one available operator, spreading the load so nobody drowns in requests.

The Breaking Point: One Server, One Limit

Imagine TicketRush, a ticket-selling site. On a single server with a modern CPU and enough RAM, it handles about 1,000 requests per second comfortably. The system has CPU headroom, the database isn't swamped, response times sit at 100–200ms. Life is good.

Then a major concert goes on sale.

10,000 requests per second arrive in the span of seconds. CPU usage hits 100%. Database query queue grows. Response times balloon to 5+ seconds. Users see 503 Service Unavailable errors. The site is online, but functionally dead.

This is not a software bug. It is physics: one machine has finite resources.

The Naive Fix: Vertical Scaling (Doesn't Work)

The first instinct is to buy a bigger server. Double the CPU cores, triple the RAM, move to a machine with faster disks.

It works—for a while.

But vertical scaling has hard limits:

Ceiling. The largest machines in any cloud region are finite. You will eventually hit the top of the catalog.
Cost. Moving from 4 cores to 32 cores doesn't cost 8x more in compute—it costs 10–15x more. The curve is steep.
Single point of failure. One server still means one server. If it crashes, if the hypervisor has a bug, if a disk dies at 3 a.m.—the entire site goes dark. No redundancy, no failover.

Vertical scaling is a short-term relief, not a long-term solution.

Horizontal Scaling: Many Small Servers

The real answer is to scale out, not up. Instead of one giant server, run three (or ten, or a hundred) small identical servers.

Each server is a complete copy of TicketRush:

Same application code
Same database client libraries
Same configuration
Same capacity: ~1,000 req/sec per server

Three servers running in parallel can now handle ~3,000 req/sec. Add a fourth, and you have ~4,000 req/sec of total capacity.

But this raises an immediate question: when a request arrives, which server should it go to?

You need a traffic cop. Enter the load balancer.

The Load Balancer: Single Entry Point, Distributed Execution

A load balancer is a piece of software (or hardware appliance) that sits in front of your servers. Every request from the internet hits the load balancer first. The load balancer then picks one of the backend servers and forwards the request to it.

From the client's perspective, there is only one endpoint: the load balancer's IP address or hostname. The load balancer is the single entry point. Behind it, the servers are invisible.

This solves two problems at once:

Traffic distribution. Each server receives only a fraction of the total traffic.
Abstraction. You can add or remove servers without changing the client's DNS records or configuration.

Round-Robin Routing: Simple Load Spreading

The simplest load balancing algorithm is round-robin. The load balancer keeps a counter. For each new request:

Request 1 → Server 1
Request 2 → Server 2
Request 3 → Server 3
Request 4 → Server 1 (loop back)
Request 5 → Server 2
And so on.

The load balancer cycles through the backend servers in order. Over time, each server gets roughly equal traffic.

Here's a simplified example of what a round-robin balancer looks like in pseudocode:

class RoundRobinLoadBalancer:
    def __init__(self, backend_servers):
        self.servers = backend_servers
        self.counter = 0
    
    def route(self, request):
        server = self.servers[self.counter % len(self.servers)]
        self.counter += 1
        return server.handle(request)

Round-robin is not always optimal (some requests are heavier than others), but it is predictable and works well when backend servers are identical.

Health Checks: Removing Dead Servers

Now you have three servers, and traffic spreads evenly. But what happens when one server dies?

Without detection, the load balancer would keep sending requests to it. Those requests would time out after 30+ seconds, users would see errors, and nobody would know why until an engineer checked the logs hours later.

The solution is health checks. The load balancer periodically pings each backend server (usually every 5 seconds) to confirm it is alive.

The health check is typically an HTTP GET request to a simple endpoint:

GET /health HTTP/1.1
Host: server-2.internal

If the server responds with HTTP 200 OK, it is marked healthy. If it times out or returns a 500 error, it is marked unhealthy.

Once a server is marked unhealthy, the load balancer stops sending requests to it. Traffic is rerouted to the remaining healthy servers. If Server 2 crashes at 2 a.m., and your team doesn't wake up until 6 a.m., users never noticed—the load balancer handled it silently.

When Server 2 comes back online, it responds to health checks again, and the load balancer gradually adds it back into the rotation.

The Restaurant Analogy (and Where It Breaks)

A load balancer works like a restaurant host. The host stands at the front and seats each arriving guest with a free waiter. If a waiter steps outside to take a smoke break, the host doesn't seat new guests with that waiter—the host sends them to an available waiter instead.

But the analogy has limits.

Sticky sessions. In the real world, a guest might want the same waiter all night (for consistency, familiarity, or because the waiter remembered their drink order). In HTTP terms, some sessions need to stick to one backend server. This is tricky with simple round-robin, because request 1 might go to Server 1 (establishing a session), but request 2 (from the same client) might go to Server 2 (which doesn't have the session data).

Load balancers handle this with session affinity or sticky sessions, which you'll see in more complex routing algorithms.

The host is a single point of failure. A restaurant needs only one host, but a website cannot afford it. If the load balancer itself fails, all traffic stops at the front door. To fix this, you run two load balancers in an active-passive or active-active configuration. A virtual IP (VIP) or DNS failover ensures traffic flows to a healthy balancer if one goes down.

Layer 4 vs Layer 7 routing. A simple restaurant host uses only visible information—table size, arrival time. But load balancers can inspect deeper. Layer 4 routing (TCP/UDP) sees only IP and port. Layer 7 routing (HTTP) can see headers, paths, and host names, allowing more sophisticated decisions like routing based on the request URL or Host header.

Load Balancer Algorithms: When Round-Robin Isn't Enough

Algorithm	How It Works	Best For	Drawback
Round-Robin	Cycles through servers in order.	Homogeneous servers with similar request weight.	Ignores actual server load; some requests are heavier than others.
Least Connections	Routes to the server with the fewest active connections.	Long-lived connections or highly variable request duration.	Requires tracking connection counts; higher LB CPU overhead.
IP Hash	Hashes the client IP to pick a server deterministically.	Session affinity without server-side session storage.	Uneven distribution if client IP range is small; adding servers causes reshuffling.
Weighted Round-Robin	Assigns a weight to each server; higher weight gets more requests.	Heterogeneous servers with different capacities.	Requires manual tuning; weights must be updated when hardware changes.
Random	Picks a server at random.	Very simple, low LB overhead.	Less predictable; can clump load on one server by chance.

Most production systems use least-connections or weighted round-robin, because they adapt to real-world variance in server performance and request weight.

Redundancy: The Load Balancer Must Not Fail

You have now distributed your site across three servers with a load balancer in front. You're safer than before, but you've introduced a new single point of failure: the load balancer itself.

To eliminate this, you run two load balancers, not one. Both listen on the same virtual IP (VIP), using a protocol like VRRP (Virtual Router Redundancy Protocol) or a cloud-native equivalent like AWS Elastic IP failover.

If the primary load balancer dies, the VIP automatically fails over to the secondary balancer in seconds. Traffic resumes. Clients don't notice.

This is called active-passive load balancing (one balancer is active, one is standby) or active-active load balancing (both are active, requests are distributed across both). Active-active is more efficient but requires careful coordination to avoid split-brain scenarios.

Real-World Load Balancers: What You Actually Use

These are the tools you'll reach for in practice:

NGINX

Layer 7 load balancer; written in C; runs on Linux
Excels at HTTP/HTTPS routing, URL-based path routing, hostname-based routing
Fast, lightweight, widely supported
Free, open-source; commercial support available
Use when: you need HTTP-level intelligence and want to run the balancer yourself

HAProxy

Layer 4 and Layer 7 load balancer; written in C; runs on Linux and BSD
Lower latency than NGINX for simple TCP forwarding; excellent for non-HTTP protocols
Mature, battle-tested, very configurable
Free, open-source
Use when: you need extreme performance or non-HTTP protocols (databases, cache, etc.)

Envoy

Layer 7 load balancer and service proxy; written in C++; designed for microservices
Observability-first architecture; integrates deeply with Kubernetes and service meshes
More complex to configure than NGINX or HAProxy
Free, open-source; backed by CNCF
Use when: you're running a service mesh or need fine-grained traffic control across services

AWS ELB / ALB / NLB

AWS managed load balancers
ELB (classic): older, Layer 4 only
ALB (Application Load Balancer): Layer 7, best for HTTP/HTTPS microservices
NLB (Network Load Balancer): Layer 4, extreme performance, best for non-HTTP or ultra-low-latency
Fully managed; no server management; integrates with auto-scaling
Use when: you're on AWS and want to avoid running your own balancer

All of these implement the same core ideas: round-robin (or better), health checks, and failover.

Load Balancing and LLM Inference Serving

When serving LLM inference, load balancing takes on a different flavor. Requests are not symmetric: a prompt that generates 2,000 tokens takes much longer than a prompt that generates 10 tokens. Round-robin will clump long requests on one GPU, leaving others idle.

For LLM serving, you want:

Least-busy scheduling based on queue depth, not request count. A server with fewer pending tokens should get the next request.
Batching awareness. If your inference server batches 32 requests together, the load balancer should know this and not overload it mid-batch.
Token budget tracking. Some setups allocate a "token budget" per backend (e.g., 500 tokens generated per second), and the load balancer respects this to prevent any one inference node from becoming the bottleneck.

Tools like vLLM and TensorRT-LLM expose metrics that a layer-7 load balancer or reverse proxy can inspect to make smarter routing decisions.

The Payoff: Silent Scaling

Your TicketRush setup now looks like this:

Two load balancers (active-passive) handling traffic
Four backend servers, each running the full TicketRush application
Health checks every 5 seconds
If one server dies, traffic reroutes instantly
If traffic spikes at 2 a.m., it spreads across all four servers; no human is paged
If you need more capacity, you spin up a fifth server, register it with the load balancer, and capacity grows with zero downtime

This is horizontal scaling. You bought six small servers instead of one giant one. You spent less money. You have redundancy. You can grow without touching the code.

A traffic flood becomes a non-event.

Verdict

Reach for NGINX when you need HTTP-level load balancing and want to manage the infrastructure yourself. Use HAProxy when you need the lowest latency or must balance non-HTTP traffic (databases, caches). Choose Envoy when you're running a service mesh or need dynamic configuration without restarts. Pick AWS ELB/ALB when you're on AWS and want fully managed, zero-ops scaling.

All of them solve the same problem: one server fails, users don't notice. Traffic spikes, they stay responsive. You add servers, capacity grows silently.

Watch the 90-second reel to see this in action.