Active-Active vs Active-Passive: You Had a Second Server, So Why Were You Down for 41 Minutes?

Prefer to watch? ▶ The full 11-minute episode ⚡ The 2-minute trailer ✈ Telegram

Two in the morning. A payments service, one database, and a second identical machine sitting right next to it, copying every change as it happens. Then the first machine's disk controller dies — no warning, no slow decline, it just stops answering.

And you had planned for this. There is a second machine, right there, fully in sync. It has been in sync for ten months.

The service is down for forty-one minutes.

Not because the standby was missing. Not because the data was gone. The standby was fine the whole time. Every one of those forty-one minutes was spent switching over to it.

That's the question this article answers: you had a second server — so why were you down for forty-one minutes?

The subject: redundancy, and its two shapes

"We have a second one" is the same sentence in both designs. It means two completely different things.

Active-passive — the second machine waits. It's kept up to date, and it gets promoted when the first one dies.
Active-active — both machines take live traffic, all the time.

Two things this is not. It isn't backups: a backup is a copy of yesterday, this is a copy of right now. And it isn't horizontal scaling — that's about adding capacity, and it's a different problem. This is the shape that sits underneath every cloud provider's version of it.

One picture first

Think of a building with an emergency generator in the basement. It's fuelled. It's serviced. Once a month somebody starts it for ten minutes with nothing plugged into it, ticks a box, and switches it off. That building is active-passive.

Now think of a building fed by two power lines from two different substations, each carrying half the load, all day. One line fails and the other picks up the rest. Nothing starts up. Nothing gets promoted. The lights don't even flicker. That building is active-active.

Here's the part that matters. In the second building you already knew the second line worked — because the lights it was powering were on. The proof was the service itself. In the first building, the generator has proved it can start. It has never once proved it can carry the building.

Active-passive, in code

Two machines in a config, one named primary and one named standby. Now look at the routing function — the thing that decides where an incoming request goes:

# config.py — addresses from RFC 5737 (documentation range)
NODES = {
    "primary": {"host": "192.0.2.10", "role": "primary"},
    "standby": {"host": "192.0.2.11", "role": "standby"},
}

def route(request):
    return NODES["primary"]          # ...that is the entire function

def health_check():
    return ping(NODES["primary"])    # nobody ever asks the standby

That's it. There is no branch in it. There is no condition. The word standby appears in the config, and then it never appears again anywhere on the path a request takes.

Look at the health check too — it only ever asks about the primary. Nobody is asking the standby whether it could serve a request, because nobody ever intends to send it one. That is what passive means: the second machine is invisible to production until the day you need it.

Failover is not an instant

People say "failover" as if it were a single moment. It's a state machine, and naming the states is worth doing:

Serving — everything normal, all traffic to A.
Suspect — health checks are failing, but you're not sure yet. Was that the machine, or the network between you and the machine?
Promoted — you've committed. B is the primary now.

Then there's the transition nobody plans for: the old machine comes back. It still has your data. It still believes it's the primary. If it starts accepting writes again, you now have two primaries and two versions of the truth. That last arrow is where most real failover incidents actually go wrong.

The four steps, and what they cost

def failover():
    # 1. DETECT — three failed checks, ten seconds apart.
    #    Three, not one: a single missed check is usually a network hiccup,
    #    and promoting on a hiccup is worse than waiting.
    wait_for(consecutive_failures=3, interval=10)      # ~30 s

    # 2. FENCE — make certain the old machine cannot accept another write.
    #    Cut its network, revoke its credentials, power it off.
    #    Skipping this is exactly how you get two primaries.
    fence(NODES["primary"])                            # ~2 s

    # 3. PROMOTE — standby applies what it hadn't caught up on, opens for writes.
    promote(NODES["standby"])                          # ~15 s

    # 4. REPOINT — every client still holds the old address.
    #    Until they let go of it, the promotion changed nothing they can see.
    update_dns(ttl=60)                                 # ~60 s

Add it up, for a failover where everything is automated and nothing goes wrong: 30 + 2 + 15 + 60 = 107 seconds. Just under two minutes, and that's the good number — the one you can defend in a design review.

The number in the story at the top was forty-one minutes. And the difference between them is not technology. The difference is that two of those four steps had never been run before, so somebody had to work out what they were while the site was down.

An idle standby is an untested standby

Your standby has been sitting there, in sync, for three hundred days. In that time your service handled about two and a half billion requests, and the standby served zero of them. What has it actually proved?

A few real things, and they're worth monitoring: the process is running, replication is keeping up, there's disk space.

Now look at what it has not proved:

That the promote command works while the machine is under load.
That the address change actually reaches your clients.
That your application reconnects, instead of sitting there holding a pool of dead connections.
That the alert wakes up a human being.

Every single one of those runs for the first time during your worst hour.

Active-active, in code

Same two machines. Now both are in the pool, both have a weight, and the routing function asks which ones are healthy:

POOL = [
    {"host": "192.0.2.10", "healthy": True, "weight": 1},
    {"host": "192.0.2.11", "healthy": True, "weight": 1},
]

def route(request):
    live = [n for n in POOL if n["healthy"]]      # asks about BOTH, every second
    return weighted_choice(live)

def on_health_failure(node):
    node["healthy"] = False                       # ...that is the whole handler

Watch what the failure handler does. It sets a flag to false. There's no promote. There's no fence. There's no address change — because the address was never pointing at a machine, it was pointing at the pool.

When node B dies at 02:14, three seconds later the healthy list has one entry in it, and you're running at 50% capacity and 100% availability.

And here's the quiet part: that same line runs every single time you deploy. The code that handled your outage had already run a thousand times this month.

That's the real argument for active-active, and it isn't the one people usually give. People say it's about capacity, or being closer to users. Both are true, and both are secondary. The actual argument is that active-active deletes the failover procedure instead of trying to improve it — there's no promotion step to get wrong because nothing is promoted, no fencing problem because there's no single writer to fence, and the second machine is continuously proven to work for the simple reason that it is working. If it were broken, you'd have found out on an ordinary Tuesday afternoon, not at two in the morning.

The bill for active-active

So why does anybody still run active-passive? Because active-active isn't free redundancy. It's a trade, and you should hear the bill before you sign it.

Off your plate: no promotion, no fencing, no address change, no path that has never been executed.

Handed to you instead:

1. Two live writers, so you own conflicts

One product, ten units in stock. Two customers each buy one, forty milliseconds apart, landing on different machines:

A: read stock=10 → 10-1 → write 9
B: read stock=10 → 10-1 → write 9      # last write wins
                              stock = 9   ← two sales, one decrement

You didn't get an error. You didn't get a conflict warning. You got data that is quietly, plausibly wrong.

-- the fix: every row carries a version, and a write must say
-- which version it believes it is replacing
UPDATE inventory
   SET stock = 9, version = 8
 WHERE id = 42 AND version = 7;   -- 0 rows updated → somebody moved first → raise a conflict

That's one extra field, plus a real decision about what your application does when it fires.

2. Shared session state

Anything you were keeping in memory on one machine has to move somewhere both of them can see it.

3. Capacity planning changes

With two sites, each has to be able to carry the whole load alone — which means each one normally runs at about half.

4. Split brain

The network between your two sites breaks, but both sites are perfectly healthy and both can still reach users. Each half looks around, sees no partner, and quite reasonably concludes the other one died. So both keep serving. Both keep writing. When the link comes back you have two divergent versions of the same data and no automatic way to say which is real.

The standard answer is a quorum: you need a majority to keep accepting writes, which is why these systems are built with three nodes rather than two — so a split always leaves one side in the minority, and the side that loses the vote stops serving.

Being briefly unavailable is something you recover from. Being briefly wrong, silently, very often is not.

The honest comparison

	Active-Passive	Active-Active
Second machine	Idle, invisible to production	Live, serving traffic
Routing function	No branch — returns the primary	Picks from the healthy pool
Failure response	Detect, fence, promote, repoint	Set a flag to false
Best-case downtime	~107 s automated (41 min if untested)	~seconds, capacity drops to 50%
Is the spare proven?	No — first run is your worst hour	Yes — it ran today
Write conflicts	None (one writer)	Yours to solve (versioning)
Session state	Can live in memory	Must be shared
Split brain	Not asked	Needs a quorum (3 nodes)
Normal utilisation	~100% of one machine	~50% of each

What this looks like when you serve an LLM

The same two shapes decide how you run inference, and the split is unusually clean — which makes it a good way to check you've understood the rule.

Your model-serving replicas are stateless: a GPU worker holding weights takes a request, returns tokens, and remembers nothing about you afterwards. That's the free case for active-active. Put every replica behind one pool, health-check them all continuously, and a dead GPU node is a flag flip and a capacity drop — not a promotion. You also get the thing an idle standby can never give you: a spare GPU that has been proven to load the weights and answer, because it answered a minute ago. A "warm standby GPU" that has never served a token is exactly the generator in the basement, and it costs the same per hour as one that's working.

Two caveats worth naming, because they're the same bill in new clothes:

The KV cache is session state. Multi-turn conversations and prefix caching make a replica feel stateful — routing a follow-up to a different node throws away the cached prefix and you pay to prefill again. That's the "shared session state" line on the bill, and the usual answer is the usual answer: session-affinity routing, or lift the cache somewhere both replicas can see. Note that this is a performance cost, not a correctness one — you get a slower answer, not a wrong one, which is exactly why the stateless tier is still the easy case.
The stateful tier behind it is not stateless. Your vector store, your metadata database, the table you write conversations and evals into — those have all the conflict and split-brain problems from the previous section. A vector index accepting writes on two sides can silently diverge in what it returns, and "the retriever quietly stopped seeing some documents" is the RAG version of the lost update: no error, plausible output, wrong answer.

Which is the general rule, arrived at from the other direction.

So which should you build? Almost certainly both

Not one for the whole company — one per layer.

Web servers, API processes, model replicas — anything that doesn't remember anything between one request and the next: run those active-active. They have no conflicts to have, so you get all of the upside and pay none of the bill.
Your main database, the single source of truth — active-passive is usually the honest answer. One writer, one version of the truth, a replica standing ready. That is not a compromise, that's the right tool.

Anyone who tells you their entire system is active-active is describing the front of it.

The sentence to keep

You are not choosing how many machines you own. In both designs, you own two.

You are choosing whether failing over is a routine or an event.

Active-passive makes it an event: a procedure that runs about once a year, at the worst possible moment, carried out by a tired person reading a document. Active-active makes it a routine: a flag that flips, on a path that already ran today.

And if you do keep an active-passive tier — which you probably should — then the only honest thing to do is to fail over to it on purpose, on a Tuesday afternoon, while everybody is awake.

The full walkthrough, with all four failover steps as real code: watch the 11-minute episode. Short on time? Here's the 2-minute version.