Chaos Engineering vs Load Testing: One Measures the Ceiling, the Other Measures the Floor

Prefer to watch? ▶ The full 11-minute episode ⚡ The 2-minute version ✈ Telegram

A service passes a load test at ten times its normal traffic. The report is green, the launch goes ahead — and the following week the whole checkout falls over, at ordinary Tuesday volume, because one non-critical downstream service got slow. Not down. Slow.

Nothing about that is a contradiction, and nothing about it is bad luck. The load test answered a question honestly. It just was not the question that took the system down.

Both practices in the title break your own system on purpose, which is why they end up filed under the same heading. They are opposites:

A load test keeps the system healthy and raises the traffic until something bends. It finds a ceiling.
A chaos experiment keeps the traffic normal and breaks one thing on purpose. It finds the floor — what still works when a part of the system does not.

Load testing validates your capacity plan. Chaos engineering validates your beliefs. Those are not the same artefact, and passing one says nothing about the other.

The subject has a name: resilience testing

The umbrella is resilience testing — deliberately putting a system under conditions it will eventually meet anyway, on a quiet weekday, while somebody is watching, instead of at 03:14 on a Sunday while a customer runs the experiment for you.

It covers load and stress testing, chaos experiments, and game days. It does not cover:

Unit and integration tests — they check that the code does what it was written to do. Resilience testing asks what happens when something else stops doing that.
Monitoring — it tells you what happened. It never chooses what to break.
A postmortem — that is the same lesson, bought at retail price, from a customer.

The uncomfortable framing: for most systems in production right now, nobody on the team knows how it fails. Not through carelessness — because nobody has ever found out.

The rope bridge

A rope bridge across a gorge: wooden planks, four ropes holding the whole thing up.

The load test walks people onto the bridge. Ten, then thirty, then fifty, watching the planks bow, until it can say: this bridge holds forty-two people. Every rope is intact the entire time. The answer is a number, and the number is a weight limit.

The chaos experiment keeps three people on the bridge — a normal load, nothing dramatic — and cuts one of the four ropes on purpose, because the drawing says four ropes and any three of them can hold the deck. That claim is written on the plan. It has never been executed. The experiment is an audit of the drawing, and the answer is not a number, it is either "the drawing was right" or "the drawing was wrong, and now we know it before it mattered".

Same bridge, same day, two questions that cannot substitute for each other.

The cheap one first: a load test is a ramp

A load test is a staircase of synthetic traffic and a set of numbers you watch while it climbs. The important thing about this file is what is absent from it: nothing anywhere is broken.

# loadtest/ramp.py  —  drive synthetic traffic up a staircase

STAGES = [                        # nothing is broken during ANY of this
    (   500, "2m"),               # warm the caches, fill the pools
    ( 5_000, "5m"),               # a busy normal day
    (50_000, "5m"),               # the number in the launch plan
]

def healthy(response, elapsed_ms):
    return response.status == 200 and elapsed_ms < 400

# what you record — never the average, it hides everything
report("p50",        percentile(latencies, 50))
report("p99",        percentile(latencies, 99))   # this one finds the knee
report("saturation", cpu, memory, db_connections_in_use)

Three details carry the whole practice.

The stages ramp, they do not jump. Starting at the target number measures your cold caches and your empty connection pools, not your system.

The average is banned. A mean latency of 120 ms is compatible with 95% of requests at 40 ms and 5% at 1.7 seconds — and the 5% is the part that generates support tickets. Percentiles, or nothing.

Saturation is recorded next to latency, because the useful output of a load test is not "it broke at 8,000". It is which resource ran out first.

The knee

requests / second	p50	p99	what is saturating
500	22 ms	88 ms	nothing — idle
5,000	24 ms	91 ms	CPU ~40%, pool half used
8,000	31 ms	340 ms	DB connections queueing ← the knee
12,000	96 ms	2.9 s	everything waits on the pool

Between 5,000 and 8,000 the p99 stops being flat and starts climbing faster than the traffic does. That bend is the answer. Notice that the p50 barely moves across the same range — a dashboard showing median latency would have called the third row healthy.

And the bend has a cause, not just a location: connections queueing. That distinction is what makes the result actionable. "We break at 8,000" is trivia. "We break at 8,000 because one pool of 40 connections is the binding constraint" is a work item.

The honest limit of every load test

A load test measures the system you have, under the traffic you imagined.

Read that again slowly, because both halves are limits. The traffic is a guess about the future shaped like the past. And everything inside the system is healthy by definition for the duration of the run — every dependency answers, every disk is present, every node is up.

So a green load test licenses exactly one sentence: at this traffic, with everything working, we are fine.

The outage in the opening paragraph did not violate that sentence. It happened in the half the test never covered: normal traffic, with something not working.

The other one: hold traffic normal, break one thing

Same system, opposite variable.

# chaos/latency_experiment.py
# traffic stays at a normal Tuesday. exactly ONE thing changes.

HYPOTHESIS = (
    "If the payments call gets 300ms slower, checkout still answers "
    "in under one second — because that call has a timeout, and a fallback."
)

def steady_state():               # measured BEFORE the fault, not after
    return orders_per_minute() > 40 and checkout_p99_ms() < 1000

FAULT = inject_latency(
    target   = "payments.internal.example.com",
    delay_ms = 300,
    share    = 0.05,              # five percent of calls. not all of them.
)

The first thing in the file is not code. It is a sentence — a claim, in plain language, with a because attached. That structure is not decoration:

A hypothesis makes the experiment falsifiable. Without it you are not testing, you are poking production and forming opinions afterwards.
The because clause names the mechanism you are actually auditing — a timeout and a fallback. If the system survives for some other reason, you have learned something different from what you set out to learn, and you would never notice without the clause.

steady_state() is measured first, and it is a business signal, not a CPU graph. Orders per minute is the thing you care about; CPU is a proxy that can look fine through a total outage. Measuring it before the fault is what turns "the system is bad now" into "the system got worse because of this".

share = 0.05 is the blast radius. Injecting 300 ms into every call is not an experiment, it is an outage you scheduled. Five percent is enough to observe and small enough that being wrong is survivable — and being wrong is a normal outcome here.

Note also what is not being injected: not 30 seconds, not a hard failure. 300 milliseconds of extra delay on a call that already succeeds. Slow is a far more interesting fault than down, because down is the case people remember to handle.

The trap: capacity plans and beliefs

This is the part that is routinely inverted.

Take any architecture diagram and read it not as a picture but as a list of claims nobody has executed:

What the diagram says	What is often true underneath
"That call has a timeout"	The timeout is on connect, not on read — so a slow response waits forever
"It retries on failure"	Three retries with no pause between them — the retry storm finishes the job the fault started
"The cache serves stale data if the service is down"	The cache warms itself from that service, so it is empty exactly when it is needed
"Search is optional on the product page"	The render awaits it, so an optional dependency is a hard one at runtime
"The queue absorbs the spike"	The consumer is what fell over; the queue absorbs it into a four-hour backlog

Every row is the same shape: a sentence that is true in the design and false in the deployment. No amount of traffic finds any of them, because at every level of load — 500 or 50,000 requests a second — the timeout is never reached, the fallback never runs, the retry never fires. The code paths under test are only entered when something is broken.

That is why a system can pass a 10× ramp on Monday and be taken down on Thursday by one non-critical service getting slower. The ramp exercised the paths that work. The outage lived in the paths that had never once executed.

The discipline: what separates an experiment from an outage

The runner is where a chaos experiment earns the right to touch anything real.

    def run(self, exp):
        if not exp.steady_state():             # is it healthy RIGHT NOW?
            return self.abort("not steady before we even started")

        exp.fault.apply(share=0.05)            # the blast radius
        try:
            for _ in range(exp.duration_s):
                time.sleep(1)
                if not exp.steady_state():     # the abort button
                    return self.stop(exp, "hypothesis disproved")
        finally:
            exp.fault.remove()                 # ALWAYS. even if this crashes.

        return "hypothesis held"

Four rules, all of them visible in ten lines:

Check steady state before starting. If the system is already degraded, the run teaches nothing and makes a bad afternoon worse. A degraded system is a reason to postpone, not to proceed carefully.
Limit the blast radius explicitly, as an argument you can read, not an implicit "well, staging is small".
The abort is automatic and continuous. A human watching a dashboard is not an abort button; the hypothesis is checked every second by the thing that caused the fault.
finally removes the fault, even if the runner itself crashes. A fault injector that can leak its own fault is strictly worse than never having run it — you have now added a failure mode instead of finding one.

And the practice has a prerequisite that is easy to skip past: the result has to be observable. If the system's health cannot be read in seconds, from the outside, in a business signal, then there is no steady state to measure and no abort condition to evaluate. You cannot run an experiment on a system you cannot see.

A disproved hypothesis is the good outcome

Result	What you walk away with
Hypothesis held	One belief is now evidence, with a date on it. Valuable, and much less than it sounds — it is true for this fault, at this share, on this build.
Hypothesis disproved	A real outage was found on a Tuesday, at 5% blast radius, with the abort already armed and the whole team present. That is the point of the exercise.

A team that only ever confirms its hypotheses is picking safe experiments. The interesting ones are the beliefs nobody wants to test.

What each one honestly costs

	Load testing	Chaos engineering
Risk to production	Low — run it off to the side	Real, by construction
Hard part	Making the traffic realistic — cache hit rates, payload sizes, the ratio of reads to writes	Organisational permission, and the observability to see the result
Fails by	Passing while being wrong — a test that hits one warm endpoint proves nothing	Being skipped, or run as theatre with no hypothesis
Prerequisite	An environment shaped like production	Health readable as a business signal in seconds
Cadence	Before launches, and on capacity changes	Scheduled, small, boring, repeated

Load testing is cheap and safe, and its cost is entirely in fidelity — a synthetic run with a 99% cache hit rate against a production reality of 60% measures a system that does not exist. Chaos is technically simpler and expensive in trust: the fault is real, in a real environment, and the first one costs a conversation, not a library.

The same two questions on an AI serving stack

None of this is a backend-only concern; an LLM serving path is just a distributed system whose slowest dependency happens to be a GPU.

The load test for an inference service is still a ramp, but the numbers change. Requests per second is nearly meaningless when one request is 40 output tokens and the next is 4,000, so the ramp is driven in concurrent streams with a realistic prompt-length mix, and what gets recorded is:

TTFT (time to first token) at p99 — the number the user actually feels, and the one that degrades first as the batch queue grows.
TPOT (time per output token) at p99 — throughput once streaming has started.
Saturation of the real binding constraint, which is usually not CPU: KV-cache memory, the batch queue depth, or the number of admitted concurrent sequences.

The knee looks exactly like the one in the table above. Continuous batching keeps TTFT flat while there is KV-cache headroom, and then requests start queueing for admission and TTFT bends sharply while TPOT stays almost unchanged — the same "median looks fine" trap, one layer up.

The chaos experiment on the same stack holds traffic at a normal Tuesday and audits the sentences in the RAG or agent diagram, which are exactly as unexecuted as any other:

"If the model provider gets slow, we fall back to the smaller model." Injected as 2 seconds of extra delay on 5% of completion calls. Frequently disproved by a timeout that is longer than the user's patience, or a fallback that is never reached because the retry consumed the budget first.
"If the vector store is unavailable, the answer degrades to the model's own knowledge." Injected as errors on 5% of retrieval calls. Frequently disproved by an exception that takes down the whole request instead.
"A rate-limit response is retried with backoff." Injected as 429s on a small share. Frequently disproved by three immediate retries that turn one rate limit into a self-inflicted outage.
"A half-finished stream is handled." Injected by cutting the connection mid-response — the case nobody wrote a branch for, because down is remembered and partial is not.

The pattern transfers whole: raising the load finds the capacity ceiling of your GPU fleet; breaking one thing finds out whether the fallback chain in the diagram exists anywhere other than the diagram.

The verdict

	Load testing	Chaos engineering
The system is	Completely healthy	Deliberately broken, in one place
The traffic is	Rising, up a ramp	Normal — a boring Tuesday
The question	How much can this take?	What happens when a part of it stops working?
The output	A number: the capacity ceiling	A verdict on a belief
It validates	Your capacity plan	Your architecture diagram
Blind to	Every failure path, because nothing fails	The volume you have never carried

One of them measures the ceiling. The other measures the floor. Most systems have measured neither — only an average Tuesday.

A practical order of operations, if both are new:

Make health readable first. One business-level signal, visible in seconds. Without it, a load test has no verdict and a chaos experiment has no abort button.
Run the ramp. It is the cheap, safe one, and its output — which resource saturates first — is the map you need before touching anything on purpose.
Write one hypothesis down, as a sentence with a because. Choose the belief you would least like to be wrong about.
Inject small, abort automatically, remove in finally. Five percent, one fault, one variable.
Expect to be wrong, and treat that as the return on the exercise rather than a failure of the exercise.

Then read your own architecture diagram again. Every arrow on it is a claim; the only question is whether it has ever been executed, and by whom — you, on a Tuesday, or a customer, at 03:14.

Want the long version? The full 11-minute episode walks all three files line by line on screen — the ramp and its knee, the hypothesis and its blast radius, and the runner's abort button — including the timeout-on-connect reveal. The 2-minute version is here if you only want the trap.