Latency vs Throughput: Time per Request vs Requests per Second (and When to Optimize Which)
Latency and throughput both measure performance, but they answer different questions: how long ONE request takes versus how MANY you can serve per second. Here's the difference that actually helps you debug — milliseconds and p99 tail latency, RPS/QPS capacity, Little's Law, why pushing utilization toward 100% makes latency explode, and how the same trade-off shows up in LLM serving.
"Just make it faster" almost always means one of two completely different things — and people keep confusing them. Someone says the system is slow; one engineer starts shaving milliseconds off a single request while another starts adding workers to serve more traffic. They're both right, and they're solving different problems.
The one-line mental model:
- Latency is how long ONE request takes — time per request.
- Throughput is how MANY requests you can serve per second — capacity.
Fast and a lot are not the same thing. Picture a highway: latency is how long one car takes to drive from end to end; throughput is how many cars pass per minute. Add lanes and more cars get through every minute — but no single car arrives any sooner.
Latency: time for one request
Latency is the delay for a single request — you click, and how long until you get a response. It's measured in milliseconds, and the number that matters is almost never the average.
The trap is reporting the mean. Averages hide your worst experiences. The number that actually reflects what users feel is the tail — the percentiles:
p50 = 40 ms ← half of requests are faster than this (the "typical" request)
p95 = 220 ms ← 1 in 20 requests is slower than this
p99 = 850 ms ← 1 in 100 requests is slower than this ← this is the one users complain about
A p50 of 40 ms looks great on a dashboard. But if your p99 is 850 ms, then one in every hundred requests is painfully slow — and a single page that fans out to 100 backend calls will almost certainly hit that slow tail on every page load. Report p95 and p99, not the average, because that's what your slowest users (and your fan-out requests) actually experience.
Rule of thumb: optimize latency for user experience. Lower latency feels snappy.
Throughput: requests per second
Throughput is how much you can push through the system in a given time. It's about capacity, not speed:
- RPS / QPS — requests (or queries) per second
- bytes/s — bandwidth
- jobs/min — batch work
The key insight: throughput is independent of single-request latency. Add more workers, more lanes, more servers, and throughput climbs — even if each individual request takes exactly as long as before.
1 worker, each request 100 ms → ~10 req/s
4 workers, each request 100 ms → ~40 req/s ← 4× the throughput, same per-request latency
Each car still takes the same time to cross the highway. You just opened more lanes.
Rule of thumb: optimize throughput for scale and cost — more capacity per machine means fewer machines for the same traffic.
How they relate: Little's Law
Latency and throughput aren't independent dials you can crank separately. They're tied together by Little's Law, one of the most useful formulas in systems:
concurrency = throughput × latency
L = λ × W
The number of requests in flight at any moment equals how fast they arrive (throughput) times how long each one stays (latency). This single relationship explains the trade-offs people keep tripping over:
- Add parallel lanes → throughput climbs, but a single car doesn't arrive any sooner. You bought capacity, not speed.
- Batch requests together → throughput climbs again (you amortize fixed costs across the batch), but each request now waits for the batch to fill, so per-request latency goes up.
You're usually trading one for the other. Almost every "performance tuning" decision is really a choice about where you want to sit on this curve.
The cliff: where it actually breaks
Here's the part that bites teams in production. As you push utilization toward 100% to get more throughput out of your hardware, latency does not rise gently. It explodes.
utilization p99 latency
50% ~ 1× (plenty of headroom)
70% ~ 2×
85% ~ 5×
95% ~ 20× ← queues are building faster than they drain
99% ~ 100× ← the cliff
This is queueing theory, not bad luck: once arrivals approach the rate at which you can serve them, requests start waiting in line, and wait time grows toward infinity as you approach the limit. Your average can still look fine while the tail falls off a cliff — which is exactly why you watch p99, not the mean.
Two practical consequences:
- Leave headroom. Running at 95% "to save money" is how you get pager alerts. The last few percent of utilization costs you enormous tail latency.
- Find the bottleneck first. Throughput is capped by your slowest stage, period. A request that's fast everywhere except one saturated database will be as slow as that database — and no amount of optimizing the other stages raises total throughput. Measure, find the constraint, fix that.
The same trade-off in LLM serving
If you work on AI systems, this is the exact tension behind every inference-serving decision — the vocabulary just changes:
- Latency splits into TTFT (time to first token — how long until the response starts streaming) and TPOT (time per output token — how fast it streams after that). This is the user-experience number.
- Throughput is tokens per second across all requests the GPU is serving — the cost number, because GPUs are expensive and you want every one saturated.
And the lever that connects them is batching. Continuous batching (as in vLLM or TGI) packs many requests through the GPU together, which massively raises tokens/sec throughput — but a larger batch means any individual request can wait longer for its turn, nudging latency up. It's Little's Law again: more concurrency in flight buys throughput at the cost of per-request latency.
So when someone asks "how many users can this model serve?" they're asking a throughput question, and "why does the chat feel laggy?" is a latency question — and the answer to both is which batch size and how much GPU headroom. Same curve, same cliff.
Latency vs throughput at a glance
| Latency | Throughput | |
|---|---|---|
| Question it answers | How long does ONE take? | How MANY per second? |
| Units | milliseconds (p50/p95/p99) | RPS · QPS · bytes/s · tokens/s |
| What it's about | Speed | Capacity |
| How to improve it | Faster code, caching, fewer hops | More workers/lanes/servers, batching |
| Optimize it for | User experience | Scale & cost |
| Watch out for | The tail (p99), not the average | The bottleneck stage; the utilization cliff |
The verdict
- Latency is time per request — optimize it for user experience, and report p95/p99, not the average. Your tail is what your slowest users feel.
- Throughput is requests per second — optimize it for scale and cost. Add workers and it climbs, even when each request takes exactly as long as before.
- They're linked by Little's Law (
concurrency = throughput × latency), so you're usually trading one for the other — widening the road or batching buys throughput at the cost of latency. - There's a cliff: as utilization approaches 100%, queues build and latency explodes while the average still looks fine. Leave headroom, and find the bottleneck before you tune anything.
Different goals, different dials. Measure both, watch your tail, and the next time someone says "make it faster," you'll know which problem they actually have.
Want the 90-second visual version? Watch the reel.