--- title: "Latency vs Throughput: Time per Request vs Requests per Second (and When to Optimize Which)" description: "Latency and throughput both measure performance, but they answer different questions: how long ONE request takes versus how MANY you can serve per second. Here's the difference that actually helps you debug — milliseconds and p99 tail latency, RPS/QPS capacity, Little's Law, why pushing utilization toward 100% makes latency explode, and how the same trade-off shows up in LLM serving." keywords: "latency vs throughput, latency, throughput, p99, tail latency, requests per second, RPS, QPS, Little's Law, utilization, queueing, bottleneck, system design, backend performance, LLM serving, time to first token, tokens per second, batching" created_at: "2026-06-28T10:00:00" post_type: "anonym_post" content_type: "technical_article" --- ![Banner](./banner.webp)

Prefer to watch? ▶ Latency vs Throughput in 90 seconds ✈ Telegram

"Just make it faster" almost always means one of two completely different things — and people keep confusing them. Someone says the system is slow; one engineer starts shaving milliseconds off a single request while another starts adding workers to serve more traffic. They're both right, and they're solving different problems. The one-line mental model: - **Latency** is **how long ONE request takes** — time per request. - **Throughput** is **how MANY requests you can serve per second** — capacity. Fast and *a lot* are not the same thing. Picture a highway: **latency** is how long one car takes to drive from end to end; **throughput** is how many cars pass per minute. Add lanes and more cars get through every minute — but no single car arrives any sooner. --- ## Latency: time for one request Latency is the delay for a single request — you click, and how long until you get a response. It's measured in **milliseconds**, and the number that matters is almost never the average. The trap is reporting the mean. Averages hide your worst experiences. The number that actually reflects what users feel is the **tail** — the percentiles: ```text p50 = 40 ms ← half of requests are faster than this (the "typical" request) p95 = 220 ms ← 1 in 20 requests is slower than this p99 = 850 ms ← 1 in 100 requests is slower than this ← this is the one users complain about ``` A p50 of 40 ms looks great on a dashboard. But if your p99 is 850 ms, then **one in every hundred requests is painfully slow** — and a single page that fans out to 100 backend calls will almost certainly hit that slow tail on *every* page load. Report **p95 and p99**, not the average, because that's what your slowest users (and your fan-out requests) actually experience. > Rule of thumb: optimize latency for **user experience**. Lower latency feels snappy. --- ## Throughput: requests per second Throughput is how much you can push through the system in a given time. It's about **capacity, not speed**: - **RPS / QPS** — requests (or queries) per second - **bytes/s** — bandwidth - **jobs/min** — batch work The key insight: **throughput is independent of single-request latency.** Add more workers, more lanes, more servers, and throughput climbs — *even if each individual request takes exactly as long as before.* ```text 1 worker, each request 100 ms → ~10 req/s 4 workers, each request 100 ms → ~40 req/s ← 4× the throughput, same per-request latency ``` Each car still takes the same time to cross the highway. You just opened more lanes. > Rule of thumb: optimize throughput for **scale and cost** — more capacity per machine means fewer machines for the same traffic. --- ## How they relate: Little's Law Latency and throughput aren't independent dials you can crank separately. They're tied together by **Little's Law**, one of the most useful formulas in systems: ```text concurrency = throughput × latency L = λ × W ``` The number of requests **in flight** at any moment equals how fast they arrive (throughput) times how long each one stays (latency). This single relationship explains the trade-offs people keep tripping over: - **Add parallel lanes** → throughput climbs, but a single car doesn't arrive any sooner. You bought capacity, not speed. - **Batch requests together** → throughput climbs again (you amortize fixed costs across the batch), but each request now waits for the batch to fill, so **per-request latency goes up.** You're usually **trading one for the other.** Almost every "performance tuning" decision is really a choice about where you want to sit on this curve. --- ## The cliff: where it actually breaks Here's the part that bites teams in production. As you push **utilization** toward 100% to get more throughput out of your hardware, latency does **not** rise gently. It explodes. ```text utilization p99 latency 50% ~ 1× (plenty of headroom) 70% ~ 2× 85% ~ 5× 95% ~ 20× ← queues are building faster than they drain 99% ~ 100× ← the cliff ``` This is queueing theory, not bad luck: once arrivals approach the rate at which you can serve them, requests start **waiting in line**, and wait time grows toward infinity as you approach the limit. Your **average** can still look fine while the **tail falls off a cliff** — which is exactly why you watch p99, not the mean. Two practical consequences: 1. **Leave headroom.** Running at 95% "to save money" is how you get pager alerts. The last few percent of utilization costs you enormous tail latency. 2. **Find the bottleneck first.** Throughput is capped by your **slowest stage**, period. A request that's fast everywhere except one saturated database will be as slow as that database — and no amount of optimizing the other stages raises total throughput. Measure, find the constraint, fix *that*. --- ## The same trade-off in LLM serving If you work on AI systems, this is the exact tension behind every inference-serving decision — the vocabulary just changes: - **Latency** splits into **TTFT** (time to first token — how long until the response *starts* streaming) and **TPOT** (time per output token — how fast it streams after that). This is the user-experience number. - **Throughput** is **tokens per second across all requests** the GPU is serving — the cost number, because GPUs are expensive and you want every one saturated. And the lever that connects them is **batching**. **Continuous batching** (as in vLLM or TGI) packs many requests through the GPU together, which **massively raises tokens/sec throughput** — but a larger batch means any individual request can wait longer for its turn, nudging **latency up**. It's Little's Law again: more concurrency in flight buys throughput at the cost of per-request latency. So when someone asks "how many users can this model serve?" they're asking a throughput question, and "why does the chat feel laggy?" is a latency question — and the answer to both is *which batch size and how much GPU headroom*. Same curve, same cliff. --- ## Latency vs throughput at a glance

	Latency	Throughput
Question it answers	How long does ONE take?	How MANY per second?
Units	milliseconds (p50/p95/p99)	RPS · QPS · bytes/s · tokens/s
What it's about	Speed	Capacity
How to improve it	Faster code, caching, fewer hops	More workers/lanes/servers, batching
Optimize it for	User experience	Scale & cost
Watch out for	The tail (p99), not the average	The bottleneck stage; the utilization cliff

--- ## The verdict - **Latency** is **time per request** — optimize it for **user experience**, and report **p95/p99**, not the average. Your tail is what your slowest users feel. - **Throughput** is **requests per second** — optimize it for **scale and cost**. Add workers and it climbs, even when each request takes exactly as long as before. - They're linked by **Little's Law** (`concurrency = throughput × latency`), so you're usually **trading one for the other** — widening the road or batching buys throughput at the cost of latency. - There's a **cliff**: as utilization approaches 100%, queues build and latency explodes while the average still looks fine. Leave headroom, and **find the bottleneck before you tune anything.** Different goals, different dials. Measure both, watch your tail, and the next time someone says "make it faster," you'll know which problem they actually have. > Want the 90-second visual version? [Watch the reel.](https://youtube.com/shorts/PxqVH7iupC0)