Continuous Batching: How One GPU Serves Hundreds of Chat Users at Once

Prefer to watch? ▶ Continuous Batching in 90 seconds ✈ Telegram

A single GPU serving 40 chat users at once is not a hardware problem—it's a scheduling decision. The key is recognizing that token generation is a loop, not a one-shot event, which means the scheduler can act every iteration instead of locking the batch at the start.

Mental model: Continuous batching is a scheduler that stops keeping empty seats reserved and instead fills them with waiting requests at each token step, changing the batch composition mid-flight.

The Foundation: Why Batching Matters

When you generate a reply, you build it one token at a time. Each step—producing token 1, then token 2, then token 3—requires reading the entire model's weights from GPU memory. That memory read dominates the cost, not the compute itself.

If you serve one user at a time, the GPU sits idle between their requests and while they think about typing more. The memory bandwidth is barely used.

Now put eight users through the same token step: read the weights once, compute output for all eight, write eight tokens. You get roughly 8× the throughput for almost the same time. That's the win of batching.

Static Batching and Its Cost

Conventional systems (static batching) lock the batch at the start of generation. You decide: "I'll process eight requests together," schedule that work, and nothing changes until all eight finish.

Here's the problem: replies have wildly different lengths. User A asks "Does 9am work?"—12 tokens. User B asks "Write a detailed migration plan for our infrastructure"—500 tokens. You didn't know that when the batch started.

At token step 20, User A is done. So are Users C, D, E, F, and G. Seven seats are now idle and reserved. They're still taking up memory, still marked as "in use" by the GPU scheduler, still burning power—but they're not doing anything. Meanwhile, requests wait in the queue outside, unable to claim those seats.

The batch is locked for 480 more steps until User B finishes. That's the waste.

The Insight: A Loop, Not a Batch

Token generation is a loop:

while not_all_done:
    logits = model(input_ids)         # Read weights, compute
    next_token = sample(logits)       # Pick the token
    input_ids.append(next_token)      # Add it to the sequence

Each iteration is a scheduling boundary. The scheduler doesn't have to wait for the entire batch to finish; it can act every step. That's the insight behind continuous batching.

Continuous Batching: Changing the Batch Mid-Flight

Continuous batching applies iteration-level scheduling: at each token step, the scheduler:

Evicts finished sequences (those that hit the end-of-sequence token or max length).
Admits waiting requests from the queue into the freed slots.
Continues the next token step with a new composition.

The batch shape changes every step. Slot 1 might hold User A for steps 1–12, then User H for steps 13–300. Slot 5 might hold User B for the full 500 steps, but User C for only steps 1–8.

No seats go cold. The GPU always has work queued, and every available slot finds a user to serve.

The Mechanism in Practice

Consider this scenario:

Step 1:    [User A, User B, User C, User D] (4 slots, in flight)
           GPU is reading weights, computing 4 users' next tokens.

Step 12:   User A finishes (12-token reply).
           [X, User B, User C, User D] → admit User E from queue
           [User E, User B, User C, User D]
           GPU continues with 4 slots, no idle.

Step 42:   Users C and D finish (both ~40 tokens).
           [User E, User B, X, X] → admit Users F and G
           [User E, User B, User F, User G]

Step 500:  User B finishes (500-token reply).
           [User E, X, User F, User G] → admit User H
           [User E, User H, User F, User G]

None of the intermediate steps have idle slots. The batch is never locked.

Static vs. Continuous: A Comparison

Dimension	Static Batching	Continuous Batching
When is batch locked?	Once at the start (request-level)	Never (iteration-level admission)
What happens when a reply finishes?	Slot stays reserved, idle, waiting for slowest	Slot is immediately refilled from queue
GPU utilization when replies vary widely?	Low (~9% if 8 slots, 1 reply left)	High (~93%, rarely has idle slots)
Queue latency (time waiting to start)?	Waits for next batch cycle	Enters at next step if space exists
Per-user token throughput (tok/s)?	High (fewer competitors per step)	Lower (more concurrent sequences)

The Trade-Off

Continuous batching is not free. Every additional sequence in flight shares the token-generation step. If 40 sequences run in parallel, each one's tokens arrive slower than if you served them one at a time or in a small batch.

The trade-off is explicit:

Per-user latency per token (TPOT) increases as you add sequences.
System throughput (total tok/s across all users) increases until memory or compute becomes the bottleneck.

A single user on a dedicated GPU might get 100 tokens/second. That same user on a GPU serving 40 concurrent users might get only 2–3 tokens/second—but the GPU is producing 80–120 tokens/second total across all 40 users. For a chat service, total throughput often matters more than individual latency, as long as each user's latency stays acceptable (typically <100ms per token).

Memory is also a constraint: the more sequences in flight, the more KV cache must be stored. Eventually, you hit a ceiling where adding more sequences just causes swapping or eviction, making things worse.

In the Context of LLM Inference

Continuous batching is fundamentally a solution to the iteration-level scheduling problem in LLM inference. Standard metrics help frame it:

TTFT (time to first token): Continuous batching improves this by admitting requests as soon as a slot opens, not waiting for the next batch cycle.
TPOT (time per output token): Continuous batching increases this per user because sequences share GPU resources, but the system total improves dramatically.
Batch utilization: Static batching wastes slots when replies finish early. Continuous batching keeps slots full, pushing utilization toward 90%+.

vLLM, Hugging Face TGI, TensorRT-LLM, and SGLang all use continuous batching as the core scheduler. Without it, serving hundreds of concurrent users would require either a very large batch (consuming huge amounts of memory) or serving users slowly one at a time.

One More Thing: Why This Isn't About Memory

You might hear about PagedAttention alongside continuous batching. They solve different problems. PagedAttention is a memory management technique that virtualizes the KV cache, letting you fit more sequences into a fixed amount of GPU memory by storing blocks non-contiguously (like paging in an OS).

Continuous batching is scheduling: deciding which sequences run at each step. You can use either, both, or neither independently. PagedAttention makes continuous batching more practical by lowering memory overhead, but continuous batching doesn't require it.

The Verdict

Reach for static batching when you have fixed, known batch sizes and all requests are similar lengths (rare in practice). Use continuous batching whenever you serve multiple LLM users concurrently and replies vary in length—which is almost always. It's the scheduling difference between a GPU sitting idle waiting for stragglers and a GPU running full speed feeding a queue.

Same GPU. Same model weights. Same compute capacity. Just a scheduler that stopped keeping empty seats warm.

Watch the 90-second reel for the visual walkthrough of how one step at a time lets the batch reshape itself.