---
title: "How Should Your LLM Stream Tokens? SSE vs WebSockets vs Polling"
description: "Your chatbot dumps the whole answer after an 8-second stare. ChatGPT types it out token by token. The difference is one transport decision — and for LLM streaming the answer is almost always Server-Sent Events, not WebSockets. Here's why, with the FastAPI code."
keywords: "LLM streaming, token streaming, SSE, Server-Sent Events, WebSockets, polling, FastAPI StreamingResponse, OpenAI streaming, real-time LLM, time to first token, nginx proxy buffering, LLMOps"
created_at: "2026-06-25T09:00:00"
post_type: "anonym_post"
content_type: "technical_article"
---
ChatGPT types its answer out one token at a time. Your chatbot makes the user stare at a spinner for eight seconds and then dumps the whole paragraph at once.
The model isn't the difference. Both finished generating at roughly the same speed. The difference is **how the tokens travel from your server to the browser** — and that's a transport choice you make once, usually without thinking about it, and then live with.
There are three options on the table: poll the server on a timer, open a WebSocket, or stream with Server-Sent Events. The interview-flashcard answer is "WebSockets are the real-time one." For LLM streaming, that answer is wrong. Here's the actual decision.
---
## Polling: the naive `/chat` endpoint
The first version everyone ships:
```codetabs
### python | Python (FastAPI)
@app.post("/chat")
async def chat(req: ChatRequest):
answer = await llm.complete(req.messages) # blocks until the FULL answer is ready
return {"answer": answer}
### typescript | TypeScript (Express)
app.post("/chat", async (req, res) => {
const answer = await llm.complete(req.body.messages); // blocks until the FULL answer is ready
res.json({ answer });
});
### java | Java (Spring)
@PostMapping("/chat")
public Map chat(@RequestBody ChatRequest req) {
String answer = llm.complete(req.getMessages()); // blocks until the FULL answer is ready
return Map.of("answer", answer);
}
```
The client POSTs, the server blocks until the model is completely done, then returns the whole thing. For long answers you make it async — hand back a `job_id` and let the client **poll** `GET /chat/{job_id}` every second until it flips to `done`.
Polling's virtue is that it's just HTTP. No persistent connection, no special server, works through every proxy, load balancer, and corporate firewall on earth. For work that's genuinely batch — an overnight eval run, a background document summarization, a "we'll email you when it's ready" flow — polling is the correct, boring answer.
## Why polling kills chat UX
For a *chat* it's the wrong answer, and the reason is **time to first token**.
The tokens already exist on your server the instant the model starts emitting them. But the user doesn't see anything until either the whole response is done (blocking) or the next poll lands (async). You're sitting on content the user could already be reading. Worse:
- Poll **faster** and you hammer your own server with mostly-empty "not done yet?" requests, each paying full HTTP + header overhead.
- Poll **slower** and the answer feels laggy and dead.
There's no poll interval that gives you a smooth typewriter. The model is streaming internally; polling throws that stream away and reassembles a lump.
## WebSockets: real-time, but two-way
So reach for the real-time tool, right? A WebSocket upgrades a single HTTP request into a **persistent, two-way channel**. The server pushes each token the instant the model emits it — no re-asking — and the client can send frames back any time.
That bidirectional part is exactly what makes WebSockets shine for chat, live trades, and multiplayer… and exactly what you **don't need** for streaming an LLM answer. The user isn't streaming data back into the middle of a generation. It's one-directional: server → client, token after token. You'd be paying for a two-way pipe to use one direction of it.
And that pipe isn't free:
- You hold **one live connection per active user** — state to track, scale, and reconnect when mobile networks drop it.
- Some proxies and load balancers need extra configuration to keep the upgraded connection alive.
- It's a whole second protocol (`ws://`) sitting next to your normal HTTP stack.
## The plot twist: you probably want SSE
Here's the part the flashcards skip. Because LLM streaming is **one-way**, there's a transport built for exactly this shape: **Server-Sent Events**.
SSE is one long-lived HTTP response that the server keeps writing to. The browser's native `EventSource` reads each chunk as it arrives. No second protocol, no socket state to manage, automatic reconnect built into the spec, and it sails through standard HTTP infrastructure because it *is* standard HTTP.
This isn't a hot take — it's what the production APIs already do. **OpenAI's and Anthropic's streaming endpoints stream over SSE**, not WebSockets. When you set `stream=True`, you're reading an SSE response.
In FastAPI it's about fifteen lines:
```codetabs
### python | Python (FastAPI)
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
app = FastAPI()
@app.post("/chat/stream")
async def chat_stream(req: ChatRequest):
async def event_stream():
async for token in llm.stream(req.messages): # provider yields tokens
yield f"data: {token}\n\n" # SSE frame: "data: \n\n"
yield "data: [DONE]\n\n"
return StreamingResponse(
event_stream(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no", # <-- the gotcha, see below
},
)
### typescript | TypeScript (Express)
app.post("/chat/stream", async (req, res) => {
res.set({
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no", // <-- the gotcha, see below
});
for await (const token of llm.stream(req.body.messages)) { // provider yields tokens
res.write(`data: ${token}\n\n`); // SSE frame: "data: \n\n"
}
res.write("data: [DONE]\n\n");
res.end();
});
### java | Java (Spring)
@PostMapping("/chat/stream")
public SseEmitter chatStream(@RequestBody ChatRequest req) {
SseEmitter emitter = new SseEmitter(0L); // no timeout
executor.execute(() -> {
try {
for (String token : llm.stream(req.getMessages())) { // provider yields tokens
emitter.send(SseEmitter.event().data(token)); // SSE frame
}
emitter.send(SseEmitter.event().data("[DONE]"));
emitter.complete();
} catch (Exception e) {
emitter.completeWithError(e);
}
});
return emitter; // Spring sets Content-Type: text/event-stream
}
```
On the client, the browser does the rest:
```javascript
const es = new EventSource("/chat/stream");
es.onmessage = (e) => {
if (e.data === "[DONE]") return es.close();
appendToken(e.data); // the typewriter effect, for free
};
```
**The one gotcha that eats an afternoon:** if your tokens still arrive in one lump after all this, it's your reverse proxy buffering the response. nginx buffers proxied responses by default and won't flush until it has a chunk it likes — which defeats streaming entirely. The `X-Accel-Buffering: no` header above tells nginx to stop; the equivalent in your nginx config is `proxy_buffering off;` on that location. Same class of bug exists behind most CDNs and API gateways — if streaming "doesn't work" in prod but works locally, it's almost always buffering, not your code.
---
## The decision rule
```
One-way, token-by-token (chat, completions, agent traces) -> SSE
True batch / fire-and-forget (eval runs, long docs) -> Polling
Genuinely bidirectional (live voice, collab, interrupt) -> WebSockets
```
| You need | Use | Why |
|---|---|---|
| Stream a chat/completion answer | **SSE** | One-way, plain HTTP, native browser support, what OpenAI/Anthropic use |
| Kick off a long job, check later | **Polling** | No persistent connection, trivially scalable, proxy-proof |
| Two-way real-time (voice agent, interrupt mid-gen, multi-user) | **WebSockets** | The only one that carries client → server traffic mid-stream |
Default to **SSE** for streaming an LLM. Reach for **WebSockets** only when you genuinely need to send data *back* during the generation — a live voice agent the user can interrupt, a collaborative session, a tool the user steers in real time. And keep **polling** in your pocket for the work that was never interactive to begin with.
The typewriter effect isn't a model feature. It's a transport decision — and now it's a fifteen-line one.