How Should Your LLM Stream Tokens? SSE vs WebSockets vs Polling

ChatGPT types its answer out one token at a time. Your chatbot makes the user stare at a spinner for eight seconds and then dumps the whole paragraph at once.

The model isn't the difference. Both finished generating at roughly the same speed. The difference is how the tokens travel from your server to the browser — and that's a transport choice you make once, usually without thinking about it, and then live with.

There are three options on the table: poll the server on a timer, open a WebSocket, or stream with Server-Sent Events. The interview-flashcard answer is "WebSockets are the real-time one." For LLM streaming, that answer is wrong. Here's the actual decision.

---

Polling: the naive /chat endpoint

The first version everyone ships:

@app.post("/chat")
async def chat(req: ChatRequest):
    answer = await llm.complete(req.messages)   # blocks until the FULL answer is ready
    return {"answer": answer}

app.post("/chat", async (req, res) => {
  const answer = await llm.complete(req.body.messages); // blocks until the FULL answer is ready
  res.json({ answer });
});

@PostMapping("/chat")
public Map<String, String> chat(@RequestBody ChatRequest req) {
    String answer = llm.complete(req.getMessages()); // blocks until the FULL answer is ready
    return Map.of("answer", answer);
}

The client POSTs, the server blocks until the model is completely done, then returns the whole thing. For long answers you make it async — hand back a job_id and let the client poll GET /chat/{job_id} every second until it flips to done.

Polling's virtue is that it's just HTTP. No persistent connection, no special server, works through every proxy, load balancer, and corporate firewall on earth. For work that's genuinely batch — an overnight eval run, a background document summarization, a "we'll email you when it's ready" flow — polling is the correct, boring answer.

Why polling kills chat UX

For a chat it's the wrong answer, and the reason is time to first token.

The tokens already exist on your server the instant the model starts emitting them. But the user doesn't see anything until either the whole response is done (blocking) or the next poll lands (async). You're sitting on content the user could already be reading. Worse:

• Poll faster and you hammer your own server with mostly-empty "not done yet?" requests, each paying full HTTP + header overhead.
• Poll slower and the answer feels laggy and dead.

There's no poll interval that gives you a smooth typewriter. The model is streaming internally; polling throws that stream away and reassembles a lump.

WebSockets: real-time, but two-way

So reach for the real-time tool, right? A WebSocket upgrades a single HTTP request into a persistent, two-way channel. The server pushes each token the instant the model emits it — no re-asking — and the client can send frames back any time.

That bidirectional part is exactly what makes WebSockets shine for chat, live trades, and multiplayer… and exactly what you don't need for streaming an LLM answer. The user isn't streaming data back into the middle of a generation. It's one-directional: server → client, token after token. You'd be paying for a two-way pipe to use one direction of it.

And that pipe isn't free:

• You hold one live connection per active user — state to track, scale, and reconnect when mobile networks drop it.
• Some proxies and load balancers need extra configuration to keep the upgraded connection alive.
• It's a whole second protocol (ws://) sitting next to your normal HTTP stack.

The plot twist: you probably want SSE

Here's the part the flashcards skip. Because LLM streaming is one-way, there's a transport built for exactly this shape: Server-Sent Events.

SSE is one long-lived HTTP response that the server keeps writing to. The browser's native EventSource reads each chunk as it arrives. No second protocol, no socket state to manage, automatic reconnect built into the spec, and it sails through standard HTTP infrastructure because it is standard HTTP.

This isn't a hot take — it's what the production APIs already do. OpenAI's and Anthropic's streaming endpoints stream over SSE, not WebSockets. When you set stream=True, you're reading an SSE response.

In FastAPI it's about fifteen lines:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.post("/chat/stream")
async def chat_stream(req: ChatRequest):
    async def event_stream():
        async for token in llm.stream(req.messages):   # provider yields tokens
            yield f"data: {token}\n\n"                  # SSE frame: "data: <payload>\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        event_stream(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",   # <-- the gotcha, see below
        },
    )

app.post("/chat/stream", async (req, res) => {
  res.set({
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    "X-Accel-Buffering": "no", // <-- the gotcha, see below
  });
  for await (const token of llm.stream(req.body.messages)) { // provider yields tokens
    res.write(`data: ${token}\n\n`);                         // SSE frame: "data: <payload>\n\n"
  }
  res.write("data: [DONE]\n\n");
  res.end();
});

@PostMapping("/chat/stream")
public SseEmitter chatStream(@RequestBody ChatRequest req) {
    SseEmitter emitter = new SseEmitter(0L); // no timeout
    executor.execute(() -> {
        try {
            for (String token : llm.stream(req.getMessages())) {  // provider yields tokens
                emitter.send(SseEmitter.event().data(token));      // SSE frame
            }
            emitter.send(SseEmitter.event().data("[DONE]"));
            emitter.complete();
        } catch (Exception e) {
            emitter.completeWithError(e);
        }
    });
    return emitter; // Spring sets Content-Type: text/event-stream
}

On the client, the browser does the rest:

const es = new EventSource("/chat/stream");
es.onmessage = (e) => {
  if (e.data === "[DONE]") return es.close();
  appendToken(e.data);   // the typewriter effect, for free
};

The one gotcha that eats an afternoon: if your tokens still arrive in one lump after all this, it's your reverse proxy buffering the response. nginx buffers proxied responses by default and won't flush until it has a chunk it likes — which defeats streaming entirely. The X-Accel-Buffering: no header above tells nginx to stop; the equivalent in your nginx config is proxy_buffering off; on that location. Same class of bug exists behind most CDNs and API gateways — if streaming "doesn't work" in prod but works locally, it's almost always buffering, not your code.

---

The decision rule

One-way, token-by-token (chat, completions, agent traces)  ->  SSE
True batch / fire-and-forget (eval runs, long docs)        ->  Polling
Genuinely bidirectional (live voice, collab, interrupt)    ->  WebSockets

You need	Use	Why
Stream a chat/completion answer	SSE	One-way, plain HTTP, native browser support, what OpenAI/Anthropic use
Kick off a long job, check later	Polling	No persistent connection, trivially scalable, proxy-proof
Two-way real-time (voice agent, interrupt mid-gen, multi-user)	WebSockets	The only one that carries client → server traffic mid-stream

Default to SSE for streaming an LLM. Reach for WebSockets only when you genuinely need to send data back during the generation — a live voice agent the user can interrupt, a collaborative session, a tool the user steers in real time. And keep polling in your pocket for the work that was never interactive to begin with.

The typewriter effect isn't a model feature. It's a transport decision — and now it's a fifteen-line one.