Perplexity — the one number that says how surprised the model is

Prefer to watch? ▶ Perplexity — the one number that says how surpri in 90 seconds ✈ Telegram

Every benchmark quotes perplexity. Everyone says "lower is better." Almost no one can say what it actually measures.

Here it is, built from the ground up.

One-line mental model: Perplexity = exp(average surprise), where surprise is how low the probability was for the true next token—expressed as an effective number of equally-likely choices the model was picking between.

The model starts with probabilities, not knowledge

A language model doesn't "know" the next word. At every position in a sequence, it outputs a probability distribution over its entire vocabulary. Every single token gets a number between 0 and 1.

When you feed it "The cat sat on the", the model doesn't think. It computes 50,000 probabilities (or however large its vocab is), one for each possible next token. Maybe [underscore] gets 0.001, mat gets 0.003, chair gets 0.15, floor gets 0.27, table gets 0.34.

The true answer—the actual next word in your test set—gets one of those probabilities. That number is all that matters for perplexity.

From probability to surprise: the negative log

If the model put probability 0.99 on the true word, it made a great guess. If it put 0.001 on it, the model was very surprised.

We need a number that captures this intuition. Enter the negative log probability:

$$\text{surprise} = -\log(p)$$

Where p is the probability the model assigned to the true next token.

If p = 1 (certainty), then −log(1) = 0. Zero surprise. ✓
If p = 0.5, then −log(0.5) ≈ 0.693. Moderate surprise.
If p = 0.01, then −log(0.01) ≈ 4.605. High surprise.
If p → 0, then −log(p) → ∞. Infinite surprise.

This is the foundation. Everything else is averaging this number.

Cross-entropy: averaging surprise over every token

You have a test set—a held-out chunk of text the model has never seen. You run the model through it token by token, compute the surprise at each position, and take the average.

That average is called cross-entropy (or negative log-likelihood):

$$\text{CE} = \frac{1}{N} \sum_{i=1}^{N} -\log(p_i)$$

Where N is the total number of tokens and p_i is the probability the model put on the true token at position i.

If your model is good, this average stays low (the model was rarely surprised). If it's bad, the average shoots up (lots of low probabilities on the true tokens).

But cross-entropy is an abstract number. CE = 4.2 doesn't mean much to a human. So we exponentiate it.

Perplexity: turning surprise into "effective choices"

$$\text{Perplexity} = e^{\text{CE}} = e^{\frac{1}{N} \sum_{i=1}^{N} -\log(p_i)}$$

Why exponentiate? Because it converts an abstract average into something you can visualize: the effective number of equally-likely words the model was choosing between.

Think of it this way: if the model sees a position and genuinely can't decide between k equally-likely next tokens, it would assign probability 1/k to each. The surprise at that position would be −log(1/k) = log(k). Average that across all positions and exponentiate, and you get k back.

So:

PPL = 1: The model is certain at every position. It assigns probability 1 to the true token every time. Total dominance.
PPL = 10: On average, the model acts as if it's choosing between roughly 10 equally-likely options. Not bad.
PPL = 100: The model is choosing between roughly 100 options. Getting worse.
PPL = vocabulary-size (say, 50,000): The model is essentially guessing randomly.

Lower perplexity = less surprised = better.

Real-world numbers

Here's what you actually see when you evaluate language models on held-out text:

Model strength	Typical perplexity (WikiText-103)	Intuition
Strong LLM (GPT-3 scale)	≈ 12–20	Very confident, usually right on English text
Medium transformer (BERT-scale)	≈ 50–150	Good, but not conversational-grade
Weak or overfit model	≈ 300–1000	Often wrong, low confidence
Untrained or random weights	≈ vocab-size	Equivalent to guessing

The two catches nobody explains

1. Perplexity is not an absolute score

Perplexity depends on two variables you control:

The tokenizer. If you use byte-pair encoding (subword), you might get fewer tokens and lower cross-entropy per token. If you use character-level tokenization, you get more tokens and higher cross-entropy. The perplexity changes, even if the model is identical.
The test set. If your test set is in-distribution (same domain, same language), perplexity will be lower. If it's out-of-distribution, perplexity shoots up.

You cannot compare a GPT-2–tokenized model's PPL 50 with a character-level model's PPL 120 and conclude anything. They're counting "tokens" differently. This is the #1 mistake in model comparison papers.

2. Perplexity measures fluency, not truth

A language model can be confidently, smoothly wrong.

Example:

Prompt: "The capital of France is"
Model output: "Berlin" (with very high probability)
Perplexity on this sequence: low (model was certain)
Correctness: zero (Berlin is wrong)

Perplexity is an intrinsic metric—it only measures how well the model predicts the next token in your test set. It says nothing about whether the output is true, useful, safe, or aligned with your task.

Whether the model's answers are correct requires a separate, extrinsic evaluation on a task-specific benchmark.

When to reach for perplexity

Perplexity is cheap, fast, and a genuine signal for language-modeling quality. Use it when:

You're training a new model and want a quick sanity check on convergence.
You're comparing two models trained on the exact same tokenizer and test set.
You care about fluency and smoothness, not task-specific correctness.
You need a number for a paper or a dashboard.

Do not use perplexity to:

Compare models with different tokenizers.
Claim a model is "better" at question-answering or reasoning without an actual task evaluation.
Evaluate safety, hallucination, or factuality.

For LLM inference: perplexity at eval time

If you're serving a language model, perplexity is your sanity check during batch evaluation. It's cheap to compute—you're already running forward passes through your test set. It won't tell you latency (TTFT, time-per-output-token) or throughput, but it will flag whether your deployed weights have drifted or been corrupted.

Some teams track running perplexity on a held-out eval stream to catch model decay in production. It's an intrinsic metric, so it catches "we broke the tokenizer" or "we accidentally downgraded the weights"—but not "our model started hallucinating."

The verdict

Reach for perplexity when you need a fast, comparable fluency score on the same test set and tokenizer. Reach for task-specific metrics (accuracy, F1, BLEU, exact-match, human eval) when you actually care whether the model's output is correct.

Watch the 90-second reel for the full walkthrough.