Quantization vs Distillation

Your model works. It's just too big to ship. You have two standard paths to shrink it — and they are nothing alike.

• Mental model: Quantization is compression; distillation is re-education. One shrinks the same brain, the other builds a smaller one that imitates the original.

---

Quantization: Same Brain, Fewer Bits

Quantization takes your trained model and re-encodes its weights at lower precision. Instead of storing each weight as a 32-bit float (FP32), you store it as a 16-bit half (FP16), an 8-bit integer (INT8), or even a 4-bit integer (INT4).

You do not retrain. The model architecture stays identical. Only the storage format changes.

This gives you:

• Up to 4× smaller model size (INT4 vs FP32).
• Faster inference on hardware that supports low-precision arithmetic (GPUs, TPUs, mobile chips).
• Lower memory bandwidth — moving fewer bits is faster.
• A small accuracy drop, especially at INT4, but often acceptable.

The trade-off is that the accuracy loss compounds with each quantization step. INT8 is usually transparent; INT4 can degrade your output quality if your model has learned fine-grained weights.

---

Distillation: Train a Student to Mimic a Teacher

Distillation is different. You take your big, trained model (the "teacher") and train a new, smaller model (the "student") to produce the same outputs.

During training, the student learns not just the original task labels, but also the soft probabilities and internal representations the teacher produces. The teacher acts as a training signal — often richer and more informative than the raw labels alone.

DistilBERT, for example, is distilled from BERT:

• 40% smaller (fewer layers, fewer dimensions).
• 60% faster inference.
• Retains ~97% of BERT's performance.
• Requires 1) the teacher model, 2) training data, and 3) time to train the student.

Because you are training a new model, you have full control over its architecture, depth, and capacity. You can make it dramatically smaller than quantization alone would allow.

---

Head-to-Head Comparison

Aspect	Quantization	Distillation
What changes	Weight precision (FP32 → INT8 → INT4)	Model architecture and learned weights
Retraining required	No	Yes
Size reduction	Up to 4×	40–50%+ (depends on student design)
Speed improvement	Moderate (with low-precision hardware support)	Significant (fewer parameters and FLOPs)
Accuracy loss	Small to none (INT8); moderate to high (INT4)	Minimal (with good teacher and data)
Implementation time	Hours to days (post-training)	Days to weeks (full training loop)
Hardware requirements	None for compression; low-precision hardware for speedup	GPU/TPU for training
Can be combined	Yes: distill a student, then quantize it

---

Why Combine Them?

Distillation and quantization are orthogonal. You can apply both to the same model:

1. Distill a small student from your large teacher.
2. Quantize the student to INT8 or INT4.

This gives you the best of both worlds: a model that is architecturally small (from distillation) and compressed to low precision (from quantization). The student, being smaller to begin with, often handles quantization better than the original teacher would.

# Pseudocode: distill then quantize

# Step 1: Train a student to mimic the teacher
student = StudentModel(hidden_dim=256, num_layers=4)
teacher = TeacherModel.load_pretrained()

for batch in train_loader:
    student_logits = student(batch)
    teacher_logits = teacher(batch)
    
    # Minimize KL divergence between student and teacher distributions
    loss = kl_divergence(student_logits, teacher_logits) + task_loss(student_logits, labels)
    loss.backward()

# Step 2: Quantize the trained student
quantized_student = quantize_to_int8(student)

# Result: small + compressed
export_model(quantized_student)

---

For LLM Inference: Latency and Throughput

When serving an LLM, think of inference latency in two phases:

• Time to First Token (TTFT): how long until the model generates the first token. Driven by the prefill pass over the full prompt.
• Time Per Output Token (TPOT): how long each subsequent token takes. Driven by the decode pass on one token at a time.

Quantization helps both:

• Smaller weights fit in GPU memory, reducing memory bandwidth bottlenecks during the decode phase.
• If your hardware supports low-precision computation (INT8 Tensor Cores on NVIDIA GPUs), you get direct compute speedup.

Distillation helps differently:

• Fewer parameters mean fewer FLOPs, so both prefill and decode are inherently faster.
• A smaller model uses less memory, allowing larger batch sizes and better throughput.

For a production LLM API, distill then quantize the result for the best latency and throughput profile.

---

The Verdict

Reach for quantization when you have a model that works well, you need a fast win, and you have no time or data to retrain. It is zero-effort compression.

Reach for distillation when you need a permanently smaller model, you have the teacher and training data available, and you can afford the training time. The result is a genuinely lighter model, not just a compressed version of the original.

Reach for both when you need to ship the smallest, fastest inference possible: distill first to pick the right architecture, then quantize to shrink it further.

Watch the 60-second reel to see this in action.