Mixture of Experts (MoE): How a 400B Model Runs Like a 40B One

A language model with 400 billion parameters shouldn't have to run like one — and with Mixture of Experts, it doesn't. By routing each token to only a subset of its knowledge, MoE models hold enormous capacity while computing like much smaller networks.

Mental model: MoE is a sparse gating mechanism that makes a model's feed-forward layer conditional: instead of every token touching every parameter, a tiny router network decides which 2 (or top-k) expert sub-networks each token can use, leaving the rest asleep.

---

The Dense Model Compute Wall

First, understand what you're optimizing away. In a standard dense transformer, every token processes through every single parameter in the model. When the model predicts the next word after "France," that token's representation multiplies against 100% of the network's weights.

This is simple and parallelizable, but it creates a scaling problem: as you grow the model from 7B to 70B to 400B parameters for more knowledge, every token still pays the full computational bill. Doubling the parameters doubles the FLOPs per token. You hit a ceiling where adding knowledge becomes prohibitively expensive at inference time.

---

How MoE Routes Tokens

Mixture of Experts splits the feed-forward (FFN) layer — typically the largest contributor to compute cost — into many smaller experts. Instead of one giant FFN, you have N independent FFNs (commonly 8, 16, or 32), each with the same shape but separate weights.

Before the experts sits a tiny router network (also called a gating network). For each input token, the router:

1. Scores all N experts (one dot-product per expert)
2. Selects the top-k experts — usually top-2
3. Sends the token to only those 2 experts
4. Concatenates the outputs back together

The other N-k experts are never activated. Their parameters stay in VRAM but contribute zero FLOPs for that token.

---

The Real Anchor: Mixtral 8×7B

Mixtral 8×7B is the clearest real-world example:

• Total parameters: 47 billion
• Per-token active parameters: ~13 billion (top-2 of 8 experts, each 7B)
• Inference speed: similar to a 13B dense model
• Training cost: similar to training a ~45B model (full forward pass during training)

So Mixtral holds 47B of knowledge but runs inference like a 13B model. That's roughly 3.6× the knowledge at similar compute cost. In the 400B case, you'd have ~400B parameters total but ~40B active per token — the 10× gap in the post title.

---

How Experts Specialize

You might ask: aren't all 8 experts identical at initialization? Yes, but during training, the router learns to route different tokens to different experts. Over time, experts quietly specialize:

• One expert might become good at math reasoning
• Another at code completion
• Another at narrative text

The router learns these preferences from data, and no explicit supervision is needed. Experts aren't hand-assigned; they emerge from gradient updates on both the router weights and the expert weights.

---

The Honest Catches

Memory vs. Compute

The biggest catch is right in the reel: you save compute, not memory. All 400B parameters must still fit in VRAM or be paged from CPU/disk. An A100 with 80GB can hold Mixtral (47B) easily, but a 400B MoE model needs multiple GPUs or larger accelerators. You've reduced the compute bill but not the infrastructure footprint.

Load Balancing

Without safeguards, the router can collapse into using the same 2 experts for nearly every token. This defeats the purpose: you want traffic spread across all experts. Training MoE models adds an auxiliary loss term that penalizes imbalance:

# Simplified load-balancing loss pseudocode
router_scores = routing_network(token)  # [batch, num_experts]
top_k_mask = top_k(router_scores, k=2)  # one-hot per token
expert_load = top_k_mask.sum(dim=0)  # counts per expert

# Penalize variance in load
load_loss = variance(expert_load) * loss_weight
total_loss = forward_loss + load_loss

This loss encourages uniform traffic. Without it, a few experts become popular while others atrophy.

Router Overhead

The router itself is tiny (usually a single linear layer from hidden size → num_experts), so its compute is negligible. But during backprop, routing decisions are discrete ("pick top-2"), which isn't differentiable. MoE training uses tricks like straight-through estimators or log-sum-exp approximations to make routing differentiable without changing the forward pass.

---

Comparison: Dense vs. MoE vs. Small Dense

Metric	Dense 400B	MoE 400B (8 experts)	Dense 40B
Total Parameters	400B	400B	40B
Active per Token	400B	~50B	40B
FLOPs/Token	800B (2× params)	~100B	80B
VRAM Required	800GB+ (weights + activations)	800GB+ (all experts loaded)	80GB
Knowledge Capacity	Maximum	Maximum (same as Dense 400B)	Lower
Inference Latency	Slowest	~10× faster than Dense 400B	Fast, but less capable

---

Why MoE for LLM Inference

From an LLM-serving perspective, MoE shifts the bottleneck:

• Dense models are memory-bandwidth-bound at inference. Token generation (autoregressive decoding) is fundamentally slow because each token requires a full forward pass.
• MoE models reduce compute per token by ~10×, which reduces memory-bandwidth requirements proportionally. This means:
  • Lower TPOT (time per output token) during decoding
  • Better throughput in batched serving (more requests per second)
  • Lower per-token cost in cloud pricing models

A dense 400B model might generate 5 tokens/second on an H100; an MoE 400B model might generate 30–50 tokens/second on the same hardware. That's the inference win.

---

When to Reach for MoE

Reach for MoE when you need:

• A large model (100B+) for inference with constrained latency or cost
• Diverse tasks that benefit from task-specific experts
• Hardware capacity for the full parameter set but not the full compute

Avoid MoE when:

• You're training from scratch and can't afford the added complexity (load-balancing loss, routing tuning)
• Your hardware is compute-bound (e.g., very short sequences, small batch sizes)
• You need deterministic routing or interpretability (MoE routing is learned and opaque)
• Your accelerators don't have enough VRAM; a smaller dense model is simpler

---

Watch the 90-second reel

This post covers the same idea as the short, but with more depth on specialization, load balancing, and the memory-compute tradeoff. The reel shows the intuition in motion; this explainer gives you the catches and when to build with it. Save both for your next system-design interview.