Embedding Dimensionality: Cut Your Vectors 4× Without Re-Embedding Anything (Matryoshka)

Prefer to watch? ▶ Embedding Dimensionality in 90 seconds ✈ Telegram

You picked 3,072 dimensions by copy-pasting it from the OpenAI docs. And now your 5 million listings are eating 61 GB of RAM.

Dimensionality is a dial, not a decision—but only if you know what you're actually paying for and how to turn it down without starting over.

Mental model: An embedding is a fixed-length list of numbers; that length is a parameter you choose at the API call. Width is not a property of the text—it's a recurring cost you pay on disk, in RAM, and on every single comparison.

The Cost of Width

Let's ground this in concrete numbers. You're running Trove, a marketplace with 5,000,000 listings. Each listing gets embedded with text-embedding-3-large at dimensions=3072.

Here's the math:

One embedding: 3,072 floats
One float: 4 bytes (standard IEEE 32-bit)
Per listing: 3,072 × 4 = 12 KB
Per 5 million listings: 12 KB × 5,000,000 = 61 GB

That 61 GB is not a one-time cost. You pay for it three times:

Disk storage. Your vector index lives on persistent storage.
RAM. The search index lives in memory for latency. All 61 GB must fit there.
Every comparison. Scoring a candidate means loading all 3,072 numbers and computing a similarity metric. The wider the vector, the more arithmetic.

So the natural question: why not ask for 32 dimensions instead? It would drop your index to 600 MB. Problem solved.

The Collapse: Why 32 Dimensions Fails

Because dimensions are room to store distinctions.

Think of embedding space as a high-dimensional room. Each dimension is an axis. The further apart two items are along those axes, the more different they are semantically.

At 32 dimensions, that room is tiny. Items with different meanings get pushed closer together by sheer geometric necessity. "Vintage leather armchair" and "leather car seat" both contain leather. Both describe furniture-like objects. In a 32-dimensional space, they might land in the same corner. Your search for armchairs returns car seats.

At 3,072 dimensions, that room is vast. You have enough axes to encode dozens of semantic properties—material, style, purpose, color, size, era—each with independent granularity.

The tradeoff is real:

Too narrow: meanings collapse. Recall suffers.
Too wide: storage and compute cost explode.

The Expensive Path: Re-Embedding Everything

You could call the embedding API again with dimensions=768. Drop your index size to 3.1 GB. Done.

Except: you need to re-embed all 5,000,000 listings. That's 5 million API calls. Depending on batch size and rate limits, that's hours of wall-clock time. And real money—especially if this is a second attempt because your first guess was wrong.

And you're doing this in the hot path. Your production index is in flux. Your search might be offline.

There's a better way.

The Mechanism: Matryoshka Representation Learning

Matryoshka representation learning (MRL) is a training technique. The model is trained so that meaning is front-loaded—the first few hundred numbers already form a complete, usable embedding on their own.

Think of a Matryoshka doll. Open the full doll and you see all the detail. Open it halfway and you still see the essential shape. The meaning is nested.

When a model is trained with MRL, the first 768 numbers are not "three quarters of the information." They are a full, dimensionally-complete embedding that just happens to use 768 coordinates instead of 3,072.

The model learns to pack semantic content front-to-back, not randomly.

The Move: Truncate and Normalize

You already have the vectors. You already ran the embedding API at full width (3,072 dimensions).

Now:

Keep only the first 768 numbers from each vector.
Recompute the norm: the length of the vector.
Divide each number by that norm.
Store the truncated, re-normalized vector.

That's it. No new API call. No re-embedding.

Here's what it looks like in pseudocode:

def truncate_embedding(embedding, new_dim=768):
    """Truncate and renormalize a Matryoshka embedding."""
    truncated = embedding[:new_dim]
    norm = np.linalg.norm(truncated)
    normalized = truncated / norm
    return normalized

If you were storing the vectors in a database, you'd run this once during a migration and save the result.

The Numbers: Storage and Quality

Here's what happens to Trove's index:

Dimensions	Index Size (GB)	Bytes/Vector	MTEB Score	Recall@10
3,072 (full)	61	12,288	64.6	96%
1,536	30.5	6,144	63.8	95.5%
768	15.25	3,072	63.1	95%
256	5.1	1,024	62.0	92%

At 768 dimensions, Trove's index shrinks from 61 GB to 15.25 GB—4x smaller—with a quality loss of less than 1.5 MTEB points. The recall stays above 95%.

At 256 dimensions, the index is 12x smaller (5.1 GB), and you lose 2.6 MTEB points. Recall is still 92%, which is solid for many applications.

These are real numbers from text-embedding-3-large, a model OpenAI trained with MRL.

The Honest Limits

This is not magic. There are two hard constraints.

1. Recall Does Not Fall Linearly

If you plot dimensionality (x-axis) against recall (y-axis), the curve is not a straight line. It's roughly flat until you hit ~512 dimensions—you barely notice the truncation. Then it bends. Below 128 dimensions, the curve falls off a cliff.

You need to test your actual recall threshold. If your application demands 97% recall on your queries, 256 dimensions might be too aggressive. If 90% is acceptable, 256 is fine.

2. This Only Works on Models Trained for It

If you take an ordinary embedding model—one trained without MRL—and you truncate its vectors, you don't get a summary. You get a random quarter of the meaning.

MRL is not baked into all embedding models. You must check the model card.

text-embedding-3-large and text-embedding-3-small support it. Many open-source models (like some variants of nomic-embed-text) are trained with MRL. Many others are not.

If you truncate the wrong model, you'll see recall crater—not because truncation is a bad idea, but because the model was never trained to pack meaning front-loaded.

Width Is One Lever; Quantization Is Another

Dimensionality is the knob you turn here. But it's not the only one.

Quantization—converting float32 to int8 or even bfloat16—is a separate lever. It can cut storage further without touching dimensions.

For this post, quantization is out of scope. Just know: if you've truncated to 256 dimensions and you still need smaller storage, quantization is your next move. The two levers work independently.

For LLM Inference: Latency and Throughput

If you're embedding text in the critical path of an LLM request (for example, retrieving context before generating), dimensionality affects both embedding latency and downstream inference latency.

A narrower vector means:

Faster embedding inference (fewer operations to compute).
Faster retrieval (fewer numbers to load, fewer comparisons to perform).
Faster re-ranking (if you're re-scoring top-k candidates with a cross-encoder, each candidate is cheaper to process).

If you're batching embeddings (the normal case), the savings scale with batch size. A batch of 1,000 vectors at 256 dimensions is much cheaper to compute, load into a GPU, and score than a batch of 1,000 vectors at 3,072 dimensions.

Time-to-first-token (TTFT) improves because retrieval is faster. Tokens-per-second (TPOT) might also improve if the reduced embedding size lets you fit more in GPU memory.

The Real Questions

If you truncate a Matryoshka embedding and recall drops far more than the benchmarks promised, ask yourself:

Did you re-normalize? Truncating without normalizing breaks similarity metrics. Always re-normalize.
Is the model actually trained for MRL? Check the model card. If not, truncation is a crapshoot.
Did you measure your own recall? Benchmarks (like MTEB) measure performance on standardized test sets. Your query distribution might be different. Always test on your data.
Are your vectors already in the index, or are you truncating on the fly? If you're truncating at query time but not at indexing time, dimension mismatch will kill you. Both must match.

Verdict

Reach for dimensionality reduction (via Matryoshka truncation) when: your index size or query latency is a real cost—you're storing millions of vectors, RAM is expensive, or every millisecond matters—and your model explicitly supports it. Test your recall on your queries first.

Reach for re-embedding when: you need to change to a fundamentally different model, or your current model doesn't support MRL and you need smaller vectors anyway. The cost is hours of API calls; the benefit is the freedom to use any model.

Reach for quantization when: you've truncated as much as recall allows, and you need to go smaller still.

Watch the 90-second reel for the one-concept version.