RAG & vector retrieval — the whole stack on one page

An embedding model maps a piece of text to a fixed-length list of floats, trained so that texts people would call similar land near each other. paid the invoice and settled the bill share no words and end up as neighbours; that is the entire trick, and everything downstream on this page is geometry over the result.

Dimensions are a storage decision, not a quality ranking. A vector is 4 bytes per dimension at float32: 768 dims is 3 KB, 1536 dims is 6 KB, 3072 dims is 12 KB. A million documents at 1536 is 6 GB of vectors before any index overhead, and the index is what has to fit in RAM. Comparing dimension counts across model families tells you nothing — a good 768-dim model beats a mediocre 1536-dim one on the same corpus. Compare on a retrieval benchmark, or better, on your own questions.

Within one model family, dimensions are now a dial rather than a fixed property. Matryoshka-trained models pack the important information into the leading dimensions, so you can truncate 3072 down to 768, renormalise, and keep most of the accuracy — a 4× cut with no re-embedding. That is the cheapest lever on this whole page and it has its own article.

🔴 The rule that costs people a weekend: two models produce two different coordinate systems. A vector from Gemini and a vector from OpenAI are not comparable, not even approximately, and a cosine between them is a meaningless number that will not error. So store the model name and dimension next to every vector, and treat a model swap as a full re-embedding migration — not a config change.

One more thing that quietly costs recall: some models are asymmetric and expect a task prefix, search_query: on the question and search_document: on the passage. Embedding both sides the same way with such a model does not fail — it just retrieves worse, forever, until someone reads the model card.

Three metrics carry almost all real usage. Cosine measures the angle between two vectors and ignores their length. Dot product (inner product) measures angle and length together. Euclidean / L2 measures the straight-line distance between the two points.

For unit-length vectors the algebra collapses: ‖a − b‖² = 2 − 2(a · b), and cosine is exactly a · b. So all three order the candidates the same way, and the choice between them is about cost — dot product is the cheapest, one multiply-add per dimension with no square roots and no norms. Most hosted models (OpenAI, Gemini) return normalised vectors already, which is why cosine "just working" everywhere is not luck.

It stops being equivalent the moment magnitude carries meaning. Some models leave vectors unnormalised so that length encodes something — confidence, document length, popularity. Then dot product rewards long or "strong" documents and cosine does not, and you have to decide which behaviour you want. Recommender systems pick maximum inner product deliberately for exactly this reason.

Beyond the three: Hamming distance on binary codes is XOR plus a popcount — an order of magnitude faster than float maths, and the reason binary quantisation in §05 is viable. Jaccard is for sets rather than dense vectors and belongs with keyword search. Manhattan / L1 shows up in specific model families and rarely in general RAG.

The cost that actually matters is not the metric, it is the multiplication by n. A cosine over 1536 dimensions is roughly 3,000 floating-point operations. Against a million vectors that is 3 GFLOP per query — perfectly fine on one core. Against a billion it is 3 TFLOP per query, and no choice of metric saves you. That is what §03 is for.

Exact k-NN is O(n · d) and there is no way around that: to prove a vector is the nearest, you have to look at all of them. A billion 768-dim float32 vectors is about 3 TB. A single exact query reads 3 TB. That is not a query, it is a batch job.

Approximate nearest neighbour structures the space in advance so a query only visits a small, well-chosen fraction of it. Three families: navigable graphs (HNSW, §04), inverted lists over clusters (IVF, §04), and hashing or tree partitions (LSH, Annoy — largely historical now). Once the vectors no longer fit in RAM, disk-resident graphs like DiskANN become the fourth.

🔴 If you do not measure recall you do not know what the index cost you. Take a few thousand real queries, compute the true top-k with a brute-force scan once, and score your index against it. Recall@10 of 0.98 is a different system from 0.80, and both will look fine in a demo. Every ANN index has a knob that trades recall for latency, and that knob has to be set against a number.

The failure mode is worth internalising: ANN error is not uniformly distributed. Queries with one obvious nearest neighbour are found reliably. Queries where the top candidates are all roughly equidistant — the genuinely ambiguous ones — are where the approximation drops the right answer. So the questions your users complain about are exactly the ones the index handles worst.

At true billion scale the index is no longer the whole answer either. You shard — by tenant, by time, by whatever boundary every query already carries — query only the shards you need, and merge the top-k. Which turns §06, metadata filtering, from a WHERE clause into an architecture decision.

And the honest version: exact search is right up to roughly 100k–1M vectors on a single box, and most projects that reach for an ANN index on day one never needed it. Adding one has a real cost: build time, memory, tuning, and a recall number you now have to defend.

HNSW builds a hierarchy of navigable small-world graphs. The top layer is sparse with long-range links, each layer below is denser, and a search greedily walks toward the query at each level before dropping down. Search behaves like O(log n). Its three parameters: m — edges per node, 16 by default, sets memory and the recall ceiling; ef_construction — how hard the builder searches while inserting, sets build quality and build time; ef_search — how wide the query search is, and the one knob you tune at runtime to trade latency for recall. Cost: the graph itself often adds 1.5–3× the raw vector size, and deletes are tombstones that degrade the graph until a rebuild.

IVF clusters the corpus with k-means into lists centroids and stores each vector in its nearest cluster. A query finds the nprobe nearest centroids and scans only those lists. Build is fast, memory overhead is tiny, and nprobe is the recall knob. 🔴 The catch: the centroids are learned from whatever data was in the table at build time. Build an IVF index on an empty or unrepresentative table and it is worthless — and it does not rebalance itself as you insert, so a growing corpus needs periodic rebuilds.

Choosing: HNSW when query latency and recall are what you are judged on and you can pay for the RAM. IVF when build time, memory or frequent full rebuilds dominate — or as IVF+PQ (§05) when the corpus is far too large to hold as floats at all.

In pgvector concretely: CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64), then SET hnsw.ef_search = 100 per session. For IVF: USING ivfflat (embedding vector_cosine_ops) WITH (lists = 1000) and SET ivfflat.probes = 20. Rule of thumb for lists: rows / 1000 up to a million rows, then √rows.

🔴 The mistake that produces a silent full scan: the operator class must match the operator you query with. An index built vector_l2_ops is simply not used by a query ordering on <=>, and Postgres will not warn you — it will seq-scan and return correct, slow results. EXPLAIN is the only thing that tells you which one you got.

Quantisation exists because memory decides whether the index fits in RAM, and RAM decides your latency. Three levels, in increasing aggression.

Scalar quantisation (int8) maps each float32 dimension onto one byte. 4× smaller, typically 1–2% recall loss, almost always worth taking. pgvector's halfvec is the gentler version of the same idea — float16, 2× smaller, loss usually indistinguishable from noise.

Binary quantisation keeps one bit per dimension — just the sign. 32× smaller: a 1536-dim vector becomes 192 bytes. Distance becomes Hamming, which is XOR plus popcount, so it is also roughly 10–40× faster to compare. On its own the recall loss is severe. Paired with rescoring it is not, and it works far better on high-dimensional models (1536+) than on 384-dim ones, because a bit per dimension is only enough information when there are many dimensions.

Product quantisation splits the vector into m sub-vectors, runs k-means with 256 centroids in each sub-space, and stores m single bytes. A 6 KB vector becomes 96 bytes at m = 96, and distances are computed from a precomputed lookup table rather than arithmetic. This is the technique that makes billion-scale indexes exist at all — FAISS's IVF-PQ is exactly this stacked on §04.

🔴 Always keep the full vectors. The pattern that makes all of this safe is two-stage: retrieve the top 100 with the compressed representation, then re-score those 100 against the original float vectors and return the top 10. It recovers most of the lost recall for one cheap extra step — and it is the same shape as the reranking in §08, one level lower down the stack.

Note that this is a different axis from Matryoshka in §01: quantisation reduces bits per dimension, Matryoshka reduces the number of dimensions. They compose — 3072 → 768 dims, then float32 → int8, is 16× total.

Post-filter: run the ANN search, get the top k, then drop what fails the predicate. Fast, trivially implemented, and 🔴 broken exactly when the filter is selective. Ask for 10 documents from a folder holding 0.1% of the corpus and the top 100 nearest vectors contain none of them — so you return nothing, from a database that has the answer.

Pre-filter: apply the predicate first, then search exactly within the survivors. Always correct. Fast when the filter is selective — and it degenerates into a full scan when it is not, because a filter that keeps 90% of the corpus has not eliminated any work.

Filtered ANN / in-graph filtering: walk the index but only accept nodes that satisfy the predicate — pgvector's iterative index scans, Qdrant's filterable HNSW, Weaviate's ACORN. This is the one that holds up under selective filters, at the cost of real complexity: under a very selective predicate the graph can become effectively disconnected, which is why these engines all fall back to brute force below some threshold.

And the fourth answer, which is usually the right one for a hard boundary: do not filter, partition. If every single query carries user_id or tenant_id, that is not a filter, it is a shard key. One collection or one partition per tenant removes the problem instead of tuning it.

What you can filter on is decided at ingest, not at query time. Anything you might want to filter by has to be extracted, stored as a real column and indexed — which means the ingest pipeline and the query surface are one design, not two.

One vector has to summarise one chunk. A large chunk covering five topics produces a vector that is the average of five things and close to none of them — it will not be retrieved for any of the five. A small chunk produces a sharp vector and arrives at the model without the context that made it meaningful. That single tension is the whole subject.

Fixed size with overlap — 1000 characters, 200 of overlap — is the baseline. It is dumb, it cuts sentences in half, and it works well enough that you should start there and only move when your evals say to.

Recursive / structural splitting respects the document's own boundaries: split on headings, then paragraphs, then sentences, packing up to a size budget. This is the correct default for prose, Markdown and HTML, and it costs nothing over fixed-size.

Document-aware splitting goes further and uses the format: code by function or class, tables kept whole with their header row repeated, PDFs cut by layout block rather than by the text stream — which matters enormously, because a two-column PDF read as a stream interleaves two unrelated columns into every chunk.

Semantic chunking embeds sentence by sentence and cuts where consecutive similarity drops. Intuitive, expensive at ingest, and in practice rarely beats good structural splitting by enough to justify itself.

Contextual retrieval / late chunking is the one with the largest measured win: before embedding, prepend a one-line, LLM-written statement of where this chunk sits and what it is about, so the vector carries the context the text lost when it was cut. Real ingest cost, real improvement.

🔴 The move that resolves the tension instead of tuning it: decouple retrieval unit from reading unit. Index small, sharp chunks — then return their parent section to the model. Small-to-big. You get the precision of a small vector and the context of a large passage, and you stop trying to find one size that is both.

When to change: read your failures, do not guess. Right document retrieved but the answer is wrong or partial → chunks are too small, or you are not returning enough around them. Right document never retrieved at all → chunks are too large and the vector is diluted, or the split cut the answer in half. Those are two different bugs and the same chunk-size slider moves them in opposite directions.

A document's vector was computed at ingest, long before the question existed. That is what lets you search a million documents in milliseconds — and it means the model never got to consider the query and the document together. This is a bi-encoder, and its blind spots are structural, not a tuning problem.

A reranker looks at the pair. It cannot scale, so it only ever sees the top 20–100 candidates from stage one, and it reorders them. Four mechanisms, in rising cost:

Reciprocal rank fusion (RRF) is not a model at all — it merges several ranked lists by summing 1/(k + rank). Free, no latency, no infrastructure, and the correct first move the moment you have both a keyword list and a vector list. Try this before anything below it.

Cross-encoder — a BERT-class model (bge-reranker, Cohere Rerank) that takes query and document concatenated and emits one relevance score, one forward pass per pair. The highest quality per token, and it needs a GPU or an API: roughly 50–200 ms for 50 candidates.

Late interaction (ColBERT) keeps a vector per token and scores by MaxSim across them. Between the two in cost and quality, and considerably heavier in storage.

LLM-as-reranker — hand the candidates to a small fast model and ask it to label them. Cheapest to build, no new infrastructure, and uniquely able to apply stated rules rather than learned similarity. This is the one to reach for when your relevance criteria are things you can write down in a sentence.

🔴 The generalisable point: a reranker is where the criteria that are not in the geometry go. Recency, entity identity, the exact reporting period, permissions, document type precedence. Cosine has no idea that a 2026 document cannot answer a 2025 question. Something has to, and stage two is where it lives.

There are three places to intervene, and they are in order of leverage — spending on the third while the first is broken is the most common way to burn a month.

One: retrieval quality. If the right passage was never retrieved, no prompt engineering downstream can fix it. Measure recall@k on a fixed question set before you touch the generation prompt.

Two: grounding at generation. Require citations down to the chunk id, so every claim points at a retrieved span. Give the model explicit permission to say the answer is not in the provided documents — a model with no escape hatch will invent one, because that is what it was trained to do. And keep the context tight: burying the answer in twenty marginal chunks measurably lowers the odds it is used.

Three: verification after the fact. A second pass that checks each claim against the retrieved text — an entailment model or an LLM judge — plus a purely mechanical citation check: the quoted span must actually appear in a retrieved chunk. That substring check is free, catches fabricated quotes outright, and almost nobody implements it.

The metrics are separable and each points at a different fix. Faithfulness — is the answer supported by the retrieved context? Answer relevance — does it address the question asked? Context precision and recall — did retrieval bring the right material, and how much noise came with it? A system scoring badly on faithfulness needs prompt and verification work; one scoring badly on context recall needs §06 and §07.

🔴 And make it a regression test, not a vibe check. A fixed set of questions with expected sources, run on every change to the chunker, the model, the prompt or the reranker. Retrieval quality drifts silently — nothing errors, the answers just get slightly worse — and a question set is the only thing that notices.