RAG

RAG conditions a model’s response on text retrieved at query time, instead of relying only on what was baked into its weights during training. The mechanics divide cleanly in two: work done at indexing time, before any query exists, and work done per request. Almost every well-known rule of thumb attaches to one of those two halves, and several of them do not survive contact with a benchmark.

Two neighbouring pages cover the ground this one leaves out. The original 2020 paper explainer covers where the idea came from and what it originally claimed. RAG at Production Scale covers what changes once the corpus updates continuously and results must be filtered by who is asking: freshness, authorisation, and the corpus as a supply chain. What follows sits between the two, and is worth reading first if neither of those yet makes sense.

The idea, before the mechanics

A language model trained on the public internet is a student sitting a closed-book exam. It answers from memory. When it remembers well, the answer is good. When it does not, it still answers, because producing fluent text is the thing it was built to do, and a confident wrong answer looks exactly like a confident right one.

RAG turns it into an open-book exam. Before the model writes anything, the system goes and finds the handful of pages most likely to contain the answer, and puts them in front of it. The model is then asked to answer using those pages.

That shift moves the hard problem. Answering is no longer the difficult part. Finding the right pages is. If the system hands over the wrong pages, a capable model will write a fluent answer grounded in the wrong material, and it will look just as convincing as a correct one. Almost everything below is about that search step, because that is where the quality actually comes from.

Consider a company handbook and the question “how much annual leave do new starters get?”. Nothing in the model’s training data knows your company’s handbook. A system that answers it well has to locate the paragraph about leave entitlement, notice that it is the paragraph for new starters and not the one for staff over five years, and hand that specific text to the model. Retrieval is doing the real work. The model is doing the writing.

What the machine is actually matching on

Software cannot compare meaning directly, so RAG converts text into something it can compare: numbers. An embedding model reads a passage and produces a long list of numbers called a vector, anywhere from a few hundred of them to a few thousand. The numbers themselves mean nothing individually. What matters is position. The model is trained so that passages about similar things end up with vectors that sit close together.

The useful mental picture is a map. Every passage in the corpus is a pin on it, and the model places the pins so that neighbouring pins are about related things. Text about annual leave lands in one neighbourhood, text about expense claims in another. A question is placed on the same map, and the system returns whatever pins are nearest.

Nearness has a specific definition. The usual one is cosine similarity, the cosine of the angle between the two vectors:

similarity(q, d) = (q · d) / (|q| × |d|)

Dividing by the two lengths is the part that matters. It takes magnitude out of the comparison and leaves only direction, so a short passage and a long one on the same subject can still score as close. Without it, longer text would drift towards looking similar to everything, which is not what anyone wants from a search. Where a model already returns vectors normalised to unit length, both lengths are 1, the division changes nothing, and a plain inner product produces the same ranking for less arithmetic.

Two consequences follow, and both come back later as failure modes.

Nearness is a measure of resemblance, not of truth. A passage can sit close to a question and still fail to answer it. Search finds text that looks like the question, which is usually, but not reliably, the text that answers it.

Only one model drew the map. Coordinates from a different embedding model, or a different version of the same one, describe a different map entirely. Mixing them puts pins at positions that were never comparable, and the distances calculated between them are arithmetic performed on unrelated numbers.

The two phases

Indexing happens once, ahead of time. Every document is split into chunks, each chunk is turned into a vector, and the vectors are stored in an index built for finding nearest neighbours quickly. This is the part that draws the map.

Where the vectors actually go

The decision here is not which product to pick from a list. It is whether the vectors live in a database you already operate or in a second system stood up beside it.

Keeping them in the existing database means similarity search is an ordinary query, in the same transaction, covered by the same backups and replication, and joinable against the rows the chunks came from. For PostgreSQL that route is pgvector, which is the one covered in depth here because this site is PostgreSQL-centred, not because it leads a field that also contains several mature specialised engines. A dedicated vector store buys index features and scaling headroom the relational route does not have, at the price of a second system to run, secure and keep consistent with the first.

Either way, one property is worth knowing before sizing anything: each vector is stored twice. One copy sits in the table next to the chunk text. A second complete copy sits inside the index, because the index computes distances as it searches and cannot afford a trip back to the table for every comparison it makes. PostgreSQL adds a wrinkle, moving the table copy out of the row once it passes a couple of kilobytes, which ordinary embedding sizes do comfortably, so reading it back can cost an extra fetch. Where the data actually lives gives the byte counts and draws both the write and the read path.

The search is approximate, and that is a choice

Comparing a query against every stored vector gives an exact answer, and across a few thousand chunks that is genuinely the right thing to do. It stops scaling because the work grows with the corpus: every query touches everything.

Larger systems use approximate nearest neighbour search instead, accepting a small chance of missing a true neighbour in exchange for not examining most of the corpus. HNSW is the common choice. It builds a layered proximity graph while indexing and searches from a sparse top layer downwards, which is how the 2016 paper behind it reaches logarithmic rather than linear scaling.

The word approximate is carrying weight there. Such an index can fail to return a chunk that really was among the nearest, and it fails silently: no error, and nothing in the result showing something was left out. The rate is tunable, since these indexes expose a knob trading recall against latency, and the default is rarely the value anyone would pick deliberately. pgvector covers those knobs, the two index types it offers, and how to measure the recall you are actually getting.

Everything below happens per request, while somebody waits. The query is embedded with the same model that built the index, the nearest chunks come back, the strongest of them are reordered, and only then does the model see anything.

The two phases have to agree. A query embedded by a different model, or a different version of the same model, is measured against a map it was not drawn on, for the reason given above. What makes this one dangerous is that nothing errors. No exception is raised and no log line appears. Results simply get quietly worse, which is how it survives long enough to reach production.

Chunking: there is no single correct size

Chunking means splitting a document into passages small enough to embed and retrieve individually. It is usually described with a specific number attached (“512 tokens,” “256 tokens”), stated as a setting and not as a decision with a real trade-off. Smaller chunks retrieve more precisely, because a query is less likely to match a chunk that is mostly about something else; larger chunks preserve more surrounding context per chunk, so a retrieved passage is less likely to be missing the sentence that would have made it useful. No chunk size is correct independent of what the documents actually look like. A chunk boundary that splits a table row or a code block in half is a worse failure than either size choice, and no fixed token count avoids it on its own.

The strategies people actually use trade precision against context in different ways:

Whichever strategy is used, what travels alongside the chunk decides what becomes possible later. A bare vector and a lump of text cannot be filtered, attributed or invalidated. Store at least the source document identifier, the section or page the passage came from, the timestamp of the version it was taken from, and a hash of the content it was embedded from. The identifier and section make citation possible, so an answer can point at where it came from. The timestamp and hash are what let an ingestion pipeline work out later which chunks are stale and which are unchanged, which is the difference between re-embedding a corpus and re-embedding the part of it that moved.

The overlap window in the first row deserves its own note, because it is the usual patch for a problem it only partly fixes. Overlapping chunks reduce the chance that a fact is severed exactly at a boundary. They also duplicate content across chunks, which inflates the index and lets several near-identical passages occupy the shortlist that should have held several different ones.

Choosing the embedding model is a commitment, not a setting

The consequence of one model drawing the map is that changing the model redraws it. Every vector in the index was produced by a specific model at a specific version, and a newer model does not produce vectors that can be compared with the old ones. Switching means re-embedding the whole corpus, then rebuilding the index over the result.

That makes the choice worth more thought than it usually gets, and it makes one piece of metadata close to mandatory: record the model name and version on every chunk. Without it, there is no way to answer whether an index is internally consistent, and a half-migrated corpus is indistinguishable from a healthy one until the answers start degrading.

Retrieval: dense, sparse, and where hybrid actually helps

Dense retrieval ranks chunks by embedding similarity, the approach described above. Sparse retrieval (commonly BM25, a lexical ranking algorithm older than embeddings) ranks by term overlap and frequency, closer to traditional keyword search. Hybrid retrieval combines both signals.

The three fail in different directions, which is the whole reason the choice matters:

That last question is the part usually skipped. Dense and sparse retrieval produce scores on unrelated scales: a cosine similarity of 0.82 and a BM25 score of 14.3 cannot be added, averaged, or compared without inventing a weighting. The standard answer avoids scores entirely. Reciprocal Rank Fusion discards the scores and merges on rank position alone, summing 1 / (k + rank) for each document across the rankings it appears in. The original SIGIR 2009 paper fixed k = 60 during a pilot and did not tune it afterwards, and reported that this beat both Condorcet Fuse and the individual systems being combined. The constant exists to stop one system’s confident top hit from dominating the merge. Its practical appeal is that it needs no training data and no score calibration, which is why it turns up as the default fusion method in search engines and vector stores rather than because it is optimal for any particular corpus.

The common advice is that hybrid retrieval is a strict improvement over either alone. A 2026 benchmark across ten retrieval strategies on a financial question-answering corpus found something more specific: a two-stage pipeline combining hybrid retrieval with neural reranking reached the highest overall result of the ten strategies tested (Recall@5 of 0.816, MRR@3 of 0.605), but on that same corpus, plain BM25 outperformed every dense retrieval method the study benchmarked - directly against the assumption that semantic search universally dominates lexical search. The paper’s own framing is specific about why: dense embeddings are trained for semantic similarity, and a financial document full of exact figures, ticker symbols and defined terms rewards exact term matching in a way that generic semantic similarity does not capture as well.

The generalisable point is not “use BM25.” The same paper’s best overall result still used hybrid retrieval with reranking, not BM25 alone. It is that the choice between dense, sparse and hybrid retrieval is corpus-dependent, and treating semantic search as universally superior means never noticing when a lexical or hybrid approach would materially outperform it on your specific documents.

Reranking

The two-stage shape of retrieval is not an optimisation bolted on afterwards. It follows from a cost difference large enough to decide the architecture.

A bi-encoder embeds the query and each passage separately, so every passage vector can be computed once, at indexing time, and reused for every query thereafter. A query then costs one embedding plus a nearest-neighbour lookup. A cross-encoder feeds the query and a passage through the model together and scores the pair, which is more accurate, because the model sees both texts at once and can attend across them. It also cannot be precomputed. Every query-passage pair is a fresh forward pass.

The Sentence-BERT paper puts a number on that gap. Finding the most similar pair in a collection of 10,000 sentences takes roughly 50 million inference computations, about 65 hours, with a BERT cross-encoder. The same task with precomputed bi-encoder embeddings takes about 5 seconds, while keeping the accuracy. That ratio is why nobody cross-encodes an entire corpus per query, and why retrieval is split in two.

A reranker is that cross-encoder, applied only to the shortlist the bi-encoder already narrowed down. Reranking 50 candidates is affordable; reranking 5 million is not. This also sets where the effort belongs. Only a handful of passages ever reach the generator, so improving the ordering at the top of a short list is worth more than improving recall far down a long one. The retriever decides what is available, and the reranker decides what actually gets used.

One consequence follows directly and is easy to miss: a reranker cannot rescue a chunk the retriever never returned. If the relevant passage is not in the candidate set, no amount of reranking quality recovers it, which makes first-stage recall the ceiling on everything downstream.

Evaluation: retrieval quality and generation quality are different questions

Two separate things can each fail independently. Evaluating only one hides which is actually broken:

• Retrieval quality: did the system surface the relevant chunks at all? Measured by the ranking metrics in the table below.
• Generation quality: given the chunks that were retrieved, did the model produce a correct, grounded answer from them. A perfect retriever feeding a model that ignores the retrieved context, or a flawed retriever whose one relevant chunk still let the model answer correctly, both look fine if only the end-to-end answer is checked.

Benchmarking libraries built specifically for RAG, such as BERGEN, evaluate these as separable components rather than only the end-to-end pipeline, for exactly this reason.

The retrieval-side metrics answer genuinely different questions, and quoting one without saying which is a common way to make a system sound better than it is:

Recall@k is the one to measure first, because it bounds everything after it. Pick k as the number of chunks that genuinely reach the generator, not a comfortable number like 50 that flatters the score.

Failure modes

• Embedding model mismatch. Indexing with one model and querying with another (or a different version of the same model) produces vector distances that do not mean what they are assumed to mean.
• Chunking that ignores document structure. A chunk boundary placed by token count alone, with no regard for tables, code blocks, or section breaks, can split the one fact a query needs across two retrieved-or-not-retrieved chunks.
• Assuming hybrid or dense retrieval is universally better without measuring on your own corpus. The benchmark cited above found the opposite of the common assumption on financial documents. There is no substitute for measuring retrieval quality on your actual documents and query patterns.
• Evaluating only the final answer. Without measuring retrieval and generation separately, a system that answers well by accident and a system that answers well because retrieval actually worked are indistinguishable until something changes and only one of them keeps working.

Glossary

Every term on this page, in one place, in the order a request meets them.

What to watch

• Whether corpus-specific retrieval benchmarking (measuring dense vs. sparse vs. hybrid on your own documents before choosing) becomes standard practice, given how directly the 2026 benchmark above contradicts the “hybrid always wins” default.
• Whether reranking becomes a default second stage in production RAG pipelines rather than an optimisation added after the fact, given how consistently it improves top-of-list ranking quality across the benchmarks cited here.