"Retrieval-Augmented Generation" - what it actually said

Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel and Kiela. Submitted to arXiv as 2005.11401 on 22 May 2020, last revised 12 April 2021 (v4); the arXiv record states “Accepted at NeurIPS 2020”. Affiliations are Facebook AI Research, University College London and New York University.

The problem it addressed

Pre-trained language models store factual knowledge in their parameters and can be fine-tuned to use it. The paper identifies three specific limitations of that arrangement, and it is worth quoting the structure of the argument because the popular summary flattens it:

1. Access and manipulation of knowledge is imprecise, so on knowledge-intensive tasks these models lag behind task-specific architectures.
2. There is no provenance. You cannot show which source supports a generated claim.
3. World knowledge cannot be updated without further training.

Prior work had attached a differentiable retrieval mechanism to pre-trained models, but the paper notes it had been investigated only for extractive downstream tasks, where the answer is a span copied out of a retrieved passage. Nothing had done it for general generation.

So the question is narrow: can you give a sequence-to-sequence generator a non-parametric memory it learns to query, and does that help on tasks where the answer must be composed rather than extracted?

What it actually proposed

A general-purpose fine-tuning recipe. The name is precise about this: retrieval-augmented generation, where a pre-trained parametric memory (a seq2seq model) and a pre-trained non-parametric memory (a dense vector index) are fine-tuned together on a downstream task.

The components, exactly as specified:

Two aspects of the method are where the popular account diverges most sharply from the paper.

The retrieved document is a latent variable, marginalised over. The paper compares two formulations, and the difference between them is not cosmetic:

• RAG-Sequence uses the same retrieved document for the whole output sequence, treating it as a single latent variable marginalised via a top-k approximation. Decoding requires running beam search once per document and then scoring hypotheses across documents.
• RAG-Token can draw a different document per generated token, summing over documents inside the transition probability. That makes it a standard autoregressive decoder over a mixture, so an ordinary beam decoder works.

Neither is “put the retrieved text in the prompt and generate once”. The output probability is a sum over retrieved documents weighted by their retrieval probability.

Training is joint but partial. The retriever and generator are trained together by minimising the negative marginal log-likelihood of the target, with no supervision on which document should be retrieved. But the document encoder and the index are held fixed: only the query encoder and the BART generator are fine-tuned. The paper is explicit that updating the document encoder would require periodically re-indexing, as REALM does, and that they “do not find this step necessary for strong performance”.

What it demonstrated, and what it only suggested

Demonstrated across four task families, all with the same Wikipedia index:

The knowledge-update experiment is the one worth understanding in detail, because it is the empirical basis for the claim that retrieval makes knowledge editable. They built an index from a December 2016 Wikipedia dump, prepared a list of 82 world leaders whose office had changed between 2016 and 2018, queried with the template “Who is {position}?”, and compared answers across indices. The model was not retrained. That is the demonstration: narrow, well-designed, and about factual substitution, not about behaviour.

Two ablations in the paper are load-bearing and rarely quoted:

• Learned retrieval matters, but not everywhere. Freezing the retriever degraded results on every task. Replacing the dense retriever with BM25 degraded open-domain QA substantially, but BM25 performed better on FEVER, which the authors attribute to FEVER claims being entity-centric and therefore well suited to word overlap.
• More retrieved documents is not uniformly better. Retrieving more at test time monotonically improved open-domain QA for RAG-Sequence, but RAG-Token peaked at 10 documents. On MS-MARCO, more documents raised Rouge-L at the expense of Bleu-1.

Suggested rather than demonstrated: the discussion proposes that the two components might be jointly pre-trained from scratch with a denoising objective, and calls the interaction of parametric and non-parametric memory a new research direction. The Broader Impact section states that grounding makes the model “hallucinate” less and offers more control and interpretability, while also stating that Wikipedia, or any external knowledge source, “will probably never be entirely factual and completely devoid of bias”.

What has held up

The core claim has held up thoroughly: attaching a non-parametric memory to a generator improves factual accuracy on knowledge-intensive tasks, and lets knowledge be updated by editing the index instead of by retraining. The comparison against T5-11B closed-book has also held up as an argument about efficiency: a 400M-parameter generator with an index outperformed a much larger parametric model on Natural Questions (Lewis et al., 2020), and that shape of trade is now standard practice.

The paper’s argument that raw text is the right memory format has held up for a reason it states directly: a memory of raw text is human-readable, giving a form of interpretability, and human-writable, so the model’s knowledge can be updated by editing the document index. That is the property every production retrieval system depends on.

What has changed: almost all of the implementation. The retriever is no longer usually a DPR bi-encoder, the index is rarely FAISS-over-Wikipedia, hybrid sparse-plus-dense retrieval and reranking are now normal, chunking is tuned per corpus and no longer fixed at 100 words, and, most significantly, the marginalisation is generally gone. Contemporary systems concatenate retrieved text into the context of a frozen instruction-tuned model and generate once.

That last change is a real departure, not a refinement. It drops the trained query encoder, the latent-variable formulation and the per-document marginalisation, which is to say it drops most of the method. It works because instruction-tuned models are far better at using supplied context than BART-large was, but the evidence for it is not this paper’s evidence.

What is commonly misattributed to it

“It invented retrieval-augmented language models.” It named RAG and it is the reference for the term, but the paper positions itself among prior work: DrQA for open-domain QA, ORQA, REALM, and DPR, whose pre-trained bi-encoder it uses to initialise its own retriever and build its index. The contribution is extending learned retrieval to general generation, not to extractive tasks.

“It proposed pasting retrieved chunks into a prompt.” It did not. There is no prompting in the paper. There is a fine-tuning objective, gradients flowing into the query encoder, and a sum over retrieved documents. If your pipeline uses a frozen model and a template, you are running a later, simpler technique that inherited the name.

“It showed retrieval eliminates hallucination.” It reported that human evaluators preferred RAG generations as more factual than BART’s in a pairwise comparison on one generation task, and its Broader Impact section explicitly warns that the knowledge source will never be entirely factual or unbiased, and that RAG can be used as a language model with the misuse risks that implies. “More factual than a comparable parametric baseline on Jeopardy question generation” is the finding.

“It showed dense retrieval beats keyword search.” On open-domain QA, yes, and the paper says retrieval quality is crucial there. On FEVER, BM25 was better. The generalisable lesson is that the right retriever depends on the query distribution, which is the opposite of the lesson usually drawn.

“It established that you should retrieve k documents.” It used k ∈ {5, 10} in training and chose k on dev data, and reported that the effect of more documents at test time differs by model formulation and by metric. There is no recommended k, and the paper contains direct evidence that more is not always better.

“The 100-word chunk is a recommended chunk size.” It was inherited from prior work for comparability on Wikipedia, and applies to disjoint chunks of encyclopaedia articles. It is not a finding about chunking, and nothing in the paper evaluates alternatives.

“It provides citations for its answers.” It provides provenance in the sense that the retrieved documents are human-readable text and the document posterior can be inspected. The paper shows an example where the posterior shifts between documents mid-generation. That is interpretability of the retrieval step, not a mechanism for attaching a citation to a claim. Producing verifiable citations is a separate engineering problem.

“It trained the retriever end to end.” Only the query side. The document encoder and the index were fixed, deliberately, to avoid periodic re-indexing. Any claim that RAG learns document representations for the task is wrong about this paper.

“It showed retrieval is better than fine-tuning.” RAG is fine-tuning - the paper’s own framing is “a general-purpose fine-tuning recipe”. The comparison it makes is against parametric-only models that were also fine-tuned. Nothing here argues against tuning; it argues for adding a non-parametric memory to a tuned model. For the decision people actually face, see retrieval or fine-tuning.

Related

• Retrieval-augmented generation - the architecture as practised now
• RAG pipeline - the components you operate
• Retrieval or fine-tuning - the decision this paper is cited in
• “Attention Is All You Need” - what it actually said