RAG at Production Scale

A retrieval prototype is a function: text in, similar text out. A production retrieval system is three things that happen to share a table: an incremental data pipeline, an access-control boundary, and an ingestion surface that anyone with write access to a source system can reach. The pattern itself is covered in RAG. What changes below is everything that follows from a corpus being large, sources being live, and readers not all holding the same permissions.

The framing that makes the rest coherent: the index is a derived, stale, and separately authorised copy of your documents. Every production failure below is one of those three words being ignored.

The pipeline, end to end

Six stages. The interesting properties live in the seams, not the stages.

The single most consequential decision is at the top. Drive the pipeline from a change feed, not a crawl. A crawl can discover new and modified documents; it cannot reliably discover deletions, because a deleted document is indistinguishable from one the crawl did not reach. That is how you get ghost retrieval: a document that was removed from the wiki six weeks ago, still in the index, still being quoted as current policy, with a citation that 404s. Deletion has to be an explicit operation against the index, propagated with the same urgency as a permission revocation. That is frequently what it is.

Stamp every chunk with the source version it was derived from, and make the retrieval path capable of returning that. Without it, you cannot answer the question a user will eventually ask, which is “where did that come from, and was it current?”

Freshness is a distribution, not a number

Indexing lag behaves like replication lag: the average is reassuring and the tail is what hurts. One document class with an extractor that times out, one source system whose webhook has been failing since a credential rotation, and the mean stays healthy while a whole category of content is weeks behind.

Measure the age of the oldest unprocessed change per source, not the mean processing time. Set a freshness budget per corpus and treat a breach as an incident. The failure mode is not “slow” but “confidently wrong about something that changed”.

A related and less obvious point: re-embedding is not free and not rare. Chunk boundaries and the embedding model are baked into the stored vectors, so changing either invalidates the entire index. A chunking strategy is therefore a schema decision with a migration cost. Treat it with the seriousness of one. Build the new index alongside the old, with the model identity in the index name, and cut over on measured results, not on the assumption that a newer embedding model is better for your corpus.

Authorisation before retrieval

This is the design decision that separates a system that can hold sensitive documents from one that cannot, and it is usually got wrong in the same way: retrieve the top k by similarity, then drop the ones the caller may not see.

That post-filter is not equivalent to a pre-filter, for reasons that have nothing to do with convenience.

• The context budget is already spent. If eight of ten candidates are dropped, the answer is built from two passages, and the model has no idea it is reasoning from a truncated view. Quality degrades exactly for the users with the narrowest permissions.
• Existence leaks. Result counts, latency and score distributions are all observable. “No results” and “results you may not see” are different, and a determined caller can tell them apart.
• The unauthorised documents influenced ranking. Relevance scores are computed relative to the candidate set, so a document the caller cannot see has already shaped what they do see.
• It fails open. A missing filter in one code path is invisible. A missing predicate in a query is a query that returns nothing.

Keeping the vectors in PostgreSQL with pgvector makes the correct version expressible directly, because similarity search is an operator inside an ordinary SQL query. The permission predicate goes in the same statement as the distance operator:

SELECT c.chunk_id, c.body, c.source_version,
       c.embedding <=> $1 AS distance
FROM   doc_chunk c
JOIN   doc_grant g ON g.doc_id = c.doc_id
WHERE  g.principal = ANY($2::text[])     -- the caller's groups, resolved per request
  AND  c.deleted_at IS NULL
  AND  c.tenant_id  = $3
ORDER  BY c.embedding <=> $1
LIMIT  10;

Row-level security is the stronger form of the same idea: a policy on doc_chunk that no query can forget, enforced by the database, not by the caller. The cost is that the policy has to be maintainable, and that a permission model expressed in a source system has to be projected into the database and kept current. That is the same freshness problem as the corpus, applied to grants, and it is the one with the shorter tolerance. A revoked grant that takes an hour to propagate is an hour of retrievable documents.

One performance consequence is worth naming before you discover it. An approximate index is built over the whole table; adding a restrictive predicate means the index scan walks candidates that the filter then discards, so a highly selective permission filter can return fewer rows than requested, or fall back to a slower plan. That is not a reason to filter afterwards. It is a reason to partition the index by tenant where the tenancy boundary is coarse, and to measure recall with the filter applied, which is the only measurement that describes what your users actually get.

The corpus is a supply chain

Retrieved content is, functionally, part of the prompt. So the set of people who can write to an indexed source system is the set of people who can influence every answer the system produces. Say that out loud during design, because the list is usually much longer than anyone expects: an onboarding wiki anyone can edit, a ticketing system open to customers, a shared drive with inherited permissions nobody has audited, a vendor’s documentation mirrored nightly.

This is a supply-chain problem, and it responds to supply-chain controls rather than to model-level ones. Note that it does not require any prompt injection: a document that contains no instructions at all, only plausible and wrong content, will be retrieved on relevance and believed on authority. Guardrails that look for imperative language in retrieved text do not address this. See vector and embedding weaknesses for the mechanisms, and supply chain for the broader category.

What actually helps:

Separate indexes by trust level. A corpus of reviewed, versioned internal policy and a corpus of customer-submitted attachments should not share an index, because a single index forces a single trust decision. Retrieve from each explicitly and label the provenance in what reaches the model.

Require provenance on every chunk. Source system, author, last editor, review status, version. This is the metadata that lets you answer “what changed?” after a bad answer, and it is the metadata that lets you exclude unreviewed content from high-stakes query paths.

Rate-limit influence, not just ingestion. A source that suddenly contributes a large share of retrieved passages is either newly relevant or newly gamed. Both are worth a look. Track retrieval share by source and by document age; a spike in retrievals of documents created in the last week is the signal that generalises.

Keep a fixed evaluation set and run it after every ingestion. Not for relevance scoring, but for change detection. If an ingestion run alters the answers to questions it had no business touching, something entered the corpus that is competing for retrieval more broadly than its content justifies. This is the cheapest poisoning detector available and it costs one scheduled job.

What to measure

Relevance metrics dominate the literature and are the least useful of these in operation, because relevance degrades slowly and the other three degrade suddenly.

Finally, build in the ability to refuse. A retrieval system that cannot find good evidence should say so, and the plumbing for that - a similarity threshold, an abstention path, a response that names what it could not find - has to exist before it is needed. The alternative is a system whose worst outputs are indistinguishable from its best ones, which is the failure that makes the whole topology untrustworthy and not merely imperfect.