What a vector database actually does, and what it does not
Retrieval, cosine similarity, and the step most explanations skip.
The word “our” is doing the work. No language model can answer this from its training, because the answer lives in an internal policy document that did not exist when the model was trained. Softmax always returns a valid distribution, so the model answers anyway, fluently and wrongly. That is not a bug someone forgot to fix. It is structural.
This part covers the machinery that fixes it, and the single most important thing to understand about that machinery: the vector database never talks to the model.
I state that plainly because most explanations imply otherwise. The database does not extend the model’s knowledge, does not touch its weights, and does not participate in inference. It fetches text. That text is pasted into the prompt as ordinary characters.
As in part one, every figure I quote from my own experiments was computed locally, on a corpus small enough to reproduce here in full.
STEP 1. Offline, Before Any Prompt Exists
Documents are split into chunks, commonly a few hundred tokens, ideally on structural boundaries so that a table or a clause is not cut in half. Each chunk goes through an embedding model, which is separate from and much smaller than the LLM, and becomes one fixed-length vector. Vector plus original text plus metadata goes into the index.
For scale, from vendor documentation: OpenAI’s text-embedding-3-small produces 1536-dimensional vectors by default and text-embedding-3-large produces 3072, with a dimensions parameter available to shorten them.
My six-document test corpus includes the chunk that should answer our question:
“Slack message retention is set to three years for all workspaces.
Legal hold overrides retention and suspends automatic deletion
until released.”
Nothing happens to this chunk at query time. It is already sitting there, already vectorized.
STEP 2. The Prompt Is Vectorized, And Not By The LLM
The first model to touch the prompt is not the language model. The question becomes a single vector. In my TF-IDF vector space, dimensionality 45, the query has just two non-zero components:
Query: What is our retention period for Slack messages?
retention 0.7733
slack 0.6341
All 43 other dimensions: 0.0
One hard constraint in production: the query must use the same embedding model as ingestion. Query and stored vectors have to live in the same space or the geometry is meaningless. Critically, a mismatch fails silently. The result is plausible garbage, not an error.
STEP 3. The Database Searches For Nearest Neighbours
Similarity is usually cosine similarity, the angle between two vectors rather than the distance between their tips.
Worth being precise here, because this detail is often overstated. OpenAI’s documentation recommends cosine similarity but adds that the choice of distance function typically does not matter much. Their embeddings are normalised to length 1, which means cosine similarity can be computed as a plain dot product, and cosine and Euclidean distance produce identical rankings. So cosine is the sensible default and the conventional choice, not a decision that will make or break retrieval quality.
Real computed scores against my corpus:
The right document ranks first. Rank 6 is the detail I did not anticipate when I wrote the corpus. The document email-retention-policy scored exactly zero, despite being a document about retention. Why? It says “retained.” The query says “retention.” Different strings, zero lexical overlap, orthogonal vectors.
That single 0.0000 is the best argument for learned embeddings I could have produced, and I got it by accident.
Production databases do not compare against every vector. They use approximate nearest neighbour indexes.
HNSW is a multi-layer proximity graph traversed greedily. Fast, high recall, memory resident, expensive to build. The common default.
IVF partitions vectors into clusters and searches only the nearest few. Cheaper on memory, tunable.
Product quantization compresses vectors into compact codes, trading accuracy for large memory savings. Often combined as IVF-PQ.
The word “approximate” is load-bearing. These indexes have a recall parameter, and setting it means explicitly accepting that some true nearest neighbours will be missed in exchange for latency. That is a tuning decision, not a default to leave alone.
STEP 4. What Lexical Matching Cannot Do
I ran one more query against the same corpus, a paraphrase that shares almost no vocabulary with the document that answers it:
Query: How long do we keep chat history before it is wiped?
Every score zero. Total retrieval failure. Not even a wrong-but-plausible ranking. The system has nothing to say.
Any human reader knows instantly that document 1 answers that question. “How long” means “three years.” “Chat history” means “message.” “Wiped” means “deletion.” Not one of those pairs shares a token.
This is precisely the gap learned embedding models fill. They place paraphrases near each other because they were trained on text where those phrasings appear in similar contexts. The retrieval mechanism, which is to vectorize, take cosines, and rank, is identical. Only the quality of the vectors changes.
Scope note: my scores come from TF-IDF, which builds vectors from word frequency. The mechanism is the same as a production vector database; the vectors are far weaker. A real embedding model would score that paraphrase well above zero. I could not run one locally, so I have shown the failure instead, which turned out to teach more than a success would have.
STEP 5. Reranking And Ordering
Vector similarity is fast but blunt. A common second pass sends the query paired with each candidate through a cross-encoder, a small model that reads both together and scores actual relevance. Far more accurate than comparing independent vectors, and far too slow for the whole corpus, which is why it runs only on the shortlist.
Ordering matters for a documented reason. Liu and colleagues, in “Lost in the Middle“, found that model performance is often highest when relevant information sits at the beginning or the end of the input context, and degrades significantly when the model must retrieve it from the middle. That held even for models explicitly built for long contexts.
Full citation: Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang (2024). Transactions of the Association for Computational Linguistics, vol. 12, pp. 157-173. DOI 10.1162/tacl_a_00638. The preprint is arXiv:2307.03172 and dates from 2023, so the year to cite depends on which version is linked.
Stated precisely: that paper tested multi-document question answering and key-value retrieval. It is strong evidence about position sensitivity, not a universal law about all prompts.
STEP 6. The Retrieved Text Is Pasted Into The Prompt
This is the step does the whole trick:
System: Answer using only the context below. Cite sources.
If the context does not contain the answer, say so.
<context>
[doc: retention-policy | section 8.2]
Slack message retention is set to three years for all workspaces.
Legal hold overrides retention and suspends automatic deletion
until released.
</context>
User: What is our retention period for Slack messages?
Plain text in the prompt. Nothing loaded, nothing learned, nothing persisted.
STEP 7. Now The Model Runs
Tokenization, embedding, attention, sampling, exactly as in part one. Unchanged. From the model’s point of view there is no database anywhere. There is only a longer prompt that happens to contain the answer.
Attention is the mechanism that makes it land. The question and the retrieved sentence now sit in the same context window, so the positions holding “three years” are available to be attended to. The model is not recalling a fact. It is reading one that was handed to it.
Two Consequences Worth Internalising
The model cannot retrieve what the database missed. My paraphrase query returned six zeros. A model handed that empty result set will still answer, fluently and wrongly. Bad retrieval does not surface as an error. It surfaces as a confident answer. A large share of what teams call “RAG hallucination” is retrieval failure in disguise.
Everything retrieved is untrusted input. Those chunks enter the prompt through the same channel as the system’s own instructions. If a document contains the words “ignore previous instructions,” a naive system may follow them. Retrieved content is data and must never be treated as instruction. In my view this is the most under-discussed risk in the stack.
And an operational third. If retrieval ignores access control, RAG becomes an exfiltration path. The model will cheerfully summarise a document the user was never permitted to open. Filtering has to happen by the caller’s identity at query time, not after.
The One-Line Version
The vector database converts the question into a vector, finds nearby text, and that text is pasted into the prompt. Retrieval is a text-fetching step that happens before inference. It is not a component of the model.
Everything else, whether that is chunking strategy, reranking, hybrid search or evaluation, is engineering around that one sentence.
What I Measured, And What I Did Not
Every retrieval score above came from that six-document corpus: the query vector and its dimensionality, both rankings, and every cosine value. I recomputed all of it from scratch a second time before publishing, because the first run is the one that flattered me. The two figures I did not compute are the embedding dimensions, which come from OpenAI’s documentation, and the position sensitivity finding, which comes from the Liu paper cited above.
What it is not: TF-IDF is not a learned embedding model, and a real one would score that paraphrase well above zero rather than at zero. Named vector-database comparisons and benchmark numbers are absent on purpose, since they date within months. Index types, which I do discuss, are stable.
Corrections are welcome. Every figure above traces to either my terminal or a other source.
Leave a Reply