{"id":278,"date":"2026-09-02T16:02:09","date_gmt":"2026-09-02T16:02:09","guid":{"rendered":"https:\/\/metaintelligencesystems.com\/blog\/?p=278"},"modified":"2026-09-02T16:02:09","modified_gmt":"2026-09-02T16:02:09","slug":"what-actually-happens-when-you-send-a-prompt","status":"publish","type":"post","link":"https:\/\/metaintelligencesystems.com\/blog\/what-actually-happens-when-you-send-a-prompt\/","title":{"rendered":"What actually happens when you send a prompt"},"content":{"rendered":"\n

One prompt, traced end to end: tokens, attention, and the part where a vector database gets involved.<\/p>\n\n\n\n

ONE PROMPT, ALL THE WAY DOWN<\/strong><\/p>\n\n\n\n

I am going to use exactly one prompt for this entire article:<\/p>\n\n\n\n

What is our retention period for Slack messages?<\/strong><\/code><\/pre>\n\n\n\n
I picked it on purpose. The word “our” is doing the work. No language model can answer this from its training, however large it is, because the answer lives in an internal policy document that did not exist when the model was trained. That one word separates “how a model generates text” from “how a model generates text grounded in private data,” and it lets me cover both without switching examples.<\/em><\/p>\n\n\n\n
Every figure I quote from my own experiments was computed locally, on a corpus small enough to reproduce here in full. Where a number comes from a published source instead, I say so. And I will be explicit throughout about what my measurements do and do not demonstrate.<\/p>\n\n\n\n
Stage 1.<\/strong> Words Becomes Numbers<\/p>\n\n\n\n
The model cannot read characters. First comes tokenization: text is split into subword units, and each unit maps to an integer ID from a fixed vocabulary.<\/p>\n\n\n\n
Most current models use Byte-Pair Encoding or a SentencePiece variant. BPE(Byte-Pair Encoding.) is built by starting from individual characters (or bytes, in byte-level variants like GPT-2’s) and repeatedly merging the most frequent adjacent pair, thousands of times, until the vocabulary hits a target size.<\/p>\n\n\n\n
Rather than quote a production tokenizer’s IDs, I implemented BPE from scratch and trained it on a 150-word corpus of policy-style text, sixteen sentences of the kind an internal-policy assistant would see. Here are the first merges it learned, in order:<\/p>\n\n\n\n
\"Article<\/figure>\n\n\n\n
Merges 12 through 14 are the ones worth pausing on. The algorithm assembles ‘re’ plus ‘te’ into ‘rete’, then ‘reten’, then joins it to the ‘tion<\/w>’ it had already discovered back at step 7. It builds the whole word “retention” out of fragments it learned separately. Nobody specified that. It is pure frequency counting, applied repeatedly.<\/p>\n\n\n\n
With 120 merges learned and a vocabulary of 148 symbols, here is our prompt tokenized:<\/p>\n\n\n\n
\"Article<\/figure>\n\n\n\n
\n
IDs: [24, 10, 3, 102, 113, 16, 22, 95, 146, 138, 125, 100, 77, 114, 36, 2, 77]<\/p>\n\n\n\n
17 tokens for 8 words, or 2.12 tokens per word<\/p>\n<\/blockquote>\n\n\n\n
Now the interesting part. Why did some words survive as single tokens while others shattered? Cross-referencing each word against how often it appears in the training corpus gives the answer:<\/p>\n\n\n\n
\"Article<\/figure>\n\n\n\n
Counts are of the exact lowercased word. “Slack” appears three times in the corpus, but twice with punctuation attached, as ‘slack.’ and ‘slack?’, so the bare form occurs once. That is exactly why it never fused into a single symbol: my merge rule requires a pair to appear at least twice, and ‘slack’ plus end-of-word only ever appeared once.<\/p>\n\n\n\n
The pattern is clean. Words the tokenizer saw often compressed to one ID. Words it barely saw or never saw decomposed into fragments. This is the entire design goal of subword tokenization: the tokenizer never fails on unfamiliar input, it just spends more tokens on it.<\/p>\n\n\n\n
That has three practical consequences:<\/p>\n\n\n\n
Whitespace and punctuation are part of the token. The symbols ‘r<\/w>’ and ‘or’ differ precisely because of the end-of-word marker. In production tokenizers a leading space works the same way, which is why “Slack” and ” Slack” are different IDs, and why a stray trailing space in a prompt can occasionally shift output.<\/p>\n\n\n\n
This is why models are bad at spelling tasks. A model asked to count the letters in a word struggles, because it never sees letters. It sees ‘messag’ and ‘es’. The limitation lives in the tokenizer, not in the intelligence.<\/p>\n\n\n\n
There is a cost asymmetry across languages. BPE merges are learned from a training corpus that is overwhelmingly English. My toy tokenizer makes this visible in miniature: “our” cost 3 tokens because the corpus never contained it, so no merge could ever assemble it. Scaled up, the same sentence in an under-represented language consumes measurably more tokens than its English equivalent. Since nearly every API prices per token, that is a real cost difference, not a curiosity.<\/p>\n\n\n\n
Note<\/em><\/strong>: This is my own small BPE, not GPT-4’s tokenizer. The algorithm is the same one production tokenizers use. The scale is not. Real vocabularies run to 100k or more symbols trained on enormous corpora, so their specific IDs and splits differ completely from mine. Anyone who wants production numbers should encode their own text with the tokenizer their provider publishes, and state which encoding was used. I built something small enough to verify end to end instead.<\/em><\/p>\n\n\n\n
Stage 2.<\/strong> Numbers Become Meaning<\/p>\n\n\n\n
Each token ID indexes a row in an embedding matrix, and that row is a vector of floats. For a mid-size model the matrix might be vocabulary size by 4096.<\/p>\n\n\n\n
The important part: nobody assigned these vectors. Nobody wrote down that “Slack” should sit near “Teams.” That geometry emerged from gradient descent over an enormous corpus. It is learned structure, not a hand-built ontology.<\/em><\/p>\n\n\n\n
That property is worth holding onto. It is the same one the vector database will exploit later, and the reason the two halves of this article are really one story.<\/p>\n\n\n\n
Attention has no inherent sense of order. Without position information, “retention policy” and “policy retention” would be identical inputs. Position gets injected either by adding positional embeddings or, in most current open-weight models, by rotating the query and key vectors by a position-dependent angle, which is rotary position embedding, or RoPE. Closed models do not disclose their architecture, so “most” is the honest word.<\/p>\n\n\n\n
Stage<\/strong> 3: Attention, Or Where – Enough Information – comes<\/p>\n\n\n\n
How does the model know what a prompt means?<\/p>\n\n\n\n
Each token’s vector is projected into three vectors: a query, a key, and a value<\/em>. Token i’s query is compared against every token’s key by dot product, scaled, and pushed through a softmax<\/a> to produce weights that sum to 1. Those weights take a weighted average of the value vectors.<\/p>\n\n\n\n
In plain language: every token asks “who here is relevant to me”, receives a set of relevance scores, and pulls in a blend of the other tokens’ information.<\/p>\n\n\n\n
In our prompt, the content-bearing tokens, “retention” and “slack”, would be expected to draw more weight at the final position than function words like “is” or “for”. I did not measure that here, since inspecting real attention maps is a separate exercise. The structural point is what matters: nobody programs which tokens matter. It falls out of the learned projections.<\/p>\n\n\n\n
Three details that matter in practice.<\/p>\n\n\n\n
Multi-head attention<\/strong><\/a>:<\/strong> This runs in parallel across many heads with separate projections, and different heads specialise. Interpretability researchers have identified heads that appear to perform specific jobs, though that is an active research area, not settled fact.<\/p>\n\n\n\n
Causal masking<\/strong><\/a>: Positions after i are set to negative infinity before the softmax, so no token attends to the future. This is what makes left-to-right generation coherent. A model trained without it would learn to peek at the answer.<\/p>\n\n\n\n
The KV cache<\/strong><\/a>: Keys and values for earlier tokens do not change as generation proceeds, so they are cached. This is why the first token of a response is slow, since the whole prompt must be processed, while later tokens are fast, one at a time. At long context lengths and large batch sizes the cache becomes a major memory consumer at serving time, and it is a big part of why long conversations get expensive.<\/p>\n\n\n\n
Each layer pairs attention with a feed-forward network<\/a>, which holds most of the model’s parameters. Current interpretability work suggests attention moves information between positions while feed-forward layers store and retrieve learned associations. I want to be careful here: that is a useful intuition from ongoing research, not a proven architectural fact.<\/p>\n\n\n\n
Stage<\/strong> 4: A Probability for Every Possible Next Token<\/p>\n\n\n\n
The final hidden state is multiplied by the unembedding matrix, producing one raw score, called a logit, per vocabulary entry. Softmax turns those into probabilities. A sampling step picks one.<\/p>\n\n\n\n
temperature   – Divides the logits before softmax. Below 1 sharpens toward  the top candidate, above 1 flattens the distribution. Zero is greedy: always the highest-probability token, fully deterministic.<\/p>\n\n\n\n
top_k     –    Keeps only the k highest-probability tokens, then renormalises.<\/p>\n\n\n\n
top_p     –    Keeps the smallest set of tokens whose cumulative probability exceeds p. Adapts to model confidence.<\/p>\n\n\n\n
frequency and presence penalties  Subtract from the logits of tokens already used, which reduces repetition.<\/p>\n\n\n\n
Here is the uncomfortable implication. Softmax always returns a valid probability distribution. If the model has no strong learned pattern for a given question, that distribution is nearly flat, but sampling still returns a token, and the resulting sentence reads exactly as fluent as a well-grounded one. The model has no separate “I do not know” output channel.<\/em><\/p>\n\n\n\n
Hallucination is not a bug someone forgot to fix. It is a structural consequence of generating text by sampling from a distribution that is always defined.<\/p>\n\n\n\n
Which brings us back to our prompt. Asked what “our” retention period is, a bare model produces something fluent, specific, and wrong, because no mechanism exists by which it could know.<\/p>\n\n\n\n
What I Measured, And What I Did Not<\/p>\n\n\n\n
Every tokenization figure above came from that 150-word corpus: the merges and their frequencies, the token IDs, the 17 tokens, the word-frequency table. I recomputed all of it from scratch a second time before publishing, because the first run is the one that flatters the author.<\/p>\n\n\n\n
What it is not: my BPE is not a production tokenizer, and I make no claim that these specific IDs resemble any real model’s. Parameter counts, context window sizes and benchmark numbers are absent on purpose, since they date within months and I would have been quoting them from memory.<\/p>\n\n\n\n
Where This Leaves Us<\/strong><\/p>\n\n\n\n
Nothing in stages 1 to 4 can fix that wrong answer. The mechanism that does fix it is not part of the model at all. It sits in front of the model, runs before inference, and is routinely described as something it is not.<\/p>\n\n\n\n
How a vector database turns a question into a vector, what cosine<\/a> similarity actually measures, the retrieval failure I did not expect to find in my own test corpus, and the one step most explanations skip.<\/p>\n\n\n\n
Corrections are welcome. Every figure above traces to either my terminal or a other sources<\/p>\n\n\n\n\n\n\n\n
What a vector database actually does, and what it does not<\/h1>\n\n\n\n
<\/p>\n\n\n\n
Retrieval, cosine<\/strong><\/a> similarity, and the step most explanations skip.<\/p>\n\n\n\n
The word “our” is doing the work. No language model can answer this from its training, because the answer lives in an internal policy document that did not exist when the model was trained. Softmax always returns a valid distribution, so the model answers anyway, fluently and wrongly. That is not a bug someone forgot to fix. It is structural.<\/p>\n\n\n\n
This part covers the machinery that fixes it, and the single most important thing to understand about that machinery: the vector database never talks to the model.<\/p>\n\n\n\n
I state that plainly because most explanations imply otherwise. The database does not extend the model’s knowledge, does not touch its weights, and does not participate in inference. It fetches text. That text is pasted into the prompt as ordinary characters.<\/p>\n\n\n\n
As in part one, every figure I quote from my own experiments was computed locally,<\/a> on a corpus small enough to reproduce here in full.<\/p>\n\n\n\n
STEP<\/strong> 1. Offline, Before Any Prompt Exists<\/p>\n\n\n\n
Documents are split into chunks, commonly a few hundred tokens, ideally on structural boundaries so that a table or a clause is not cut in half. Each chunk goes through an embedding model, which is separate from and much smaller than the LLM, and becomes one fixed-length vector. Vector plus original text plus metadata goes into the index.<\/p>\n\n\n\n
For scale, from vendor documentation: OpenAI’s text-embedding-3-small produces 1536-dimensional vectors by default and text-embedding-3-large produces 3072, with a dimensions parameter available to shorten them.<\/a><\/p>\n\n\n\n
My six-document test corpus includes the chunk that should answer our question:<\/p>\n\n\n\n
“Slack message retention is set to three years for all workspaces.<\/p>\n\n\n\n
Legal hold overrides retention and suspends automatic deletion<\/p>\n\n\n\n
until released.”<\/p>\n\n\n\n
Nothing happens to this chunk at query time. It is already sitting there, already vectorized.<\/p>\n\n\n\n
STEP<\/strong> 2. The Prompt Is Vectorized, And Not By The LLM<\/p>\n\n\n\n
The first model to touch the prompt is not the language model. The question becomes a single vector. In my TF-IDF vector space, dimensionality 45, the query has just two non-zero components:<\/p>\n\n\n\n
Query: What is our retention period for Slack messages?<\/p>\n\n\n\n
\n
retention      0.7733<\/p>\n\n\n\n
slack          0.6341<\/p>\n\n\n\n
All 43 other dimensions: 0.0<\/p>\n<\/blockquote>\n\n\n\n
One hard constraint in production: the query must use the same embedding model as ingestion. Query and stored vectors have to live in the same space or the geometry is meaningless. Critically, a mismatch fails silently. The result is plausible garbage, not an error.<\/p>\n\n\n\n
STEP<\/strong> 3. The Database Searches For Nearest Neighbours<\/p>\n\n\n\n
Similarity is usually cosine similarity, the angle between two vectors rather than the distance between their tips.<\/p>\n\n\n\n
Worth being precise here, because this detail is often overstated. OpenAI’s documentation recommends cosine similarity but adds that the choice of distance function typically does not matter much. Their embeddings are normalised to length 1, which means cosine similarity can be computed as a plain dot product, and cosine and Euclidean distance produce identical rankings. So cosine is the sensible default and the conventional choice, not a decision that will make or break retrieval quality.<\/p>\n\n\n\n
Real computed scores against my corpus:<\/p>\n\n\n\n
\"Article<\/figure>\n\n\n\n
The right document ranks first. Rank 6 is the detail I did not anticipate when I wrote the corpus. The document email-retention-policy scored exactly zero, despite being a document about retention. Why? It says “retained.” The query says “retention.” Different strings, zero lexical overlap, orthogonal vectors.<\/p>\n\n\n\n
That single 0.0000 is the best argument for learned embeddings I could have produced, and I got it by accident.<\/p>\n\n\n\n
Production databases do not compare against every vector. They use approximate nearest neighbour indexes.<\/p>\n\n\n\n
HNSW<\/a> is a multi-layer proximity graph traversed greedily. Fast, high recall, memory resident, expensive to build. The common default.<\/p>\n\n\n\n
IVF partitions<\/a> vectors into clusters and searches only the nearest few. Cheaper on memory, tunable.<\/p>\n\n\n\n
Product quantization compresses vectors into compact codes, trading accuracy for large memory savings. Often combined as IVF-PQ.<\/p>\n\n\n\n
The word “approximate” is load-bearing. These indexes have a recall parameter, and setting it means explicitly accepting that some true nearest neighbours will be missed in exchange for latency. That is a tuning decision, not a default to leave alone.<\/p>\n\n\n\n
STEP<\/strong> 4. What Lexical Matching Cannot Do<\/p>\n\n\n\n
I ran one more query against the same corpus, a paraphrase that shares almost no vocabulary with the document that answers it:<\/p>\n\n\n\n
Query: How long do we keep chat history before it is wiped?<\/p>\n\n\n\n
\"Article<\/figure>\n\n\n\n
Every score zero. Total retrieval failure. Not even a wrong-but-plausible ranking. The system has nothing to say.<\/p>\n\n\n\n
Any human reader knows instantly that document 1 answers that question. “How long” means “three years.” “Chat history” means “message.” “Wiped” means “deletion.” Not one of those pairs shares a token.<\/p>\n\n\n\n
This is precisely the gap learned embedding models fill. They place paraphrases near each other because they were trained on text where those phrasings appear in similar contexts. The retrieval mechanism, which is to vectorize, take cosines, and rank, is identical. Only the quality of the vectors changes.<\/p>\n\n\n\n
Scope note: my scores come from TF-IDF, which builds vectors from word frequency. The mechanism is the same as a production vector database; the vectors are far weaker. A real embedding model would score that paraphrase well above zero. I could not run one locally, so I have shown the failure instead, which turned out to teach more than a success would have.<\/p>\n\n\n\n
STEP<\/strong> 5. Reranking And Ordering<\/p>\n\n\n\n
Vector similarity is fast but blunt. A common second pass sends the query paired with each candidate through a cross-encoder, a small model that reads both together and scores actual relevance. Far more accurate than comparing independent vectors, and far too slow for the whole corpus, which is why it runs only on the shortlist.<\/p>\n\n\n\n
Ordering matters for a documented reason. Liu and colleagues, in “Lost in the Middle<\/a>“, found that model performance is often highest when relevant information sits at the beginning or the end of the input context, and degrades significantly when the model must retrieve it from the middle. That held even for models explicitly built for long contexts.<\/p>\n\n\n\n
Full citation: Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang (2024). Transactions of the Association for Computational Linguistics, vol. 12, pp. 157-173. DOI 10.1162\/tacl_a_00638. The preprint is arXiv:2307.03172 and dates from 2023, so the year to cite depends on which version is linked.<\/p>\n\n\n\n
Stated precisely: that paper tested multi-document question answering and key-value retrieval. It is strong evidence about position sensitivity, not a universal law about all prompts.<\/p>\n\n\n\n
STEP<\/strong> 6. The Retrieved Text Is Pasted Into The Prompt<\/p>\n\n\n\n
This is the step does the whole trick:<\/p>\n\n\n\n
System<\/strong>: Answer using only the context below. Cite sources.<\/p>\n\n\n\n
If the context does not contain the answer, say so.<\/p>\n\n\n\n
\n
<context><\/em><\/p>\n\n\n\n
[doc: retention-policy | section 8.2]<\/em><\/p>\n\n\n\n
Slack message retention is set to three years for all workspaces.<\/em><\/p>\n\n\n\n
Legal hold overrides retention and suspends automatic deletion<\/em><\/p>\n\n\n\n
until released.<\/em><\/p>\n\n\n\n
<\/context><\/em><\/p>\n\n\n\n
User<\/strong>: What is our retention period for Slack messages?<\/p>\n<\/blockquote>\n\n\n\n
Plain text in the prompt. Nothing loaded, nothing learned, nothing persisted.<\/p>\n\n\n\n
STEP<\/strong> 7. Now The Model Runs<\/p>\n\n\n\n
Tokenization, embedding, attention, sampling, exactly as in part one. Unchanged. From the model’s point of view there is no database anywhere. There is only a longer prompt that happens to contain the answer.<\/p>\n\n\n\n
Attention is the mechanism that makes it land. The question and the retrieved sentence now sit in the same context window, so the positions holding “three years” are available to be attended to. The model is not recalling a fact. It is reading one that was handed to it.<\/p>\n\n\n\n
Two Consequences Worth Internalising<\/strong><\/p>\n\n\n\n
The model cannot retrieve what the database missed. My paraphrase query returned six zeros. A model handed that empty result set will still answer, fluently and wrongly. Bad retrieval does not surface as an error. It surfaces as a confident answer. A large share of what teams call “RAG hallucination” is retrieval failure in disguise.<\/p>\n\n\n\n
Everything retrieved is untrusted input. Those chunks enter the prompt through the same channel as the system’s own instructions. If a document contains the words “ignore previous instructions,” a naive system may follow them. Retrieved content is data and must never be treated as instruction. In my view this is the most under-discussed risk in the stack.<\/p>\n\n\n\n
And an operational third. If retrieval ignores access control, RAG becomes an exfiltration path. The model will cheerfully summarise a document the user was never permitted to open. Filtering has to happen by the caller’s identity at query time, not after.<\/p>\n\n\n\n
The One-Line Version<\/strong><\/p>\n\n\n\n
The vector database converts the question into a vector, finds nearby text, and that text is pasted into the prompt. Retrieval is a text-fetching step that happens before inference. It is not a component of the model.<\/p>\n\n\n\n
Everything else, whether that is chunking strategy, reranking, hybrid search or evaluation, is engineering around that one sentence.<\/p>\n\n\n\n
What I Measured, And What I Did Not<\/strong><\/p>\n\n\n\n
Every retrieval score above came from that six-document corpus: the query vector and its dimensionality, both rankings, and every cosine value. I recomputed all of it from scratch a second time before publishing, because the first run is the one that flattered me. The two figures I did not compute are the embedding dimensions, which come from OpenAI’s documentation, and the position sensitivity finding, which comes from the Liu paper cited above.<\/p>\n\n\n\n
What it is not: TF-IDF is not a learned embedding model, and a real one would score that paraphrase well above zero rather than at zero. Named vector-database comparisons and benchmark numbers are absent on purpose, since they date within months. Index types, which I do discuss, are stable.<\/p>\n\n\n\n
Corrections are welcome. Every figure above traces to either my terminal or a other source.<\/p>\n","protected":false},"excerpt":{"rendered":"
One prompt, traced end to end: tokens, attention, and the part where a vector database gets involved. ONE PROMPT, ALL THE WAY DOWN I am going to use exactly one prompt for this entire article: I picked it on purpose. The word “our” is doing the work. No language model can answer this from its […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10,21],"tags":[18,19,20],"class_list":["post-278","post","type-post","status-publish","format-standard","hentry","category-ai","category-prompts","tag-prompts","tag-tokens","tag-vector"],"_links":{"self":[{"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/posts\/278","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/comments?post=278"}],"version-history":[{"count":1,"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/posts\/278\/revisions"}],"predecessor-version":[{"id":279,"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/posts\/278\/revisions\/279"}],"wp:attachment":[{"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/media?parent=278"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/categories?post=278"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/metaintelligencesystems.com\/blog\/wp-json\/wp\/v2\/tags?post=278"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}