One prompt, traced end to end: tokens, attention, and the part where a vector database gets involved.
ONE PROMPT, ALL THE WAY DOWN
I am going to use exactly one prompt for this entire article:
What is our retention period for Slack messages?I picked it on purpose. The word “our” is doing the work. No language model can answer this from its training, however large it is, because the answer lives in an internal policy document that did not exist when the model was trained. That one word separates “how a model generates text” from “how a model generates text grounded in private data,” and it lets me cover both without switching examples.
Every figure I quote from my own experiments was computed locally, on a corpus small enough to reproduce here in full. Where a number comes from a published source instead, I say so. And I will be explicit throughout about what my measurements do and do not demonstrate.
Stage 1. Words Becomes Numbers
The model cannot read characters. First comes tokenization: text is split into subword units, and each unit maps to an integer ID from a fixed vocabulary.
Most current models use Byte-Pair Encoding or a SentencePiece variant. BPE(Byte-Pair Encoding.) is built by starting from individual characters (or bytes, in byte-level variants like GPT-2’s) and repeatedly merging the most frequent adjacent pair, thousands of times, until the vocabulary hits a target size.
Rather than quote a production tokenizer’s IDs, I implemented BPE from scratch and trained it on a 150-word corpus of policy-style text, sixteen sentences of the kind an internal-policy assistant would see. Here are the first merges it learned, in order:
Merges 12 through 14 are the ones worth pausing on. The algorithm assembles ‘re’ plus ‘te’ into ‘rete’, then ‘reten’, then joins it to the ‘tion</w>’ it had already discovered back at step 7. It builds the whole word “retention” out of fragments it learned separately. Nobody specified that. It is pure frequency counting, applied repeatedly.
With 120 merges learned and a vocabulary of 148 symbols, here is our prompt tokenized:
IDs: [24, 10, 3, 102, 113, 16, 22, 95, 146, 138, 125, 100, 77, 114, 36, 2, 77]
17 tokens for 8 words, or 2.12 tokens per word
Now the interesting part. Why did some words survive as single tokens while others shattered? Cross-referencing each word against how often it appears in the training corpus gives the answer:
Counts are of the exact lowercased word. “Slack” appears three times in the corpus, but twice with punctuation attached, as ‘slack.’ and ‘slack?’, so the bare form occurs once. That is exactly why it never fused into a single symbol: my merge rule requires a pair to appear at least twice, and ‘slack’ plus end-of-word only ever appeared once.
The pattern is clean. Words the tokenizer saw often compressed to one ID. Words it barely saw or never saw decomposed into fragments. This is the entire design goal of subword tokenization: the tokenizer never fails on unfamiliar input, it just spends more tokens on it.
That has three practical consequences:
Whitespace and punctuation are part of the token. The symbols ‘r</w>’ and ‘or’ differ precisely because of the end-of-word marker. In production tokenizers a leading space works the same way, which is why “Slack” and ” Slack” are different IDs, and why a stray trailing space in a prompt can occasionally shift output.
This is why models are bad at spelling tasks. A model asked to count the letters in a word struggles, because it never sees letters. It sees ‘messag’ and ‘es’. The limitation lives in the tokenizer, not in the intelligence.
There is a cost asymmetry across languages. BPE merges are learned from a training corpus that is overwhelmingly English. My toy tokenizer makes this visible in miniature: “our” cost 3 tokens because the corpus never contained it, so no merge could ever assemble it. Scaled up, the same sentence in an under-represented language consumes measurably more tokens than its English equivalent. Since nearly every API prices per token, that is a real cost difference, not a curiosity.
Note: This is my own small BPE, not GPT-4’s tokenizer. The algorithm is the same one production tokenizers use. The scale is not. Real vocabularies run to 100k or more symbols trained on enormous corpora, so their specific IDs and splits differ completely from mine. Anyone who wants production numbers should encode their own text with the tokenizer their provider publishes, and state which encoding was used. I built something small enough to verify end to end instead.
Stage 2. Numbers Become Meaning
Each token ID indexes a row in an embedding matrix, and that row is a vector of floats. For a mid-size model the matrix might be vocabulary size by 4096.
The important part: nobody assigned these vectors. Nobody wrote down that “Slack” should sit near “Teams.” That geometry emerged from gradient descent over an enormous corpus. It is learned structure, not a hand-built ontology.
That property is worth holding onto. It is the same one the vector database will exploit later, and the reason the two halves of this article are really one story.
Attention has no inherent sense of order. Without position information, “retention policy” and “policy retention” would be identical inputs. Position gets injected either by adding positional embeddings or, in most current open-weight models, by rotating the query and key vectors by a position-dependent angle, which is rotary position embedding, or RoPE. Closed models do not disclose their architecture, so “most” is the honest word.
Stage 3: Attention, Or Where – Enough Information – comes
How does the model know what a prompt means?
Each token’s vector is projected into three vectors: a query, a key, and a value. Token i’s query is compared against every token’s key by dot product, scaled, and pushed through a softmax to produce weights that sum to 1. Those weights take a weighted average of the value vectors.
In plain language: every token asks “who here is relevant to me”, receives a set of relevance scores, and pulls in a blend of the other tokens’ information.
In our prompt, the content-bearing tokens, “retention” and “slack”, would be expected to draw more weight at the final position than function words like “is” or “for”. I did not measure that here, since inspecting real attention maps is a separate exercise. The structural point is what matters: nobody programs which tokens matter. It falls out of the learned projections.
Three details that matter in practice.
Multi-head attention: This runs in parallel across many heads with separate projections, and different heads specialise. Interpretability researchers have identified heads that appear to perform specific jobs, though that is an active research area, not settled fact.
Causal masking: Positions after i are set to negative infinity before the softmax, so no token attends to the future. This is what makes left-to-right generation coherent. A model trained without it would learn to peek at the answer.
The KV cache: Keys and values for earlier tokens do not change as generation proceeds, so they are cached. This is why the first token of a response is slow, since the whole prompt must be processed, while later tokens are fast, one at a time. At long context lengths and large batch sizes the cache becomes a major memory consumer at serving time, and it is a big part of why long conversations get expensive.
Each layer pairs attention with a feed-forward network, which holds most of the model’s parameters. Current interpretability work suggests attention moves information between positions while feed-forward layers store and retrieve learned associations. I want to be careful here: that is a useful intuition from ongoing research, not a proven architectural fact.
Stage 4: A Probability for Every Possible Next Token
The final hidden state is multiplied by the unembedding matrix, producing one raw score, called a logit, per vocabulary entry. Softmax turns those into probabilities. A sampling step picks one.
temperature – Divides the logits before softmax. Below 1 sharpens toward the top candidate, above 1 flattens the distribution. Zero is greedy: always the highest-probability token, fully deterministic.
top_k – Keeps only the k highest-probability tokens, then renormalises.
top_p – Keeps the smallest set of tokens whose cumulative probability exceeds p. Adapts to model confidence.
frequency and presence penalties Subtract from the logits of tokens already used, which reduces repetition.
Here is the uncomfortable implication. Softmax always returns a valid probability distribution. If the model has no strong learned pattern for a given question, that distribution is nearly flat, but sampling still returns a token, and the resulting sentence reads exactly as fluent as a well-grounded one. The model has no separate “I do not know” output channel.
Hallucination is not a bug someone forgot to fix. It is a structural consequence of generating text by sampling from a distribution that is always defined.
Which brings us back to our prompt. Asked what “our” retention period is, a bare model produces something fluent, specific, and wrong, because no mechanism exists by which it could know.
What I Measured, And What I Did Not
Every tokenization figure above came from that 150-word corpus: the merges and their frequencies, the token IDs, the 17 tokens, the word-frequency table. I recomputed all of it from scratch a second time before publishing, because the first run is the one that flatters the author.
What it is not: my BPE is not a production tokenizer, and I make no claim that these specific IDs resemble any real model’s. Parameter counts, context window sizes and benchmark numbers are absent on purpose, since they date within months and I would have been quoting them from memory.
Where This Leaves Us
Nothing in stages 1 to 4 can fix that wrong answer. The mechanism that does fix it is not part of the model at all. It sits in front of the model, runs before inference, and is routinely described as something it is not.
How a vector database turns a question into a vector, what cosine similarity actually measures, the retrieval failure I did not expect to find in my own test corpus, and the one step most explanations skip.
Corrections are welcome. Every figure above traces to either my terminal or a other sources
Leave a Reply