How a model reads a sentence

A processor holds no notion of a word. Give it rabbit and it receives six byte values in a row; nothing in that sequence records that a rabbit is an animal, or is small, or is anything at all. Whatever a model appears to know about language, it holds entirely as numbers. So the first job in any LLM system is turning text into numbers, and the last is turning numbers back into text.

This page follows one short sentence through that sequence. It is worth reading once end to end, because the steps are usually encountered separately, out of order, and as unrelated problems: a billing surprise, a retrieval that returns the wrong document, an answer that changes between two identical requests. They are the same pipeline seen from different sides.

Tokens, not words

The first thing to unlearn is that models read words. They read tokens, which are pieces of text drawn from a fixed vocabulary. A token may be a whole word, part of one, a single character, or a punctuation mark, and spaces affect how the split falls (OpenAI).

Storing whole words is not a realistic option. English alone would need separate entries for run, runs, ran, running, runner and runners, before any other language, identifier, URL or emoji is considered. A vocabulary of reusable fragments covers far more text for the same size, because fragments recombine.

The method behind that split is byte pair encoding, introduced for machine translation by Sennrich, Haddow and Birch. Training starts from individual characters, repeatedly finds the adjacent pair that occurs most often, and merges it into a new symbol. What survives as a single token is whatever the corpus repeats often enough to be worth one; anything unusual is left assembled from smaller parts. Nobody writes the vocabulary by hand, and nothing in it encodes a decision about grammar. It is the output of a frequency count.

Three consequences follow, and each surprises people the first time:

• A leading space is part of the token. red, Red and red are three different pieces of text and an encoding may represent all three differently.
• Unfamiliar words are assembled from familiar parts rather than rejected, which is why a model can process a word it has never seen.
• Token IDs are specific to an encoding. An ID from one model’s vocabulary means nothing in another’s, so an example ID should never be treated as a property of the word.

The rules of thumb, and their limits

For English prose, OpenAI gives approximately 4 characters per token, roughly three quarters of a word, and 100 tokens to about 75 words, while stating plainly that these are estimates rather than counts.

A long repeated character run, an identifier like getUserAccountById, or the same passage in a language the vocabulary was not built around will all consume more tokens than the ratio predicts. Where the number matters, count it rather than estimating it.

Why the count is worth caring about

Token count is not trivia. It is the unit of four separate things:

• Billing. Charges are per token, split between input and output, and reasoning models also charge for internal reasoning tokens that never appear in the answer.
• The context window. A stated limit is a token limit, so a 128,000-token window holds roughly 96,000 English words, and less of anything else.
• The retrieval budget. Whatever context is retrieved occupies room the answer would otherwise have had, which is the pressure behind chunking, compression and reranking in a RAG pipeline.
• Latency. A shorter input is less to process before generation starts.

An ID locates a token; it does not describe one

Every token has an integer attached: the row it occupies in the vocabulary. It is tempting to read something into those integers, and there is nothing there to read.

Those figures are made up, and deliberately so: as noted above, an ID belongs to one encoding and means nothing outside it, so quoting real ones from a particular model would invite exactly the mistake this section is warning about. What is realistic is the scatter. Two animals as closely related as rabbit and wolf sit more than twenty thousand rows apart, because the vocabulary was ordered by how often fragments occur during training and not by what they refer to. Whatever occupies row 11,205, next door to wolf, is far more likely to be a punctuation mark or the tail of an unrelated word than anything lupine.

Note the last row too. guinea pig is a single animal but not a single token, so the vocabulary has no entry for the thing itself, only for two fragments that happen to appear together.

The consequence is that no arithmetic on these integers means anything. wolf is not nearer to bear than to rabbit by virtue of 11,204 being closer to 8,957 than to 34,682. This is precisely why the next step has to exist: one number per token cannot express relatedness, and neither can two. What is needed is a description with enough separate components that similar things end up with similar descriptions.

Many numbers instead of one

Suppose each animal were described not by one number but by several, each standing for a measurable property: how many legs, whether it has fur, whether it lives wild, whether people keep it, and roughly how heavy it is.

The size column is an invented 1 to 10 scale, not a measurement; the point is the shape of the rows rather than their accuracy. Rabbit and guinea pig agree on four of five components. Wolf and bear agree on three and are far from the other two on the columns that separate them. Nobody declared which pairs belong together, and it falls out of the descriptions anyway. Rows like these can be compared arithmetically, and that is the whole of the idea: a row of numbers placing something in a space where distance corresponds to relatedness. OpenAI’s documentation defines an embedding the same way, as a vector whose distance from another measures how related the two are, with small distances meaning high relatedness.

The columns are not chosen by hand. This is the part most often misread. Nothing in the training process assigns a meaning to the third position, or to any other. The values begin as noise and are adjusted during training, by the ordinary mechanism of being wrong and being corrected, until words that occur in similar contexts end up with similar vectors. The organising principle is distributional: things used alike are positioned alike. Nothing guarantees a given dimension corresponds to anything a human would name, and generally none of them does.

How many dimensions

Published models differ, and the number is a deliberate trade rather than a constant. OpenAI’s text-embedding-3-small returns 1,536 values by default and text-embedding-3-large returns 3,072, with a dimensions parameter to request fewer.

More dimensions can separate more distinct concepts. They also cost more to store, more to compare, and more to move, and the cost applies to every vector in the corpus rather than once. The right number is whatever holds the distinctions your task depends on, which is a question to answer by measurement on your own data rather than by preference. Choosing one commits you: changing the embedding model later means re-embedding everything, since vectors from two models are not comparable.

Attention, and why context changes the numbers

Everything so far is a lookup. The vector for apple from the embedding matrix is the same whether the sentence is about fruit or about a phone, which is not enough to be useful.

Attention is the step that fixes this. Each token is recomputed as a weighted combination of the others, with the weights learned rather than fixed, so a token’s representation depends on the company it keeps. After attention, apple in “I ate an apple” and apple in “Apple released a phone” are different vectors, though they began as the same row of the same table.

This is the mechanism introduced by Vaswani et al. in 2017, and it is worth being precise about what that paper did and did not claim: it was a machine translation result, and the page on it sets out what has been attributed to it since that it does not actually say.

The practical consequence is that meaning is computed per request, in context, rather than retrieved. It is also why prompt wording has effects that seem out of proportion to the change: altering surrounding text alters the vectors of tokens you did not touch.

Choosing the next token

One vector emerges from the final layer. Comparing it against the entire vocabulary produces a raw score for each possible continuation, and those scores are normalised into a probability distribution.

At this point the model has not decided anything. It has produced a ranked set of possibilities, and the choice among them is a separate, configurable step. Sampling parameters, temperature among them, control how much probability mass reaches the lower-ranked options. At the conservative end the most likely token wins almost every time. Further along, the distribution flattens until candidates well down the ranking are drawn often enough to notice.

Two things follow that matter more than they first appear:

• Identical prompts can produce different answers, by design, and that is a configuration choice rather than a defect. It is also why an evaluation run on a single sample measures very little, as the evaluation page sets out.
• Fluency is not confidence. The distribution reflects what is likely to follow given the training data, not whether the statement is true. A wrong answer and a right one are produced by the same mechanism, and the model has no separate signal marking one as unsupported.

Then the chosen token is appended to the input and the whole sequence runs again, from tokenisation to sampling, once per token produced. Generation is that loop.

What each step decides

Reading the table upwards is a reasonable diagnostic habit. An answer that varies between runs is a sampling question. An answer that is fluent and wrong is a scoring question, not a sampling one. Retrieval that returns plausible but unhelpful passages is an embedding question. A request that costs more than expected is a tokenisation question. Confusing one for another is the most common way to spend effort on the wrong layer.

What this page is not

It is not a description of any specific model’s internals. Layer counts, attention variants, positional encoding, normalisation and the differences between architectures are all omitted, because none of them changes the sequence above and all of them obscure it on a first pass.

It is also not sufficient for tuning anything. Knowing that sampling controls repeatability does not tell you what to set it to for your task; that is an evaluation question, and the answer depends on data this page knows nothing about.