AI
Token
The atomic unit a language model reads and generates - a word, subword, or byte depending on the tokeniser.
Tokens set the granularity of everything: context window size, pricing, latency and what the model can represent. A word in English is roughly 1.3 tokens, but the ratio varies by language, by domain (code is more fragmented) and by tokeniser. The tokeniser is a fixed preprocessing step, not learned, so its biases are baked in.