Generative AI

A generative model produces new content (a sentence, an image, a line of code) instead of a label or a score attached to something that already exists. That distinction is the whole category. Older, “predictive” or “discriminative” machine learning takes an input and returns a judgement about it: is this transaction fraudulent, does this scan show a tumour, what is this email’s spam score. A generative model instead learns the statistical shape of its training data closely enough to produce new examples of it. A fraud model recognises a pattern; a generative model could, in principle, write a plausible-looking fraudulent transaction itself. What a model of this kind can do well, and specifically where it fails, both follow from that single design choice: it was built to generate, not to judge.

Two model families do almost all of it

Generative systems in production today are built on one of two mechanisms, and most of what looks like a bewildering number of products collapses into these two once you know which is underneath.

Autoregressive transformers generate one token at a time: given everything produced so far, the model estimates a probability for every possible next token and samples one, then repeats. Every current large language model works this way (GPT, Claude, Gemini, Llama, and the rest), which is also why a chat model can be interrupted mid-sentence and still finish coherently: there is no plan for the whole answer, only a running estimate of what should come next. The architecture that made this practical at scale is the Transformer, introduced in 2017 by Vaswani et al. and described in depth on this site’s own explainer. In short, it lets a model weigh every earlier token against every other one directly, instead of passing information through a chain of previous steps the way older recurrent models did.

Diffusion models work in the opposite direction. Training starts from a real image (or, more recently, audio or video) and adds noise to it in stages until it is indistinguishable from static, then trains a model to reverse each stage: remove a little noise, look more like the target, repeat. Generation runs that reverse process starting from pure noise, so an image forms gradually across many denoising steps instead of being produced token by token. This is the mechanism behind most image generators, including Stable Diffusion and OpenAI’s DALL-E family. It took over from the two earlier image-generation approaches, variational autoencoders (2013) and generative adversarial networks (2014), because it gives finer control over the output at the cost of a slower generation process.

A growing number of systems are multimodal: one model handling text, images and audio together, usually by converting everything into the same kind of token stream so a single autoregressive model can generate across media. Where that convergence ends up, and whether diffusion remains the better mechanism specifically for high-fidelity images and video, is not settled; see the note under “what to watch” below.

How a model gets from raw data to a usable assistant

Training a model capable of the behaviour above happens in stages, and conflating them is a common source of confusion about what a given number (parameter count, training cost, benchmark score) is actually describing.

1. Pretraining builds the foundation model: a huge, general-purpose network trained on the “fill in the blank” task described above, over as much data as the lab can justify the cost of. This step is what is compute- and cost-intensive enough to require dedicated clusters and produces a model that has absorbed the statistical patterns of its training data but has no built-in sense of which of its outputs a person actually wants.
2. Tuning narrows that general capability toward being useful and safe to talk to. Supervised fine-tuning trains the model on examples of the desired input-output format directly; reinforcement learning from human feedback (RLHF), or the newer variants built on AI-generated preference judgements, adjusts the model based on which of two candidate answers a rater or another model preferred. This is the stage that turns a raw next-token predictor into something that reliably answers a question instead of just continuing the text pattern it was given.
3. Generation is what happens at inference time, and it is genuinely probabilistic: a setting usually called “temperature” controls how sharply the model favours its single most likely next token versus sampling more broadly across plausible ones. This is the direct, mechanical reason the same prompt sent twice can produce two different answers - there is no cached “correct” response being looked up, only a fresh probability distribution being sampled each time.

What this mechanism does not give you

Because generation is sampling from a learned probability distribution rather than retrieving a stored fact, a generative model has no built-in way to distinguish a confident correct recollection from a confident fabrication - both come out of the same process and can look identical in tone. OWASP names this directly as a top risk category for LLM applications, under the heading of misinformation arising from generation itself rather than from any attack. Three consequences follow, and each is why a separate page on this site exists to address it rather than being folded into “just prompt it better”:

• The model has no privileged access to information outside its training data unless something explicitly supplies it at generation time - see retrieval-augmented generation for the standard way to do that, and context engineering for what else competes for space in that same input alongside retrieved facts.
• Getting a model to behave a specific way is an instruction problem, not a training problem, for most practical use - see prompt engineering for what is actually established about that versus what is folklore.
• Because output quality cannot be judged by inspection alone at any scale, “it looks better” is not evidence - see evaluating LLM systems for what a real measurement looks like.

None of this is a defect to be patched away; it is the direct consequence of what generation mechanically is. Treating a generative model as a search engine, a database, or a deterministic function is the single most common source of surprise for anyone new to working with one.