Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin. Submitted to arXiv as 1706.03762 on 12 June 2017 and last revised 2 August 2023 (v7); published as “Attention is All you Need” in Advances in Neural Information Processing Systems 30 (NIPS 2017). Author affiliations are Google Brain and Google Research; the paper states equal contribution with a random listing order.
The problem it addressed
Sequence transduction (mapping one sequence to another, of which translation is the canonical case) was done with recurrent encoder-decoder models, usually with an attention mechanism connecting the two. Recurrence has a structural cost the paper is precise about: a recurrent layer requires O(n) sequentially executed operations for a sequence of length n. Each step depends on the previous one, so the computation cannot be parallelised along the sequence. Training time is bounded by sequence length regardless of how much hardware you have.
Convolutional alternatives of the period (the paper names the Extended Neural GPU, ByteNet and ConvS2S) removed the sequential dependency but replaced it with a distance problem: the number of operations needed to relate two positions grows with the distance between them, linearly for ConvS2S and logarithmically for ByteNet. That makes long-range dependencies harder to learn.
So the problem is narrower than “understanding language”. It is: can you get the parallelism of a convolutional model and a constant path length between any two positions at the same time?
What it actually proposed
The Transformer, described by the paper as the first transduction model relying entirely on self-attention to compute representations of its input and output, without sequence-aligned recurrence or convolution. Note the phrasing the authors chose: first transduction model relying entirely on self-attention. Not the first use of attention, and not the first use of self-attention, both of which the paper attributes to prior work.
The architecture is an encoder-decoder stack, and the specifics matter because they are what is usually paraphrased away:
| Component | As specified in the paper |
|---|---|
| Encoder | Stack of N = 6 identical layers, each with multi-head self-attention and a position-wise feed-forward sub-layer |
| Decoder | Stack of N = 6 identical layers, adding a third sub-layer performing multi-head attention over the encoder output |
| Sub-layer wiring | Residual connection then layer normalisation: LayerNorm(x + Sublayer(x)) |
| Model width | d_model = 512 for all sub-layers and embeddings, so residual connections compose |
| Feed-forward inner width | d_ff = 2048 |
| Heads | h = 8 parallel attention layers, with d_k = d_v = d_model / h = 64 |
| Attention | Scaled dot-product: dot products scaled by 1/√d_k before the softmax |
| Position information | Sinusoidal positional encodings added to the input embeddings, wavelengths in geometric progression from 2π to 10000·2π |
| Regularisation | Dropout at 0.1 on sub-layer outputs and on embedding-plus-position sums; label smoothing at 0.1 |
| Optimiser | Adam with β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹, and an inverse-square-root schedule with warmup_steps = 4000 |
Three design details deserve more attention than they usually get.
Multi-head attention exists to repair a defect the authors identified in their own mechanism. They state that reducing the path between positions to a constant number of operations comes “at the cost of reduced effective resolution due to averaging attention-weighted positions”. A single attention head is a weighted average and therefore blurs. Multi-head attention is described explicitly as the counter-measure. It is not primarily a capacity or parallelism trick.
The scaling by 1/√d_k is a specific fix, not decoration. Without it, dot products grow with dimension and push the softmax into regions of small gradient.
Sinusoidal position encoding was a choice, and the paper reports the alternative was equivalent. They also tried learned positional embeddings and found “nearly identical results”, choosing the sinusoidal version on a hypothesis that it might extrapolate to sequence lengths longer than those seen in training. The paper presents this as a hypothesis, not a demonstrated property.
What it demonstrated, and what it only suggested
Demonstrated, with numbers, on three tasks:
| Result | Value reported in arXiv v7 |
|---|---|
| WMT 2014 English-to-German, big model, newstest2014 BLEU | 28.4, described as improving over existing results including ensembles by over 2 BLEU (Vaswani et al., 2017) |
| WMT 2014 English-to-French, big model, single-model BLEU | 41.8 (Vaswani et al., 2017) |
| Training time, big model | 300,000 steps, 3.5 days, on one machine with eight NVIDIA P100 GPUs (Vaswani et al., 2017) |
| Training time, base model | 100,000 steps, about 12 hours, same hardware (Vaswani et al., 2017) |
| WSJ section 23 constituency parsing F1, 4-layer model | 91.3 trained on WSJ only, 92.7 semi-supervised (Vaswani et al., 2017) |
Two facts about that table are worth stating because they get lost.
The numbers are version-dependent. The NIPS 2017 proceedings abstract reports a single model of 165 million parameters achieving 27.5 BLEU on English-to-German and 41.1 on English-to-French, improving on the previous single-model result by 0.7 BLEU. The arXiv v7 abstract reports 28.4 and 41.8. If you are citing a figure from this paper, cite which version you took it from.
The datasets were small by later standards, and the training was from scratch. English-German was about 4.5 million sentence pairs with a shared byte-pair vocabulary of about 37,000 tokens; English-French was 36M sentences with a 32,000 word-piece vocabulary (Vaswani et al., 2017). There is no pre-training stage and no transfer. Each model was trained on its task.
Suggested rather than demonstrated, and the paper is honest about which is which:
- Generality across modalities. The conclusion states a plan to extend the Transformer to input and output modalities other than text, and to investigate local, restricted attention for large inputs such as images, audio and video. That is stated as future work.
- Extrapolation to longer sequences. Offered as the hypothesis behind the sinusoidal choice, not tested.
- Efficiency on long sequences. The paper notes self-attention could be restricted to a neighbourhood of size r to improve computational performance on very long sequences, and puts that restricted variant in its complexity table. It did not implement it.
- Making generation less sequential. Named as a research goal. Decoding remained autoregressive.
The complexity argument is also narrower than it is usually quoted. Self-attention is faster than recurrence per layer when the sequence length n is smaller than the representation dimensionality d. The paper says this is most often the case for the sentence representations used in translation. That is a conditional claim about a regime, and the regime it describes is sentences, not documents.
What has held up
The architecture, largely unchanged in its load-bearing parts. Scaled dot-product attention, multi-head attention, residual connections around each sub-layer with layer normalisation, a position-wise feed-forward sub-layer roughly four times the model width, learning-rate warmup followed by decay, and dropout plus label smoothing are all still standard. The complexity argument also held: constant path length between positions is what made depth and width scale as they later did.
The prediction in the conclusion about other modalities held up too, and faster than the phrasing suggests. Applying self-attention to images and audio is now routine.
Several specifics did not survive contact with scale, which is a normal outcome for a paper reporting one configuration:
- Post-normalisation wiring. The paper places layer normalisation after the residual addition. Deep models are now generally trained with normalisation before the sub-layer instead, because the published ordering is harder to train at depth without warmup.
- Sinusoidal absolute positions. The paper itself found learned embeddings equivalent; later work moved largely to relative and rotary schemes.
- Fixed encoder-decoder shape. Most current language models keep only the decoder stack. That configuration is not in this paper.
- The specific hyperparameters. N = 6, d_model = 512, h = 8 describe a 2017 translation model on eight P100 GPUs. Nothing in the paper claims these transfer.
- Quadratic attention cost. The paper flagged this and suggested restricted attention as the remedy; the intervening years produced a large literature on the problem, which is evidence the concern was real rather than that the paper solved it.
What is commonly misattributed to it
This is the part worth reading carefully, because most of these claims are made confidently.
“It invented attention.” It did not, and the paper does not say so. Attention in neural machine translation is due to Bahdanau, Cho and Bengio (2014), cited in the paper. Self-attention, which the paper notes is “sometimes called intra-attention”, is also credited to prior work on reading comprehension, abstractive summarisation, textual entailment and sentence representations. The contribution is building a transduction model entirely out of it.
“It introduced large language models.” There is no language model in the paper. There is no pre-training, no fine-tuning, no unsupervised objective and no text generation task beyond translation and parsing. Pre-training on unlabelled text and transferring to downstream tasks arrived in subsequent, separate work. The Transformer is the substrate those papers used; the idea is not in this one.
“It proposed the decoder-only architecture.” The opposite. Section 3 specifies six encoder layers and six decoder layers with cross-attention between them, and the paper describes competitive sequence transduction models as having an encoder-decoder structure. Decoder-only stacks are a later simplification.
“It showed that attention is all you need.” The title is a slogan, and the model contradicts it if read literally. Each layer contains a position-wise feed-forward network - two linear transformations with a ReLU between them, at d_ff = 2048 against d_model = 512, so a substantial fraction of the parameters is not attention at all. Residual connections, layer normalisation, embeddings, a softmax output layer and positional encodings are all load-bearing. What the paper demonstrated is that you can remove recurrence and convolution, which is a different and more precise statement.
“It proved Transformers are more efficient than RNNs.” It reported that its models trained faster to a given quality on two translation datasets, which is a parallelism result on a specific regime. It made no claim about inference efficiency, and attention’s cost in sequence length is worse than recurrence’s, as the paper’s own complexity table shows. Conflating “more parallelisable during training” with “cheaper” is the error.
“It predicted scaling laws” or “showed that bigger is better”. Section 6.2 varies model dimensions on a translation task and reports which variations help. There is no scaling law, no compute-optimal analysis and no claim about emergent capability. Those are later results by other authors.
“It introduced subword tokenisation.” It used byte-pair encoding and word-piece vocabularies, both cited to earlier work. Tokenisation is an inherited experimental detail here, not a contribution.
“It was immediately recognised as the foundation of modern AI.” The paper presents itself as a machine translation result with a generality hypothesis attached. Reading the conclusion (extend to other modalities, investigate restricted attention, make generation less sequential) gives you the authors’ own sense of what was unfinished, which is a more useful picture than the retrospective one.
Related
- “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” - what it actually said
- Retrieval or fine-tuning - a decision the architecture makes possible
- Retrieval-augmented generation