AI
Attention
A mechanism that lets each position in a sequence attend to every other position, weighting their relevance dynamically rather than relying on fixed-distance connections.
Self-attention computes query, key and value projections for each token, then uses the query-key dot products to weight the values. Multi-head attention runs several of these in parallel, each learning different relationship patterns. The quadratic cost in sequence length is why context windows have practical limits.