Transformer (neural network architecture)

A neural network architecture built on self-attention, introduced in 2017, that processes sequences in parallel and underlies most large language models.

Connect

Self-attention: A mechanism where each token computes weighted averages of all other tokens' representations, based on learned compatibility scores. The core operation every other transformer component serves; the whole architecture exists to run it efficiently.
Attention Is All You Need: The 2017 Vaswani et al. paper at Google that introduced the transformer for machine translation. The founding publication, whose title claims attention alone suffices without recurrence.
Ashish Vaswani: Lead author of Attention Is All You Need, the 2017 paper introducing the transformer. First author on the founding paper and a driving force behind the original design.
BERT: A 2018 bidirectional transformer encoder pretrained on masked tokens, defining a generation of language models. The encoder half turned into the standard pretrained model for language understanding tasks.
State space model: A sequence model using linear recurrences, such as Mamba, offering constant-memory inference over long contexts. The leading post-transformer rival, trading attention's exact recall for linear-time processing.
Multi-head attention: Running several attention computations in parallel with separate learned projections, letting different heads track different relationships. How transformers get multiple distinct attention patterns from a single layer.
Recurrent neural network: A network that processes sequences token by token through a hidden state carried forward in time. The dominant architecture the transformer replaced, constrained by sequential processing.
Ian Goodfellow: Researcher known for generative adversarial networks and the widely used deep learning textbook. Wait — not a transformer figure; exclude.
GPT: A family of decoder-only transformers trained to predict the next token, scaling to large language models. The decoder half scaled up became the basis of most modern generative language models.
Linear attention: Approximations of attention that replace the quadratic token-by-token matrix with cheaper kernel forms. Attempts to keep the transformer's form while removing its quadratic cost in sequence length.