Transformer (neural network architecture)

A neural network architecture that processes sequences entirely through self-attention, replacing recurrence and convolution, introduced in the 2017 paper Attention Is All You Need.

Self-attention: A mechanism where each token weighs every other token in the sequence to build its own representation. The core operation a Transformer runs; every block is built around it.
Attention Is All You Need: The 2017 Vaswani et al. paper at Google that introduced the Transformer for machine translation. The founding publication defining the architecture and its eight-person author team.
Ashish Vaswani: First author of Attention Is All You Need and a Google Brain researcher on sequence modeling. Led the team that wrote the founding Transformer paper.
BERT: A 2018 bidirectional encoder-only Transformer pretrained with masked-language modeling on text. The first Transformer variant to reset the state of the art across language tasks.
Recurrent neural network: Networks that process sequences one step at a time while carrying a hidden state forward. The sequential processing model Transformers abandoned for parallel attention.
Multi-head attention: Running several attention operations in parallel with different learned projections, then combining their outputs. Lets each layer attend to different relationships, like syntax and coreference, simultaneously.
Seq2seq: Encoder-decoder recurrent networks mapping sequences to sequences, dominant in translation before 2017. The architecture the Transformer directly replaced, bottlenecked by its fixed-length vector.
Jakob Uszkoreit: Google researcher who argued attention alone could outperform recurrent networks. Pushed the hypothesis that self-attention should replace recurrence entirely.
GPT: A family of decoder-only Transformer language models trained to predict the next token. Scaled the architecture to hundreds of billions of parameters and conversational use.
Convolutional neural network: Networks applying learned filters across local windows of an input grid. Built-in locality and weight sharing versus the Transformer's global, position-agnostic attention.

Linked from 10 pages