Artificial intelligence · 2017

Transformer

'Attention Is All You Need': drop recurrence, use attention, and train in parallel.

Eight Google researchers replaced RNNs with stacked self-attention and feedforward layers. The transformer set translation records while training far faster on GPUs. It became the architecture of nearly every major AI system that followed, from BERT and GPT to AlphaFold 2.

See it in the living treeWikipedia