Artificial intelligence · 2014

Attention mechanism

Let the decoder look back at every input word, weighted by relevance.

Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio added learned soft alignment to encoder-decoder translation, removing the fixed-length bottleneck. Attention became the core operation of the transformer three years later.

See it in the living tree Wikipedia

Arcs above join it to what it stood on, arcs below to what it led to. The bar places this stretch on the tree's whole timeline.

What it built on 1

Sequence to sequence 2014

What it led to 3

Google Neural Machine Translation 2016
Transformer 2017
DeepSeek sparse attention 2025

Related measurements

The dashed line marks when it first worked.

Training compute of notable AI models Source: Epoch AI, Notable AI Models (CC BY 4.0)

ML hardware FLOP/s per dollar Source: Epoch AI, ML hardware (CC BY 4.0)