Artificial intelligence · 2018
BERT
A bidirectional transformer pretrained by filling in blanks sets 11 NLP records.
Jacob Devlin and colleagues at Google trained BERT with masked-word prediction on 3.3 billion words. It raised the GLUE benchmark by 7.7 points, and in 2019 Google Search began using it on English queries. Pretrain-then-fine-tune became standard.
What it built on 3
What it led to 2
Related measurements
The dashed line marks when it first worked.
Training compute of notable AI models Source: Epoch AI, Notable AI Models (CC BY 4.0)
ML hardware FLOP/s per dollar Source: Epoch AI, ML hardware (CC BY 4.0)