Artificial intelligence · 2020
Neural scaling laws
Loss falls as a smooth power law in parameters, data and compute.
Jared Kaplan and colleagues at OpenAI measured language model loss across seven orders of magnitude of compute and found predictable power laws. The finding turned AI progress into an engineering forecast and justified billion-dollar training runs.
What it built on 1
What it led to 6
Related measurements
The dashed line marks when it first worked.
Training compute of notable AI models Source: Epoch AI, Notable AI Models (CC BY 4.0)
ML hardware FLOP/s per dollar Source: Epoch AI, ML hardware (CC BY 4.0)