Artificial intelligence · 2022
Chinchilla scaling
DeepMind shows models were undertrained: use about 20 tokens per parameter.
Jordan Hoffmann and colleagues trained over 400 models and found compute-optimal training scales data and parameters equally. The 70B Chinchilla beat the 280B Gopher. The result reshaped every later training run, including Llama.
What it built on 2
What it led to 1
Related measurements
The dashed line marks when it first worked.
Training compute of notable AI models Source: Epoch AI, Notable AI Models (CC BY 4.0)
ML hardware FLOP/s per dollar Source: Epoch AI, ML hardware (CC BY 4.0)