Artificial intelligence · 2018
GPT-1
OpenAI pretrains a transformer on books, then fine-tunes it for many tasks.
Alec Radford's Generative Pre-trained Transformer had 117 million parameters trained on about 7,000 unpublished books. Generative pretraining then fine-tuning improved the state of the art on 9 of 12 tasks, establishing the recipe that GPT-2, GPT-3 and ChatGPT scaled.
What it built on 2
What it led to 1
Related measurements
The dashed line marks when it first worked.
Training compute of notable AI models Source: Epoch AI, Notable AI Models (CC BY 4.0)
ML hardware FLOP/s per dollar Source: Epoch AI, ML hardware (CC BY 4.0)