Artificial intelligence · 2015
Knowledge distillation
A small student network learns from a large teacher's soft predictions.
Geoffrey Hinton, Oriol Vinyals and Jeff Dean showed that training on a big model's output probabilities transfers much of its skill to a compact model. Distillation now produces most fast, cheap production models, including the small DeepSeek-R1 variants.
What it built on 1
What it led to 1
Related measurements
The dashed line marks when it first worked.
Training compute of notable AI models Source: Epoch AI, Notable AI Models (CC BY 4.0)
ML hardware FLOP/s per dollar Source: Epoch AI, ML hardware (CC BY 4.0)