Artificial intelligence · 1991
Mixture of experts
A gating network routes each input to specialist subnetworks.
Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton proposed adaptive mixtures of local experts. The idea returned at scale in 2017 as sparse MoE layers, letting today's largest language models activate only a fraction of their parameters per token.
What it built on 1
What it led to 6
Related measurements
The dashed line marks when it first worked.
Training compute of notable AI models Source: Epoch AI, Notable AI Models (CC BY 4.0)
Computation per constant dollar, 1850-2006 Latest: 7.3B million units of computer power per 2006 $ (2003) Source: Nordhaus 2007, Two Centuries of Productivity Growth in Computing, J. Econ. Hist. 67(1), Table 6