Artificial intelligence · 2003

Neural language model

Bengio predicts the next word with a network that learns word vectors.

Yoshua Bengio and colleagues' neural probabilistic language model learned a distributed representation for each word jointly with a next-word predictor. It introduced learned embeddings and the next-token objective that GPT models would later scale.

See it in the living tree Wikipedia

Arcs above join it to what it stood on, arcs below to what it led to. The bar places this stretch on the tree's whole timeline.

What it built on 2

Statistical machine translation 1990
Backpropagation 1986

What it led to 2

word2vec 2013
GPT-1 2018

Related measurements

The dashed line marks when it first worked.

Training compute of notable AI models Source: Epoch AI, Notable AI Models (CC BY 4.0)

Computation per constant dollar, 1850-2006 Latest: 7.3B million units of computer power per 2006 $ (2003) Source: Nordhaus 2007, Two Centuries of Productivity Growth in Computing, J. Econ. Hist. 67(1), Table 6