Artificial intelligence · 2023

Mixtral

An open sparse mixture-of-experts model matches GPT-3.5 at a fraction of the cost.

Mistral's Mixtral 8x7B (December 2023) routed each token to 2 of 8 experts, using about 13B active of 47B total parameters. It matched or beat GPT-3.5 and Llama 2 70B and made MoE the standard design for large open models.

See it in the living treeWikipedia