How Mixture of Experts Architecture Evolved from 1991 Research to DeepSeek-V3

Mixture of Experts (MoE) is an AI architecture that activates only a small subset of a model's parameters for each token, dramatically reducing compute costs compared to traditional dense models. The concept dates back to a 1991 paper by Jacobs, Jordan, Nowlan, and Hinton, which introduced a gating network to route inputs across multiple smaller expert networks, though all experts still ran on every input, offering no compute savings. The critical breakthrough came in 2017 when Google Brain researchers introduced sparse gating in their paper on Sparsely-Gated Mixture-of-Experts, where only the top-scoring experts are selected and executed per token. This sparse routing mechanism forms the foundation of modern large language model MoE designs. DeepSeek-V3 exemplifies the approach at scale, carrying 671 billion total parameters while activating only around 37 billion per forward pass.