Mixture of Experts: How Sparse Routing Scales Model Size Without Matching Compute Costs

Dense transformers spend the same compute on every parameter for every token, so making them larger also makes them more expensive. Mixture of Experts (MoE) instead keeps a large pool of parameters but activates only a small subset for each token. A learned router scores the experts and selects the top few, so total model size can grow far faster than per-token cost. This design introduces a risk: the router can concentrate traffic on a few experts while others go unused, a problem typically countered with an auxiliary load balancing loss. The article explains the routing mechanics, why balancing matters, and when MoE may be the wrong architectural choice.