New Research Shows Optimal AI Training Depends on Cluster Hardware, Not Just Compute

A new academic paper argues that standard compute-optimal scaling laws fail to account for real-world cluster infrastructure constraints. The researchers propose integrating systems-level factors directly into the scaling-law analysis, a framework they call cluster-optimal training. Under this approach, the ideal architecture for a model — such as the sparsity level of a Mixture-of-Experts (MoE) system — varies depending on the specific hardware cluster used for training. The findings suggest that recommendations derived purely from theoretical compute efficiency may lead to suboptimal design choices in practice. The paper is available as a preprint on arXiv.