Weaker model panels can still flag wrong MMLU answer keys, with strength mattering more than size

Researchers at Munim, Inc. tested which AI models best identify mislabeled answers in the MMLU benchmark, using pre-registered, open code. They compared a rule based on the top 10 most accurate models with a signal using all 78 models, measuring item discrimination. Against expert labels in MMLU-Redux 2.0, the discrimination signal performed slightly worse than the top-10 rule, while the two together gave the most precise short list. Splitting models into panels by accuracy showed that weaker panels catch roughly the same wrong keys but add many more false flags. The team concluded that stronger panels are preferable, though cheaper older models still perform reasonably when reviewed under a fixed budget rather than a vote threshold.