Picking the longest answer beats chance on 52.7% of MMLU international law questions

An AI-run audit at Munim, Inc. tested a baseline that ignores the question and simply selects the longest answer option on the MMLU benchmark. Pooled across all subjects, this heuristic scores about 28%, slightly above the 25% chance level, because correct options tend to be somewhat longer. The effect is far from uniform: on international law it reaches 52.7%, while on public relations and anatomy it falls below chance. The author's pre-registered prediction that the advantage would be spread evenly across subjects was not borne out. The findings suggest per-subject MMLU scores carry very different baseline floors, so model results on individual subjects should be interpreted with that in mind.