Software Bugs Masqueraded as AI Personality Traits in Model Benchmarking Study

A developer running AI matches on the Kai! Arena platform discovered that four software bugs in the evaluation harness were creating false impressions of distinct model behaviors. One bug caused DeepSeek V4-Pro to appear to bid without looking at its dice nearly 40% of the time, a rate that dropped to just 6% after the flaw was fixed. In total, more than half of the most visible behavioral differences between models across the first 60 matches shrank once all four bugs were corrected. The findings highlight that arena-style benchmarks measure the combined system of model and harness, not the model alone. The author concludes that any evaluation must verify the measurement infrastructure did not inadvertently supply hints, hide inputs, or alter outputs before attributing results to the model itself.