Developer Uncovers Three Layers of Broken Metrics Days Before Hackathon Deadline

A developer building RedCouncil, an adversarial multi-agent AI system, discovered critical flaws in their evaluation metrics just two days before a hackathon deadline. While attempting to prove measurable efficiency gains over single-agent baselines, they found that a key metric tracking missed findings was structurally empty by definition and could never return meaningful data. Further investigation revealed a plumbing bug where one part of the system wrote conflict data to one dictionary key while another part read from a different, never-populated key, keeping conflict counts permanently at zero. The root cause turned out to be a similarity-based disagreement detector that assumed agents would use overlapping vocabulary, fundamentally contradicting the system's design premise that specialist agents think differently. The developer ultimately fixed the detector by trusting each agent's own rebuttal signal rather than inferring disagreement from language overlap, and drew broader lessons about silent eval failures, structural metric blind spots, and non-determinism in multi-agent systems.