100% Model Agreement, 75% Accuracy: Why Shadow Traffic Metrics Can Mislead

A developer building SuperRouter, an open-source AI routing tool, found that a cheaper model agreed with a reference model 100% of the time during shadow traffic testing, yet was only correct 75% of the time. The finding highlights a core flaw in using inter-model agreement as a quality proxy: two models trained on overlapping data tend to fail in the same direction, making agreement highest where it is least protective. The author argues that evaluation must be scored against known ground truth — using deliberately planted faults with pre-determined correct answers — rather than against another model's output. Additional pitfalls uncovered include undetectable planted defects inflating scores, trivial test cases suppressing false-alarm rates, and published leaderboard rankings transferring poorly to product-specific tasks. The takeaway is that agreement rate is the most intuitive but most misleading metric when routing between AI models to cut costs.