Spring Boot AI Agent Uses LLM Judge to Catch Bad Prompt Changes Before Production

A developer building a production AI agent in Spring Boot discovered that a seemingly obvious system prompt fix — adding a plain-text instruction to stop broken markdown tables — actually made the agent worse when tested rigorously. Running a pairwise comparison across 40 standardized cases revealed the old prompt won 18 pairs against 10 losses, because the new wording inadvertently made responses too terse, dropping order summaries customers needed. The incident highlighted a core limitation of relying on manual spot-checks or nightly scoring alone: prompt changes are experiments with a sample size of one until tested against a fixed dataset. The solution adopted is the pairwise LLM-as-a-judge pattern, where a single judge model compares two prompt versions head-to-head on the same dataset on the same day, rather than scoring each against an absolute rubric. This approach, documented in Spring AI's evaluation guide, is argued to produce more reliable verdicts because relative judgment is inherently easier and less prone to drift than absolute scoring.