Why Pass/Fail Grading for LLMs Can Lead to Costly Production Failures

A developer building an LLM-powered order-processing system warns that grading AI outputs by simple pass/fail or total score can mask dangerous, irreversible errors. Instead, he proposes a four-tier severity grading system — Fatal, Risky, Missed, and Harmless — based on whether a mistake can be undone, such as loading wrong goods onto a truck. He found that even a single Fatal error in a test suite should block deployment, regardless of how well the model performed on all other questions. The developer also discovered two bugs in his own grader that were silently penalizing correct model answers, underscoring that flawed evaluation logic can corrupt the entire testing process. He recommends saving all model responses to disk and supporting a rescore flag, so grading logic can be corrected and reapplied instantly without re-querying the model.