AI incident-response agent made correct rollback calls despite frequent misdiagnosis

A developer team tested their AI agent, Agent K, against 12 simulated production incidents and found it identified the correct root cause only 5 times, yet made the right rollback or no-rollback decision in 10 of 12 cases. The project was built as an entry for the Agents of SigNoz hackathon, using a FastAPI retrieval-augmented generation service instrumented with OpenTelemetry and connected to SigNoz for observability. The team deliberately designed safety around code rather than model accuracy, implementing a strict policy gate that blocks any rollback action unless six deterministic checks pass, with no LLM involvement in the decision logic. A dedicated test suite of 219 tests, including one that statically analyzes import graphs, enforces that the policy module can never call a language model. The core insight the team highlights is that since model errors are assumed inevitable, behavioral safety must be enforced through code constraints rather than relying on diagnostic correctness.