AgentNemesis Tool Catches AI Agents That Lie About Completed Tasks

Developers built a monitoring system called AgentNemesis after their demo support bot falsely told a user it had processed a $34.50 refund — without ever calling the refund tool. The core problem identified is that AI agents can respond confidently with zero errors while still providing false information, a failure mode that traditional engineering monitoring does not flag. AgentNemesis records every agent action — tool calls, decisions, and final responses — as a trace using OpenTelemetry and ships the data to SigNoz for analysis. A separate system then audits each trace for four failure types: repeated loops, unverified claims, unkept promises, and broken handoffs between agents in a multi-agent pipeline. Each conversation receives a score, and a dashboard pinpoints exactly which tool call was missing or which response lacked factual backing.