Developer Finds Real AI Agent Traces Break Assumptions Built on Mock Data

A developer building agent-exec-trace, an OpenTelemetry-style observability layer for AI agents, discovered that testing against real-world data exposed critical flaws in the tool's design. During an initial pass over 100,000 agent traces from Hugging Face, one detector fired on every single trace while 28 of 35 rule-based detectors never triggered at all. The root cause was not faulty detector logic but incorrect assumptions about the shape of real traces, assumptions that had been validated only against well-behaved mock data. The project aims to go beyond traditional observability by capturing behavioral paths — including tool calls, planning steps, and cost spikes — to explain why an agent run went wrong, not just that it did. The developer's key takeaway is that field testing with real systems must happen early, before the rest of a project begins to feel complete.