Dev team builds AI agent monitor after silent LangChain failures cost client $2,400

A software team discovered that a LangChain agent deployed for a B2B client had been silently failing on roughly 30% of sessions for two weeks before anyone noticed. The agent continued running and returning responses without throwing errors, making it appear healthy in standard observability tools like LangSmith. The root cause was the agent retrieving incorrect context and generating plausible but wrong answers — a semantic failure invisible to request-level tracing. By the time the client flagged the issue through unusual business metrics, approximately $2,400 in LLM costs had already been wasted. In response, the team built an internal monitoring tool called AgentWatch, which tracks session outcomes as explicit fields, flags retry patterns as signals, and attributes costs per client to catch such silent failures earlier.