Developer shares hard-won lessons from wiring AI agents to self-hosted SigNoz

A developer built ArcNet, an AI agent monitoring system, using self-hosted SigNoz during the Agents of SigNoz hackathon to answer a key question: how do you verify a fix worked after an AI agent misbehaves in production. A critical early mistake involved assuming Agno instrumentation would emit OpenTelemetry's gen_ai.* attributes, when it actually uses the different OpenInference convention, leaving all pre-built dashboards silently empty. The developer recommends sending a real request first and manually reading raw span attributes in the trace viewer before writing any queries or dashboards. Guardrail verdicts were made observable by writing structured span attributes — such as checkpoint, action, rule, and risk score — enabling aggregation, grouping, and alerting on blocked events. This approach revealed that a prompt-injection attempt was blocked at the tool-call stage rather than at content ingest, a distinction only visible because the checkpoint was recorded as a traceable attribute.