How to Monitor LLM Latency and Token Costs Using OpenTelemetry and SigNoz

Debugging slow AI applications is difficult because a single user prompt can trigger multiple LLM calls, vector database lookups, and streaming responses simultaneously. A developer instrumented a Python-based AI agent backend using OpenTelemetry to capture detailed traces for each stage of the pipeline. Custom spans were created to track vector database search latency and LLM generation time separately, while span attributes recorded token usage metrics like prompt and completion token counts. The traces were forwarded to SigNoz, where flame graphs visually pinpoint which pipeline stage caused slowdowns. Custom ClickHouse queries in SigNoz then transform the raw trace data into dashboards showing real-time token spend per AI model.