How to Monitor an LLM System in Production

The article explains how engineers should monitor LLM systems at three levels: individual requests, multi-step traces, and overall service health. For each request, it recommends logging identifiers, prompt versions, model names, token counts, latency, cost, and structured outcomes, while redacting personal data. Multi-step workflows such as agents need distributed traces that show the order of model, retrieval, and tool calls. Dashboards and alerts should focus on sustained error rates, p95 latency, and cost spikes, alongside quality checks like sampled human review and user feedback. The piece also warns against common mistakes, including storing raw PII in logs and relying on aggregate metrics alone.