Why Your LLM Telemetry Table May Be Comparing Incompatible Metrics

A detailed analysis of LLM telemetry reporting reveals a common but dangerous flaw: metrics grouped under the same model label often count fundamentally different things, making direct comparisons misleading. Core process metrics were attributed at the epoch level within threads, while completion proxies were calculated once per thread and only for sufficiently pure main sessions, meaning sample sizes differed across tables without either being incorrect. Main sessions and sidechains were also found to have distinct structures and ending conditions, so combining their re-edit rates conflates model behavior with session type. The report further noted that a model name can persist across relaunches or configuration changes even when the underlying treatment has changed, invalidating before-and-after comparisons. The key takeaway is that before interpreting any difference in a telemetry table, analysts must first establish what a single row actually represents — an epoch fragment, a full thread, or a delegation edge.