Developer finds 96 routing policy violations in AI agent logs, all off the main thread
A software developer auditing their AI agent's token usage discovered that 96 out of 425 routing decisions — about 22.6% — deviated from their stated policy, costing an extra $1,248.13. All deviations occurred in subagent threads, not on the main thread, revealing a gap between intended and actual model selection behavior. The developer maintains a 30-line YAML routing policy and an append-only trace log, comparing expected versus actual model tiers per decision unit. An early policy draft was found to be misaligned with actual intent, prompting a correction before measurements were taken to avoid scoring behavior against a flawed baseline. Separately, trace logs revealed an 18-day gap in Claude Fable 5 usage that appeared to reflect natural session drainage rather than a deliberate choice to stop using the model.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in