Token Price Alone Is a Poor Metric for Choosing a Customer Support LLM API
A software team discovered that selecting an LLM API based solely on token price can be misleading when building customer support chatbots, as factors like retry behavior, tool calls, escalation rates, and context length significantly affect true cost per resolved conversation. A production incident revealed that a naive retry mechanism caused the same internal case note to be written twice across 37 conversations, highlighting the danger of conflating model-generation retries with business-operation retries. The team now evaluates LLMs using a replay set of real support scenarios and measures metrics such as cost per resolved conversation, latency percentiles, tool-operation success rate, and duplicate-effect count — with zero tolerance for duplicate side effects. They also stress-test OpenAI-compatible APIs with contract suites rather than assuming compatibility based on matching request formats alone. The key takeaway is that no single LLM API is permanently the cheapest for every use case, and idempotency boundaries must be established before model selection becomes meaningful.
