How One AI Firm Cut Model Spend by Routing Tasks Away From GPT-4o

An AI company discovered that GPT-4o was handling 77% of its production traffic while consuming 97% of its total model spend, revealing a costly imbalance in how tasks were assigned to language models. The disparity arose not from deliberate choices but from three systemic forces: demos built on the strongest model becoming permanent defaults, no per-task cost visibility in monthly bills, and asymmetric blame that punished cheap-model failures but never questioned frontier-model overuse. To address this, the company introduced a written routing policy that reserves frontier models for open-ended, high-stakes, or judgment-heavy tasks, while directing structured, verifiable work to cheaper alternatives. The key distinction driving the policy is whether a task executes an existing plan — which cheaper models handle well — or requires deciding the plan, where errors are costly to detect and reverse. The company also cautions that the right success metric is cost per completed task, not cost per call, since a cheaper model requiring multiple retries can negate its savings.