40 Lines of Go Slashed One Team's LLM Costs by 71% Using Cheap-First Routing

A development team discovered they were routing nearly all LLM requests — including trivial tasks like naming files or generating titles — through an expensive frontier model by default. After reviewing their traffic mix, they built a lightweight Go-based router that sends requests to a cheaper model first and only escalates to the stronger model if the output fails a quality gate. The system moved 81% of requests off the expensive model, cutting the overall bill by 71%, with only a 15% escalation rate. The team found that pre-classifying prompts by difficulty was unreliable, since short prompts can be complex and long ones trivial, making output-based judgment more effective than input-based prediction. They noted that latency, not cost, proved to be the more significant constraint in practice.