Smart Routing cuts LLM costs vs premium models but trails cheaper fixed options

LLM Gateway published a benchmark testing its Smart Routing system against three fixed-model baselines across 80 standardised prompts covering arithmetic, reasoning, and instruction-following tasks. Smart Routing cost 33.2% less than the fixed premium baseline but was 35.6 times more expensive than the fixed low-cost option. On task success, Smart Routing scored 75 out of 80, comparable to the low-cost baseline at 74 and the mid-priced at 76, while the premium model also scored 74 under strict grading. The cost savings against the premium baseline carried wide statistical uncertainty, with the confidence interval ranging from a 4.3% increase to a 66.1% decrease, meaning no reliable reduction was established. The benchmark was authored by an LLM Gateway employee and prepared with AI assistance, which the publication disclosed upfront.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in