Benchmark Tests 8 LLMs on Real Analytics Tasks, Finds Cheap Models Often Hallucinate

A developer building InsightTrack, an open-source web analytics tool called Pulse, designed a 480-question benchmark to objectively evaluate which large language model best handles real-world analytics queries. The benchmark covers four tasks: tool selection, data reading, traffic diagnosis, and SQL generation, each with standard and harder edge-case questions. The test prioritizes qualities absent from public leaderboards, including restraint in uncertain situations, precise threshold application, and accurate reading of actual tool output formats. Results showed that lower-cost models frequently either invented plausible-sounding but false explanations or failed to flag genuine problems, both outcomes considered worse than admitting uncertainty. The benchmark was submitted to the Kaggle Benchmarking Challenge and aims to help developers choose AI models based on task-specific evidence rather than general performance scores.