Vendor's 200x Speed Claim Shrinks to 3–12x Under Real Workload Testing

A software team tested TypeSafe AI's model 'Jev' against their existing open model using a real production task: classifying email accounts on an admin dashboard. End-to-end, the new model proved 12x faster at the median, narrowing to roughly 3x once accounting for unnecessary output from the old model, and 7x cheaper — primarily because outbound tokens are free on the new platform. The team admits they produced two confident but incorrect benchmark figures before arriving at valid results, including repeating the very measurement error they had previously criticized the vendor for. The most meaningful finding was not speed but consistency: the new model showed just 38ms of latency variance compared to 2,353ms on the old one, eliminating parse failures that had been causing error pages. The team concludes that benchmarks are only useful when run on your own workload, close to production, with a reliable correctness check that does not rely on another model's judgment.