Cheaper LLM beat flagship model by 5.8x cost margin in niche domain benchmark

A developer building a BaZi birth-chart reading app benchmarked eight large language models on real production workloads before launch, finding that generic leaderboards were useless for niche domain evaluation. The most expensive flagship model was excluded because its hybrid reasoning mode could not be disabled, adding significant latency and billing for internal "thinking" tokens on top of an already high per-call price. Three mid-tier models were also eliminated for flipping domain-specific terminology — such as rendering Yang Wood as Yin Wood — which the developer treated as categorical, not marginal, errors. A smaller, cheaper model ultimately won the free tier slot, while a mid-tier model and its previous generation handled paid requests, forming a cost-ordered fallback chain. The developer concluded that for niche-domain LLM apps, custom evals on actual workloads and a structured routing layer with fallback chains matter far more than headline benchmark scores.