LLM Evals Are Just Parameter Sweeps — Existing Tools Already Solve Most of It

A technical analysis argues that most LLM evaluation workflows reduce to a well-understood data science pattern: a Cartesian product of cases, scorers, models, prompt variants, and repetitions — commonly called a parameter sweep. The author contends that the LLM industry has largely rebuilt this solved infrastructure as metered SaaS, despite decade-old tooling already handling caching and recomputation efficiently. While acknowledging that newer platform features like agent trace visualization and span-level debugging are genuinely novel and worth paying for, the piece maintains that the core evaluation matrix and persistence layer are not LLM-specific problems. The author proposes a composable open-source stack — pydantic-evals for scoring, oryxflow for the parameter matrix and provenance, and Logfire for traces — with each layer handling a single responsibility. A worked example using an intent classifier illustrates the approach, also warning that aggregate accuracy metrics can mask critical per-class failures in imbalanced evaluation datasets.