LLM eval tools offer 50+ metrics, but the hard parts of evaluation remain unsolved

A developer spent a week reviewing the metric catalogs of five widely-used LLM evaluation libraries — Arize Phoenix, DeepEval, Future AGI, Langfuse, and Ragas — to assess what they actually offer beyond their marketing claims. All five tools provide ready-made metrics, with counts ranging from around 16 to 72, and every one supports LLM-as-a-judge scoring. However, the author argues that the metrics themselves represent only the easier 20 percent of the evaluation problem. The genuinely difficult tasks — selecting a metric that matches a specific failure mode and quantifying uncertainty in results — are left entirely to the user. Notably, the catalogs are converging on the same core metrics, such as Faithfulness and Tool Correctness, suggesting that metric coverage is fast becoming a commodity rather than a differentiator.