Engineer tests six LLM eval frameworks in CI merge queue; only two proved reliable

A software engineer ran six open-source LLM evaluation frameworks inside a real GitHub Actions merge queue over eight months, testing them against production pull requests. The core criterion was determinism: a gate had to return the same pass or fail verdict on unchanged inputs every single time. Frameworks relying heavily on LLM-as-judge scoring failed this test, as judge-assigned scores drifted between runs without any code changes, causing false blocks and eroding team trust in the gate. Promptfoo and DeepEval were the only two frameworks that consistently met the bar, largely because they kept deterministic checks in the blocking path and treated judge scores as non-blocking signals. The author concludes that a flaky CI gate is worse than no gate, since teams learn to bypass it, rendering it meaningless.