pass^k, not pass@k: Measuring whether AI agents succeed reliably every time
A DEV Community author argues that the common pass@k metric, which counts success if at least one of k attempts works, hides unreliability in AI agents. The alternative pass^k requires all k attempts to succeed, giving a truer picture of trustworthiness. The piece notes that per-step failures compound across long tool-call sequences, so a 99% per-call success rate can still yield only about 60% task success over 50 calls. The author recommends reporting pass^k alongside pass@k, evaluating full trajectories rather than final answers, and using a deterministic verifier instead of another model to check results. The author also discloses that they run an AI-ops store and write about agent verification.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in