Developer stress-tests a stdlib eval tool's substring scoring and finds it inflates accuracy
A developer built AIEval, a small offline CLI that compares recorded model answers to expected strings and exits non-zero when accuracy falls below a configured threshold. While testing its scoring rule, they found that short expected strings like 'yes' or '15' can match wrong answers through simple substring matching, so the accuracy figure partly reflects vocabulary rather than model quality. Running the tool also revealed that missing responses count as failures and that the latency threshold setting is not enforced. The author recommends choosing expected strings that wrong answers cannot accidentally contain, and wiring the exit code into a CI gate.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in