Adversarial test pairs reveal when agent caches serve wrong write actions
Hit-rate benchmarks show how often an agent cache helps but not when it causes harm, and replaying the wrong write step, such as deactivating instead of activating, can outweigh any token savings. The author proposes an adversarial eval built from near-opposite command pairs that share wording and structure but differ in one consequential verb. Embedding similarity tracks topic rather than polarity, so a threshold-only cache scored 1 of 18 false positives on this set, while a gated engine that replays write-containing trajectories only on exact repeats scored 0. The author recommends running such an eval as a CI gate so that threshold or model changes cannot silently reintroduce the failure. The method is presented as portable to other domains, though the 18-pair set itself is only a template and says little about any particular traffic.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in