SShortSingh.
Back to feed

Adversarial test pairs reveal when agent caches serve wrong write actions

0
·13 views

Hit-rate benchmarks show how often an agent cache helps but not when it causes harm, and replaying the wrong write step, such as deactivating instead of activating, can outweigh any token savings. The author proposes an adversarial eval built from near-opposite command pairs that share wording and structure but differ in one consequential verb. Embedding similarity tracks topic rather than polarity, so a threshold-only cache scored 1 of 18 false positives on this set, while a gated engine that replays write-containing trajectories only on exact repeats scored 0. The author recommends running such an eval as a CI gate so that threshold or model changes cannot silently reintroduce the failure. The method is presented as portable to other domains, though the 18-pair set itself is only a template and says little about any particular traffic.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

FINDR AI Open-Source Tool Targets GPS Spoofing in Parcel Delivery Tracking

FINDR AI is an open-source system built to detect suspicious location manipulation in e-commerce parcel delivery, where GPS coordinates can be misleading. It flags GPS spoofing, physically impossible teleportation jumps, and gradual location drift, then assigns risk scores to suspicious telemetry. The system adds investigation workflows, human approval for critical rescue actions, and hash-chain audit logging for traceable decisions. Built in Python with Streamlit, FastAPI, MongoDB Atlas, and SQLite, it was submitted to the Hacktoberfest Open-Source AI Challenge and reports 105 passing regression tests.

0
ProgrammingDEV Community ·

Debugging Malformed JSON From an Email Send API After Payment

The article advises validating the exact request payload against the provider's documented schema before sending, rather than assuming the response is at fault. It recommends checking the HTTP status before parsing the body, since an HTML gateway error or empty response can cause a JSON decoding failure. Retries should use a stable idempotency key and a policy keyed to status codes, to avoid duplicate sends and redundant payloads. The author also proposes splitting retention, keeping operational metadata and a payload hash longer while shortening retention of sensitive receipt content. The overall guidance is to log request bytes, status, content type, and a size-limited raw body separately from parser errors.

0
ProgrammingDEV Community ·

Measuring a Server: Latency Knees, Misleading Percentiles, and Little's Law

The author built a deliberately simple single-worker server to study how latency behaves under load. Ramping request rates revealed a 'knee' where tail latency rises sharply well before the server is fully utilised. At about 70% busy, the 99th percentile latency had already more than doubled. The author also argues that averages hide slow outliers, so percentiles better reflect what users actually experience. A further experiment showed that fan-out amplifies this effect, since a page is only as fast as its slowest dependency.

0
ProgrammingDEV Community ·

Developer Struggles With Calculator App Logic, Plans to Resume Work Tuesday

A developer posting on DEV Community says they are stuck on the logic of a calculator app they are building. They plan to continue the project on Tuesday because they have other commitments on Monday. The developer had expected the task to be simple but found it more difficult than anticipated. They recalled previously building a basic calculator webpage with AI assistance in about half an hour, and reflected that they have not retained the earlier learning.