Agent replays only prove something if they run with network access blocked
Several recent developer posts describe replaying agent traces to debug production failures, but they share an unchecked assumption: that tool calls are served from the recording. Many harnesses silently fall back to live functions when a cache key misses, so a replay can quietly re-fetch live data. The article proposes a simple check, running the replay with OS-level network egress blocked, which tests the trace from outside the process. If the failure still reproduces offline, the trace is complete; if it disappears, the replay was contaminated. The author also notes that a step-by-step diff cannot reveal whether inputs came from live sources, and that new OpenTelemetry export from Anthropic's platform does not replace storing response data locally.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in