Replay test shows on-call agent's incident accuracy rising from 33% to 83% with memory
Déjà Vu, an on-call SRE agent built on Hindsight, was evaluated with a chronological replay harness that restricts each prediction to incidents that occurred earlier. Across 22 graded incidents, the agent reached 50% top-1 accuracy and 64% top-3 accuracy. Its accuracy rose from 33% on the first six graded alerts to 83% on the last six. The evaluation uses a deterministic BM25 ranker without an LLM grader, so the improvement can be attributed to retrieval. The project also displays the cumulative learning curve and sample predictions to show how the agent's guesses change as its memory grows.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in