Developer Builds CI/CD Failure Agent With Structured Memory to Prevent AI Overconfidence

A developer created PipelineSage, a Python-based tool with a Streamlit dashboard, designed to diagnose CI/CD pipeline failures using an LLM backed by structured agent memory. The project uses Hindsight Cloud, an open-source memory system by Vectorize, to store and retrieve incident data through two core operations: retaining experiences and recalling relevant past incidents. A key design decision was restricting what the agent is allowed to remember — only structured, verified incident records are stored, not raw model recommendations, to prevent unverified conclusions from compounding into false precedents. Each stored incident follows a fixed schema including deployment ID, root cause, resolution, and outcome, with unconfirmed fields explicitly marked to maintain honesty in recalled context. Retrieval goes beyond simple semantic search, using multiple differently-phrased queries, deduplication, and heuristic re-ranking to ensure the most relevant — not just the most textually similar — past incident informs the current diagnosis.