Developer Builds AI-Maintained Failure Log to Close the ML Eval Feedback Loop

A developer working on an MLX-based classifier that maps work sessions to Jira tickets found that running evaluations was easy, but tracking and diagnosing recurring failures was not. After accumulating 62 failures across three eval runs with no reliable way to spot patterns, they designed a structured solution using a Claude Code skill invoked manually after each evaluation. The workflow writes failure data to a machine-maintained file called FEEDBACK.json, storing runs, individual observations, and named failure classes that persist across multiple eval cycles. To keep context usage manageable, the skill queries only targeted slices of the file using jq rather than loading it entirely. The approach aims to turn evaluation results into an actionable engineering tool rather than a static scoreboard.