Engineer Builds Three-Tier Self-Healing System After Service Restarted 847 Times in 4 Hours

A software engineer developed a structured three-tier self-healing framework called Jarvis after witnessing a production service restart 847 times in four hours due to a slow downstream dependency triggering false health checks. The first tier uses circuit breakers to absorb minor blips without escalation, stopping unnecessary restarts before they bury real issues in logs. The second tier deploys a constrained LLM that classifies failures into one of eight predefined categories, preventing the model from generating arbitrary or unsafe repair commands. A third surgical tier allows an LLM patch agent to attempt code fixes, but only within a strict scope guard tied to the affected component, with all patches required to pass existing tests before deployment. The author emphasizes that maintaining clear boundaries between tiers is critical, as poorly tuned thresholds can mask genuine bugs or allow AI agents to make overly broad, risky changes.