AI Made the Same 3 Planning Mistakes Across 170 Goals, Regardless of Model Size

A developer built a small engine called PlannerCritic that paired one LLM to write plans with a second to review them, then tested it across 170 change-planning goals spanning 40 domains at a total cost of $0.49. Out of 132 concrete blockers identified, 121 fell into just three recurring failure categories: unverified dependencies, unsafe sequencing, and weak rollback logic. Switching to a more powerful model like GPT-4o produced better-written plans but did not eliminate these structural defects. The author found that deterministic validation gates — which ran without any LLM calls — were more effective at catching bad plans than upgrading the underlying model. The key takeaway is that AI planning failures in multi-step tasks are structural problems, not parameter or model-size problems.