How One Developer Built a Trap-Heavy Exam to Stress-Test an LLM Order System

A developer building an LLM-powered order-processing program designed a 29-question exam where only 4 questions covered normal orders, with the majority targeting edge cases, ambiguous inputs, and non-order queries like price or stock inquiries. The rationale was that language models rarely fail on clean, standard inputs — failures occur on irregular data such as order changes, typos, and questions mistaken for orders. The developer also stressed the importance of using messy, near-identical product data in tests, arguing that a clean catalog will pass any exam while production data causes the system to collapse. A late addition of seven post-learning scenarios revealed a critical risk: once the model learned shortcuts like mapping '250' to a specific product, it could misapply that knowledge to ship wrong items or misread questions as orders. The full 29-question test suite and product catalog have been made publicly available on GitHub.