Kaggle Benchmark Tests AI Models on Booking Conflict Replay Logic

A new Kaggle benchmark called Reservation Replay challenges AI models to correctly handle booking conflicts, specifically testing whether a failed request remains failed even after conditions change. The benchmark uses fictional rooms and integer time intervals across 8 base traces and 4 dependent metamorphic variants, scored by exact trace success without an LLM judge. Google's Gemini 3.7 Flash achieved a perfect 12/12 score, while Gemini 2.5 Flash scored only 4/12, primarily due to output formatting failures rather than logical errors. A scoring policy update was introduced mid-evaluation after Gemini 2.5 Flash triggered a ResponseParsingError, highlighting how format compliance can significantly skew benchmark results. The author cautions that the small pilot size and dependent variants are insufficient to establish stable model rankings, but the reproducible fixtures and preserved run evidence make the findings verifiable.