Benchmark Tests AI Models on Payment Verification Logic With Synthetic Data

A diagnostic benchmark called 'Promise Is Not Payment' was submitted to the Kaggle Benchmarking Challenge to evaluate how well AI models distinguish between payment claims, pending states, and verified receipts. The benchmark consists of 24 scored cases across 12 counterfactual pairs, each altering a key piece of evidence such as transaction status, recipient, or refund type. Two models — Google Gemini 3.7 Flash and Claude Haiku 4.5 — completed the task on Kaggle on September 27, 2026, while a third model, Qwen3-Next-80B, failed due to server overload and was not scored. Additionally, two locally run quantized models, Llama3:8b and Qwen3.5:9b, were tested under controlled settings, though these results are separate from the official Kaggle submission. The benchmark is a controlled diagnostic pilot using entirely fictional records and does not interact with real payment systems or accounts.