New Benchmark Tests If AI Models Know When to Admit They Can't Answer

A developer has created a 200-item benchmark called ESCALATE to measure whether AI models can recognize the limits of their own knowledge, not just whether they answer correctly. Each task includes a deliberate "unanswerable" scenario in roughly one in five items, where the only correct response is to escalate rather than guess. The benchmark spans four task types — routing, classification, judgment, and grounded question-answering — and scores models on both accuracy and false-confidence rate. The project compares frontier models hosted on Kaggle against small open-source models ranging from 1B to 8B parameters running locally on a single laptop. The author has pre-registered predictions before results are in, including the hypothesis that some small local models may show lower false-confidence rates than certain frontier models.