Kaggle Challenge Benchmarks LLM Judgment on 36 Data Science Scenarios

A new benchmark was created to test how large language models handle practical data science judgment calls, based on 36 measured scenarios from Kaggle competitions. The benchmark tasks models with both multiple-choice and open-ended responses to these realistic situations. Models from Google, Anthropic, OpenAI, xAI, DeepSeek, and Alibaba were evaluated, though some could not be fully tested due to technical or cost limitations. The study found that while LLMs can answer textbook questions, their advice on nuanced, competition-specific decisions often diverges from measured best practices. The benchmark is designed to log not just accuracy but the specific types of mistakes different models make.