Satellite Geo QCM Leaderboard Shows How Transparent LLM Benchmarking Works

The Satellite Geo QCM benchmark evaluates large language models on geolocation tasks using satellite images, asking each model to pick the correct location from four fixed options. Unlike many leaderboards, it uses fully deterministic scoring with no LLM judge, meaning results are raw accuracy counts rather than subjective assessments. On the public leaderboard, DeepSeek V4 Flash Vision scored 98.18%, while the submitting team's own model, GLM 5.2, scored 78.0%, a gap of roughly 20 percentage points. The benchmark also breaks down scores by difficulty level, allowing users to see whether a model performs consistently or merely excels on easier items. A publicly accessible verbatim response trail lets anyone inspect the exact image, options, and answer each model provided, adding a layer of accountability rare in LLM evaluation.