Two API Settings Tripled GPT-5.6 Sol's ARC-AGI-3 Benchmark Score, OpenAI Says

·1 views

OpenAI reported that enabling two Responses API settings — retained reasoning and compaction — raised GPT-5.6 Sol's score on the ARC-AGI-3 public benchmark from 13.3% to 38.3%. The initial evaluation harness prevented the model from carrying forward reasoning across steps of the benchmark's interactive 2D puzzle tasks, limiting its performance. Alongside the score improvement, the new configuration also reduced output token usage by approximately six times. OpenAI emphasized that this outcome highlights how API configuration, not just model capability, can significantly influence both agent performance and token efficiency. The findings serve as a practical reminder that benchmark scores reflect the combined effect of the model, its settings, and the evaluation setup rather than model weights alone.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

RE-call vs Mem0: No-LLM memory system trades accuracy for cost and privacy

Developer benchmarks comparing RE-call, an open-source memory system that never calls an LLM, against Mem0 show Mem0 outperforms RE-call on temporal reasoning tasks, scoring 0.567 versus 0.408 on the BEAM benchmark. The gap exists because Mem0 distills memories into clean summaries at ingest using an LLM, while RE-call stores raw conversation turns, causing date-related queries to retrieve wrong instances of the same fact. However, RE-call's no-LLM design means zero ingest cost compared to Mem0's $7.29 and 2.6 million tokens spent processing the same workload. RE-call also completed ingestion in 67 seconds versus Mem0's 288 seconds, and keeps all data within the user's own database. The author concludes that neither system is universally better — the right choice depends on whether a user prioritises accuracy on time-sensitive queries or prefers lower cost, faster ingest, and local data control.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

OpenAI Urges Clearer Reporting of AI Benchmark Conditions and Setup Variables

OpenAI has published new guidance calling on researchers, evaluators, and buyers to treat AI benchmark scores as measurements tied to specific testing conditions rather than fixed capability limits. The company argues that factors such as the evaluation harness, compute budget, available tools, and memory management can significantly alter observed model performance, particularly on complex multi-step tasks. OpenAI distinguishes between different types of evaluation claims — including strong elicitation, controlled comparison, and safeguard robustness — noting that each requires different evidence and cannot be used interchangeably. The guidance recommends that benchmark reports disclose the full setup behind a score so that comparisons between models remain meaningful and reliable. OpenAI cited its own GPT-5.5 cyber-range evaluations as an example, where performance visibly improved when the harness used context compaction across extended tasks.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Actually Verify If an AI Crawler Is Legitimate, Not Just Labeled

AI crawlers like GPTBot identify themselves via user-agent strings, but these can be easily faked by anyone in seconds, making them unreliable as sole proof of identity. Verification requires independently confirming a crawler's claimed identity using methods the operator controls, such as reverse DNS lookup, published IP ranges, or cryptographic HTTP message signatures. Reverse DNS remains the most robust method, while IP-range checks are the most widely supported, used by 20 of 41 documented crawlers including OpenAI's GPTBot. Cryptographic signing via HTTP Message Signatures is emerging as the most secure approach, with infrastructure support from Cloudflare, Akamai, and AWS, though operator-side adoption remains limited. The stakes have grown significantly by 2026, as verified crawler identity now governs real privileges like search representation, pay-per-crawl access, and bot-challenge exemptions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI memory systems cannot reliably detect what they don't know, new benchmark shows

A developer investigating AI memory agent reliability found that the ability to abstain from unanswerable questions is not a single capability but varies with how far removed a question is from its supporting evidence. A new benchmark was built with questions generated at controlled 'excision distances,' revealing that discrimination between answerable and unanswerable queries improves sharply as more context is deleted, from near-chance accuracy at close range to near-perfect at full topic removal. This finding explains why two public benchmarks, LOCOMO and BEAM, appeared to contradict each other: both were measuring different points on the same hidden curve and reporting them as a flat capability score. Further analysis showed that for roughly two-thirds of test questions, deleting the gold evidence caused no change in retrieval scores, meaning the correct answer was never the top-ranked result in the first place. The author concludes that abstention failures in AI memory systems are structural rather than tunable, and that benchmark comparisons are misleading without specifying the difficulty distance of unanswerable questions.

0 comments Read more at DEV Community