SShortSingh.
Back to feed

GPT-5.6 Sol jumps from 13.3% to 38.3% on ARC-AGI-3 with config changes alone

0
·1 views

OpenAI tested GPT-5.6 Sol on the ARC-AGI-3 benchmark under two different API configurations without changing the model itself, and the score jumped from 13.3% to 38.3% in Relative Human Action Efficiency while using one-sixth the output tokens. The key differences were retaining the model's reasoning across turns and using history compaction instead of truncating older context once conversations exceeded 175,000 characters. The default harness discarded private reasoning after every move and dropped earlier actions as context filled, effectively forcing the model to re-derive its understanding of game rules from scratch each turn. With the adjusted configuration, GPT-5.6 Sol solved all six levels of a game where no frontier model had previously passed the first level on the public leaderboard. The findings suggest that a significant portion of the performance gap between frontier models and human testers may stem from agent loop design rather than model capability.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

AI Speeds Up Code Generation but Makes Verification Harder, Developers Find

AI coding tools have split developers into two camps — those who feel far more productive and those who abandoned the tools after getting confidently wrong outputs. A key insight from experienced developers is that while generating code has become nearly effortless, verifying that code remains just as demanding as before. The biggest practical gain lies not in writing new code but in quickly understanding unfamiliar or legacy codebases, reducing hours of manual investigation to seconds. However, AI-generated code poses a stealthy review risk: unlike junior developer mistakes, AI errors are stylistically polished and harder to flag on instinct. Teams that adopt these tools without rethinking their code review processes risk shallower oversight and a higher volume of subtle, hard-to-spot bugs reaching production.

0
ProgrammingDEV Community ·

Memory Engines Poised to Become Enterprise Infrastructure, Analysts Argue

A vendor analysis published on DEV Community argues that AI memory engines are at the same inflection point relational databases were in the mid-1970s, predicting they will become standard enterprise infrastructure. The piece contends that as AI tools spread across business functions — from sales and customer service to software development — organisations are generating valuable institutional knowledge that currently disperses across chat logs and personal notes with no centralised store. The authors, who disclose they have built their own memory engine called NylonME, surveyed existing public memory solutions and identified four critical gaps: poor write performance and latency, single-user architectures ill-suited to multi-tenant enterprise use, over-reliance on LLMs in the read-write pipeline, and data-sovereignty risks from managed cloud deployments. They argue current products are well-designed for personal AI assistants but break down under enterprise-scale concurrent usage and compliance requirements. The central claim is that enterprises will eventually require a shared, centralised memory layer connecting all AI agents — much as they now rely on a shared database cluster for business systems.

0
ProgrammingDEV Community ·

How updating your AWS CLI config can eliminate repeated SSO login prompts

Developers using AWS IAM Identity Center frequently encounter token expiry errors that force them to run 'aws sso login' multiple times a day. The root cause is often an outdated AWS CLI config file that uses a legacy format, which does not support refresh tokens properly. AWS SSO actually relies on two tokens — a short-lived access token (~8 hours) and a longer-lived refresh token (up to 90 days) — where the latter is meant to silently renew the former. The legacy config style repeats SSO settings inside every profile and never shares a refreshable session across them. Migrating to the modern 'sso-session' block in ~/.aws/config centralises session settings, enables refresh token support, and allows a single login command to cover all configured profiles.

0
ProgrammingDEV Community ·

Beyond Detection: How Edge AI People Counters Actually Work End to End

Building a functional edge AI people counter requires five distinct stages beyond simple person detection: frame capture, image preprocessing, detection, multi-frame tracking, and conversion of movement into entry or exit events. Each stage introduces potential errors, from uneven frame delivery and resizing artifacts to trackers assigning new IDs mid-doorway, all of which compound into inaccurate final counts. A tracker assigns temporary IDs to keep detections linked to the same individual across frames, enabling a line-crossing algorithm to register a visit only when a tracked path genuinely transitions from one side to the other. Occupancy figures are derived as running state — prior occupancy plus entries minus exits — meaning a single missed event corrupts the count until manually corrected. The article emphasizes that most demos stop at detection and frame-rate display, obscuring the substantial engineering needed before a system can reliably tell a retailer how many visitors entered a store.