SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

AWS Strands Agent Lifts Claude Opus 5 from 30% to 99.95% on ARC-AGI-3 Benchmark

0
·1 views

AWS engineers used Claude Opus 5 paired with the open-source Strands Agents SDK to build an AI agent that completed all 183 levels across ARC-AGI-3's 25 public environments, achieving a 99.95% relative human action efficiency score in a single 8-hour run costing roughly $830 in tokens. ARC-AGI-3, developed by ARC Prize, tests AI systems by dropping them into unfamiliar environments without explaining the rules or objectives, requiring the agent to experiment and adapt autonomously. The same Claude Opus 5 model scored only 30.16% when evaluated by ARC Prize under standard conditions, highlighting how critically the surrounding agent architecture — or 'harness' — influences real-world performance. NVIDIA reported a comparable outcome with its AVO agent, also built on Opus 5, scoring 100% on the same public game set. Developers note that the public benchmark results, while not the official competition evaluation, offer valuable insight into which agent design choices enable AI to handle complex, long-horizon tasks.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Whoosh library lets Python developers add search result highlighting without Elasticsearch

The pure-Python full-text search library Whoosh includes a built-in highlighting engine that displays relevant snippets around matched query terms, similar to how Google and GitHub present search results. Developers can install the maintained fork, Whoosh3, and use its highlighter by passing terms=True during a search query to accurately track which terms matched each document. The library offers multiple fragmenter options — including ContextFragmenter and SentenceFragmenter — to control snippet length and boundaries, as well as formatters for HTML output or plain-text CLI displays. For web interfaces, HtmlFormatter wraps matched terms in configurable HTML tags, while UppercaseFormatter or custom ANSI-coded formatters serve terminal-based applications. The approach requires no external search infrastructure, making it a lightweight option for adding contextual search highlighting to Python projects.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Indian student's solo-built code intelligence tool beats rivals in head-to-head benchmarks

Arihant, a student in India, has built Aletheore, a code intelligence platform that parses repositories into dependency graphs to power features like security scanning, PR review, and semantic code search. Six weeks after its public launch, the tool has recorded over 2,600 unique downloads on PyPI. In benchmark tests against RepoWise across seven open-source codebases, Aletheore achieved higher code-location accuracy in five of seven corpora at zero indexing cost, compared to RepoWise's $1.85 spend on LLM-generated wikis. Against Graphify on a roughly one-million-line Python codebase, Aletheore matched coverage while using 36% fewer tokens per query, though it lagged on initial graph-build setup time. The full methodology and raw benchmark results have been made publicly available by the developer.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Nvidia's Data Center Revenue Surged 62% YoY in 2024 Amid AI GPU Demand

Nvidia recorded $13.5 billion in data center revenue in Q2 2024, a 62% year-on-year increase, driven by intense global demand for high-performance GPUs used to train large AI models. The company's H100, A100, and RTX 4090 chips dominate the market due to their performance, energy efficiency, and compatibility with major AI frameworks like PyTorch and TensorFlow. A chip shortage combined with rapidly growing demand for large-scale AI models has allowed Nvidia to raise prices and maintain margins exceeding 50%. Nvidia's business spans hardware sales, software licensing through tools like CUDA and AI Enterprise, and cloud services in partnership with AWS and Azure. The European Union is reportedly examining Nvidia's growing market concentration in AI infrastructure, which could affect pricing and availability in coming years.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds NetPulse, a .NET 8 and Next.js SaaS Boilerplate to Cut Setup Time

A developer at PulseLabs spent two months building NetPulse, an open-source SaaS boilerplate aimed at reducing repetitive project setup from weeks to under five minutes. The template combines a .NET 8 ASP.NET Core backend with a Next.js frontend, using PostgreSQL 16 and Entity Framework Core 8 for data management. A key feature is a multi-gateway billing engine that abstracts Stripe, Polar.sh, and Lemon Squeezy behind a single interface, standardizing subscription and webhook handling. Authentication is handled via ASP.NET Core Identity with JWT tokens and OAuth 2.0 support for Google and GitHub. The entire stack, including the database and a Stripe webhook tunnel, can be launched locally with a single Docker Compose command.

0 comments Read more at DEV Community
AWS Strands Agent Lifts Claude Opus 5 from 30% to 99.95% on ARC-AGI-3 Benchmark · ShortSingh