Giving AI Coding Agents More Time Does Not Improve Success Rates, Study Finds

·2 views

A benchmark called Real-SWE tested leading AI coding agents against private enterprise codebases covering billing, tax, and multi-service workflows. The study found that extending agent runtime made virtually no difference — failure rates hovered around 71–73% regardless of whether a task ran under or over 10 minutes. The top-performing system, Fable 5.1 paired with Claude Code, resolved only 38.8% of tasks, while GPT-6 Astra on Codex CLI reached 33.8%. Researchers noted that agents tend to fail not due to insufficient compute time, but because of gaps in contextual understanding or inadequate scaffolding. The study also highlighted that benchmark scores reflect a specific model-plus-harness combination, warning that comparing results across different scaffolding setups leads to misleading conclusions.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Run Debian 13 and a Local LLM on Apple Silicon Using Apple's Container CLI

A developer has published a technical guide detailing how to set up a Debian 13 virtual machine using Apple's container CLI on an Apple Silicon Mac, while offloading AI inference to the Mac's GPU via Ollama. Because Linux VMs lack direct Metal GPU access, the guide proposes running the application inside the Debian VM while routing model calls to Ollama running on the host macOS system at 192.168.64.1. A key pitfall documented is that the official Debian 13 image lacks /sbin/init, causing the machine to silently fail to boot — a fix involving a custom Dockerfile with systemd-sysv is provided. The setup was tested on an Apple M3 Mac running macOS 27.0 with container runtime 1.4.1 and Ollama 0.33.3, achieving 45.2 tokens per second with the Gemma 4 model loaded entirely on the GPU. Supporting files and configuration tips have been published on GitHub to help others replicate the workflow.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Solon AI Sandbox Gives Java AI Agents Real OS-Level Filesystem and Network Isolation

Solon AI has released a Java module called solon-ai-sandbox that enforces genuine operating system-level isolation for AI coding agents, rather than relying on prompt instructions. The module is a Java port of Claude Code's sandbox-runtime and works natively on macOS, Linux, and Windows using each platform's built-in sandboxing tools. Unlike container-based solutions, it imposes a narrow security boundary without requiring a container runtime, making it suitable for interactive agents running directly on a developer's machine. Filesystem rules follow opposite defaults for reads and writes — writes are denied by default while reads are permitted unless explicitly restricted — allowing fine-grained control over sensitive paths like SSH keys or cloud credentials. Developers can wrap any agent-issued shell command in a single method call, with platform-specific isolation applied transparently under the hood.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Six practices to make Playwright email-verification tests reliable and trustworthy

Playwright tests can falsely pass email-verification checks by reading stale or mismatched messages from a shared inbox, making green browser assertions insufficient on their own. A developer and technical co-founder of gettemp.email has outlined six practices to improve test reliability, including creating a fresh short-lived inbox per test run and matching messages by subject to avoid false positives. Tests should poll for email delivery with a fixed interval and deadline, validate extracted URLs strictly against the expected hostname, and delete inboxes in a finally block to ensure cleanup even after failures. Playwright trace artifacts should be treated as sensitive since they may capture addresses and verification URLs. The guidance includes a working code example using the MIT-licensed gettemp-email-testing package at v0.2.0, with a noted defect in its Playwright adapter that users are advised to avoid.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Four Key Architecture Patterns Shaping Modern AI Agent Design

AI agents are autonomous systems capable of perceiving their environment, making decisions, and executing multi-step tasks using external tools and APIs. Developers currently rely on four main architectural patterns: ReAct, which loops through reasoning and action; Standard Operating Procedure (SOP), which follows predefined decision trees for consistent outcomes; Reflection, where agents self-critique and revise their outputs; and Multi-Agent, where specialized agents collaborate on large-scale tasks. Each pattern suits different use cases, ranging from complex reasoning to repetitive workflows and quality-critical applications. Experts anticipate that future AI agents will feature more advanced planning, better tool integration, and improved memory and collaboration capabilities.

0 comments Read more at DEV Community