Study finds AI code agents ignore existing repo context more than they hallucinate

·1 views

A pre-registered study examined three merged GitHub Copilot pull requests from major .NET organizations using a 12-reviewer automated pipeline called review-pro. The researcher, who maintains the tool, set out to test whether AI-authored code primarily fails through hallucination — such as invented APIs or undefined config keys — as is commonly assumed. Instead, the dominant failure pattern found was agents overlooking knowledge already present in the repository, rather than fabricating nonexistent elements. The pipeline required each specialist reviewer to locate evidence within the repository before making any claim, making unsupported assertions inadmissible. The study's methodology, corpus criteria, and per-case records were pre-registered before results were analyzed, and negative findings were included in the published report.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Orca Orchestrator Ditches Secret Managers, Stores Encrypted Secrets in Git

Orca, a lightweight single-binary orchestrator, has replaced its managed Infisical sidecar with a dependency-free secrets solution using SOPS and the age encryption tool. The previous approach created a bootstrap problem: the secret manager was itself a service requiring deployment, credentials, and uptime, undermining Orca's promise of no hidden dependencies. Secrets are now stored as an encrypted JSON file committed directly to the config repository, with only values encrypted and keys left in plaintext, enabling readable git diffs during code review. Decryption happens in-process via a Rust library at runtime, with no external binaries invoked, and multi-recipient age keys allow local recovery independent of whether the orchestrator is running. The design keeps the git repository as the single source of truth, with the master automatically committing and pushing re-encrypted files after any secret update.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

ML Neuron Basics Explained Through a Pizza Store Analogy

A developer tutorial published on DEV Community uses a pizza store scenario to explain the foundational mechanics of a machine learning neuron. In the analogy, the number of customers represents input, the two-pizzas-per-person rule represents the model's weight, and the gap between pizzas aimed to sell and actually sold represents the error. The core training loop involves calculating an adjustment by multiplying the error by the input, then updating the weight using a learning rate. This mirrors gradient descent, where the weight is iteratively corrected to minimize error. The tutorial is aimed at beginners who have some basic familiarity with neural networks but struggle with the underlying math.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How pytest module-scoped fixtures silently share mutable state across tests

A technical article on DEV Community highlights a common pytest pitfall where fixtures set to 'scope=module' share the same object across all tests in a module rather than providing each test its own copy. This means if one test mutates a shared dict or object, subsequent tests receive the already-modified version, causing hard-to-trace failures that only appear when tests run in certain orders. The author recommends three fixes: dropping the scope to the default function level, keeping expensive shared setup while handing each test a deep copy, or using an immutable type like MappingProxyType so mutations raise an immediate error. The article also suggests installing the pytest-randomly plugin to shuffle test execution order in CI, which helps surface order-dependent failures early. The core advice is to treat scope widening as a performance optimisation — only justified when the shared object is proven to be read-only.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Codename One shifts from Maven Central to self-hosted Cloudflare R2 repository

Codename One, an open-source Java/Kotlin framework for cross-platform app development, is migrating its package distribution from Maven Central to a self-operated repository hosted on Cloudflare R2. The move is driven by the project's unusually high usage of Maven Central infrastructure, exceeding Sonatype's guidelines by over 2,600% in storage and nearly 2,000% in file count. A phased rollout begins July 31, 2026, with dual publishing to both platforms, followed by new projects automatically receiving the new repository URL from August 7. Codename One plans to stop publishing new versions to Maven Central entirely by August 28, 2026, provided the dual-publish period runs smoothly. Existing projects will need to manually add the new repository configuration to continue receiving future updates after the cutover.

0 comments Read more at DEV Community