SWE-Bench Study Shows Top LLMs Resolve Under 2% of Real GitHub Issues

·1 views

A developer series on DEV Community highlights findings from a Princeton ArXiv study showing that even state-of-the-art large language models struggle to resolve real-world GitHub issues on the SWE-bench benchmark. Claude 2, the top-performing model tested, managed to fix only 1.96% of issues, with all models limited to the simplest problem types. The study reveals that current LLMs break down when tasks require multi-file reasoning, dependency awareness, or systems-level understanding. In response, the author is developing a hybrid multi-persona AI approach targeting medium-complexity issues — the tier where existing models consistently fail. The project uses three specialized reasoning personas covering evaluation, systems analysis, and agentic patch strategies, with further details promised in upcoming posts.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Builds Group Expense App Where the Algorithm Proved Harder Than the UI

A developer created Pay Splitter, a group expense management app designed to simplify cost-sharing after trips or shared outings. The core challenge was not building the interface but designing a settlement algorithm that minimizes the number of transactions between participants. Rather than generating multiple individual payments, the algorithm matches debtors with creditors to consolidate transfers while ensuring all balances are settled correctly. The project also involved handling additional complexities such as multiple currency support. The developer continues to refine the app based on user feedback and has made it available on both web and Android platforms.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

A Practical Checklist for Debugging Common JWT Errors Fast

JSON Web Token (JWT) errors such as malformed tokens, invalid signatures, and expiration failures are among the most common authentication issues developers face. Debugging them effectively requires isolating token structure, signature verification, and claim validation rather than making random configuration changes. A malformed JWT often stems from simple issues like a missing Bearer prefix strip, extra whitespace, or a truncated environment variable rather than cryptographic problems. Signature mismatches typically occur when the verifier and issuer disagree on the algorithm, key, or how the secret is encoded. Beyond signature checks, claims like exp, nbf, iss, and aud must also be explicitly validated on the backend to ensure tokens are fully trustworthy.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Google Launches Gemma 2 Open Models With Competitive Performance at Lower Cost

Google has released Gemma 2, its next-generation open AI model family, available in 9 billion and 27 billion parameter sizes. The 27B model is built on a redesigned architecture featuring a hybrid attention mechanism and is capable of running on a single NVIDIA H100 or A100 80GB GPU, significantly reducing deployment costs. Google claims the 27B version delivers performance competitive with proprietary models more than twice its size, while the 9B model reportedly outperforms rivals like Meta's Llama 3 8B. The models are accessible via Hugging Face, Kaggle, and Google AI Studio, with support for PyTorch, JAX, and TensorFlow frameworks. Google also announced plans to open-source its SynthID text watermarking technology for Gemma models, adding a layer of AI safety and content provenance tooling.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Study finds zero of 2,204 AI-authored GitHub PRs declared machine-readable scope

A developer scanned 2,204 recently merged pull requests authored by AI coding agents — including Devin, Copilot, Codex, Claude Code, and Cursor — on public GitHub repositories. Using a deterministic, checkout-free policy engine that reads only PR metadata and file contents via the GitHub API, the analysis found that not a single PR included a machine-readable declaration of its intended scope. Around 7% of fully analyzed PRs triggered at least one boundary finding, with workflow-touching PRs showing the highest risk concentration, including unpinned actions and escalated permissions. Notably, 3.9% of PRs modified agent control-plane files such as AGENTS.md or CLAUDE.md, which effectively shape the behavior of all future agent PRs in a repository. The author argues that since AI agents generate rich task context at creation time, a simple standardized scope contract embedded in PR bodies could make intent machine-verifiable rather than leaving reviewers to guess.

0 comments Read more at DEV Community