Twin-run spike: testing whether a task suits an AI agent in 90 minutes

·2 views

A DEV Community post proposes a 90-minute protocol to judge whether a coding task is suitable for an AI agent, by running two independent agent attempts from the same base commit. The core idea is that if the two runs diverge sharply, the weakness lies in the task description rather than the model. The method compares which code regions each run changes using a hunk-overlap score, and gates the outcome on whether both branches pass the existing test suite. A decision table maps overlap and test results to actions such as shipping, narrowing the task, or killing it. The author also advises recording token usage and hunks produced per run, and warns about pitfalls like shared base drift, passing tests that hide wrong fixes, and mid-run prompt edits.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Claude Code Action Can Automatically Review GitHub Pull Requests

The anthropics/claude-code-action GitHub Action lets Claude review pull requests and answer @claude mentions in issues without a dedicated server, running on the user's own GitHub runner. It selects its behavior from the workflow context, handling code questions, PR reviews, and simple fixes, and sends API calls to a chosen provider such as Anthropic, Amazon Bedrock, Google Vertex AI, or Microsoft Foundry. Setup is done by running claude /install-github-app from a repo terminal, which requires repository admin rights and is documented for direct Anthropic API users, with other providers covered in the repo's cloud-providers guide. The project's Solutions Guide lists review patterns such as automatic PR review, path-specific checks, and OWASP-aligned security reviews, and the article recommends starting small and limiting tool access. The author stresses that the AI review should be treated as a first pass rather than a replacement for human approval, and notes they did not run the action on a live repository.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

A 65B Local Model Outscored a 501B Model on Identical Eval Questions

A developer reported that a 65-billion-parameter model run locally scored 80.52 on their evaluation suite, higher than a 501-billion-parameter frontier model on the same test. Both models answered the same 200 multiple-choice questions at temperature 0 with no system-prompt changes, using the same scoring script. The local model was quantized to Q4_K_M and ran on a single consumer GPU, while the larger model was accessed through its hosted API with default settings. The smaller model reportedly won 14 of 20 categories. The author noted that verifying the larger model's result costs about twelve dollars per full run, compared with roughly forty minutes of GPU time and electricity for the local run, and shared a short video walkthrough of the setup.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developers Explore Running Local LLMs in Browsers Using WebGPU and Transformers.js

A technical article from DEV Community describes how WebGPU gives browsers direct access to local GPU compute, enabling quantized open-weight models to run client-side. It names Qwen 2.5, Phi-3.5, and Llama 3.2 as examples of models that can execute within the browser sandbox. The author highlights a code example using the Transformers.js pipeline API and notes challenges such as shader compilation latency during initial load. Caching weights through the browser's Cache Storage API is suggested so models are downloaded only once. The piece argues that keeping prompts on the device avoids server round trips and improves data privacy for sensitive information.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Deduplizieren einer Kundenliste in Python, wenn keine Zeilen exakt identisch sind

Doppelte Einträge in Kundenlisten sind selten identische Zeilen, weil sich etwa das Bestelldatum, die Stadt oder die Schreibweise der E-Mail unterscheiden. Welche Zeilen als Duplikate gelten, hängt daher vom gewählten Vergleichsschlüssel ab, der bestimmt, wie viele Zeilen entfallen und welche erhalten bleiben. Der Artikel zeigt dies anhand einer Beispielliste mit 412 Zeilen in reinem Python und vergleicht die Ergebnisse mit dem Befehl Remove Duplicates im Spreadsheet Editor von osFoundry. Ein Schlüssel über die ganze Zeile entfernt keine Einträge, die E-Mail allein entfernt alle 38 Wiederholungen, und E-Mail plus Stadt entfernt 25. Zudem behält die übliche Konvention die erste Zeile, was bei den meisten Dubletten nicht die neueste Bestellung ist, weshalb die Sortierung vor dem Deduplizieren wichtig sein kann.

0 comments Read more at DEV Community