Developer Runs 6-Week AI Code Review Experiment Pitting Claude Against Gemini

·1 views

A developer built a multi-agent code review pipeline using Claude and Gemini as parallel adversarial reviewers on GitHub Actions, tracking results across 95 pull requests over six weeks on a real multi-tenant codebase. A key discovery was that Gemini had inadvertently configured its own GitHub Action to only read diffs, not the full repository — a limitation it set up itself. Once given full repository access, Gemini's reviews became less reliable, shifting from honest uncertainty to confidently fabricating file names, functions, and API calls that did not exist. Over the trial, both reviewers together caught 200 real issues with zero escapes and zero false consensus when they agreed, but their findings overlapped only about 12 percent of the time. Claude's findings were accurate roughly 81 percent of the time, while Gemini's accuracy stood at approximately 45 percent, even after excluding the weeks it was reviewing without repository access.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Watching AI Agents Step-by-Step Is a Design Failure, Not Governance

Most engineering teams in 2026 are deploying AI agents with heavy, continuous human oversight, requiring more attention than the manual processes they were meant to replace. A growing argument in the developer community holds that this approach reflects a fundamental design failure, not responsible governance. The proposed alternative is an exception-based monitoring model, where teams define measurable outcomes, constraints, and escalation thresholds upfront, then only intervene when those conditions are breached — similar to how CI/CD pipelines and observability platforms already operate. The core shift is to treat AI agents as process executors whose outcomes need verifying, rather than autonomous decision-makers whose every step needs approval. Critics of this model acknowledge that the real challenge lies in the upfront work of defining precise, automatable success criteria — an investment most teams currently avoid by defaulting to dense oversight instead.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds AI Quiz Tool That Verifies Every Question Against Source Text

A developer created QuizPaste after repeatedly finding that popular AI quiz generators produced questions based on facts not present in the source material he provided. Unlike conventional tools that simply prompt a language model to generate questions, QuizPaste requires the model to cite the specific sentence from the user's text that each question is based on. The system then programmatically checks whether that sentence actually exists in the original input, discarding any question it cannot verify before the user ever sees it. Users can paste notes or YouTube transcripts and receive a quiz with flashcards in roughly five seconds, with no account required for the first two daily sessions. The tool, launched about four days before the article was written, supports export to Anki-compatible TSV and CSV formats and is free to try at quizpaste.com.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Passing Browser Tests Can Still Hide Critical Application Failures

Browser tests can show green results while masking real failures caused by environment mismatches, outdated mocks, or undocumented feature flags. A test may pass locally but fail in staging simply because a feature flag altered the component tree or UI flow without changing the URL. Experts recommend logging the exact configuration state — including active flags, service versions, and seed data — alongside every test run rather than relying on generic environment labels. Adding a lightweight environment verification phase before the main test suite can catch setup issues early and prevent teams from wasting time debugging phantom failures. The core problem is that reliable browser testing demands proof that the environment, application state, and execution path match what the team actually intends to test.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why SaaS Testing Must Cover Cross-Tab, Multi-Tenant, and Email Workflows

Modern SaaS applications involve complex user journeys that span multiple browser tabs, OAuth popups, MFA prompts, email verification links, and regional data boundaries, making isolated page-level tests insufficient. A breakdown at any handoff point — such as a failed token exchange or a misrouted callback URL — can cause the entire workflow to fail even when individual steps pass. Tenant isolation testing is equally critical, requiring verification that users cannot access another organization's data through direct URLs, API calls, or browser navigation tricks. Multi-step sign-up flows further complicate coverage by involving external systems like email providers, billing platforms, and fraud checks beyond the visible form. Thorough SaaS QA must model authentication as a state machine and systematically test failure paths across all integrated systems and communication channels.

0 comments Read more at DEV Community