AI Code Review Test Reveals Which Tools Actually Read Your Rules Files

A developer designed a controlled experiment to determine whether AI code review tools genuinely read project-specific rule files like CLAUDE.md and AGENTS.md, or simply rely on general best practices. Two tools, CodeRabbit and Qodo, were tested against a small Cloudflare Worker API that deliberately violated two rules: one widely-known security bad practice and one obscure internal naming convention. Both tools flagged the security violation, but that test proved little since any standard scanner would catch it regardless of file access. Only Qodo successfully identified the camelCase route naming violation — a purely internal convention with no external reference — and traced its functional impact on real requests. The experiment highlights a meaningful gap in AI code review reliability when enforcing custom, repo-specific conventions at scale.