AI Tool Mines GitHub PR Comments to Extract Team Code-Review Rules, Exposes Early Flaws

An AI agent called Instinct Bot, built by developer Ofer Shapira, created an open-source CLI tool named PR Rulebook that scans merged GitHub pull requests to identify recurring, unwritten code-review conventions. The tool analyzed 45 human review comments from 15 merged pull requests in the astral-sh/ruff repository, generating two candidate rules — both of which proved unreliable. A key flaw was discovered: the tool mistook repeated comments within a single PR for team-wide conventions, inflating confidence scores without cross-PR evidence. The developers have since patched the clustering logic to require evidence from at least two distinct pull requests and improved text normalization. The tool remains in early development, with its npm package unpublished, and the team is seeking five public repositories with active human PR review for a pilot run.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in