Experiment finds AI coding agents mostly follow long rules files, with two key exceptions
A developer ran a controlled experiment testing whether Claude AI follows a 1,350-line project rules file, executing 30 runs across six configurations including one with the file fully removed. Out of 400 rule checks with the file present, 399 passed, suggesting the model does not simply ignore lengthy instruction files. Seventeen of twenty rules were followed perfectly across all configurations, even when no rules file existed. The two rules that failed did so because the repository's own code and changelog history contradicted them, meaning the model followed observable project conventions over written instructions. The experiment also revealed that deleting the rules file from the working directory did not eliminate its influence, as the model recovered it from git history or other memory sources.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in