robots.txt AI crawler blocks often fail due to RFC 9309 rule many miss

A common robots.txt configuration error causes AI crawlers like GPTBot to ignore wildcard Disallow rules entirely, because RFC 9309 specifies that a bot matching a named group uses only that group and not the wildcard fallback. A developer discovered this flaw while cataloguing 42 open-source projects aimed at helping ecommerce stores — specifically Magento — become more visible and transactable within AI systems. The project list revealed that llms.txt discovery files attracted 11 implementations, while harder layers like crawler policy received only 3 each, reflecting how the ecosystem tends to tackle easier problems first. OpenAI's in-chat Instant Checkout, which launched with promises of over a million merchants, had pulled back significantly by March 2026, with live merchants estimated between a dozen and thirty. Experts suggest that product feed integration remains the practical near-term path to AI visibility, while agentic checkout protocols are still maturing and not yet revenue-ready.