AI search growth puts pressure on the web indexes that answer engines depend on
AI-driven search traffic has grown sharply since 2024 and now makes up a small but rising share of website visits, while Google's share of search has slipped below 90 percent. Every AI answer engine draws on an index, and a few companies control most of those indexes, so pages missing from them are effectively invisible. Automated traffic already exceeds half of all web traffic, according to a 2026 bot report. The independent search engine cl0q.com described how a surge of requests from a single scraper fleet caused thousands of server errors in mid-November, prompting it to introduce rate limits and disk caching. The company argues that smaller, independent indexes could reduce the web's dependence on any single company's crawl.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in