Crawling six docs sites into RAG chunks shows a few pages can dominate the token budget

The author crawled 25 pages from each of six popular documentation sites on 3 November 2026 and split the content into token-counted chunks. Across 150 pages, the crawl produced about 394,000 tokens, and on four of the six sites a single page held between 39% and 69% of the total. Pages such as FastAPI's release notes, Stripe's testing reference and n8n's changelog are rarely useful for a support bot and mostly add near-duplicate chunks to a vector index. FastAPI's breadth-first crawl also spent much of its budget on translated home pages, which can return answers in the wrong language. The author notes that tokens-per-word ratios vary widely between sites, so budgets estimated from word counts can be off by about a factor of two.