Filter First, Allocate Later: Why Selective Attention Beats Truncation

A technical analysis argues that when managing token budgets in AI systems, filtering irrelevant content before allocating space is far more effective than truncating documents to fit. Using a 30,000-token budget example, the piece shows that filtering down to three relevant complete documents outperforms cramming ten truncated ones, while using the same token count. The core principle is to prefer fewer whole documents over many partial ones, since truncated documents lose their conclusions and become unreliable. A proper filter requires three components: a scoring function, a relevance threshold, and a cap on the number of admitted documents. The article also provides a cost-benefit formula to determine when running a filter model is economically justified compared to simply passing all candidates to the main model.