Routing LLM Jobs by Difficulty Can Cut Product Catalog Processing Costs

A software developer working on RAG and agent features in Python has outlined a cost-reduction strategy for using large language models to process product catalogs. The approach centers on measuring cost per accepted record rather than per token, then routing each task to a small or large model based on complexity. A token-counting gate determines whether a description fits within budget before a model call is made, with oversized inputs flagged for markup stripping, splitting, or manual review. Results are logged in an append-only ledger that tracks tenant ID, model route, token counts, validation status, and retry counts to ensure rejected outputs remain visible and accountable. The method aims to prevent cheap models from appearing more accurate simply because failed or invalid outputs go unrecorded in cost reports.