Why local RAG systems succeed or fail on chunking, retrieval, and index freshness

Self-hosted models such as Ollama, llama.cpp, and vLLM have made private inference affordable, but retrieval-augmented generation still breaks in familiar ways, including poor chunks, stale indexes, and citations to documents the model never used. The article argues that local RAG is mainly a product decision, since chunking, embedding choice, and refresh cadence determine whether users trust the system. It recommends structure-aware chunking, hybrid dense and keyword retrieval for identifiers and proper nouns, and freshness commitments backed by reindexing. It also advises requiring quoted supporting spans and rejecting answers drawn from empty retrieval sets. Local deployment is favored for privacy, predictable cost, and regulated environments, though some teams pair it with cloud models for demanding reasoning tasks.