RAG vs. Full-Context AI Reading: A Developer Tests Both on Real Documents

A developer built an open-source pipeline to compare two AI answering approaches — Retrieval-Augmented Generation (RAG) using BGE-M3 and Qwen3, versus direct full-document context — on a research paper and a full-length book. The pipeline was designed to be deliberately basic, using fixed-size chunking and plain cosine similarity, to expose where standard RAG setups fail before any optimizations are applied. On a SIGUL 2024 paper about English-Nepali legal machine translation, both methods produced accurate answers, though the direct-context approach returned more precise numeric detail. However, when tested on the book 'Hands-On Large Language Models,' the RAG method retrieved irrelevant chunks and returned a completely incorrect summary, while the full-context approach correctly identified the document. The experiment highlights a key weakness of vanilla RAG: retrieval quality is heavily dependent on chunk relevance, and without reranking or smarter chunking, the system can confidently return wrong answers.