IBM's Chunkless RAG Challenged: Does Ditching Document Chunks Actually Help?

IBM has been promoting a 'Chunkless RAG' approach that uses AI agents to navigate document structure — preserving headings, tables, and hierarchies — instead of splitting text into fixed-size chunks for embedding. Critics argue the method is overhyped, noting that real-world document corpora — including scanned PDFs, legal contracts, and Slack exports — are too messy for reliable structure parsing. When a parser fails to extract a clean document tree, the navigating agent simply traverses noisy data with added latency, swapping one failure mode for another that is harder to detect. Analysts also point out that the core retrieval problem in most pipelines is poor query-to-passage semantic overlap, which Chunkless RAG does not inherently solve. Techniques like hypothetical document embeddings, query expansion, and hybrid BM25-plus-dense-vector retrieval are seen as more reliably moving the needle on retrieval precision.