Go RAG Citations Point to Wrong Page: Fixing Chunk Offset Metadata
A citation is only trustworthy when its text, byte offsets, page label, and document version all come from the same parse of a source. The article argues that ingestion should reject any run that cannot keep this lineage intact, rather than patching page numbers after retrieval. A common failure occurs when a chunker works on normalized text while a separate utility computes offsets from raw extraction, so small differences like removed headers or ligatures cause citations to drift silently. The recommended fix is to re-parse each document version once, derive all chunk and citation metadata from that parse, store the version with every chunk, and atomically swap indexes while keeping the previous version for rollback. Because this is a corpus-wide transformation rather than a minor metadata change, rollout capacity should account for re-parsing, validation, vector writes and retained old versions.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in