Using MinHash and LSH to catch near-duplicate documents before RAG indexing
Document collections often contain near-duplicates, such as re-exported files, pages with altered footers, or revised contract versions, which crowd out varied passages in RAG top-k results. MinHash helps detect them by comparing word-shingle overlap via Jaccard similarity, compressing each document into a short signature that approximates that similarity. LSH banding then narrows comparisons to candidate pairs, avoiding exhaustive checks across large collections. A test showed that two policy versions with changed figures scored about 66% similar, so a loose threshold could wrongly flag a revised version as a duplicate. The author therefore recommends removing true copies while keeping distinct versions, since discarding an updated document could leave outdated information in the index.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.



Discussion (0)
Log in to join the discussion and vote.
Log in