RAG System Silently Indexed Only 62% of Corpus Due to NFS Blind Spot
A developer discovered that their self-hosted RAG knowledge base had only indexed 62% of its intended corpus, despite all monitoring dashboards showing healthy green statuses. The root cause was the indexer's reliance on inotify filesystem events, which do not propagate across NFS mounts, meaning documents written from remote machines were never detected. Because the system always returns an answer from whatever it has indexed, the coverage gap went unnoticed — the failure was in ingestion, not in the language model itself. The developer also highlighted a broader measurement pitfall: reporting only documents successfully indexed without defining the total eligible corpus turns a metric into a misleading figure. The fix involved replacing event-driven watching with periodic polling using stat(), which works across network filesystems by actively querying rather than waiting for notifications.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in