Cursor-based pagination for multi-tenant RAG ingestion at 10k documents
The article argues that ingestion for a multi-tenant help center should use tenant-scoped cursor pagination over a stable snapshot rather than offset pagination, which becomes a correctness and index-cost problem as content grows. The author describes ingesting documents in bounded pages ordered by update time and document ID, then chunking and upserting vectors with deterministic keys so repeated runs stay idempotent. Page size and chunk size are presented as tunable decisions to be measured against the document mix, not fixed constants. Metadata such as tenant, revision, and access controls is kept with each chunk so retrieval can enforce tenant boundaries before ranking. The piece is written from the perspective of a logistics SaaS workload with carrier rules, customs notes, and warehouse procedures.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in