LioranDB Uses LSM-Based Secondary Indexes to Separate Write and Index Paths

LioranDB, a document database developed by Swaraj Puppalwar at Lioran Group, implements secondary indexing using an LSM-style architecture to handle queries beyond simple document ID lookups. When a document is updated, the system generates index mutations that are written to a log and applied asynchronously through a pipeline ending in compaction. The design converts random index updates into sequential batches using immutable sorted segments, reducing the cost of repeated rewrites. LioranDB assigns separate worker groups for secondary indexes, text indexes, flushing, and compaction, since text mutations are significantly more expensive than secondary ones. To manage the lag introduced by asynchronous indexing, the system monitors pending mutations and compaction debt, applying backpressure when hard limits are reached to prevent the write pipeline from outpacing indexing.