SetrixDB: Go-based set engine uses MPHF and AVX-512 for exact ID intersection
A developer has built SetrixDB, an open-source set engine written in Go, designed to perform exact membership checks and intersections over uint64 identifiers at microsecond speeds. The engine uses a Minimal Perfect Hash Function (CHD v2) to map terms to collision-free uint64 IDs, achieving zero collisions across 50 million keys while consuming just 0.5 bytes per key — roughly 44 times less memory than a standard Go map. Intersection operations are accelerated using AVX-512 SIMD instructions via cgo, with runtime dispatch and a scalar fallback for broader hardware compatibility. Benchmarks show the bitset AND kernel completing in around 6 microseconds with AVX-512, outperforming hash joins and sorted merges by a wide margin for dense ID sets. The author notes clear trade-offs: for sparse or randomly distributed 64-bit ID universes too large to fit in RAM, compressed bitmap libraries like Roaring64 remain more memory-efficient alternatives.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in