SetrixDB: Go-based set engine delivers microsecond ID intersection with AVX-512

·4 views

A developer has built SetrixDB, an open-source set engine written in Go, designed to perform exact set intersections over uint64 identifiers at microsecond speeds. The engine uses a Minimal Perfect Hash Function (CHD v2) to eliminate key collisions, achieving 0 collisions across 50 million keys at just 0.5 bytes per key — far more memory-efficient than a standard Go map at 22.3 bytes per key. Intersection operations are accelerated using AVX-512 SIMD instructions via cgo, with a scalar fallback for broader hardware compatibility. Benchmarks conducted on a 2-vCPU AMD EPYC (Zen4) server show bitset AND operations completing in as little as 6 microseconds with AVX-512, compared to 91.6 milliseconds for a hash join approach. The author acknowledges that SetrixDB underperforms Roaring bitmaps in sparse, large-universe scenarios where memory is constrained, but outperforms them significantly for dense ID sets with random 64-bit identifiers.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Secure Kubernetes Services Using Gateway API, Traefik, and OAuth2 Proxy

A technical guide details how to rebuild Kubernetes service authentication using Gateway API, Traefik, OAuth2 Proxy, and Pocket ID, replacing the older ingress-nginx annotation-based approach. The migration is prompted by ingress-nginx being retired, with Kubernetes now recommending Gateway API for traffic management. The setup exposes three HTTPS hostnames under a single domain, allowing OAuth2 Proxy to manage session cookies with a narrow, scoped domain. Traefik's Middleware CRD handles browser-based OIDC login and external authentication filtering, since Gateway API does not standardize these functions natively. The guide was validated on a local K3s cluster and uses standard Kubernetes and Helm commands, making it adaptable to other environments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

SetrixDB: Go-based set engine uses MPHF and AVX-512 for exact ID intersection

A developer has built SetrixDB, an open-source set engine written in Go, designed to perform exact membership checks and intersections over uint64 identifiers at microsecond speeds. The engine uses a Minimal Perfect Hash Function (CHD v2) to map terms to collision-free uint64 IDs, achieving zero collisions across 50 million keys while consuming just 0.5 bytes per key — roughly 44 times less memory than a standard Go map. Intersection operations are accelerated using AVX-512 SIMD instructions via cgo, with runtime dispatch and a scalar fallback for broader hardware compatibility. Benchmarks show the bitset AND kernel completing in around 6 microseconds with AVX-512, outperforming hash joins and sorted merges by a wide margin for dense ID sets. The author notes clear trade-offs: for sparse or randomly distributed 64-bit ID universes too large to fit in RAM, compressed bitmap libraries like Roaring64 remain more memory-efficient alternatives.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Tanzanian Self-Taught Developer Seeks Community Among African Coders

A young self-taught developer from Tanzania has shared his coding journey, highlighting that he learned programming through AI tools, YouTube, documentation, and personal projects rather than formal education. He describes himself as a beginner who still encounters everyday challenges but finds genuine enjoyment in the learning process. His primary motivation for posting was the sense of isolation that comes with learning independently, particularly on platforms like GitHub. He is actively seeking connections with fellow beginner and intermediate developers across Tanzania, Kenya, Uganda, Rwanda, and the broader African continent. His goal is to build a informal peer network where developers can learn, experiment, and build projects together.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Decrypted 5G NR trace reveals dual security layers in a real attach session

A publicly shared 5G NR packet capture, released alongside its decryption keys as part of Wireshark work-item 19757, offers a rare look at genuine radio security in action. The 151-frame trace, spanning roughly 49 seconds, uses real AES-based ciphering and integrity protection rather than the null cipher common in lab captures. The session shows two separate security procedures running in parallel: one between the device and the core network, and another between the device and the radio base station, each using different keys and peers. A single frame illustrates both states simultaneously, with the radio layer decrypted and readable while the core network payload remains an opaque encrypted blob. A third security layer, IPsec protecting IMS signaling within the user plane, adds further complexity beneath the 5G stack.

0 comments Read more at DEV Community