samkhya Unifies Query Statistics Across DuckDB, DataFusion, Polars, and Postgres

Modern data platforms often run multiple query engines — such as DuckDB, DataFusion, Polars, and Postgres — against the same underlying tables, but each engine computes and stores its own statistics independently. This fragmentation means expensive histograms and sketches built during an ELT pipeline are invisible to other engines' query optimizers, leading to poor query plans and wasted computation. An open-source tool called samkhya addresses this by serializing statistical sketches into versioned Iceberg Puffin sidecar files that any compatible engine adapter can read without re-deriving the data. The shared statistics include HyperLogLog distinct-count sketches, Bloom and Count-Min filters, equi-depth histograms, and 2D correlated histograms for join-key pairs. To guard against stale or incorrect sketches, samkhya applies a provable pessimistic bound ensuring that bad statistics can never cause a query plan to perform worse than engine defaults.