How Apache Iceberg's Puffin Format Solves Query Optimization Gaps in Large Tables

Apache Iceberg's Puffin is a lightweight container file format designed to store statistics that do not fit inside Iceberg's standard manifest files. Manifests efficiently hold per-file scalar metrics like row counts and value bounds, but cannot accommodate larger structures such as distinct-value sketches or row-level delete bitmaps. Without distinct-value estimates, query optimizers are forced to guess join strategies, potentially causing unnecessary shuffling of terabytes of data. Puffin addresses this by housing two blob types: Theta sketches for estimating distinct column values, and deletion vectors for tracking row-level deletes introduced in Iceberg format version 3. The format uses a simple structure — a magic number, opaque data blobs, and a JSON footer — allowing query engines to access costly statistics without bloating the manifests read on every query.