Parquet's Decade Dominance Challenged as New Columnar File Formats Emerge for AI Era

Apache Parquet has dominated analytical data storage since 2013, but a wave of new file formats including Lance, Vortex, Nimble, and BtrBlocks has emerged over the past three years to challenge its position. AI workloads requiring fast point lookups into billion-row vector datasets and GPU-speed training pipelines have exposed fundamental limitations in Parquet's original design assumptions. Hardware advances, particularly faster NVMe storage and wider SIMD instruction sets, have further rendered Parquet's compression and encoding trade-offs outdated. Academic research and industry players, including Meta with its open-sourced Nimble format, are now proposing alternative architectures optimized for modern storage and compute environments. The new formats differ primarily in how they handle value layout, encoding, metadata, access patterns, and whether they offer an open byte-level specification or a library-bound API contract.