How ORC Files Achieve Speed: A Developer Breaks Down the Format From Scratch

A developer built a small ORC file parser from scratch and generated 5 million rows of synthetic order data to investigate exactly how the format handles selective column reads. The core mechanism relies on three principles: storing column values contiguously, encoding values using neighbour-similarity compression, and maintaining a compact file map so queries can skip irrelevant sections entirely. An ORC file begins with a PostScript — never compressed — that tells the query engine the codec, footer length, and metadata location, enabling schema and stripe-level statistics to be retrieved in just three reads before any row data is accessed. Each stripe is a self-contained unit with its own index, data, and footer, allowing parallel processing by distributed engines like Spark with no cross-stripe dependency. Within a stripe, each column is broken into typed streams stored contiguously, meaning reading one column requires a single range read rather than scattered seeks across the file.