How HDFS Erasure Coding Cuts Storage Overhead From 200% Down to 50%

HDFS traditionally replicates each data block across three machines, adding 200% storage overhead for every terabyte of useful data. Erasure coding offers the same fault tolerance at a fraction of the cost by splitting data into k fragments and computing m parity fragments using finite-field algebra. Based on Reed-Solomon mathematics published in 1960, any k of the total k+m fragments are sufficient to reconstruct the original file, regardless of which fragments are lost. This approach is already used in systems like HDFS, S3-style object storage, and RAID 6 arrays to survive disk failures without duplicating every byte. Choosing the right k and m values for a given infrastructure allows engineers to reduce storage overhead to roughly 50% while maintaining robust fault tolerance.