Apache Iceberg Tables Can Become 'Data Landfills' Without Regular Maintenance

Apache Iceberg, a popular open-source table format for data lakes, silently accumulates snapshots, orphaned files, and metadata overhead with every transaction if left unmanaged. Frequent streaming writes or micro-batching create hundreds of tiny files, forcing query engines to spend more effort resolving table state than actually scanning data. Without scheduled compaction and snapshot expiry, metadata layers can balloon to hundreds of thousands of entries, causing Spark out-of-memory errors and skyrocketing cloud storage costs. Engineers are advised to use Iceberg's built-in rewriteDataFiles and expire_snapshots procedures on a regular schedule, targeting file sizes between 128MB and 512MB. A strict snapshot retention policy — typically seven days — alongside a retain_last safety net is recommended to balance performance, cost, and recoverability.