Small S3 files and deep partitions can quietly drive up Athena query costs and latency

The article argues that data lakes built on many tiny files in deeply nested S3 partitions slow Amazon Athena queries and raise costs, since the engine spends more time on metadata calls than on processing data. It illustrates the problem with a logging pipeline that writes thousands of small files per hour under date-based prefixes. The author proposes Apache Iceberg, which tracks files through manifests and supports hidden partitioning such as days(event_time) instead of brittle Hive-style directory paths. The piece also notes a tradeoff between fast ingestion, which produces small files, and query efficiency, which favors buffering into larger Parquet or ORC files of roughly 128MB to 512MB. Compaction through Iceberg's rewrite_data_files procedure is presented as the fix, though it carries its own compute cost.