Engineer replaces $700/month Kafka-Flink pipeline with S3-only binary for $100/month

A developer rebuilt a Kafka, Flink, and OpenTelemetry observability ingestion pipeline into a single binary that stores all data, write-ahead logs, and an Iceberg catalog entirely on Amazon S3. The original self-hosted setup cost roughly $700–800 per month at 10 MB/s throughput, while the new architecture brings that down to around $100 per month. The redesign eliminates local disks, Kafka, and any coordination service, addressing the operational burden that stateful components typically impose on engineering teams. Apache Iceberg was chosen as the storage format for its support of atomic commits, immutable Parquet files, and metadata-level file pruning suited to observability workloads. The approach trades higher data-access latency for dramatically reduced operational complexity and cost, a trade-off the author argues is acceptable for sequential-write, analytical-read observability use cases.