How a mass IoT device reconnect silently overwhelmed a downstream data pipeline

When thousands of field IoT devices lose connectivity simultaneously, they buffer their readings locally and flush all stored data at once upon reconnecting — creating a massive traffic spike far beyond normal steady-state levels. In one real-world pipeline, this burst passed cleanly through a managed MQTT broker and Kinesis stream, but overwhelmed the Lambda consumer layer, causing iterator age — the measure of how far behind real-time processing has fallen — to climb by tens of minutes. The situation worsened when a single malformed record caused an entire Kinesis shard to stall, since the service enforces strict in-order processing. Instinctive fixes like adding shards or raising Lambda concurrency failed because the true bottleneck was not throughput capacity but the cost of individual database writes to MySQL under sudden load. The incident highlights how pipeline stability is often an assumption tied to arrival rate, not architecture, and that store-and-forward behavior at the edge can become a serious ingest hazard during mass reconnect events.