How a 155 KiB row silently broke writes in CockroachDB despite healthy infrastructure
A collaborative education platform experienced write failures on a production worksheet despite all system metrics appearing normal, with healthy pods, low CPU, and no database crashes. The root cause traced back to CockroachDB's MVCC architecture, which retains every historical version of a row under a timestamped key rather than overwriting data in place. The app was storing entire CRDT document snapshots as a single BYTEA column on every save, causing one document's storage range to balloon to 1 GiB. This triggered a backpressure error when CockroachDB could not find a valid split key for the oversized range, silently blocking further writes. Two unrelated issues — a CPU spike and flapping readiness probes — masked the real error, which was buried at low volume in the logs the entire time.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in