Standby database received all changes but applied none, delaying a planned switchover by 41 minutes
During a planned two-minute switchover to patch a primary Postgres server's kernel, the promoted standby took 41 minutes to begin accepting writes, halting every service that writes data. The standby also served finance reports, and a setting that told it to wait indefinitely before cancelling conflicting queries let replay stall for about five hours. The standby kept receiving changes and saving them to disk without applying them. Promotion must replay everything received before accepting writes, so the backlog caused the delay. The team's pre-switchover check compared the wrong position, which showed the standby as current, and they have since separated reporting replicas from the failover candidate and added replay-based lag alerts and checks.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in