How one shared staging environment cost nine teams years of wasted debugging
A software engineering team at an unnamed company spent years sharing a single staging environment among nine teams, leading to chronic release delays and misattributed failures. A post-mortem review of eight weeks of staging incidents revealed that 31 of 74 failures were caused by another team's changes, yet were investigated by the wrong team each time. The team solved the problem by building per-pull-request environments using their existing infrastructure-as-code setup, each taking six minutes to spin up and costing under £300 a month in total. A small number of genuinely non-duplicable dependencies, such as a partner test account and a mainframe link, were placed behind an automated pipeline lock instead of a manual booking process. The key takeaway, as noted by engineer Sergey Shinder, was that years of effort had been spent speeding up automation when the real bottleneck was always the shared physical resource that no one had thought to measure.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in