Hidden Network Policy Caused 503 Errors That Four Teams Failed to Catch Together

A misconfigured NetworkPolicy in a separate infrastructure repository blocked a newly deployed pod from reaching the tax-rate-cache service, causing 11% of invoice-page requests to fail with 503 errors. All individual team dashboards appeared healthy — the container was running, the database migration had succeeded, and feature flags were enabled — but no single team had visibility across the full release chain. By the time the three critical facts were shared in one incident channel, 63 deployment messages had obscured the relevant one. In response, the team began consolidating release dependencies — including network policies, migrations, feature flags, and smoke tests — into the service repository itself. The goal is to make handoff requirements explicit and machine-checkable before code reaches production, rather than relying on scattered runbooks and informal contracts across teams.