How High-Availability Systems Are Designed to Survive Failure, Not Prevent It

Modern high-availability software is built on the assumption that failure is inevitable, treating it as a routine, recoverable event rather than an outage. Simply running duplicate nodes is insufficient, as it risks 'split-brain' scenarios where two nodes independently accept conflicting writes, requiring consensus protocols like Raft or Paxos — used by tools such as etcd and Consul — to establish a single agreed-upon leader. Systems must also define explicit fallback behaviour when dependencies become unavailable, rather than allowing requests to hang indefinitely; circuit breakers address this by halting calls to repeatedly failing services and failing fast instead. Rolling deployments help avoid downtime during updates, but introduce a window where old and new code versions operate simultaneously on shared data, making backward-compatible schema changes essential. The broader principle is that resilience requires deliberate upfront design decisions about failure modes, not reactive fixes discovered during post-incident reviews.