Key takeaways
- In 99 of 184 documented outages, redundancy existed – and failed anyway. N+1 on paper is not N+1 in practice.
- The most common documented failure mode is boring: transfer switches (ATS/STS) that did not transfer when utility power dropped – followed by UPS batteries exhausted under load, and cooling that could not restart after a power event.
- Root causes by category: electrical 26%, network 22%, software/control 20%, fire 15%, cooling 14% – the classic causes are stable, but every rising category is software and control.
- Evidence base: 506 public signals filtered to 184 documented incidents across 30 countries (2003–2026), 59 with full post-mortems, median outage ~7 hours. Full sourced report attached.
Every data center sells N+1. And every outage post-mortem starts the same way: “despite redundant systems…” I wanted to know what actually happens in that gap between the promise and the failure. So we compiled 184 documented outages – 506 public signals, 59 high-impact events with full post-mortems – and asked one question: when redundancy fails, how exactly does it fail?
The uncomfortable headline: in 99 of the 184 documented outages – 53.8% – the systems that were supposed to provide backup were present and still failed. Redundancy did not save these facilities; it gave them a false sense of safety.
The most common documented failure modes are unglamorous.
| Failure mode | Typical chain | Trend |
|---|---|---|
| ATS/STS failed to transfer | utility power drops → transfer switch does not switch → IT load loses power | stable |
| UPS/battery exhausted under load | power fails → batteries degrade or run out before generators pick up | rising |
| Cooling could not restart/ride through | power blip → chillers fail to restart → thermal shutdown | rising |
| BMS/control dependency | one control system fails → both “independent” paths die together | rising |
| Maintenance/switching error | routine work → redundancy accidentally taken offline | stable |
| Common-mode design | one shared component (switchgear, fuel, control) → everything shares the failure | falling |
| Network config / BGP change | one configuration push → global outage | rising |
Look at the real incidents and a pattern emerges. In the OVH SBG2 fire (2021), there was no automatic extinguishing system in the battery/UPS rooms and the building design let the fire spread – a 43-day recovery. At Cloudflare (Nov 2023), generators could not be restarted manually because the access-control system had also lost power – the security system killed the security. At Equinix LD8 (2020), “A+B” feeds that were supposed to be independent were routed through a single UPS. Different companies, same lesson: in each of these cases, the redundancy appears to have been on the diagram, not in reality.
One more finding worth sitting with: “human error” appears in 182 of 184 incidents, but that number is misleading. Humans rarely cause outages out of nowhere – they trip a trap that bad design and weak process already built. The maintenance mistake that takes down a facility is only possible because there was no interlock, no isolation, no second check. Blaming the technician is easier than fixing the system that made the mistake fatal.
The trend line is clear. The classic physical causes – power, cooling, fire – are stable or falling as engineering matures. Everything that is rising is software and control: BMS dependencies, configuration errors, BGP changes that propagate globally in seconds. The modern data center increasingly fails not from hardware, but from code.
A note on method, because averages lie and vendor uptime claims lie more: every incident in this dataset is tied to a public, verifiable source, with confidence ratings where evidence was thin. The data skews toward large, well-documented operators and English-language reports – documented evidence follows the incidents that get published. This is not a complete census of every outage; it is an evidence base of the ones we can actually verify.
📄 Full research report (184 incidents, all sources): Download the PDF
When did your last failover test actually run – and did the switch actually transfer? 👇
#DataCenters #Uptime #CriticalInfrastructure #PowerInfrastructure
Related articles: