CameleeringCameleering / Headroom
Cameleering · capacity, recovery and efficiency
Headroom: what is left when servers drop out
On 12 and 13 November 2026, Microsoft 365 went down around the world. Microsoft said the load that strained its infrastructure was not new traffic. It was the same traffic, landing on fewer servers. How much a fleet survives losing part of itself depends on what each request costs and how fast a server can come back. That is an efficiency property, and it can be measured before an incident.
No claim without a number.
What Microsoft has said so far
Read this first. As of 13 November 2026, Microsoft has published no post-incident review and no root cause. We don't know the cause, and nothing on this page is a guess at it. We will update the page when Microsoft publishes its review.
The outage affected Exchange Online and Outlook, Teams, SharePoint Online, Copilot for Microsoft 365, Defender XDR and other services. Microsoft tracked it as advisory MO1550821, with a related Exchange incident EX1550543 [1] [2], and outage trackers filled with reports from users who could not reach mail or meetings [5].
Microsoft's status updates, in order:
- The elevated load "isn't driven by an increase in traffic", but by servers being unexpectedly unavailable, "straining the remaining infrastructure" [1].
- Engineers were restarting and validating servers, and assessing whether reverting "a small number of recent configuration updates" could help [2].
- Servers returned to service "can become unresponsive again once they receive traffic", so Microsoft was temporarily limiting which types of traffic could reach the affected infrastructure [3].
Reports say servers were failing automated health checks [4]. Microsoft said there was no indication of a cyberattack [5] [6].
Two things in those updates are general, whatever the cause turns out to be. Load from lost servers moved onto the ones still running. And servers coming back fell over again when traffic reached them. Every fleet faces both, and both can be tested.
The same traffic on fewer servers
When servers drop out, the requests they were serving do not go away. Load balancers send them to the survivors. Each survivor's utilisation rises by the ratio of the old fleet to the new one.
| normal utilisation | lose a quarter | lose a third | lose half |
|---|---|---|---|
| 40% | 53% | 60% | 80% |
| 50% | 66% | 75% | 100% |
| 60% | 80% | 90% | 120% |
| 70% | 93% | 105% | 140% |
Arithmetic, not a measurement: utilisation divided by the fraction of servers left. It assumes load spreads evenly and that nothing else changes. In practice it gets worse: latency climbs long before 100%, queues grow, and clients retry.
A fleet at 60% that loses a third of its servers runs the survivors at 90%. That is tight but alive. The same fleet at 70% is past 100%. The difference between those two rows is the headroom.
Headroom is usually bought with more servers. But utilisation is load times cost per request. Halve the CPU, memory and connection setup each request needs, and the same fleet runs at half the utilisation. The margin is the same as doubling the servers, and it stays there on the day some of them disappear. That is why per-request cost is a reliability number, not only a cost number.
Coming back under load
Microsoft's third update describes a known trap. A restarted server is at its weakest when it first joins. Its caches are empty. A JIT-compiled runtime has not compiled its hot paths yet. Connection pools to databases and back ends are empty. TLS session caches are cold, so every client does a full handshake. If that server gets its full share of traffic at once, it can fail its health check and leave again. The rest of the fleet then carries the load for one more cycle.
Two properties decide how this goes:
- Start-up time. How long from process start to serving at full speed (cold start is a recovery problem). Seconds of warm-up per node add up across a fleet restart.
- The cost of a new connection. After an outage, clients reconnect together. A full TLS handshake costs far more than a request on an open connection, so the reconnection wave can be heavier than normal traffic (the cost of a TLS handshake).
The standard mitigations are well known, and Microsoft used one of them:
The load balancer ramps a new server's share up over a window instead of sending it a full share at once.
A server refuses work it cannot finish in time, cheaply and early, rather than accepting everything and timing out on all of it.
Limit concurrent work per server and per traffic type. Limiting which types of traffic reach the affected servers, as Microsoft did, is this.
Clients cap retries as a share of requests and back off with jitter, so a failure does not multiply the load.
Measure it before the incident
Headroom is easy to assume and easy to measure. Three tests, run on a staging fleet that matches production in shape, tell you where you stand (the full procedure is in how to run the degraded-fleet test). Record the results next to the date and the build, as the Cameleering method asks for every number.
| test | what to do | what to record |
|---|---|---|
| 1. Degraded fleet | Take N servers at production-shaped load. Remove a third of them. Replay the same load for at least as long as your longest past incident. | p99 latency, error rate, CPU and memory on the survivors; the point where p99 breaks your target. |
| 2. Cold restart under load | While the degraded fleet is under load, restart one node and let it rejoin with and without slow start. | Time from process start to passing health checks; time to full speed; whether it fails again. |
| 3. Connection storm | Drop all client connections at once and let clients reconnect, with full and with resumed TLS handshakes. | Handshakes per second per core, p99 during the wave, time until normal latency returns. |
Run them again after every change that touches the hot path, and keep the history. A regression in per-request cost is a regression in headroom, even when no test fails. The playbook describes how to turn such numbers into budgets that fail a build.
What our own measurements say
Cameleering's projects measure the properties above against the software they replace. These numbers come from our own runs, each with its baseline. They say nothing about Microsoft's systems, and we do not claim any of it would have prevented this outage.
| property | result | baseline |
|---|---|---|
| start-up | Cameleer: 3 ms | Apache Camel 4.23 on Java 21: 1.1 s; on Quarkus native: 30 ms |
| throughput on the same CPUs | Cameleer: 1.7 to 8.7x | Apache Camel 4.23 on Java 21, same DigitalOcean c-4 machines |
| peak memory | Cameleer: 4.8 to 54x less | the same |
| CPU per request | simoom: about 13 µs, 43% less | cameleer-http, the server it replaced: about 23 µs |
| throughput | simoom: 1.7 to 1.8x | cameleer-http |
| TLS 1.3 handshakes per second | simoom: 9,396 full, 12,388 resumed | 1.39x and 1.56x H2O; 24% and 9% behind axum + rustls |
| overload, 195k req/s offered | simoom: 10th of 19 for answers within 1 ms | about as many answers in total as the others; actix-web answered 22x more in time |
Sources: the ledger, Cameleer, the ledger, simoom and simoom benchmarks [7].
Where it helps
Lower CPU per request and lower memory mean lower utilisation for the same load, which is headroom. A 3 ms start-up shortens the time before a restarted node can take traffic at all, though caches and pools still have to fill. Fast resumed TLS handshakes make a reconnection wave cheaper.
Where it does not, yet
Overload behaviour is hard, and our own numbers show it. Offered far more than it could serve, simoom answered about as many requests as the other servers but few of them within 1 ms. That is the failure mode load shedding exists for, and we have work to do there. On full TLS handshakes, axum with rustls is ahead of us.
Efficient software buys headroom. It is not a substitute for capacity planning, load shedding and safe rollouts. It makes each of them easier, because there is more margin to work with.
A note on the people
Somewhere, engineers spent the night restarting servers, watching them fall over, and trying again with less traffic. That is hard, careful work, done in public, under pressure. Their updates were clear about what they knew and what they did not. This page is not a verdict on them.
The thing to fight is waste, not the people. Every request that costs less leaves a little more room for the day the fleet is smaller than planned. For the cost side of the same idea, see code efficiency as a FinOps lever; for how we measure, the method; for every result, good and bad, the ledger.
Common questions
Was the Microsoft 365 outage a cyberattack?
Microsoft said there was no indication of a cyberattack, as reported by ABC News and BleepingComputer.
What caused the outage?
Microsoft has not published a root cause or a post-incident review yet. Its status updates described servers that were unexpectedly unavailable and the strain that put on the rest. We will update this page when the review is out.
What is headroom?
The spare capacity a fleet has between its normal load and the load at which it stops answering in time. It is what absorbs lost servers, traffic spikes and the extra work of recovery.
How do I test my own headroom?
Run a degraded-fleet test: remove a third of your servers, replay production-shaped load, and record p99 latency, error rate, CPU, memory and how long a restarted node takes to become healthy. Add a cold restart under load and a storm of new TLS connections.
Would a more efficient server have prevented it?
No one can say, and we do not claim it. Efficient software buys margin, not immunity. It does not replace capacity planning, load shedding and safe rollouts.
Sources
Checked on 2026-11-13.
- Microsoft 365 Status, status update on elevated load, 12 November 2026.
- Microsoft 365 Status, status update on restarts and configuration reverts, 12 November 2026.
- Microsoft 365 Status, status update on servers becoming unresponsive under traffic, 13 November 2026.
- Forbes, "Microsoft 365 down: tech giant says server failures causing disruption", 12 November 2026.
- BleepingComputer, "Massive Microsoft 365 outage disrupts Outlook, Teams, Copilot worldwide".
- ABC News, "Microsoft outage affects users worldwide", 13 November 2026.
- Cameleering measurements: the ledger and simoom benchmarks.
Updates
- 2026-11-13: first version, written before Microsoft's post-incident review.