How to test your headroom before an incident: the degraded-fleet test

Why run it, and what you will learn

Outages often bring no new traffic. They bring the same traffic onto fewer servers, and restarted servers that fall over again when the load reaches them. The headroom essay explains why. This page is the procedure.

At the end you will know four things, each as a number with a date. Where the capacity curve bends (the knee) with N servers and with two thirds of N. How long a restarted node takes to become healthy under load, and whether it stays healthy. What a wave of reconnecting clients costs. And what your service does when it is offered more than it can serve.

Set-up that makes the numbers believable

A capacity number is only as good as the test that produced it. Most of the work is here.

The open-loop point matters most. A closed-loop generator waits for each answer before sending the next request, so when the server slows down, the generator slows down with it and never measures the requests that should have been sent during the stall. Gil Tene called this coordinated omission; it can make the tail look many times better than users see it ^{[1] [2]}.

Tools that schedule by the clock include wrk2, built to produce a constant throughput load and measure latency from the intended send time ^[2]; k6 with its constant-arrival-rate executor, an open-model executor that starts iterations regardless of how the system responds ^[3]; and Vegeta, which attacks at a constant request rate ^[4]. Check that your tool reports latency from when a request was due, not from when it was finally sent.

Test 1: the degraded fleet

What to record at every step

How to read it

Compare C′/C with the ideal 2/3. Close to 2/3 means the work spreads cleanly. Well below means something shared is the limit: a dependency, a lock, a connection pool sized per fleet rather than per server. Look at the shape too. A sharp knee gives little warning before p99 breaks; a gentle one gives you time to react. Note the step where errors begin, and which resource ran out first.

Invented numbers, to show the reading. This fleet looks comfortable at 62% of capacity on a normal day. With a third gone it is past its knee at today's peak, and CPU is not the reason: the database pool is. The pass line fails, and the fix is a pool setting, not more servers.

Test 2: restart under load

Why this test matters more than start-up time alone: cold start is a recovery problem.

With the degraded fleet under steady load, restart one node and let it rejoin. Do it twice: (a) it gets its full share at once, (b) the load balancer ramps it up with slow start.

What to look at: warm-up. Caches are empty, connection pools to databases are empty, a JIT runtime has not compiled its hot paths, and the TLS session cache is cold, so every client does a full handshake. The SRE book lists slow start-up and cold caches as a cause of cascading failure ^[5].

Health-check design decides whether a slow node recovers or flaps. In Kubernetes, a failed liveness probe restarts the container, a failed readiness probe removes the pod from its Services' endpoints, and a startup probe holds both off until the application has started ^[6]. A liveness probe that fails under load turns a slow node into a restart loop. Readiness should mean "can serve now"; liveness should mean "is stuck and needs a restart".

Slow start exists in the common load balancers, with limits worth knowing:

• AWS Application Load Balancer: the target group attribute slow_start.duration_seconds ramps a healthy target's requests up linearly; it cannot be used with the least outstanding requests or weighted random algorithms ^[7].
• NGINX: the slow_start parameter of server in an upstream recovers weight from zero; it is part of the commercial subscription and cannot be used with hash, ip_hash or random ^[8].
• Envoy: slow start mode scales a new endpoint's weight over a window, for round robin, least request and client-side weighted round robin ^[9].

Test 3: the connection storm

After a failure, clients reconnect together. Drop all client connections at once, for example by restarting the load balancer's listeners or the generators' connection pools, and let clients reconnect. Run it twice: with clients that back off with jitter, and with clients that reconnect at once.

• Handshakes per second per core, and CPU time per handshake.
• p99 on connections that were already open during the wave. A storm that starves existing users is worse than one that only delays new ones.
• Time until latency is normal again.
• Full and resumed TLS. Run once with session resumption off and once with it on. A full handshake costs more than a resumed one (about a third more in our own test, details and limits), and after a restart the session cache may be empty. The cost of a TLS handshake explains what to measure.

Your load generator must open new connections at the rate you configure, not reuse them. k6 has noConnectionReuse to turn off keep-alive ^[10]; Vegeta has -keepalive, true by default ^[4]. Count the handshakes on the server side to confirm.

Test 4: overload

Offer about 120% of C and hold it. Count the answers that came back within the latency target, not only the total. AWS calls the useful part goodput: responses without errors and fast enough for the client to use. A response that arrives after the client gave up is wasted work ^[11].

• Does load shedding or admission control engage, and does it refuse work early and cheaply?
• Do clients retry, and how much? Retries can multiply load. Google's SRE book describes a per-client retry budget of 10% of requests ^[12]; AWS describes capped retries, exponential backoff with jitter and a local token bucket ^[13].
• When the offered load drops back to normal, does the fleet recover by itself? Both the SRE book and AWS advise testing to the breaking point and beyond it ^{[5] [11]}.

Put it together

One page per run, in the shape of the crossing report. Fill it in and keep it in the repository.

Then turn each pass line into a budget, as step 6 of the playbook describes: a smaller version of test 1 in CI that fails the build when C′ drops or p99 at the target rate rises past the line. Rerun the full set after any change to the hot path, after changes to instance types or dependencies, and before every peak season.

Common mistakes

• Closed-loop load. The generator slows down with the server, and the tail disappears from the results.
• A saturated load generator. If its CPU is at 100%, you measured the generator.
• An empty cache, or a perfect one. Measure the hit rate production has, and test cold starts on purpose.
• Only the happy fleet size. Capacity with N servers says little about N minus a third.
• Different builds, unsaid. Compare runs only with the build, flags and date written next to each.
• Averages. Use p99 and p99.9 from the client side.
• No pass line beforehand. Then any result can be read as fine.

Our own numbers

Parts of this test we have measured on simoom, our HTTP engine. All of these come from simoom's campaign 4 on DigitalOcean c-4 droplets, 2026-11-12, against 20 other HTTP servers on the same machines (the ledger).

How we generated the load

The advice above is open-loop. Our own harnesses are not all open-loop, so here is what each figure rests on.

• Cameleer against Apache Camel (the ledger’s Cameleer rows) is closed-loop. N worker threads each send the next request as soon as the last one is answered, with the same driver on both sides. Throughput is what each side sustains at saturation. The p99 values are latencies at that saturation, timed from when each request was sent, so they are not corrected for coordinated omission. Both sides use the same driver, so the comparison is like for like, but read those p99 figures as relative, not as what an open-loop test at a fixed rate would show (BENCHMARKS).
• simoom’s request-rate and latency tests are open-loop. They hold a constant rate and time each request from when it was due. The overload test offers a fixed absolute rate over 1,024 connections.
• simoom’s TLS handshake, HTTP/3 and bulk-download tests are closed-loop. The handshake figures above are handshakes per second at saturation, from 192 workers each opening a new connection per request, not a rate we offered.

These were measured on one droplet type, not on a fleet. We have not yet run the full degraded-fleet test on our own projects. It is the next result we will publish, with the losses, in the ledger.

Sources

Checked on 2026-11-13.

1. Gil Tene, "How NOT to Measure Latency", SpringOne2GX 2013 recording.
2. wrk2, README: constant throughput load and coordinated omission.
3. Grafana k6, constant-arrival-rate executor.
4. Vegeta, README: constant request rate and the -keepalive option.
5. Google, Site Reliability Engineering, chapter 22, "Addressing Cascading Failures".
6. Kubernetes, liveness, readiness and startup probes.
7. AWS, Application Load Balancer target group attributes: slow start mode.
8. NGINX, ngx_http_upstream_module: slow_start.
9. Envoy, slow start mode.
10. Grafana k6, options reference: noConnectionReuse.
11. Amazon Builders' Library, "Using load shedding to avoid overload".
12. Google, Site Reliability Engineering, chapter 21, "Handling Overload".
13. Amazon Builders' Library, "Timeouts, retries, and backoff with jitter".