Cameleering · the method

How to measure, so the numbers hold up

What we measure, how we measure it, what counts as a fair baseline, a ten-point check to run before each release, and the one-page report every ledger row links to. Use it on any software, in any language.

No claim without a number.

What to measure

metric	how	why it matters
Startup	time from process start to the first unit of work being ready (a started context, a listening port answering a request); many runs, report the median	scale to zero, restarts, serverless cold starts
Memory at startup	resident set size once ready and idle	how many instances fit on a machine
Peak memory under load	maximum resident set size during the throughput run	what you must provision
Throughput	completed work per second at a fixed concurrency, after a warm-up, with the load generator on a separate machine	how much hardware the job needs
Tail latency	p99 (and p50) from the load generator, at the same load as the throughput run	what the slowest users wait
CPU	CPU time per unit of work, or CPU percentage at a fixed rate	the cost of each request
Size	binary or image size, compressed and uncompressed	deploy time, storage, attack surface
Dependencies and idle threads	count of third-party packages compiled in; threads alive when idle	supply chain, audit cost, idle overhead

What counts as a fair baseline

Do

Compare with the version most people run today, with the exact version, runtime and flags written down.
Include the strongest variant of the baseline too, not only the easy one. Cameleer is compared with Apache Camel on the JVM and on Quarkus native.
Run both sides on identical machines, swapping roles between them, with the load generator on its own machine.
Run each scenario more than once and say how the figures were combined.
Round against yourself: your times and memory up, your ratios down.

Don't

Compare a tuned build with a default one, or a warm run with a cold one.
Quote the best ratio without the worst ("up to 54x" without "4.8x").
Leave out the scenarios you lose.
Present a target in the same style as a result.
Borrow one project's numbers for another.

Headroom: the degraded-fleet test

An outage often brings no new traffic: it is the same traffic on fewer servers, and restarted servers that fall over again when the load returns (headroom, explained). Measure that margin before an incident, with the same harness and baseline rules as above. The full procedure, with what to record and how to read it, is in the degraded-fleet test.

test	what you do	what you record
Lose a third	run production-shaped load on N servers, then take a third of them out of rotation without changing the load	p99, error rate, CPU and memory on the survivors; the load at which p99 passes your target
Restart under load	restart one server while the others carry full traffic, and send it its share at once	time to its first answer, time until its p99 is back within target, and whether it fails its health check again
Connection storm	open many new TLS connections at once, as clients do when they reconnect after a failure	handshakes per second, CPU per handshake, and p99 for requests on connections that were already open
Overload	offer more load than the fleet can serve	how many requests are still answered within target, not only in total; whether load shedding or admission control kicks in

Report it like any other result: both sides, the same hardware, the losses. Our own overload result is a loss: simoom answers as many requests in total as the others but few within 1 ms (the ledger).

The water check: ten questions before each release

Do we know each number, and did any of them get worse? Keep the answers in the repository, where the team sees them.

Cold start time
Memory when idle
Peak memory under load
Binary or image size
Threads alive when idle
p99 latency at the expected load
CPU per request
Number of dependencies
Time a user waits behind a loader or spinner
Is each result written down, with the date and the machine?

The crossing report: one page per ledger row

section	what it holds
Load	the workload: the routes, requests or queries, and why they are representative
Baseline	the software compared with, exact versions, runtime and flags
Terrain	hardware, operating system, kernel, build profile
Water	startup, idle and peak memory, size, CPU
Distance	throughput and p50/p99 latency
Losses	where the new version is slower or bigger, and why
Repeat it	the scripts, commands, checksums and raw results
Date	when it was measured; a result older than 12 months is stale

Cameleer's report in this shape: BENCHMARKS, produced by scripts/cloud-bench from the raw results.

Cameleered or Cameleering: the two labels

"Cameleered"

A project may say it has been cameleered when all of these hold:

a before-and-after comparison against a named, current baseline version;
both sides on the same hardware;
a script anyone can rerun, and the raw results;
the losses disclosed;
measured within the last 12 months.

"Cameleering"

Work in progress: the baseline may be published and targets set, but nothing is measured yet. A cameleering project shows its targets as targets and no results. A project that has not started is listed as planned, with neither label.

The labels never carry another project's name or trademark, and they say nothing about the projects they are compared with beyond the measured numbers.