Performance testing
The numbers worth watching, why the average lies to you, and the tools that produce a figure somebody can act on.
Performance testing goes wrong in a predictable way: somebody measures an average, the average looks fine, and the complaints continue. Response times are not normally distributed. They have a long right tail, and every user in that tail is having a bad time while the mean sits comfortably in the middle.
So the figures to watch are percentiles. The 95th tells you what one user in twenty experiences; the 99th tells you about the ones who write in. A change that improves the mean and worsens the 99th has made your product worse for exactly the people most likely to notice.
The other half of the subject is front-end performance, which is a different discipline with different instruments. Server latency is about throughput and contention; page performance is about what the browser is blocked on. Core Web Vitals exist to make that measurable in a way that correlates with what a person perceives — how long until the main content appears, how quickly the page responds to a tap, and whether the layout jumps around while you are reading it.
A load test against a single container on a laptop produces a real number about an unreal system. Either run it against something shaped like production, or report it as a comparison against a previous run on the same hardware and refuse to quote the absolute figure. Both are honest. Quoting a laptop number as a capacity claim is not.
What the browser is doing
3 entries-
Core Web Vitals
Official docsThe three metrics Google publishes and measures in the field: largest contentful paint, interaction to next paint, cumulative layout shift. Defined precisely enough to be a test assertion rather than an opinion.
web.dev -
Lighthouse
Official docsAudits from the browser you already have, scriptable so a regression fails a build rather than being noticed six weeks later. Read the section on lab versus field data before treating the score as truth.
developer.chrome.com -
Chrome DevTools: performance panel
Official docsHow to read a flame chart, which is the skill that turns “the page is slow” into “this function runs 400 times on scroll”. The single most transferable front-end debugging skill there is.
developer.chrome.com
What the server is doing
3 entries-
k6
Official docsLoad tests written as JavaScript, run from a Go binary, with thresholds that make a run pass or fail. The documentation's explanation of test types — smoke, load, stress, soak, spike — is a good short education in the subject on its own.
grafana.com -
Apache JMeter
Official docsThe long-standing Java option, with a GUI for building plans and a very large body of existing knowledge. Still the default in a lot of enterprises, and worth recognising even if you would start elsewhere.
jmeter.apache.org -
Gatling
Official docsLoad testing in Scala, Java or Kotlin, with a simulation DSL and detailed HTML reports. The natural choice where the load tests should live in the same repository, and be reviewed in the same pull request, as the service they point at.
docs.gatling.io
Reading the results honestly
2 entries-
Gil Tene's talk, and the source of “coordinated omission” — the measurement error that makes almost every load test report a latency figure better than reality. If you are going to publish a percentile, watch this first.
www.youtube.com -
The orders of magnitude: cache, memory, disk, network, datacentre round trip. Calibrates your sense of whether a number is surprising before you spend a week optimising something that was never the problem.
gist.github.com
What to take away
- Quote percentiles, not averages.The mean describes a request that hardly anybody makes.
- Front-end and back-end performance are different jobs.One is about what the browser is blocked on; the other is about contention. Different tools, different numbers.
- Say what you measured on.A figure without an environment is a number, not a result.
Reading is the slow way to learn how a suite behaves. Record one against your own app, free, and watch what the report says.