Flaky tests: research and reading
A test that fails without anything having changed. What the research says, and how to tell three different problems apart.
A flaky test is one that passes and fails without the code changing. That is the definition, and it hides the actual difficulty: “element not found after 30 seconds” is the same message whether the page was slow, the button was renamed, or the application silently did nothing at all.
This is why retry-until-green is so appealing and so corrosive. It works, in the sense that the build turns green. It also erases the only evidence distinguishing a slow environment from a genuine race condition in the product. A step that passed on the third attempt because a request took four seconds and a step that passed on the third attempt because your application drops one click in three look identical in the summary. One of those is a test problem. The other is a customer-facing bug you have just agreed not to see.
The research is consistent on two points. Flakiness is not rare — Google publish figures for their own monorepo and report the problem as continuous rather than occasional — and it correlates with how much a test touches rather than with the quality of the code under test. Browser tests lead the table for exactly that reason: a single end-to-end test depends on a network, a renderer, a backend and a clock, and any of them can have a bad second.
A test nobody trusts is worse than no test, because it still costs time to run and read. Move it out of the blocking suite the day you notice, keep it running somewhere visible, and give it an owner and a date. Deleting it hides a real bug about a third of the time; leaving it in place teaches the team to ignore red.
What the data says
3 entries-
The post that put a number on the problem, and described the response: detect, quarantine, and treat the flake rate itself as a metric somebody owns. Short, and the source most other writing on this subject cites.
testing.googleblog.com -
The follow-up, with data on which kinds of test flake most. The headline finding is that flake correlates with test size and with the number of dependencies a test reaches, not with the code under test being bad.
testing.googleblog.com -
Luo, Hariri, Eloussi and Marinov's paper: 201 flaky-test fixes from open-source projects, categorised by root cause. Async wait, concurrency and test-order dependency account for most of them. If you only read one academic paper about testing, this is a good candidate.
mir.cs.illinois.edu
The mechanics that actually help
4 entriesNothing here is a retry.
-
Playwright: auto-waiting and actionability
Official docsThe reason a well-written Playwright test rarely needs an explicit wait: every action re-checks a list of conditions until they hold. Understanding this list is most of the cure for timing flake.
playwright.dev -
Playwright: timeouts
Official docsWhich timeout applies where — test, expect, action, navigation, global — and how they interact. Half of all “it hangs in CI” questions are answered on this page.
playwright.dev -
Playwright: retries
Official docsThe official mechanism, including the important detail that a test which passes on retry is reported as flaky rather than as a pass. That distinction is what makes retries survivable: the build goes green and the record of how survives.
playwright.dev -
Playwright: trace viewer
Official docsThe single most useful debugging tool in browser testing. A trace from a failed CI run gives you the DOM snapshot at every step, the network log, the console and the source line — so “it only fails on CI” stops being unanswerable.
playwright.dev
Ours
1 entry-
How we separate “not there yet” from “gone” from “nothing happened” using evidence the run already has — a request in flight, a navigation counter that moved — rather than a heuristic about how flaky a step has been in the past. Also the argument for why a wait budget should widen and never narrow.
qaright.com
What to take away
- A timeout is three different failures wearing one message.Any process that treats them the same — including a blanket retry — learns nothing from any of them.
- Report a retried pass as flaky, never as a pass.The build can still go green. The evidence has to survive.
- Sometimes the flaky thing is the product.When waits were honoured and the locator resolved and it still fails one run in three, the honest verdict is a race in the application.
Reading is the slow way to learn how a suite behaves. Record one against your own app, free, and watch what the report says.