Flaky tests: telling a slow app from a broken one
A flaky test is one that fails without anything having changed. That is how it feels, anyway. Almost always, something did change — the app took 3.8 seconds to do what it used to do in 1.2 — and the test had no way to say so. It had a timeout, the timeout expired, and "element not found" covered three completely different situations with one message.
What "flaky" usually means
The standard cure is retry-until-green: run the failed test again, up to three times, and count it as a pass if any attempt passes. It works, in the sense that the build goes green. It also destroys the one piece of information the failure carried. A step that passed on the third attempt because the page was slow, and a step that passed on the third attempt because a race in the application happens one time in three, look identical in the summary. One of those is a test problem. The other is a bug your users are hitting.
Three failures that look identical in a log
When a step times out waiting for an element, one of three things is true, and the log line is the same for all of them.
Not there yet
The element is coming. The page is mid-navigation, a request is in flight, the node is attached but hasn't painted. The wait was simply shorter than the app was slow. More time would have fixed it, and the test is the thing at fault.
Gone
The page arrived, and the element isn't on it — renamed, restyled, replaced. No amount of waiting helps. This is a broken locator, and it is self-healing's problem, not a timing one.
Nothing happened
The click landed and the application did nothing: no request, no navigation, no change to the DOM. Nothing is coming, however long you wait. Retrying makes this case slower, not fixed, and a tool that grants extra time by default turns a two-second failure into a two-minute one with the same answer at the end.
How QAright tells them apart
The distinction is made with evidence the run already has, not with a heuristic about how flaky the step has been historically.
Recorded waits carry a measurement
When the recorder captures a click that navigates, it measures how long the page actually took to settle and stores that with the step. So the wait budget on playback is not a global 30-second default or a number someone guessed; it is what the app did on the day the test was written, with headroom. That measurement is what makes everything below legitimate. A wait a person typed into the step by hand gets none of it, because nothing backs it.
Grace is granted on evidence, never by default
When a recorded wait expires, the engine looks at whether the page is visibly working: a document request still in flight, a navigation counter that moved, the target attached but not yet painted. Only then does it wait again, and only for a bounded extra budget. Without that evidence the step fails exactly as promptly as it would have before. This is the line between "the app got slower" and "the app swallowed the click", and it is why a genuinely broken step does not get slower under this feature.
Budgets widen, and never narrow
A step that needed grace and then succeeded gets its budget widened for next time — a multiple of the cost actually measured, capped, and written back to the step so the correction persists. A run never narrows a budget, because that re-introduces the flake the first time the app has a slow day, and an over-long budget costs nothing on a run that passes. The report marks the step SLOW · WAIT WIDENED rather than hiding the recovery, so a slow endpoint shows up as a trend instead of a pass.
The failure report proves which case it was
When a step does fail, the report doesn't say "element not found". It shows whether the element ever appeared during the wait, what the page was doing when time ran out, and a full-page screenshot — not just the viewport, because the error banner is usually below the fold. If the same test ran in another browser in the same run and passed, that verdict is weighed too: a step that fails in one browser and passes in its sibling is not "the page is broken".
What a retry is actually for
Retries still exist, because some things genuinely are polled: a background job that finishes in a few seconds, a queue that drains. Those get a step type built for it — poll until a condition holds, at an interval you set — rather than a sleep with a hopeful number in it. A step can also carry its own retry count, capped at five, and an attempt that succeeds on the third try is reported with its attempt number, not folded into a plain pass. The build can still go green. The record of how it went green survives.
Sometimes the flaky thing is the product
When the waits were honoured, the locator resolved, the same failure reproduces in a second browser and the app still did nothing on one run in three, the honest verdict is that the application has a race. A test tool's instinct is to blame the test, add a retry, and move on; the report here says what it saw and lets that conclusion stand. A suite that never surfaces a product bug because it has been tuned to pass isn't a stable suite. It's a quiet one.
- Never widen a hand-authored wait.No measurement backs it, so there is nothing to extrapolate from.
- Never narrow a budget.The first slow day would bring the flake straight back.
- Never grant grace without evidence.A step where nothing is happening fails as fast as it always did.
- Never turn a retry into a silent pass.The attempt count and the widened budget are in the report, every time.
Keep reading. The research on flaky tests is collected in flaky tests: research and reading. How a step whose element moved is repaired is self-healing test automation, and test drift detection finds that kind of change before a run fails on it.
Point it at the suite you trust least, on the free plan, and read what the report says about the steps that used to be "just flaky".