Resources

Running browser tests in CI

A suite that only runs when somebody remembers is documentation. This is how it runs on every push and still finishes before anyone gives up.

9 resources9 from official documentationTools

There is a moment in every automation effort where the tests work on the author's machine and nowhere else. Getting past it is mostly unglamorous: pin the browser versions, install the system libraries, stop depending on the author's local database, and make the result readable by somebody who was not watching.

The second problem arrives later and is more interesting. A suite that takes forty minutes will not be run on every pull request, whatever the policy says, so the honest target is not “more tests” but “the same tests, finished before the reviewer has moved on”. Two mechanisms do nearly all of that work: running tests in parallel inside one machine, and splitting the suite across several machines.

Sharding: four machines each run a quarter of the suite and emit a partial report, which merges into one.

# .github/workflows/tests.yml
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci
      # Installs the browsers AND the system libraries they link against.
      - run: npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shard }}/4

The shape of it. The official CI guide has the full version, including artefact upload.

Everything after that is about what the pipeline says when it goes red. A failure that reports only “1 failed” costs somebody twenty minutes of local reproduction; a failure that uploads a trace costs them ninety seconds. Attach the artefacts.

fail-fast: false, deliberately

The default in most matrix runners is to cancel the remaining jobs as soon as one fails. For a build that is fine. For a test shard matrix it means you learn about one failure and then have to run everything again to find out whether there were five. Turn it off.

Getting it running on every push

3 entries

Playwright: continuous integration
Official docs

Worked configurations for GitHub Actions, Azure Pipelines, GitLab, Jenkins and CircleCI, including the Docker image and the browser dependency flags. Start here rather than adapting somebody's gist.
playwright.dev
Playwright: Docker
Official docs

The official image, what it contains, and why running as a non-root user with the right IPC settings matters. The answer when “it works in Actions but not in our Kubernetes runner”.
playwright.dev
GitHub Actions documentation
Official docs

If your tests do not run on every push they are documentation. This is the shortest path to changing that, and the matrix and artefact pages are the two that matter for test suites.
docs.github.com

Making it finish in time

3 entries

Parallelism and workers
Official docs

How Playwright runs files in parallel by default, how to opt individual files in or out, and why fully parallel mode requires tests that do not share state. The constraint is the feature.
playwright.dev
Sharding
Official docs

Splitting one suite across several machines, and merging the partial reports back into one with the blob reporter. The official page covers the merge step, which is the part people miss and then conclude sharding is not worth it.
playwright.dev
Authentication
Official docs

Listed here as well as under Playwright because in CI it is a performance feature: signing in once in a setup project and reusing the storage state removes a login from every test in the suite.
playwright.dev

Making the result readable

3 entries

Reporters
Official docs

The built-in reporters, including the HTML report, the blob reporter that sharding needs, JUnit XML for systems that expect it, and how to run several at once. Also how to write your own, which is easier than it sounds.
playwright.dev
Trace viewer
Official docs

Upload the trace as a build artefact and a failure becomes a recording anyone on the team can scrub through, without reproducing anything.
playwright.dev
Test annotations
Official docs

Skip, fail, fixme and slow, with a reason attached. A skipped test with a written reason is a known gap; a commented-out test is a mystery for whoever finds it in six months.
playwright.dev

What to take away

Install browsers with their system dependencies.--with-deps, or the Docker image. The commonest first-run failure on a clean Linux box has nothing to do with your tests.
Shard, then merge.Sharding without merging the reports gives you four partial answers and no summary, which is why people try it once and stop.
Upload the trace.It is the difference between a twenty-minute reproduction and a ninety-second read.

Missing something? If there is a resource you send to every new tester on your team, tell us and we will read it. The list stays short on purpose; each entry has to earn its place, and entries that stop earning it get removed rather than left to rot.

Reading is the slow way to learn how a suite behaves. Record one against your own app, free, and watch what the report says.

Get started free See how it works