Categories
Playwright

Playwright’s “retry with video”

Another feature I love in Playwright Test is the retry with video option.

In your playwright.config.js file you specify:

module.exports = {
  use: {
    headless: true,
    screenshot: 'only-on-failure',
    video: 'retry-with-video'
  },
  retries: 1
}

The video option combined with the retry option means that if a test fails Playwright will re-run it whilst recording a video of that re-run.

If the test fails on the second attempt it will capture the video in the test results (alongside the screenshot if you config it to capture screenshots also).

It’s a webm video that is lightweight, easy to capture during CI, and easy to share:

Categories
Playwright

Ensuring consistent Playwright tests with “repeat each”

My workflow to ensure any new Playwright Test test I write is deterministic and repeatable:

  1. Mark the new test as .only temporarily locally during development
  2. Run npx playwright test --repeat-each 100 where 100 is the number of times you want to run it in parallel locally
  3. Make sure it passes 100% of the times

Here’s an example of how it looks (using 10 times for conciseness):

Playwright Test results showing consistent test runs

I’ve used this to discover a test failed about 5% of the times I ran it which was one of the only Playwright tests I’ve written so far that needed an explicit wait for element.

Categories
e2e Testing Playwright

Playwright Test Runner

I’d previously shared how to set up Playwright with Jest as a test runner which enabled us to do some cool things like:

  • Parallel test execution,
  • Automatic retries,
  • HTML reports (using Jest Stare), and
  • Screenshots when failing

It turns out that Playwright now supports all of the features without having to use Jest! It’s called the Playwright Test Runner.

I went ahead and created a fork of my existing code to show how to set this up with the Playwright Test Runner.

Specifying Tests

Fortunately the syntax to specify tests is almost identical to Jest, but you don’t need to worry about spawning browser contexts yourself 😎

This Jest test:

jest.retryTimes(1)

test('can use xpath selectors to find elements', async () => {
  global.page = await pages.spawnPage()
  await nav.visitHomePage(global.page)
  await home.clickScissors(global.page)
}, jestTimeoutMS)

becomes:

const { test } = require('@playwright/test')

test('can use xpath selectors to find elements', async ({ page }) => {
  await nav.visitHomePage(page)
  await home.clickScissors(page)
})

Running in Parallel

Like Jest, Playwright automatically runs your tests in parallel, sharing isolated browser pages within browsers, and spawning browsers across test files. The time taken was pretty much identical:

Jest

 PASS  scenarios/example4.spec.js
 PASS  scenarios/example3.spec.js
 PASS  scenarios/example5.spec.js
 PASS  scenarios/api.spec.js
 PASS  scenarios/example2.spec.js
 PASS  scenarios/example.spec.js

Test Suites: 6 passed, 6 total
Tests:       10 passed, 10 total
Snapshots:   2 passed, 2 total
Time:        5.514 s

Playwright Test

Running 10 tests using 6 workers

  ✓ scenarios/api.spec.js:5:1 › [chromium] can GET a REST API and check response using approval style (590ms)
  ✓ scenarios/example.spec.js:4:1 › [chromium] can wait for an element to appear (3s)
  ✓ scenarios/example2.spec.js:4:1 › [chromium] can handle alerts (4s)
  ✓ scenarios/api.spec.js:12:1 › [chromium] can GET a REST API and check response using assertion style (587ms)
  ✓ scenarios/example3.spec.js:5:1 › [chromium] can check for errors when there should be none (1s)
  ✓ scenarios/example4.spec.js:5:1 › [chromium] can check for errors when there are present (1s)
  ✓ scenarios/api.spec.js:25:1 › [chromium] can POST a REST API and check response using approval style (536ms)
  ✓ scenarios/example5.spec.js:6:1 › [chromium] can use xpath selectors to find elements (988ms)
  ✓ scenarios/api.spec.js:32:1 › [chromium] can POST a REST API and check response using assertion style (545ms)
  ✓ scenarios/example.spec.js:9:1 › [chromium] can use an element that appears after on page load (220ms)


  10 passed (5s)

Automatic Retries

Jest supports automatic retries in the test itself:

jest.retryTimes(1)

In Playwright Test you can configure it globally or per run:

// playwright.config.js
module.exports = {
  use: {
    retries: 2
  }
}
npx playwright test --retries=3

HTML Reports

I couldn’t see HTML report output for Playwright Test – but there’s heaps of others like JSON and Junit (for CI).

Screenshots

Screenshots are easily configured for Playwright Test: for each test never, always or on failure (my preference).

module.exports = {
  use: {
    screenshot: 'only-on-failure'
  }
}

But wait there’s more

The Playwright Test Runner also has these very cool features:

Visual comparisons (visdifs)

test('can wait for an element to appear', async ({ page }) => {
  await nav.visitHomePage(page)
  await page.waitForSelector('#elementappearschild', { visible: true, timeout: 5000 })
  expect(await page.screenshot()).toMatchSnapshot('element-appears.png')
})

At any point in your tests you can take a screenshot and store this to visually compare with future runs with fine grained tweaking of matches. The best part about this, in my opinion, is these visuals are stored alongside your tests (not in some third party system) so you know exactly what you’re expecting to see 😎

The one downside is the screenshots are platform specific so when I checked in those generated on my Mac then CI failed as it was running on Linux and didn’t have the baseline files to compare to. I just downloaded the captured files from CircleCI and stored them as a baseline.

And it also supports other comparisons – similar to Jest snapshots I have previously demonstrated. You just need to stringify them first:

test('can GET a REST API and check response using approval style', async () => {
  const request = supertest('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io')
  const response = await request.get('/posts')
  expect(response.status).toBe(200)
  expect(JSON.stringify(response.body)).toMatchSnapshot('posts.txt')
})

Closing Thoughts

I was pretty blown away by the Playwright Test runner. It offers everything Jest provides with less code I had to write, plus it has in built visual comparison tools that can also be extended to do API approval snapshot testing. Playwright is well becoming my e2e test automation tool of choice.

Show me the Code

Of course: code is here: https://github.com/alisterscott/playwright-test-demo

Passing CI here: https://app.circleci.com/pipelines/github/alisterscott/playwright-test-demo/7/workflows/fe13f579-1f53-4a01-834e-6a4e7388e22c