I’d previously shared how to set up Playwright with Jest as a test runner which enabled us to do some cool things like:
- Parallel test execution,
- Automatic retries,
- HTML reports (using Jest Stare), and
- Screenshots when failing
It turns out that Playwright now supports all of the features without having to use Jest! It’s called the Playwright Test Runner.
I went ahead and created a fork of my existing code to show how to set this up with the Playwright Test Runner.
Specifying Tests
Fortunately the syntax to specify tests is almost identical to Jest, but you don’t need to worry about spawning browser contexts yourself 😎
This Jest test:
jest.retryTimes(1)
test('can use xpath selectors to find elements', async () => {
global.page = await pages.spawnPage()
await nav.visitHomePage(global.page)
await home.clickScissors(global.page)
}, jestTimeoutMS)
becomes:
const { test } = require('@playwright/test')
test('can use xpath selectors to find elements', async ({ page }) => {
await nav.visitHomePage(page)
await home.clickScissors(page)
})
Running in Parallel
Like Jest, Playwright automatically runs your tests in parallel, sharing isolated browser pages within browsers, and spawning browsers across test files. The time taken was pretty much identical:
Jest
PASS scenarios/example4.spec.js PASS scenarios/example3.spec.js PASS scenarios/example5.spec.js PASS scenarios/api.spec.js PASS scenarios/example2.spec.js PASS scenarios/example.spec.js Test Suites: 6 passed, 6 total Tests: 10 passed, 10 total Snapshots: 2 passed, 2 total Time: 5.514 s
Playwright Test
Running 10 tests using 6 workers ✓ scenarios/api.spec.js:5:1 › [chromium] can GET a REST API and check response using approval style (590ms) ✓ scenarios/example.spec.js:4:1 › [chromium] can wait for an element to appear (3s) ✓ scenarios/example2.spec.js:4:1 › [chromium] can handle alerts (4s) ✓ scenarios/api.spec.js:12:1 › [chromium] can GET a REST API and check response using assertion style (587ms) ✓ scenarios/example3.spec.js:5:1 › [chromium] can check for errors when there should be none (1s) ✓ scenarios/example4.spec.js:5:1 › [chromium] can check for errors when there are present (1s) ✓ scenarios/api.spec.js:25:1 › [chromium] can POST a REST API and check response using approval style (536ms) ✓ scenarios/example5.spec.js:6:1 › [chromium] can use xpath selectors to find elements (988ms) ✓ scenarios/api.spec.js:32:1 › [chromium] can POST a REST API and check response using assertion style (545ms) ✓ scenarios/example.spec.js:9:1 › [chromium] can use an element that appears after on page load (220ms) 10 passed (5s)
Automatic Retries
Jest supports automatic retries in the test itself:
jest.retryTimes(1)
In Playwright Test you can configure it globally or per run:
// playwright.config.js
module.exports = {
use: {
retries: 2
}
}
npx playwright test --retries=3
HTML Reports
I couldn’t see HTML report output for Playwright Test – but there’s heaps of others like JSON and Junit (for CI).
Screenshots
Screenshots are easily configured for Playwright Test: for each test never, always or on failure (my preference).
module.exports = {
use: {
screenshot: 'only-on-failure'
}
}
But wait there’s more
The Playwright Test Runner also has these very cool features:
Visual comparisons (visdifs)
test('can wait for an element to appear', async ({ page }) => {
await nav.visitHomePage(page)
await page.waitForSelector('#elementappearschild', { visible: true, timeout: 5000 })
expect(await page.screenshot()).toMatchSnapshot('element-appears.png')
})
At any point in your tests you can take a screenshot and store this to visually compare with future runs with fine grained tweaking of matches. The best part about this, in my opinion, is these visuals are stored alongside your tests (not in some third party system) so you know exactly what you’re expecting to see 😎
The one downside is the screenshots are platform specific so when I checked in those generated on my Mac then CI failed as it was running on Linux and didn’t have the baseline files to compare to. I just downloaded the captured files from CircleCI and stored them as a baseline.
And it also supports other comparisons – similar to Jest snapshots I have previously demonstrated. You just need to stringify them first:
test('can GET a REST API and check response using approval style', async () => {
const request = supertest('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io')
const response = await request.get('/posts')
expect(response.status).toBe(200)
expect(JSON.stringify(response.body)).toMatchSnapshot('posts.txt')
})
Closing Thoughts
I was pretty blown away by the Playwright Test runner. It offers everything Jest provides with less code I had to write, plus it has in built visual comparison tools that can also be extended to do API approval snapshot testing. Playwright is well becoming my e2e test automation tool of choice.
Show me the Code
Of course: code is here: https://github.com/alisterscott/playwright-test-demo
Passing CI here: https://app.circleci.com/pipelines/github/alisterscott/playwright-test-demo/7/workflows/fe13f579-1f53-4a01-834e-6a4e7388e22c