Categories
e2e Testing Playwright

Playwright Test Runner

I’d previously shared how to set up Playwright with Jest as a test runner which enabled us to do some cool things like:

  • Parallel test execution,
  • Automatic retries,
  • HTML reports (using Jest Stare), and
  • Screenshots when failing

It turns out that Playwright now supports all of the features without having to use Jest! It’s called the Playwright Test Runner.

I went ahead and created a fork of my existing code to show how to set this up with the Playwright Test Runner.

Specifying Tests

Fortunately the syntax to specify tests is almost identical to Jest, but you don’t need to worry about spawning browser contexts yourself 😎

This Jest test:

jest.retryTimes(1)

test('can use xpath selectors to find elements', async () => {
  global.page = await pages.spawnPage()
  await nav.visitHomePage(global.page)
  await home.clickScissors(global.page)
}, jestTimeoutMS)

becomes:

const { test } = require('@playwright/test')

test('can use xpath selectors to find elements', async ({ page }) => {
  await nav.visitHomePage(page)
  await home.clickScissors(page)
})

Running in Parallel

Like Jest, Playwright automatically runs your tests in parallel, sharing isolated browser pages within browsers, and spawning browsers across test files. The time taken was pretty much identical:

Jest

 PASS  scenarios/example4.spec.js
 PASS  scenarios/example3.spec.js
 PASS  scenarios/example5.spec.js
 PASS  scenarios/api.spec.js
 PASS  scenarios/example2.spec.js
 PASS  scenarios/example.spec.js

Test Suites: 6 passed, 6 total
Tests:       10 passed, 10 total
Snapshots:   2 passed, 2 total
Time:        5.514 s

Playwright Test

Running 10 tests using 6 workers

  ✓ scenarios/api.spec.js:5:1 › [chromium] can GET a REST API and check response using approval style (590ms)
  ✓ scenarios/example.spec.js:4:1 › [chromium] can wait for an element to appear (3s)
  ✓ scenarios/example2.spec.js:4:1 › [chromium] can handle alerts (4s)
  ✓ scenarios/api.spec.js:12:1 › [chromium] can GET a REST API and check response using assertion style (587ms)
  ✓ scenarios/example3.spec.js:5:1 › [chromium] can check for errors when there should be none (1s)
  ✓ scenarios/example4.spec.js:5:1 › [chromium] can check for errors when there are present (1s)
  ✓ scenarios/api.spec.js:25:1 › [chromium] can POST a REST API and check response using approval style (536ms)
  ✓ scenarios/example5.spec.js:6:1 › [chromium] can use xpath selectors to find elements (988ms)
  ✓ scenarios/api.spec.js:32:1 › [chromium] can POST a REST API and check response using assertion style (545ms)
  ✓ scenarios/example.spec.js:9:1 › [chromium] can use an element that appears after on page load (220ms)


  10 passed (5s)

Automatic Retries

Jest supports automatic retries in the test itself:

jest.retryTimes(1)

In Playwright Test you can configure it globally or per run:

// playwright.config.js
module.exports = {
  use: {
    retries: 2
  }
}
npx playwright test --retries=3

HTML Reports

I couldn’t see HTML report output for Playwright Test – but there’s heaps of others like JSON and Junit (for CI).

Screenshots

Screenshots are easily configured for Playwright Test: for each test never, always or on failure (my preference).

module.exports = {
  use: {
    screenshot: 'only-on-failure'
  }
}

But wait there’s more

The Playwright Test Runner also has these very cool features:

Visual comparisons (visdifs)

test('can wait for an element to appear', async ({ page }) => {
  await nav.visitHomePage(page)
  await page.waitForSelector('#elementappearschild', { visible: true, timeout: 5000 })
  expect(await page.screenshot()).toMatchSnapshot('element-appears.png')
})

At any point in your tests you can take a screenshot and store this to visually compare with future runs with fine grained tweaking of matches. The best part about this, in my opinion, is these visuals are stored alongside your tests (not in some third party system) so you know exactly what you’re expecting to see 😎

The one downside is the screenshots are platform specific so when I checked in those generated on my Mac then CI failed as it was running on Linux and didn’t have the baseline files to compare to. I just downloaded the captured files from CircleCI and stored them as a baseline.

And it also supports other comparisons – similar to Jest snapshots I have previously demonstrated. You just need to stringify them first:

test('can GET a REST API and check response using approval style', async () => {
  const request = supertest('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io')
  const response = await request.get('/posts')
  expect(response.status).toBe(200)
  expect(JSON.stringify(response.body)).toMatchSnapshot('posts.txt')
})

Closing Thoughts

I was pretty blown away by the Playwright Test runner. It offers everything Jest provides with less code I had to write, plus it has in built visual comparison tools that can also be extended to do API approval snapshot testing. Playwright is well becoming my e2e test automation tool of choice.

Show me the Code

Of course: code is here: https://github.com/alisterscott/playwright-test-demo

Passing CI here: https://app.circleci.com/pipelines/github/alisterscott/playwright-test-demo/7/workflows/fe13f579-1f53-4a01-834e-6a4e7388e22c

Categories
API Testing Automated Testing

API “approval” testing using Jest

I recently added some API integration testing examples to my Playwright/Jest e2e automated testing repository using Supertest.

I wanted to also explore the use of Jest for approval testing of REST JSON APIs.

Jest has in built support for snapshot testing which is most commonly used to test how React components are rendered – however Jest snapshot testing can also be used to test any serializable values.

Snapshot tests are an example of what are are also known as approval tests: whereby the expected output of a test is generated the first time it is run and compared against this on subsequent runs. Changes to the output can be “approved” which become the new baseline.

These are no where near as common as “assertion” based tests whereby an automated test performs some action and asserts some results.

Assertion tests can be used to assist test driven development, whereas approval tests are generated after the fact so they can be useful for outside-in software development, API integration testing of known endpoints, and for legacy code where no assertion based or lower level tests exist.

I wrote the same integration tests in both approval and assertion style to show how these differ.

Say we have an endpoint that returns all the blog posts we have seeded into our system, we could write some assertions on the content:

const response = await request.get('/posts')
expect(response.status).toBe(200)
expect(response.body.length).toBe(3)
expect(response.body[0].id).toBe(1)
expect(response.body[0].title).toBe('Post 1')
expect(response.body[1].id).toBe(2)
expect(response.body[1].title).toBe('Post 2')
expect(response.body[2].id).toBe(3)
expect(response.body[2].title).toBe('Post 3')

Since we’re testing an existing endpoint with known “approved” behaviour we can generate and test a snapshot of it:

  const response = await request.get('/posts')
  expect(response.status).toBe(200)
  expect(response.body).toMatchSnapshot()

The first time we run this the following is generated in a separate file:

// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`can GET a REST API and check response using approval style 1`] = `
Array [
  Object {
    "id": 1,
    "title": "Post 1",
  },
  Object {
    "id": 2,
    "title": "Post 2",
  },
  Object {
    "id": 3,
    "title": "Post 3",
  },
]
`;

We commit this to our codebase and if any of the body returned for this endpoint is changed our test fails (with an option to approve the new result as the new baseline).

The benefit of these approval tests is they are quicker to write for an existing endpoint (since they are generated automatically) and are complete in that any change to the body returned will be highlighted – compared to our assertion based tests which only test specific things.

The downside to these tests is they are generated after the fact – so aren’t compatible with test driven development – and need to take into account if there is any dynamic content that changes between test runs – so these generally aren’t good for non-deterministic systems.

Feel free to check out the source code for these API integration tests and let me know what you think and whether you’ve had experience, good or bad, with approval testing.

Categories
e2e Testing

Playwright + Jest = 💖

We were looking at a possible replacement for our dated Protractor + Cucumber e2e testing framework. As we move away from Angular to React microapps we have found that Protractor doesn’t work very well/efficiently and cucumber isn’t giving us any benefits.

It was a good opportunity to do some research/tinkering to answer the question lingering in my mind: in 2020 what e2e testing tool would I use by default for a dynamic react based web application?

After some experimentation it came down to Puppeteer + Jest or Playwright + Jest. I’ll compare those in this post.

Why Jest?

As explained previously we don’t need a BDD framework, but we do need something that allows us to specify our tests, create assertions and run these tests (in parallel). Jest, particularly when using the Jest Circus test runner, seems the most mature tool in this regard in 2020. Whilst Jest is often associated with React as it handles the snapshot test results also, it doesn’t need to be used with React and it’s possible and easy to use it as a standalone Node.js testing library.

Parallel Support

Jest by default runs tests across files in parallel and uses the available resources to scale appropriately using processes/threads. I’ve found this particularly good as I write independent e2e tests which can scale through parallelism, and using Puppeteer and Playwright you can spawn new incognito browser contexts to run these.

Auto-Retry Support

Whilst I believe in writing deterministic e2e automated tests, since e2e tests are full-stack there are often external dependencies and services beyond our control (like a third party domain provider) and I’d rather prioritize test reliability over test perfection. With this mind I think it’s important to be able to automatically retry a single failing test scenario before failing a build, and the Jest Circus test runner supports this:

jest.retryTimes(1)

test('can wait for an element to appear', async () => {
  global.page = await pages.spawnPage()
  await nav.visitHomePage(global.page)
  await global.page.waitForSelector('#elementappearschild', { visible: true, timeout: 5000 })
}, jestTimeoutMS)

HTML Reports and Screenshots

Nice looking HTML reports are easy to achieve by using jest-stare and screenshots are easy to generate using Jest Circus hooks.

Playwright or Puppeteer?

I’m a fan of Puppeteer however Playwright is a much nicer browser automation library. Whilst it adds support for Firefox and Webkit, even if you’re running your e2e tests in one browser (Chromium) I’d still recommend Playwright over Puppeteer any day of the week. Here’s why:

Automatic Waiting

I’ve found the automatic waiting in Playwright just works™️ Especially when dealing with dynamically rendered react web apps I’ve found my Puppeteer code looks is full of page.waitFor calls to make it run reliably:

await page.waitFor('#loadedchild', { visible: true, timeout: 5000 })

Whilst this is occasionally necessary in Playwright (in particular waiting for an iFrame to switch into), I’ve found it’s almost never required which reminds me of good old Watir days.

Nicer API

Whilst the APIs are similar, Playwright is just nicer to use. Take grabbing some text from a div, in Puppeteer:

await page.goto(`${config.get('baseURL')}`)
await page.waitFor('#loadedchild', { visible: true, timeout: 5000 })
const element = await page.$('#loadedchild')
const text = await (await element.getProperty('textContent')).jsonValue()
expect(text).toBe('Loaded!')

In Playwright:

await page.goto(`${config.get('baseURL')}`)
const text = await page.textContent('#loadedchild')
expect(text).toBe('Loaded!')

Summary

If I was writing an e2e web testing framework from scratch in 2020 I would use Playwright + Jest. Playwright offers automatic waiting and a nice API, whilst Jest offers a solid runner with parallel support, automatic retries and the ability to easily generate HTML reports and capture screenshots.

I’ve created one repository for Playwright + Jest #
And another for Puppeteer + Jest to compare #

As an aside we put changing node libraries on hold for our e2e test framework and will look at some test infrastructure improvements instead.