Categories
Automated Testing Playwright

TypeScript!

I recently changed jobs πŸŽ‰

One of the biggest things I noticed since changing jobs nearly 3 years ago was the prevalence of TypeScript: JavaScript With Syntax For Types. Most, if not all, of the jobs I were looking at were dev teams using TypeScript – driven by the increasing use full stack Node.js.

It’s only week 2 in my new job but I’ve already set up an e2e testing framework using Playwright and it’s all in TypeScript 😊 I thought I’d share the main differences and I’ve created a sample repository to do so!

To use TypeScript on a front-end codebase running in a browser you need to compile/transpile it into JavaScript (which a browser understands).

Fortunately Playwright doesn’t require this step as Playwright runs your TypeScript code directly.

The first thing to do is add typescript to your project and add a tsconfig.json file to your project.

I use Playwright Test – and the main difference between a TypeScript spec file (*.spec.ts) and a JavaScript one (*.spec.js) is the import statements:

import { test, expect } from '@playwright/test'
import { visitHomePage } from '../lib/actions/nav'

test('can wait for an element to appear', async ({ page }) => {
  await visitHomePage(page)
  await page.waitForSelector('#elementappearschild', { state: 'visible', timeout: 5000 })
})

test('can use an element that appears after on page load', async ({ page }) => {
  await visitHomePage(page)
  const text = await page.textContent('#loadedchild')
  expect(text).toBe('Loaded!')
})

whereas in JavaScript the first two lines were:

const nav = require('../lib/actions/nav')
const { test, expect } = require('@playwright/test')

The main differences are in the library code.

import { Page } from '@playwright/test'
import config from 'config'

export async function visitHomePage (page: Page) {
  return await page.goto(`${config.get('baseURL')}`)
}

export async function goToPath (page: Page, path: string) {
  return await page.goto(`${config.get('baseURL')}/${path}`)
}

You will see the types being added, fortunately Playwright provides type definitions which are imported here.

Previously, this same file looked like:

const config = require('config')

async function visitHomePage (page) {
  return await page.goto(`${config.get('baseURL')}`)
}

async function goToPath (page, path) {
  return await page.goto(`${config.get('baseURL')}/${path}`)
}

module.exports = {
  visitHomePage,
  goToPath
}

The other main thing to update is if you use classes you define types against all the properties.

Everything else is about the same.

My fully working TypeScript example of Playwright Test is here: https://github.com/alisterscott/playwright-test-ts-demo

Enjoy TypeScripting πŸ˜€

Categories
CSharp e2e Testing

Playwright in C# (.NET Core)

Whilst I was doing some reading of the Playwright docs I noticed they have C# bindings (as well as Python & Java, but not Ruby) – and since it’s been a couple of years since I’ve used C# I thought I’d take a look at how it works – especially considering .NET Core has support for Mac which makes working in C# .NET so much easier for me.

First I downloaded Visual Studio for Mac Community Edition which was pretty easy to install and this included the .NET Core framework which includes the dotnet command line tool.

One thing about .NET Core is there’s a lot more command line options to do things.

Installing Playwright?

dotnet tool install --global Microsoft.Playwright.CLI
playwright install

Creating a new NUnit Project?

dotnet new nunit -n PlaywrightNunitDemo

Adding Playwright, Building and Running Your Tests?

dotnet add package Microsoft.Playwright.NUnit
dotnet build
dotnet test

My tests end up looking like this:

using System.Threading.Tasks;
using Microsoft.Playwright.NUnit;
using NUnit.Framework;
using PlaywrightNunitDemo.lib;

namespace PlaywrightNunitDemo
{
    [Parallelizable(ParallelScope.Self)]
    public class Scenario02 : PageTest
    {
        [Test]
        public async Task CanCheckForErrors()
        { 
            string errors = await AppHelpers.VisitURLGetErrors(Page, "/error");
            Assert.AreEqual(": Purple Monkey Dishwasher Error", errors);
        }

        [Test]
        public async Task CanCheckForNoErrors()
        {
            string errors = await AppHelpers.VisitURLGetErrors(Page);
            Assert.AreEqual(string.Empty, errors);
        }
    }
}

and my reusable Playwright code can live in a class with static methods:

using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
using Microsoft.Playwright;

namespace PlaywrightNunitDemo.lib
{
    public class AppHelpers
    {
        public static async Task<IResponse> VisitURL(IPage page, string path = "/")
        { 
            var config = new ConfigurationBuilder().AddJsonFile("appsettings.json").Build();
            string url = config["BASE_URL"] + path;
            return await page.GotoAsync(url);  
        }

        public static async Task<string> VisitURLGetErrors(IPage page, string path = "/")
        {
            var errors = "";
            page.PageError += (_, exception) => { errors = errors + exception; };
            await VisitURL(page, path);
            return errors;
        }
    }
}

Summary

I quite enjoyed working in C# in previous roles. There’s something nice about typed languages which makes you confident when writing the code that things will do what you expect. Having .NET Core easily accessible on a Mac and being able to use Playwright makes C# even nicer.

Example Code

My example code is: https://github.com/alisterscott/PlaywrightNunitDemo

I’ll see if I can get this running on Linux in CircleCI soon.

Categories
Playwright

Playwright’s “retry with video”

Another feature I love in Playwright Test is the retry with video option.

In your playwright.config.js file you specify:

module.exports = {
  use: {
    headless: true,
    screenshot: 'only-on-failure',
    video: 'retry-with-video'
  },
  retries: 1
}

The video option combined with the retry option means that if a test fails Playwright will re-run it whilst recording a video of that re-run.

If the test fails on the second attempt it will capture the video in the test results (alongside the screenshot if you config it to capture screenshots also).

It’s a webm video that is lightweight, easy to capture during CI, and easy to share:

Categories
Playwright

Ensuring consistent Playwright tests with “repeat each”

My workflow to ensure any new Playwright Test test I write is deterministic and repeatable:

  1. Mark the new test as .only temporarily locally during development
  2. Run npx playwright test --repeat-each 100 where 100 is the number of times you want to run it in parallel locally
  3. Make sure it passes 100% of the times

Here’s an example of how it looks (using 10 times for conciseness):

Playwright Test results showing consistent test runs

I’ve used this to discover a test failed about 5% of the times I ran it which was one of the only Playwright tests I’ve written so far that needed an explicit wait for element.

Categories
e2e Testing Playwright

Playwright Test Runner

I’d previously shared how to set up Playwright with Jest as a test runner which enabled us to do some cool things like:

  • Parallel test execution,
  • Automatic retries,
  • HTML reports (using Jest Stare), and
  • Screenshots when failing

It turns out that Playwright now supports all of the features without having to use Jest! It’s called the Playwright Test Runner.

I went ahead and created a fork of my existing code to show how to set this up with the Playwright Test Runner.

Specifying Tests

Fortunately the syntax to specify tests is almost identical to Jest, but you don’t need to worry about spawning browser contexts yourself 😎

This Jest test:

jest.retryTimes(1)

test('can use xpath selectors to find elements', async () => {
  global.page = await pages.spawnPage()
  await nav.visitHomePage(global.page)
  await home.clickScissors(global.page)
}, jestTimeoutMS)

becomes:

const { test } = require('@playwright/test')

test('can use xpath selectors to find elements', async ({ page }) => {
  await nav.visitHomePage(page)
  await home.clickScissors(page)
})

Running in Parallel

Like Jest, Playwright automatically runs your tests in parallel, sharing isolated browser pages within browsers, and spawning browsers across test files. The time taken was pretty much identical:

Jest

 PASS  scenarios/example4.spec.js
 PASS  scenarios/example3.spec.js
 PASS  scenarios/example5.spec.js
 PASS  scenarios/api.spec.js
 PASS  scenarios/example2.spec.js
 PASS  scenarios/example.spec.js

Test Suites: 6 passed, 6 total
Tests:       10 passed, 10 total
Snapshots:   2 passed, 2 total
Time:        5.514 s

Playwright Test

Running 10 tests using 6 workers

  βœ“ scenarios/api.spec.js:5:1 β€Ί [chromium] can GET a REST API and check response using approval style (590ms)
  βœ“ scenarios/example.spec.js:4:1 β€Ί [chromium] can wait for an element to appear (3s)
  βœ“ scenarios/example2.spec.js:4:1 β€Ί [chromium] can handle alerts (4s)
  βœ“ scenarios/api.spec.js:12:1 β€Ί [chromium] can GET a REST API and check response using assertion style (587ms)
  βœ“ scenarios/example3.spec.js:5:1 β€Ί [chromium] can check for errors when there should be none (1s)
  βœ“ scenarios/example4.spec.js:5:1 β€Ί [chromium] can check for errors when there are present (1s)
  βœ“ scenarios/api.spec.js:25:1 β€Ί [chromium] can POST a REST API and check response using approval style (536ms)
  βœ“ scenarios/example5.spec.js:6:1 β€Ί [chromium] can use xpath selectors to find elements (988ms)
  βœ“ scenarios/api.spec.js:32:1 β€Ί [chromium] can POST a REST API and check response using assertion style (545ms)
  βœ“ scenarios/example.spec.js:9:1 β€Ί [chromium] can use an element that appears after on page load (220ms)


  10 passed (5s)

Automatic Retries

Jest supports automatic retries in the test itself:

jest.retryTimes(1)

In Playwright Test you can configure it globally or per run:

// playwright.config.js
module.exports = {
  use: {
    retries: 2
  }
}
npx playwright test --retries=3

HTML Reports

I couldn’t see HTML report output for Playwright Test – but there’s heaps of others like JSON and Junit (for CI).

Screenshots

Screenshots are easily configured for Playwright Test: for each test never, always or on failure (my preference).

module.exports = {
  use: {
    screenshot: 'only-on-failure'
  }
}

But wait there’s more

The Playwright Test Runner also has these very cool features:

Visual comparisons (visdifs)

test('can wait for an element to appear', async ({ page }) => {
  await nav.visitHomePage(page)
  await page.waitForSelector('#elementappearschild', { visible: true, timeout: 5000 })
  expect(await page.screenshot()).toMatchSnapshot('element-appears.png')
})

At any point in your tests you can take a screenshot and store this to visually compare with future runs with fine grained tweaking of matches. The best part about this, in my opinion, is these visuals are stored alongside your tests (not in some third party system) so you know exactly what you’re expecting to see 😎

The one downside is the screenshots are platform specific so when I checked in those generated on my Mac then CI failed as it was running on Linux and didn’t have the baseline files to compare to. I just downloaded the captured files from CircleCI and stored them as a baseline.

And it also supports other comparisons – similar to Jest snapshots I have previously demonstrated. You just need to stringify them first:

test('can GET a REST API and check response using approval style', async () => {
  const request = supertest('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io')
  const response = await request.get('/posts')
  expect(response.status).toBe(200)
  expect(JSON.stringify(response.body)).toMatchSnapshot('posts.txt')
})

Closing Thoughts

I was pretty blown away by the Playwright Test runner. It offers everything Jest provides with less code I had to write, plus it has in built visual comparison tools that can also be extended to do API approval snapshot testing. Playwright is well becoming my e2e test automation tool of choice.

Show me the Code

Of course: code is here: https://github.com/alisterscott/playwright-test-demo

Passing CI here: https://app.circleci.com/pipelines/github/alisterscott/playwright-test-demo/7/workflows/fe13f579-1f53-4a01-834e-6a4e7388e22c

Categories
e2e Testing

Playwright + Jest = πŸ’–

We were looking at a possible replacement for our dated Protractor + Cucumber e2e testing framework. As we move away from Angular to React microapps we have found that Protractor doesn’t work very well/efficiently and cucumber isn’t giving us any benefits.

It was a good opportunity to do some research/tinkering to answer the question lingering in my mind: in 2020 what e2e testing tool would I use by default for a dynamic react based web application?

After some experimentation it came down to Puppeteer + Jest or Playwright + Jest. I’ll compare those in this post.

Why Jest?

As explained previously we don’t need a BDD framework, but we do need something that allows us to specify our tests, create assertions and run these tests (in parallel). Jest, particularly when using the Jest Circus test runner, seems the most mature tool in this regard in 2020. Whilst Jest is often associated with React as it handles the snapshot test results also, it doesn’t need to be used with React and it’s possible and easy to use it as a standalone Node.js testing library.

Parallel Support

Jest by default runs tests across files in parallel and uses the available resources to scale appropriately using processes/threads. I’ve found this particularly good as I write independent e2e tests which can scale through parallelism, and using Puppeteer and Playwright you can spawn new incognito browser contexts to run these.

Auto-Retry Support

Whilst I believe in writing deterministic e2e automated tests, since e2e tests are full-stack there are often external dependencies and services beyond our control (like a third party domain provider) and I’d rather prioritize test reliability over test perfection. With this mind I think it’s important to be able to automatically retry a single failing test scenario before failing a build, and the Jest Circus test runner supports this:

jest.retryTimes(1)

test('can wait for an element to appear', async () => {
  global.page = await pages.spawnPage()
  await nav.visitHomePage(global.page)
  await global.page.waitForSelector('#elementappearschild', { visible: true, timeout: 5000 })
}, jestTimeoutMS)

HTML Reports and Screenshots

Nice looking HTML reports are easy to achieve by using jest-stare and screenshots are easy to generate using Jest Circus hooks.

Playwright or Puppeteer?

I’m a fan of Puppeteer however Playwright is a much nicer browser automation library. Whilst it adds support for Firefox and Webkit, even if you’re running your e2e tests in one browser (Chromium) I’d still recommend Playwright over Puppeteer any day of the week. Here’s why:

Automatic Waiting

I’ve found the automatic waiting in Playwright just worksℒ️ Especially when dealing with dynamically rendered react web apps I’ve found my Puppeteer code looks is full of page.waitFor calls to make it run reliably:

await page.waitFor('#loadedchild', { visible: true, timeout: 5000 })

Whilst this is occasionally necessary in Playwright (in particular waiting for an iFrame to switch into), I’ve found it’s almost never required which reminds me of good old Watir days.

Nicer API

Whilst the APIs are similar, Playwright is just nicer to use. Take grabbing some text from a div, in Puppeteer:

await page.goto(`${config.get('baseURL')}`)
await page.waitFor('#loadedchild', { visible: true, timeout: 5000 })
const element = await page.$('#loadedchild')
const text = await (await element.getProperty('textContent')).jsonValue()
expect(text).toBe('Loaded!')

In Playwright:

await page.goto(`${config.get('baseURL')}`)
const text = await page.textContent('#loadedchild')
expect(text).toBe('Loaded!')

Summary

If I was writing an e2e web testing framework from scratch in 2020 I would use Playwright + Jest. Playwright offers automatic waiting and a nice API, whilst Jest offers a solid runner with parallel support, automatic retries and the ability to easily generate HTML reports and capture screenshots.

I’ve created one repository for Playwright + Jest #
And another for Puppeteer + Jest to compare #

As an aside we put changing node libraries on hold for our e2e test framework and will look at some test infrastructure improvements instead.

Categories
Automated Acceptance Testing Automated Testing Selenium

Playing with Playwright

Playwright is a new browser automation library from Microsoft:

Playwright is a Node library to automate the Chromium, WebKit and Firefox browsers with a single API. It enables cross-browser web automation that is ever-green, capable, reliable and fast.

https://github.com/microsoft/playwright

I’m a big fan of Puppeteer, so this section in their FAQ stood out to me:

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer project is active and is maintained by Google.

We are the same team that originally built Puppeteer at Google, but has since then moved on. Puppeteer proved that there is a lot of interest in the new generation of ever-green, capable and reliable automation drivers. With Playwright, we’d like to take it one step further and offer the same functionality for all the popular rendering engines. We’d like to see Playwright vendor-neutral and shared governed.

https://github.com/microsoft/playwright#q-how-does-playwright-relate-to-puppeteer

Playwright uses similar concepts to Puppeteer:

“Due to the similarity of the concepts and the APIs, migration between the two should be a mechanical task.”

https://github.com/microsoft/playwright#q-how-does-playwright-relate-to-puppeteer

Luckily I have a demo test suite written in Puppeteer which I have cloned and converted to use Playwright to see how it works, and compares.

Here are my thoughts:

I really, really like the BrowserContext concept

In Puppeteer, and WebDriverJs, you have Browsers and Pages. Each Page in a Browser share the state across the Browser, so to create isolated tests using the same Browser (to avoid the inefficiencies of spawning a Browser per test) you need custom code to delete all cookies and local storage between tests. Playwright solves this with the BrowserContext object which is a new incognito window where its pages are created: each test can use the same browser but a different BrowserContext. Super cool πŸ‘Œ

It automatically waits to click, and supports xpath expressions

Playwright automatically waits for elements to be available and visible before clicking, by default, and also has the same API for xpath expressions, which means this Puppeteer code:

await page.goto( `${ config.get( 'baseURL' )}` );
await page.waitForXPath( '//span[contains(., "Scissors")]' );
const elements = await page.$x( '//span[contains(., "Scissors")]' );
await elements[0].click();
await page.waitForXPath( '//div[contains(., "Scissors clicked!")]' );

becomes a lot cleaner:

await page.goto( `${ config.get( 'baseURL' )}` );
await page.click( '//span[contains(., "Scissors")]' );
await page.waitFor( '//div[contains(., "Scissors clicked!")]' );

It supports three “browsers” but not as you know them

Q: Does Playwright support new Microsoft Edge?

The new Microsoft Edge browser is based on Chromium, so Playwright supports it.

https://github.com/microsoft/playwright#q-does-playwright-support-new-microsoft-edge

Playwright supports three “browsers” but not as you know them. I’d say it supports three rendering engines (Chromium, WebKit & Gecko) rather than Browsers as you can only use the (somewhat modified) browsers that come bundled with Playwright over using an already installed browser on your operating system (like Selenium does). This makes it easier to ensure consistency of test runs since the library is bundled with the browsers, but there are some risks your tests could pass on the bundled browsers but fail on “real” browsers. I would say that the claim it supports running on Microsoft Edge is a little misleading.

I’m unsure of CircleCI Support for WebKit and Firefox

I was able to get my tests running against Chromium on CircleCI using the same configuration as Puppeteer, however I couldn’t get the WebKit or Firefox tests to run on CircleCI even when having the default CircleCI browsers installed. I didn’t want to invest the time, but it is probably due to some headless Linux dependencies missing which could be solved in the project config.

Conclusion

If the only thing Playwright did better than Puppeteer was also supporting WebKit and Gecko then I wouldn’t suggest using it over Puppeteer, since Puppeteer is closely aligned with Chromium, and I’m going to run my tests solely in Chrome/Chromium anyway. I don’t believe in running the same e2e tests in multiple browsers: the maintenance overhead outweighs the benefits in my experience.

However, Playwright offers a much nicer BrowserContext concept, and the xpath support is much nicer (although I rarely use xpath expressions anyway).

If anything I am hoping Puppeteer adds support for BrowserContexts – I’ve raised a feature request here so feel free to comment on it if you think it would be a good idea.

All the sample code is available here: https://github.com/alisterscott/playwright-demo