Categories
Playwright

Consistently waiting for network responses in Playwright

One of the neat features I like about Playwright is how easily it is to wait for network responses that are triggered by actions like clicking an element in a browser. This is the way a lot of modern web applications work so it’s important to be able to handle this.

But I noticed the way I was writing code for this example scenario was problematic and that it could result in non-deterministic (flaky) test results. For example:

  test('can wait for network responses when clicking', async ({ page }) => {
    await page.goto('https://webdriverjsdemo.github.io/dynamic/')
    await page.click('#show')
    await page.waitForResponse('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io/posts')
    await expect(page.locator('#content')).toHaveText('[ { "id": 1, "title": "Post 1" }, { "id": 2, "title": "Post 2" }, { "id": 3, "title": "Post 3" } ]')
  })

I noticed in the example above there can be a race condition between Playwright clicking and waiting for the response, resulting in the waitForResponse to timeout as though it never responded when in fact it did but just before the click finished!

Thankfully Playwright makes it easy to handle these scenarios in a promise wrapper they suggest via their documentation:

// Note that Promise.all prevents a race condition
// between clicking and waiting for the response.
const [response] = await Promise.all([
  // Waits for the next response with the specified url
  page.waitForResponse('https://example.com/resource'),
  // Triggers the response
  page.click('button.triggers-response'),
]);

We can use the Promise.all call in our test like so, noting that there’s no awaits on the calls within Promise.all:

test('can wait for network responses when clicking', async ({ page }) => {
    await page.goto('https://webdriverjsdemo.github.io/dynamic/')
    await Promise.all([
      page.waitForResponse('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io/posts'),
      page.click('#show')
    ])
    await expect(page.locator('#content')).toHaveText('[ { "id": 1, "title": "Post 1" }, { "id": 2, "title": "Post 2" }, { "id": 3, "title": "Post 3" } ]')
  })

Whilst this works well, I find it a bit harder to write and remember not to just call these sequentially, so if we’re going to clicking things and waiting for responses a lot we can move it into a shared function like so:

export async function clickAndWait (page: Page, locator: string, expectResponseURL: string) {
  const [response] = await Promise.all([
    page.waitForResponse(expectResponseURL),
    page.click(locator)
  ])
  return response
}

This way our test becomes simpler and easier to read again:

test('can wait for network responses when clicking', async ({ page }) => {
    await page.goto('https://webdriverjsdemo.github.io/dynamic/')
    await clickAndWait(page, '#show', 'https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io/posts')
    await expect(page.locator('#content')).toHaveText('[ { "id": 1, "title": "Post 1" }, { "id": 2, "title": "Post 2" }, { "id": 3, "title": "Post 3" } ]')
  })

Have you had to use this feature? I hope this helps if you’ve been having problems with page.waitForResponse like me.

Categories
Playwright

A couple of cool new Playwright features

Version 1.19.1 of Playwright has been released and by reading the release notes I noticed a couple of cool new features (one of which was introduced in 1.18)

The first allows you to pass a message as an optional second parameter to an expect() call.

Say you’re calling an API and expect a certain status code:

test('can POST a REST API and check response using approval style', async ({ request }) => {
    const response = await request.post('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io/posts', { data: { title: 'Post 4' } })
    expect(response.status()).toBe(202)
  })

If this fails you don’t have much info:

1) scenarios/alltests.spec.ts:62:3 › All tests › can POST a REST API and check response using approval style 

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: 202
    Received: 201

With this new second parameter we can provide more info:

  test('can POST a REST API and check response using approval style', async ({ request }) => {
    const response = await request.post('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io/posts', { data: { title: 'Post 4' } })
    await expect(response.status(), `Response: ${await response.text()}`).toBe(202)
  })

and get more detailed output

Error: Response: {
      "title": "Post 4",
      "id": 4
    }

    Expected: 202
    Received: 201

The second new feature is the ability to use .toBeOK() assertions instead of having to assert on status codes. This asserts the response is in the 2xx status range which means the request was OK.

We can now do this:

test('can POST a REST API and check response using approval style', async ({ request }) => {
    const response = await request.post('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io/posts', { data: { title: 'Post 4' } })
    await expect(response, `Response: ${await response.text()}`).toBeOK();
  })

note: you can also call .not.toBeOK() if you are expecting an error.


Have you found any useful new Playwright features being released recently that you now can’t live without?

Categories
Playwright

Playwright Developer Advocate Advertised at Microsoft

A long time reader of this site kindly sent me a link to this tweet thinking I may be interested in the new Playwright Developer Advocate role at Microsoft.

https://twitter.com/JamesMontemagno/status/1490796849304854533

I am not looking for a remote working role (and I’m not even sure that job is remote outside of the USA anyway), but I wanted to share to my other readers in case they’d find this interesting!

Categories
Continuous Integration Playwright

Setting timezones for consistent Playwright results

In the system I am working on we have some tests for leave balances which are timezone dependent. I noticed these would fail on CI before 10am local time, and pass for the rest of our (work) day.

Since our local timezone is UTC+10 I realised that our CI system was using UTC and therefore wasn’t accurate in its estimations.

I discovered there are two ways to ensure a consistent timezone in our CI system.

Firstly we set the timezoneId for Playwright to our timezone using the list of timezones is available here.

This is our Playwright config file (playwright.config.ts):

use: {
        headless: true,
        locale: 'en-AU',
        timezoneId: 'Australia/Brisbane',
    }

And secondly we make sure the timezone is set correctly on the CI docker images. We use Bitbucket Pipelines are the config file (bitbucket-pipelines.yml) line looks like:

script:
  - cp -f /usr/share/zoneinfo/Australia/Brisbane /etc/localtime # set timezone
  - npm ci
  - npm test

Setting both the system and browser ensures consistent timezone execution on CI and we’re eliminated our inconsistencies by implementing this.

Categories
Playwright

Demonstrating the value of our Playwright tests

I don’t think anyone would disagree that creating and maintaining a set of e2e automated tests (like Playwright) takes a lot of time and effort, and therefore costs a lot of money.

So how does one demonstrate the value of investing time/effort/money into automated e2e testing?

My goal for automated e2e tests like Playwright is to have just enough that the team/project/product has zero manual regression testing required to release frequently.

I’ve written about this before, and it doesn’t mean no human testing, it just means no manual regression tests or test scripts a human follows to make sure existing functionality isn’t being broken by changes introduced into your system.

When looking at metrics I like the GQM approach where you start with goals, devise some questions to determine whether you’re achieving your goals, and create some metrics to answer those questions.

Our goal is already articulated above:

Goal: Zero manual regression testing

Some questions we could ask to see whether we’re achieving our goal would be:

  1. How good are our Playwright tests at catching regressions?
  2. How much manual regression testing do we perform?

And finally some metrics to answer our questions:

  1. Number of regression bugs found by the Playwright test suite
  2. Number of regression bugs not caught by the Playwright test suite
  3. Time spent performing manual regression testing

I like keeping things really simple when it comes to collecting and displaying metrics.

I created a Confluence page where I simply recorded regressions as they happened, in a table like such:

No.DateRegressionPlaywright/ManualRaised byStatus
1.10 JanWelcome screen doesn’t displayPlaywrightAlisterWIP
2.
Sample table to collect data

And using the table I created two simple graphs within the Confluence page to show our metrics:

Trend of regressions found
Playwright vs Manual

I think these metrics answer the question “how good are our Playwright tests at catching regressions?

To answer the other question “how much manual regression testing do we perform?” I can ask our QAs this question in our fortnightly catch up and record the results similar to the data above.

By using the answers to our questions we can determine whether or not we’re meeting our goal of having “zero manual regression testing” quite easily and whether Playwright is helping us achieve this goal.

What metrics do you collect around automated e2e testing?

Categories
Playwright

Debugging Playwright Tests with VS Code

I use VS Code as my text editor/IDE for writing Playwright tests. I can use use VS Code for debugging since it offers full debugging functionality like breakpoints and being able to see variables etc.

To enable this, there’s a couple of things you do:

  1. I created a debug task in my package.json file: "debug": "npx playwright test --headed --timeout=0" which means I can use npm run debug to execute a test without a timeout and showing the browser – by either adding a .only to a specific test, or telling it a file, eg. npm run debug ./scenarios/test1.spec.ts
  2. In VSCode use the “View → Debug Console” menu option, choose “Terminal” and make sure “JavaScript Debug Terminal” is set as the terminal type.
  3. Add a breakpoint in your code using the red dot in the left margin
  4. You can then use the npm run debug command which starts a debugging session where you can step through and see variables etc.
Debugging Playwright scripts in VS Code

Happy Debugging & Happy New Year! 🥳

Categories
e2e Testing Playwright

10 tips for successful e2e web app test automation

  1. Write independent automated tests: you should try remove dependencies on other tests or test data – this allows tests to be consistent, repeatable and to run in parallel (see #6).
  2. Set up data/state for each test via API calls: calling APIs is quick and efficient and can set up exactly what you need.
  3. Clean up data/state for each test using “after” hooks: this ensures test environments are kept clean and tidy and unwanted test data doesn’t cause issues with exploratory testing.
  4. Re-use browser authentication so you only need to log in once: this speeds up tests, see this post on how to do this with Playwright.
  5. Generate and use consistent (static) test data for each test: only generate unique/randomised values to satisfy uniqueness constraints, else use hard-coded known good values. Further reading here.
  6. Run all tests in parallel locally and in CI: hardware is powerful and there’s really no reason not to (unless you use Cypress and can’t 😝)
  7. Run new/updated tests at least 10 times locally in parallel before committing: this helps with reducing and removing non-deterministic tests and race conditions from your test suite.
  8. Use your automated test scripts to assist with manual/exploratory testing: for example you can easily set up state/accounts/sessions for testing – create npm commands so you can run npm run newuser for example to generate and log in as a brand new user ready for testing.
  9. Use linting/code autoformatting: such as JavaScript Standard Style, for consistently formatted code and not having to make decisions.
  10. Focus on reducing the need for manual regression testing, rather than code coverage: when you can confidently release your web application with no manual regression testing you know you have enough e2e automated tests.

What are your tips for successful web app test automation?

Categories
e2e Testing Playwright

Writing automated e2e tests for known buggy systems

Every system I’ve worked on, old or new, is full of known bugs (and unknown bugs for good measure 🤪). These known bugs are the ones that have never made it to the top of the bug backlog to be fixed because there’s always other more important work to do.

But what do you do with automated e2e tests that exercise such code and demonstrate such bugs?

Imagine a very simple example of a test that visits our page and asserts the title is correct.

Our page looks like this:

super simple webpage

And our Playwright code looks like this:

test.only('can have a test for a known bug in the system', async ({ page }) => {
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leave Page');
})

You can see our test has different text it asserts than what is displayed. The text is our test is what we actually want to display, however the system displays it differently so our test fails when we run it.

What do we do with such tests? There’s a few different options all with their own advantages and disadvantages.

Option One: Commit the failing test as it is

Advantages: test is pure and correct, test is still run on every build highlighting the functionality that is wrong

Disadvantages: each build will fail until this functionality is fixed, creating red/failed builds and not giving immediate feedback on other potential issues found in the builds and resulting in people losing confidence in overall build results.

I personally wouldn’t recommend this approach as I think the noise of the failing builds outweighs any benefits it has.

Option Two: Mark the failing test as skipped

test.skip('can have a test for a known bug in the system', async ({ page }) => {
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leave Page');
})

Advantages: no noise in builds since test no longer runs

Disadvantages: test can be forgotten about since it never runs and other issues could be introduced in the feature. For example if the text was changed to something else that is also wrong we wouldn’t know since the test is not being run.

Whilst this is preferable to option one this option often results in forgotten tests so I would also not recommend it.

Option Three: Update the assertion to be incorrect (with a comment)

test.only('can have a test for a known bug in the system', async ({ page }) => {
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leaving Page'); // BUG: This text should be WebDriverJs Demo Leave Page
  })

Advantages: if the text changes to any value (whether now correct, or still incorrect) the test will fail alerting us to a change in functionality

Disadvantages: the tests are no longer representative of what is expected of the system – the expectations contradict what is actually expected.

I probably prefer this to having a pending test but something doesn’t feel right about a false assertion.

Option Four:

Playwright actually offers a solution for scenarios like this, it’s the test.fail() syntax which marks a test as being expected to fail, so it is still run but if it fails it passes, and if it passes it fails 🙃

We can write the test like this:

test.only('can have a test for a known bug in the system', async ({ page }) => {
  test.fail() // BUG: The text is presently wrong
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leave Page');
  })

And when it fails it “passes” with a green cross:

If the system was fixed this test would then fail, and we’d know to remove the test.fail()line.

Advantages: if the text changes to the correct value we will know as this test will pass when we don’t expect it to. We can keep our assertions correct/pure.

Disadvantages: if the test was to fail in a different way we wouldn’t know about it since all the test cares about is that it fails (which we’re expecting).

Whilst this can hide other test failures, since I aim to write independent tests I can live with it potentially hiding other issues so this is my preferred approach to known failures.

How do you deal with known failures? Any of these ways or another I’ve missed?

Categories
Playwright

Reusable Authentication Across Playwright Tests

Most web apps require user authentication which requires the user to login at the start of an e2e test. A very basic example is:

import { test, expect } from '@playwright/test'

test.describe.parallel('Unauthenticated tests', () => {
  test('can view as guest', async ({ page }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Hello Please Sign In')).toBeVisible()
  })
})

test.describe.parallel('Authenticated tests', () => {
  test('can view as admin', async ({ page }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await page.type('#firstname', 'Admin')
    await page.type('#surname', 'User')
    await page.click('#ok')
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=AdminUser')).toBeVisible()
  })

  test('can view as standard user', async ({ page }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await page.type('#firstname', 'Standard')
    await page.type('#surname', 'Person')
    await page.click('#ok')
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=StandardPerson')).toBeVisible()
  })
})

Whilst it’s easy to move the common code which authenticates (sets the cookies/tokens) into a login function that uses Playwright to visit a login page which is called from each test, Playwright offers something much better in that it can save browser storage state and re-use it. The idea is to login once and re-use the resulting browser state for every test that requires that role to work.

If the cookies/tokens don’t expire, you can capture them once, commit them to your code repository and simply re-use them:

First to capture:

  test('can view as admin', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await page.type('#firstname', 'Admin')
    await page.type('#surname', 'User')
    await page.click('#ok')
    await context.storageState({ path: 'storage/admin.json' });
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=AdminUser')).toBeVisible()
  })

  test('can view as standard user', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await page.type('#firstname', 'Standard')
    await page.type('#surname', 'Person')
    await page.click('#ok')
    await context.storageState({ path: 'storage/user.json' });
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=StandardPerson')).toBeVisible()
  })

And then to re-use the captured files:

import { test, expect } from '@playwright/test'

test.describe.parallel('Unauthenticated tests', () => {
  test('can view as guest', async ({ page }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Hello Please Sign In')).toBeVisible()
  })
})

test.describe.parallel('Administrator tests', () => {
  test.use({storageState: './storage/admin.json'})
  test('can view as admin', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=AdminUser')).toBeVisible()
  })
})

test.describe.parallel('User tests', () => {
  test.use({storageState: './storage/user.json'})
  test('can view as standard user', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=StandardPerson')).toBeVisible()
  })
})

But what if like most apps your authentication cookies/tokens do expire?

Fortunately you can dynamically create the session state once per test run – in the global hooks – then simply refer to those same local storage files in each test.

You can create a file like global-setup.ts which generates our storage state files once per test run:

// global-setup.ts
import { Browser, chromium, FullConfig } from '@playwright/test'

async function globalSetup (config: FullConfig) {
  const browser = await chromium.launch()
  await saveStorage(browser, 'Standard', 'Person', 'storage/user.json')
  await saveStorage(browser, 'Admin', 'User', 'storage/admin.json')
  await browser.close()
}

async function saveStorage (browser: Browser, firstName: string, lastName: string, saveStoragePath: string) {
  const page = await browser.newPage()
  await page.goto('http://webdriverjsdemo.github.io/auth/')
  await page.type('#firstname', firstName)
  await page.type('#surname', lastName)
  await page.click('#ok')
  await page.context().storageState({ path: saveStoragePath })
}

export default globalSetup

which is referenced in playwright.config.ts

// playwright.config.ts
module.exports = {
  globalSetup: require.resolve('./global-setup'),
  reporter: [['list'], ['html']],
  retries: 0,
  use: {
    headless: true,
    screenshot: 'only-on-failure',
    video: 'retry-with-video',
    trace: 'on-first-retry'
  }
}

Once you have this set up our tests remain the same but the local storage values are captured and set once per test run:

test.describe.parallel('Admin tests', () => {
  test.use({ storageState: './storage/admin.json' })
  test('can view as admin', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=AdminUser')).toBeVisible()
  })
})

test.describe.parallel('User tests', () => {
  test.use({ storageState: './storage/user.json' })
  test('can view as standard user', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=StandardPerson')).toBeVisible()
  })
})

You can call test.use({ storageState: './storage/user.json' }) for a file or a test.describe block, so if all your tests in your test file use the same authentication role place it outside a test.describe block, otherwise place it within a test.describe block of tests that use the same authentication role. Different roles? Use different test.describe blocks with different test.use calls to different files in each.

What do you think of Playwright’s ability to capture and use browser storage state?

Categories
Playwright

Five reasons why Playwright is better than Cypress

It’s pretty obvious I am not a Cypress fan – heck I wrote a whole post about it 2 years ago.

I’ve since become very keen on Playwright and I thought I’d revisit Cypress to compare it with Playwright and see whether any of my long-standing gripes about Cypress have been addressed in 2+ years since I’ve used it.

Versions compared:

Cypress: 8.7.0 on Electron 93
Playwright: 1.16.0

Reason 1: Playwright is so much faster than Cypress

Up to 4x faster.

Take this simple example taken directly from the Cypress doco:

describe('My First Test', () => {
    it('clicking "type" shows the right headings', () => {
      cy.visit('https://example.cypress.io')
  
      cy.contains('type').click()
  
      // Should be on a new URL which includes '/commands/actions'
      cy.url().should('include', '/commands/actions')
  
      // Get an input, type into it and verify that the value has been updated
      cy.get('.action-email')
        .type('fake@email.com')
        .should('have.value', 'fake@email.com')
    })
})

Takes 8 seconds running on my M1 Macbook Air

 ✔  demo-spec.js                             00:08

I wrote the same example in Playwright:

import { test, expect } from '@playwright/test'

test.describe.parallel('My First Test', () => {
  test('clicking "type" shows the right headings', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    // Should be on a new URL which includes '/commands/actions'
    await page.waitForURL(/.+\/commands\/actions$/)
    // Get an input, type into it and verify that the value has been updated
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
})

Running 1 test using 1 worker

  ✓  scenarios/compare.cypress.spec.ts:4:3 › My First Test › clicking "type" shows the right headings (1s)


  1 passed (2s)

4 times faster 😎

Reason 2: Playwright has first-class support for parallel running of tests on a single machine both locally and on CI without a subscription or account required

Cypress on the other hand requires you to set your tests to “Record” everything they do which sends everything to Cypress (or Sorry Cypress) if you want to run in parallel, and using a single machine is not recommended (despite it working perfectly in Playwright using my Macbook Air M1)

“While parallel tests can also technically run on a single machine, we do not recommend it since this machine would require significant resources to run your tests efficiently.”

https://docs.cypress.io/guides/guides/parallelization#Overview

Either this is true and Cypress is so bloated that running in parallel locally needs crazy expensive hardware, or the Cypress team want users to pay for their parallel execution service. Or both?

Reason 3: Playwright supports parallel tests within a single test file – to run Cypress tests in parallel you need to split them across files.

For example, in Playwright I can run these 8 tests in parallel:

import { test, expect } from '@playwright/test'

test.describe.parallel('My First Test', () => {
  test('clicking "type" shows the right headings 1', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 2', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 3', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 4', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 5', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 6', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 7', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 8', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
})

which takes only 6 seconds on my Macbook Air 😎

> npx playwright test "./scenarios/compare.cypress.bulk.spec.ts"

Running 8 tests using 4 workers

  ✓  scenarios/compare.cypress.bulk.spec.ts:4:3 › My First Test › clicking "type" shows the right headings 1 (4s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:11:3 › My First Test › clicking "type" shows the right headings 2 (4s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:18:3 › My First Test › clicking "type" shows the right headings 3 (4s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:25:3 › My First Test › clicking "type" shows the right headings 4 (4s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:46:3 › My First Test › clicking "type" shows the right headings 7 (1s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:32:3 › My First Test › clicking "type" shows the right headings 5 (1s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:39:3 › My First Test › clicking "type" shows the right headings 6 (1s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:53:3 › My First Test › clicking "type" shows the right headings 8 (1s)

  8 passed (6s)

To possibly run these in parallel (using a server) in Cypress I’d need to structure these as individual files each with a single spec:

And to run them on the same machine I ran these 8 scenarios in Playwright in 6 seconds takes 45 seconds! 7.5 times slower!

Reason 4: Playwright fully supports async/await syntax for clean, readable code.

This is the Playwright code:

import { test, expect } from '@playwright/test'

test.describe.parallel('My First Test', () => {
  test('clicking "type" shows the right headings', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
})

No chaining, no hard to understand code, no magic. You can see where test and expect come from, you can see where page comes from.

The same test in Cypress:

describe('My First Test', () => {
    it('clicking "type" shows the right headings', () => {
      cy.visit('https://example.cypress.io')
    
      cy.contains('type').click()
  
      // Should be on a new URL which includes '/commands/actions'
      cy.url().should('include', '/commands/actions')
  
      // Get an input, type into it and verify that the value has been updated
      cy.get('.action-email')
        .type('fake@email.com')
        .should('have.value', 'fake@email.com')
    })
})

Note how the function calls chain together – in this simple example it’s not too bad but it quickly gets out of hand. Also note there’s no indication that this is synchronous code – no await on any calls. Also where does this cy thing come from – oh yes that’s magic. Same for describe, again magic. No imports but lots of magic.

Reason 5: Playwright doesn’t need plugins

There’s so many limitations in Cypress there’s pretty much a plugin for everything. And there’s still so may trade-offs that can’t be be fixed by a plugin.

Need to send the tab key to the browser? There’s a plugin for that. And it doesn’t work well at all according to the comments on the Github issue for lack of tab key support open since 2016 (and still open since I last wrote about Cypress).

Need to upload a file? There’s also a plugin for that.

Need to run tests n-times in a row to measure repeatability? There’s a plugin for that.

Fortunately Playwright supports all of these things, and many more, natively. No plug-ins needed 😎

Summary

As you can see I’m still not a Cypress fan all these years later 😀

It’s just that now there’s a tool which is so much better and freely available to use.

So I’ll stand by my summary from 2 years ago which I can repeat here:

From a distance Cypress looks like a polished tool for automated testing – I just think it’s incorrectly marketed as an end-to-end testing tool when it’s really only good for component testing. There are too many limitations in the tool in acting like a real user to use it to create true end-to-end automated tests.

https://alisterbscott.com/2019/07/06/my-thoughts-on-cypress-io/