Categories
e2e Testing Playwright

Writing automated e2e tests for known buggy systems

Every system I’ve worked on, old or new, is full of known bugs (and unknown bugs for good measure ๐Ÿคช). These known bugs are the ones that have never made it to the top of the bug backlog to be fixed because there’s always other more important work to do.

But what do you do with automated e2e tests that exercise such code and demonstrate such bugs?

Imagine a very simple example of a test that visits our page and asserts the title is correct.

Our page looks like this:

super simple webpage

And our Playwright code looks like this:

test.only('can have a test for a known bug in the system', async ({ page }) => {
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leave Page');
})

You can see our test has different text it asserts than what is displayed. The text is our test is what we actually want to display, however the system displays it differently so our test fails when we run it.

What do we do with such tests? There’s a few different options all with their own advantages and disadvantages.

Option One: Commit the failing test as it is

Advantages: test is pure and correct, test is still run on every build highlighting the functionality that is wrong

Disadvantages: each build will fail until this functionality is fixed, creating red/failed builds and not giving immediate feedback on other potential issues found in the builds and resulting in people losing confidence in overall build results.

I personally wouldn’t recommend this approach as I think the noise of the failing builds outweighs any benefits it has.

Option Two: Mark the failing test as skipped

test.skip('can have a test for a known bug in the system', async ({ page }) => {
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leave Page');
})

Advantages: no noise in builds since test no longer runs

Disadvantages: test can be forgotten about since it never runs and other issues could be introduced in the feature. For example if the text was changed to something else that is also wrong we wouldn’t know since the test is not being run.

Whilst this is preferable to option one this option often results in forgotten tests so I would also not recommend it.

Option Three: Update the assertion to be incorrect (with a comment)

test.only('can have a test for a known bug in the system', async ({ page }) => {
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leaving Page'); // BUG: This text should be WebDriverJs Demo Leave Page
  })

Advantages: if the text changes to any value (whether now correct, or still incorrect) the test will fail alerting us to a change in functionality

Disadvantages: the tests are no longer representative of what is expected of the system – the expectations contradict what is actually expected.

I probably prefer this to having a pending test but something doesn’t feel right about a false assertion.

Option Four:

Playwright actually offers a solution for scenarios like this, it’s the test.fail() syntax which marks a test as being expected to fail, so it is still run but if it fails it passes, and if it passes it fails ๐Ÿ™ƒ

We can write the test like this:

test.only('can have a test for a known bug in the system', async ({ page }) => {
  test.fail() // BUG: The text is presently wrong
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leave Page');
  })

And when it fails it “passes” with a green cross:

If the system was fixed this test would then fail, and we’d know to remove the test.fail()line.

Advantages: if the text changes to the correct value we will know as this test will pass when we don’t expect it to. We can keep our assertions correct/pure.

Disadvantages: if the test was to fail in a different way we wouldn’t know about it since all the test cares about is that it fails (which we’re expecting).

Whilst this can hide other test failures, since I aim to write independent tests I can live with it potentially hiding other issues so this is my preferred approach to known failures.

How do you deal with known failures? Any of these ways or another I’ve missed?

Categories
Playwright

Reusable Authentication Across Playwright Tests

Most web apps require user authentication which requires the user to login at the start of an e2e test. A very basic example is:

import { test, expect } from '@playwright/test'

test.describe.parallel('Unauthenticated tests', () => {
  test('can view as guest', async ({ page }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Hello Please Sign In')).toBeVisible()
  })
})

test.describe.parallel('Authenticated tests', () => {
  test('can view as admin', async ({ page }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await page.type('#firstname', 'Admin')
    await page.type('#surname', 'User')
    await page.click('#ok')
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=AdminUser')).toBeVisible()
  })

  test('can view as standard user', async ({ page }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await page.type('#firstname', 'Standard')
    await page.type('#surname', 'Person')
    await page.click('#ok')
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=StandardPerson')).toBeVisible()
  })
})

Whilst it’s easy to move the common code which authenticates (sets the cookies/tokens) into a login function that uses Playwright to visit a login page which is called from each test, Playwright offers something much better in that it can save browser storage state and re-use it. The idea is to login once and re-use the resulting browser state for every test that requires that role to work.

If the cookies/tokens don’t expire, you can capture them once, commit them to your code repository and simply re-use them:

First to capture:

  test('can view as admin', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await page.type('#firstname', 'Admin')
    await page.type('#surname', 'User')
    await page.click('#ok')
    await context.storageState({ path: 'storage/admin.json' });
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=AdminUser')).toBeVisible()
  })

  test('can view as standard user', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await page.type('#firstname', 'Standard')
    await page.type('#surname', 'Person')
    await page.click('#ok')
    await context.storageState({ path: 'storage/user.json' });
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=StandardPerson')).toBeVisible()
  })

And then to re-use the captured files:

import { test, expect } from '@playwright/test'

test.describe.parallel('Unauthenticated tests', () => {
  test('can view as guest', async ({ page }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Hello Please Sign In')).toBeVisible()
  })
})

test.describe.parallel('Administrator tests', () => {
  test.use({storageState: './storage/admin.json'})
  test('can view as admin', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=AdminUser')).toBeVisible()
  })
})

test.describe.parallel('User tests', () => {
  test.use({storageState: './storage/user.json'})
  test('can view as standard user', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=StandardPerson')).toBeVisible()
  })
})

But what if like most apps your authentication cookies/tokens do expire?

Fortunately you can dynamically create the session state once per test run – in the global hooks – then simply refer to those same local storage files in each test.

You can create a file like global-setup.ts which generates our storage state files once per test run:

// global-setup.ts
import { Browser, chromium, FullConfig } from '@playwright/test'

async function globalSetup (config: FullConfig) {
  const browser = await chromium.launch()
  await saveStorage(browser, 'Standard', 'Person', 'storage/user.json')
  await saveStorage(browser, 'Admin', 'User', 'storage/admin.json')
  await browser.close()
}

async function saveStorage (browser: Browser, firstName: string, lastName: string, saveStoragePath: string) {
  const page = await browser.newPage()
  await page.goto('http://webdriverjsdemo.github.io/auth/')
  await page.type('#firstname', firstName)
  await page.type('#surname', lastName)
  await page.click('#ok')
  await page.context().storageState({ path: saveStoragePath })
}

export default globalSetup

which is referenced in playwright.config.ts

// playwright.config.ts
module.exports = {
  globalSetup: require.resolve('./global-setup'),
  reporter: [['list'], ['html']],
  retries: 0,
  use: {
    headless: true,
    screenshot: 'only-on-failure',
    video: 'retry-with-video',
    trace: 'on-first-retry'
  }
}

Once you have this set up our tests remain the same but the local storage values are captured and set once per test run:

test.describe.parallel('Admin tests', () => {
  test.use({ storageState: './storage/admin.json' })
  test('can view as admin', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=AdminUser')).toBeVisible()
  })
})

test.describe.parallel('User tests', () => {
  test.use({ storageState: './storage/user.json' })
  test('can view as standard user', async ({ page, context }) => {
    await page.goto('http://webdriverjsdemo.github.io/auth/')
    await expect(page.locator('text=Welcome name=StandardPerson')).toBeVisible()
  })
})

You can call test.use({ storageState: './storage/user.json' }) for a file or a test.describe block, so if all your tests in your test file use the same authentication role place it outside a test.describe block, otherwise place it within a test.describe block of tests that use the same authentication role. Different roles? Use different test.describe blocks with different test.use calls to different files in each.

What do you think of Playwright’s ability to capture and use browser storage state?

Categories
Playwright

Five reasons why Playwright is better than Cypress

It’s pretty obvious I am not a Cypress fan – heck I wrote a whole post about it 2 years ago.

I’ve since become very keen on Playwright and I thought I’d revisit Cypress to compare it with Playwright and see whether any of my long-standing gripes about Cypress have been addressed in 2+ years since I’ve used it.

Versions compared:

Cypress: 8.7.0 on Electron 93
Playwright: 1.16.0

Reason 1: Playwright is so much faster than Cypress

Up to 4x faster.

Take this simple example taken directly from the Cypress doco:

describe('My First Test', () => {
    it('clicking "type" shows the right headings', () => {
      cy.visit('https://example.cypress.io')
  
      cy.contains('type').click()
  
      // Should be on a new URL which includes '/commands/actions'
      cy.url().should('include', '/commands/actions')
  
      // Get an input, type into it and verify that the value has been updated
      cy.get('.action-email')
        .type('fake@email.com')
        .should('have.value', 'fake@email.com')
    })
})

Takes 8 seconds running on my M1 Macbook Air

 โœ”  demo-spec.js                             00:08

I wrote the same example in Playwright:

import { test, expect } from '@playwright/test'

test.describe.parallel('My First Test', () => {
  test('clicking "type" shows the right headings', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    // Should be on a new URL which includes '/commands/actions'
    await page.waitForURL(/.+\/commands\/actions$/)
    // Get an input, type into it and verify that the value has been updated
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
})

Running 1 test using 1 worker

  โœ“  scenarios/compare.cypress.spec.ts:4:3 โ€บ My First Test โ€บ clicking "type" shows the right headings (1s)


  1 passed (2s)

4 times faster ๐Ÿ˜Ž

Reason 2: Playwright has first-class support for parallel running of tests on a single machine both locally and on CI without a subscription or account required

Cypress on the other hand requires you to set your tests to “Record” everything they do which sends everything to Cypress (or Sorry Cypress) if you want to run in parallel, and using a single machine is not recommended (despite it working perfectly in Playwright using my Macbook Air M1)

“While parallel tests can also technically run on a single machine, we do not recommend it since this machine would require significant resources to run your tests efficiently.”

https://docs.cypress.io/guides/guides/parallelization#Overview

Either this is true and Cypress is so bloated that running in parallel locally needs crazy expensive hardware, or the Cypress team want users to pay for their parallel execution service. Or both?

Reason 3: Playwright supports parallel tests within a single test file – to run Cypress tests in parallel you need to split them across files.

For example, in Playwright I can run these 8 tests in parallel:

import { test, expect } from '@playwright/test'

test.describe.parallel('My First Test', () => {
  test('clicking "type" shows the right headings 1', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 2', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 3', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 4', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 5', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 6', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 7', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 8', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
})

which takes only 6 seconds on my Macbook Air ๐Ÿ˜Ž

> npx playwright test "./scenarios/compare.cypress.bulk.spec.ts"

Running 8 tests using 4 workers

  โœ“  scenarios/compare.cypress.bulk.spec.ts:4:3 โ€บ My First Test โ€บ clicking "type" shows the right headings 1 (4s)
  โœ“  scenarios/compare.cypress.bulk.spec.ts:11:3 โ€บ My First Test โ€บ clicking "type" shows the right headings 2 (4s)
  โœ“  scenarios/compare.cypress.bulk.spec.ts:18:3 โ€บ My First Test โ€บ clicking "type" shows the right headings 3 (4s)
  โœ“  scenarios/compare.cypress.bulk.spec.ts:25:3 โ€บ My First Test โ€บ clicking "type" shows the right headings 4 (4s)
  โœ“  scenarios/compare.cypress.bulk.spec.ts:46:3 โ€บ My First Test โ€บ clicking "type" shows the right headings 7 (1s)
  โœ“  scenarios/compare.cypress.bulk.spec.ts:32:3 โ€บ My First Test โ€บ clicking "type" shows the right headings 5 (1s)
  โœ“  scenarios/compare.cypress.bulk.spec.ts:39:3 โ€บ My First Test โ€บ clicking "type" shows the right headings 6 (1s)
  โœ“  scenarios/compare.cypress.bulk.spec.ts:53:3 โ€บ My First Test โ€บ clicking "type" shows the right headings 8 (1s)

  8 passed (6s)

To possibly run these in parallel (using a server) in Cypress I’d need to structure these as individual files each with a single spec:

And to run them on the same machine I ran these 8 scenarios in Playwright in 6 seconds takes 45 seconds! 7.5 times slower!

Reason 4: Playwright fully supports async/await syntax for clean, readable code.

This is the Playwright code:

import { test, expect } from '@playwright/test'

test.describe.parallel('My First Test', () => {
  test('clicking "type" shows the right headings', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
})

No chaining, no hard to understand code, no magic. You can see where test and expect come from, you can see where page comes from.

The same test in Cypress:

describe('My First Test', () => {
    it('clicking "type" shows the right headings', () => {
      cy.visit('https://example.cypress.io')
    
      cy.contains('type').click()
  
      // Should be on a new URL which includes '/commands/actions'
      cy.url().should('include', '/commands/actions')
  
      // Get an input, type into it and verify that the value has been updated
      cy.get('.action-email')
        .type('fake@email.com')
        .should('have.value', 'fake@email.com')
    })
})

Note how the function calls chain together – in this simple example it’s not too bad but it quickly gets out of hand. Also note there’s no indication that this is synchronous code – no await on any calls. Also where does this cy thing come from – oh yes that’s magic. Same for describe, again magic. No imports but lots of magic.

Reason 5: Playwright doesn’t need plugins

There’s so many limitations in Cypress there’s pretty much a plugin for everything. And there’s still so may trade-offs that can’t be be fixed by a plugin.

Need to send the tab key to the browser? There’s a plugin for that. And it doesn’t work well at all according to the comments on the Github issue for lack of tab key support open since 2016 (and still open since I last wrote about Cypress).

Need to upload a file? There’s also a plugin for that.

Need to run tests n-times in a row to measure repeatability? There’s a plugin for that.

Fortunately Playwright supports all of these things, and many more, natively. No plug-ins needed ๐Ÿ˜Ž

Summary

As you can see I’m still not a Cypress fan all these years later ๐Ÿ˜€

It’s just that now there’s a tool which is so much better and freely available to use.

So I’ll stand by my summary from 2 years ago which I can repeat here:

From a distance Cypress looks like a polished tool for automated testing โ€“ I just think itโ€™s incorrectly marketed as an end-to-end testing tool when itโ€™s really only good for component testing. There are too many limitations in the tool in acting like a real user to use it to create true end-to-end automated tests.

https://alisterbscott.com/2019/07/06/my-thoughts-on-cypress-io/

Categories
Playwright

API Testing using Playwright

I’m a big fan of being able to call APIs during e2e tests to set up data, verify things, do things and all that jazz.

Up until now I’ve been mixing in Supertest with Playwright to call APIs within tests, until now…

Playwright 1.16 includes the ability to call APIs both independently and using the page browser object (which sends the currently stored cookies for API requests).

I’ve updated my example TypeScript project to include these API calls directly using both page and request which looks like:

test('can POST a REST API and check response using approval style', async ({ request }) => {
    const response = await request.post('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io/posts', { data: { title: 'Post 4' } })
    expect(response.status()).toBe(201)
    const body = await response.text()
    expect(body).toMatchSnapshot('post4.txt')
  })

  test('can POST a REST API and check response using assertion style (using page)', async ({ page }) => {
    const response = await page.request.post('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io/posts', { data: { title: 'Post 4' } })
    expect(response.status()).toBe(201)
    const body = JSON.parse(await response.text())
    expect(body.id).toBe(4)
    expect(body.title).toBe('Post 4')
  })

It’s nice to be able to remove another dependency from my e2e tests and still allow calling APIs using a nice API and reusing the existing objects that Playwright provides. I am continually blown away by how Playwright continues getting more and more awesome ๐Ÿ˜Ž

Categories
Playwright

Selecting hard to identify elements with Playwright

Say you have a web app which displays a table of dynamic data that looks like this:

Admire my beautiful design skills!

And the DOM looks like:

<table>
    <theader>
      <tr>
        <th>Status</th>
        <th>Name</th>
        <th>Actions</th>
      </tr>
    </theader>
    <tbody>
      <tr>
        <td>OPEN</td>
        <td>Cactus</td>
        <td>
          <div>
            <button>CLICK ME</button>
          </div>
        </td>
      </tr>
      <tr>
        <td>CLOSED</td>
        <td>Succulent</td>
        <td>
          <div>
            <button>CLICK ME</button>
          </div>
        </td>
      </tr>
      <tr>
        <td>OPEN</td>
        <td>Aloe</td>
        <td>
          <div>
            <button>CLICK ME</button>
          </div>
        </td>
      </tr>
      <tr>
        <td>OPEN</td>
        <td>Agave</td>
        <td>
          <div>
            <button>CLICK ME</button>
          </div>
        </td>
      </tr>
    </tbody>
  </table>

Say you’re writing an e2e test that adds an item to the table dynamically (with a new name each time) and you need to click the button on the corresponding line to complete your e2e test.

How do you select the button based upon the name of the item you just created? Let’s save Agave?

I feel like the best possible approach is to add a data- attribute to the buttons so you can know exactly which one to click. Something like <button data-name="Agave">Agave</button"

A previous article I wrote on using data attributes

But if you can’t add that value to each row – what’s the next best thing to do?

Playwright has a pretty neat test recorder (even though I don’t like test recorders) – so let’s see what it generates

I launch Playwright Inspector (which has the Record function) with

npx playwright open https://webdriverjsdemo.github.io/table/

After clicking Record, and then clicking the four buttons on the table in order from top-bottom, the Inspector tool gives me:

Clicking the four buttons here
  await page.click('text=CLICK ME');
  await page.click('text=Succulent CLOSED CLICK ME >> button');
  await page.click('text=Aloe OPEN CLICK ME >> button');
  await page.click('text=Agave OPEN CLICK ME >> button');

The issue with this generated code is it relies on the status values which can change, also if the status column was removed in a future change it would fail which makes the tests brittle.

We can use the Playwright Inspector “Explore” feature to enter a selector value and see whether it finds what we’re looking for in the DOM.

If we took one of the selectors above and tried to remove the bit we don’t care about from the text you can see it doesn’t find what we want:

text=Agave >> button

Doing some research I found there’s an option to find any element containing some text, we can choose the row containing the text “Agave” (or whatever the name of the record we created is) and then choose the button belonging to that row:

tr:has-text("Agave") >> button

The important thing to note here is we are selecting the row, even though the cell contains the text we’re looking for.

I think this is probably the best option if we don’t have ability to change the DOM and add data attributes – it’s the least brittle as any additional columns could be added to the table and it would still work.

Some other options I investigated were locators based on position

For example, you can write a selector that finds the buttons to the right of the text we’re looking for:

button:right-of(:text("Agave"))

But as you can see from the image above it locates all the buttons as they’re all technically to the right of the text – just not immediately to the right.

Summary

Whilst it’s good practice to add testability features to your app, Playwright also offers powerful ways to select elements based on text and DOM layout if you can’t. You can read all about the Playwright selectors here.

Categories
Playwright

Running parallel Playwright Tests within a single spec file

Playwright by default runs all tests within a single spec file in order using the same worker, and runs tests in different spec files in parallel.

You can now use the test.describe.parallel block to specify that all tests contained in the block are independent and can be run in parallel. This can mean faster test runs.

There’s also test.describe.serial to specify the tests contained should be run one after another, and the subsequent tests won’t be run if an earlier test fails.

There’s also test.step which can be used to break tests down further into a series of steps.

I’ve updated my example TypeScript tests to use the parallel format.

Categories
Aside

Microsoft Teams

Most of my meetings nowadays are held online.

I hate being late to meetings, but I also like getting things done. When my calendar gives me the reminder about a meeting 10 mins before I’ve developed a habit of joining the meeting straight away so I am already in the meeting when other people join. That way I can continue my work without forgetting to join right on time. I’ve found if I don’t do that I’ll dismiss the notification, get caught up in my work (flow) and forget to join.

This was all well and good at my previous company where we used Google Meet. But my current company uses Microsoft Teams and when I join 10 mins early everyone else invited to the meeting gets a notification telling them I’ve started the meeting! Who designed Microsoft Teams thinking that was a good idea!?!

Any ideas how I can work with this?

Categories
Automated Testing Playwright

Playwright Trace Viewer

One of the things that people really like about Cypress is the ability to see a step by step replay of what happened with snapshots of the DOM as it ran.

Well Playwright also offers this ๐Ÿ˜Ž

An easy way to get access to these is to set your Playwright config to rerun failed tests and to capture a trace on the second attempt:

playwright.config.ts:

module.exports = {
  use: {
    headless: true,
    screenshot: "only-on-failure",
    video: "retry-with-video",
    trace: "on-first-retry",
  },
  retries: 1,
};

A trace zip file will be generated on a failed 2nd attempt.

I uploaded an example here. You simply run

npx playwright show-trace ./doco/exampletrace.zip

which displays an interactive step by step trace viewer ๐Ÿ˜Ž

An example trace viewer
Categories
Automated Testing Playwright

TypeScript!

I recently changed jobs ๐ŸŽ‰

One of the biggest things I noticed since changing jobs nearly 3 years ago was the prevalence of TypeScript: JavaScript With Syntax For Types. Most, if not all, of the jobs I were looking at were dev teams using TypeScript – driven by the increasing use full stack Node.js.

It’s only week 2 in my new job but I’ve already set up an e2e testing framework using Playwright and it’s all in TypeScript ๐Ÿ˜Š I thought I’d share the main differences and I’ve created a sample repository to do so!

To use TypeScript on a front-end codebase running in a browser you need to compile/transpile it into JavaScript (which a browser understands).

Fortunately Playwright doesn’t require this step as Playwright runs your TypeScript code directly.

The first thing to do is add typescript to your project and add a tsconfig.json file to your project.

I use Playwright Test – and the main difference between a TypeScript spec file (*.spec.ts) and a JavaScript one (*.spec.js) is the import statements:

import { test, expect } from '@playwright/test'
import { visitHomePage } from '../lib/actions/nav'

test('can wait for an element to appear', async ({ page }) => {
  await visitHomePage(page)
  await page.waitForSelector('#elementappearschild', { state: 'visible', timeout: 5000 })
})

test('can use an element that appears after on page load', async ({ page }) => {
  await visitHomePage(page)
  const text = await page.textContent('#loadedchild')
  expect(text).toBe('Loaded!')
})

whereas in JavaScript the first two lines were:

const nav = require('../lib/actions/nav')
const { test, expect } = require('@playwright/test')

The main differences are in the library code.

import { Page } from '@playwright/test'
import config from 'config'

export async function visitHomePage (page: Page) {
  return await page.goto(`${config.get('baseURL')}`)
}

export async function goToPath (page: Page, path: string) {
  return await page.goto(`${config.get('baseURL')}/${path}`)
}

You will see the types being added, fortunately Playwright provides type definitions which are imported here.

Previously, this same file looked like:

const config = require('config')

async function visitHomePage (page) {
  return await page.goto(`${config.get('baseURL')}`)
}

async function goToPath (page, path) {
  return await page.goto(`${config.get('baseURL')}/${path}`)
}

module.exports = {
  visitHomePage,
  goToPath
}

The other main thing to update is if you use classes you define types against all the properties.

Everything else is about the same.

My fully working TypeScript example of Playwright Test is here: https://github.com/alisterscott/playwright-test-ts-demo

Enjoy TypeScripting ๐Ÿ˜€

Categories
Automated Testing

“Never Trust an Automated Test You Haven’t Seen Fail”

My number one philosophy when developing automated tests is:

“Never Trust an Automated Test You Haven’t Seen Fail”

I’ve found it can be easy to write a test that always passes, so forcing a test to fail as a starting point gives me confidence that it can fail.

I know of two ways to make a test fail:

  1. Change the implementation to fail and make sure the test fails at first – fixing the implementation fixes the test
  2. Change the assertion to make a test fail, make sure the actual result is correct and then update the assertion to make the test pass

I don’t really care what method I use to know a test fails, but I make sure I always make it fail. Never trust a test that hasn’t failed.