Categories
e2e Testing Playwright

Writing automated e2e tests for known buggy systems

Every system I’ve worked on, old or new, is full of known bugs (and unknown bugs for good measure 🤪). These known bugs are the ones that have never made it to the top of the bug backlog to be fixed because there’s always other more important work to do.

But what do you do with automated e2e tests that exercise such code and demonstrate such bugs?

Imagine a very simple example of a test that visits our page and asserts the title is correct.

Our page looks like this:

super simple webpage

And our Playwright code looks like this:

test.only('can have a test for a known bug in the system', async ({ page }) => {
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leave Page');
})

You can see our test has different text it asserts than what is displayed. The text is our test is what we actually want to display, however the system displays it differently so our test fails when we run it.

What do we do with such tests? There’s a few different options all with their own advantages and disadvantages.

Option One: Commit the failing test as it is

Advantages: test is pure and correct, test is still run on every build highlighting the functionality that is wrong

Disadvantages: each build will fail until this functionality is fixed, creating red/failed builds and not giving immediate feedback on other potential issues found in the builds and resulting in people losing confidence in overall build results.

I personally wouldn’t recommend this approach as I think the noise of the failing builds outweighs any benefits it has.

Option Two: Mark the failing test as skipped

test.skip('can have a test for a known bug in the system', async ({ page }) => {
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leave Page');
})

Advantages: no noise in builds since test no longer runs

Disadvantages: test can be forgotten about since it never runs and other issues could be introduced in the feature. For example if the text was changed to something else that is also wrong we wouldn’t know since the test is not being run.

Whilst this is preferable to option one this option often results in forgotten tests so I would also not recommend it.

Option Three: Update the assertion to be incorrect (with a comment)

test.only('can have a test for a known bug in the system', async ({ page }) => {
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leaving Page'); // BUG: This text should be WebDriverJs Demo Leave Page
  })

Advantages: if the text changes to any value (whether now correct, or still incorrect) the test will fail alerting us to a change in functionality

Disadvantages: the tests are no longer representative of what is expected of the system – the expectations contradict what is actually expected.

I probably prefer this to having a pending test but something doesn’t feel right about a false assertion.

Option Four:

Playwright actually offers a solution for scenarios like this, it’s the test.fail() syntax which marks a test as being expected to fail, so it is still run but if it fails it passes, and if it passes it fails 🙃

We can write the test like this:

test.only('can have a test for a known bug in the system', async ({ page }) => {
  test.fail() // BUG: The text is presently wrong
  await goToPath(page, 'leave')
  expect(page.locator('#leavepage')).toHaveText('WebDriverJs Demo Leave Page');
  })

And when it fails it “passes” with a green cross:

If the system was fixed this test would then fail, and we’d know to remove the test.fail()line.

Advantages: if the text changes to the correct value we will know as this test will pass when we don’t expect it to. We can keep our assertions correct/pure.

Disadvantages: if the test was to fail in a different way we wouldn’t know about it since all the test cares about is that it fails (which we’re expecting).

Whilst this can hide other test failures, since I aim to write independent tests I can live with it potentially hiding other issues so this is my preferred approach to known failures.

How do you deal with known failures? Any of these ways or another I’ve missed?

3 replies on “Writing automated e2e tests for known buggy systems”

Working in a Jest environment, one thing I miss about `pytest` is the `test.xfail` syntax which describes Option 4. I used it quite heavily both in my own projects and also in previous gigs that used Python.

There is an issue that has been open at facebook/jest since May 2020 that has amassed a non-trivial amount of likes, but it looks like the Jest team is choosing not to acknowledge it[1].

[1] https://github.com/facebook/jest/issues/10030

this is a very neat soluition to an unavoidable problem. I usually end up going for the option of adding comments using the jetbrains TODO or FIXME tag but it’s easy to ignore these and they end up never being updated. Using this test.fail feature kind of forces the issue on you and will I assume encourage me to update the tests more regulary

test.fixme() is another really nice one, though it is better used for tests that are failing as a result of the test code being wrong rather than the application under test.

Leave a Reply

Your email address will not be published.