Categories
Playwright

Five reasons why Playwright is better than Cypress

It’s pretty obvious I am not a Cypress fan – heck I wrote a whole post about it 2 years ago.

I’ve since become very keen on Playwright and I thought I’d revisit Cypress to compare it with Playwright and see whether any of my long-standing gripes about Cypress have been addressed in 2+ years since I’ve used it.

Versions compared:

Cypress: 8.7.0 on Electron 93
Playwright: 1.16.0

Reason 1: Playwright is so much faster than Cypress

Up to 4x faster.

Take this simple example taken directly from the Cypress doco:

describe('My First Test', () => {
    it('clicking "type" shows the right headings', () => {
      cy.visit('https://example.cypress.io')
  
      cy.contains('type').click()
  
      // Should be on a new URL which includes '/commands/actions'
      cy.url().should('include', '/commands/actions')
  
      // Get an input, type into it and verify that the value has been updated
      cy.get('.action-email')
        .type('fake@email.com')
        .should('have.value', 'fake@email.com')
    })
})

Takes 8 seconds running on my M1 Macbook Air

 ✔  demo-spec.js                             00:08

I wrote the same example in Playwright:

import { test, expect } from '@playwright/test'

test.describe.parallel('My First Test', () => {
  test('clicking "type" shows the right headings', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    // Should be on a new URL which includes '/commands/actions'
    await page.waitForURL(/.+\/commands\/actions$/)
    // Get an input, type into it and verify that the value has been updated
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
})

Running 1 test using 1 worker

  ✓  scenarios/compare.cypress.spec.ts:4:3 › My First Test › clicking "type" shows the right headings (1s)


  1 passed (2s)

4 times faster 😎

Reason 2: Playwright has first-class support for parallel running of tests on a single machine both locally and on CI without a subscription or account required

Cypress on the other hand requires you to set your tests to “Record” everything they do which sends everything to Cypress (or Sorry Cypress) if you want to run in parallel, and using a single machine is not recommended (despite it working perfectly in Playwright using my Macbook Air M1)

“While parallel tests can also technically run on a single machine, we do not recommend it since this machine would require significant resources to run your tests efficiently.”

https://docs.cypress.io/guides/guides/parallelization#Overview

Either this is true and Cypress is so bloated that running in parallel locally needs crazy expensive hardware, or the Cypress team want users to pay for their parallel execution service. Or both?

Reason 3: Playwright supports parallel tests within a single test file – to run Cypress tests in parallel you need to split them across files.

For example, in Playwright I can run these 8 tests in parallel:

import { test, expect } from '@playwright/test'

test.describe.parallel('My First Test', () => {
  test('clicking "type" shows the right headings 1', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 2', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 3', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 4', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 5', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 6', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 7', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
  test('clicking "type" shows the right headings 8', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
})

which takes only 6 seconds on my Macbook Air 😎

> npx playwright test "./scenarios/compare.cypress.bulk.spec.ts"

Running 8 tests using 4 workers

  ✓  scenarios/compare.cypress.bulk.spec.ts:4:3 › My First Test › clicking "type" shows the right headings 1 (4s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:11:3 › My First Test › clicking "type" shows the right headings 2 (4s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:18:3 › My First Test › clicking "type" shows the right headings 3 (4s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:25:3 › My First Test › clicking "type" shows the right headings 4 (4s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:46:3 › My First Test › clicking "type" shows the right headings 7 (1s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:32:3 › My First Test › clicking "type" shows the right headings 5 (1s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:39:3 › My First Test › clicking "type" shows the right headings 6 (1s)
  ✓  scenarios/compare.cypress.bulk.spec.ts:53:3 › My First Test › clicking "type" shows the right headings 8 (1s)

  8 passed (6s)

To possibly run these in parallel (using a server) in Cypress I’d need to structure these as individual files each with a single spec:

And to run them on the same machine I ran these 8 scenarios in Playwright in 6 seconds takes 45 seconds! 7.5 times slower!

Reason 4: Playwright fully supports async/await syntax for clean, readable code.

This is the Playwright code:

import { test, expect } from '@playwright/test'

test.describe.parallel('My First Test', () => {
  test('clicking "type" shows the right headings', async ({ page }) => {
    await page.goto('https://example.cypress.io')
    await page.click('a:has-text("type")')
    await page.waitForURL(/.+\/commands\/actions$/)
    await page.fill('.action-email', 'fake@email.com')
    await expect(page.locator('.action-email')).toHaveValue('fake@email.com')
  })
})

No chaining, no hard to understand code, no magic. You can see where test and expect come from, you can see where page comes from.

The same test in Cypress:

describe('My First Test', () => {
    it('clicking "type" shows the right headings', () => {
      cy.visit('https://example.cypress.io')
    
      cy.contains('type').click()
  
      // Should be on a new URL which includes '/commands/actions'
      cy.url().should('include', '/commands/actions')
  
      // Get an input, type into it and verify that the value has been updated
      cy.get('.action-email')
        .type('fake@email.com')
        .should('have.value', 'fake@email.com')
    })
})

Note how the function calls chain together – in this simple example it’s not too bad but it quickly gets out of hand. Also note there’s no indication that this is synchronous code – no await on any calls. Also where does this cy thing come from – oh yes that’s magic. Same for describe, again magic. No imports but lots of magic.

Reason 5: Playwright doesn’t need plugins

There’s so many limitations in Cypress there’s pretty much a plugin for everything. And there’s still so may trade-offs that can’t be be fixed by a plugin.

Need to send the tab key to the browser? There’s a plugin for that. And it doesn’t work well at all according to the comments on the Github issue for lack of tab key support open since 2016 (and still open since I last wrote about Cypress).

Need to upload a file? There’s also a plugin for that.

Need to run tests n-times in a row to measure repeatability? There’s a plugin for that.

Fortunately Playwright supports all of these things, and many more, natively. No plug-ins needed 😎

Summary

As you can see I’m still not a Cypress fan all these years later 😀

It’s just that now there’s a tool which is so much better and freely available to use.

So I’ll stand by my summary from 2 years ago which I can repeat here:

From a distance Cypress looks like a polished tool for automated testing – I just think it’s incorrectly marketed as an end-to-end testing tool when it’s really only good for component testing. There are too many limitations in the tool in acting like a real user to use it to create true end-to-end automated tests.

https://alisterbscott.com/2019/07/06/my-thoughts-on-cypress-io/

Categories
Automated Testing

My Thoughts on Cypress.io

Run asks…

I’ve just started using Cypress.io. As someone who initially learned about testing conventions years ago through your blog, cypress seems to want to burn all old conventions to the ground in a way that immediately turned me off. After playing with it a bit and watching a talk by one of its founders, I’m a little more convinced now. It’s a great tool. Do you have an opinion on Cypress yet, and do you think old testing conventions are becoming obsolete thanks to much better reporting tools around testing?

There certainly seems to be a lot of hype and enthusiasm around Cypress.io. I recently saw another (rather evangelical) talk about Cypress.io here in Brisbane so I thought it was time to share my thoughts.

What exactly is Cypress.io?

Looking at Cypress.io it is described as a “JavaScript End to End Testing Framework” and “Fast, easy and reliable testing for anything that runs in a browser“. Some other descriptions on Cypress.io include “A complete end-to-end testing experience.” and “Cypress is the new standard in front-end testing that every developer and QA engineer needs.” 

Screen Shot 2019-07-06 at 2.54.48 pm.png

What is end-to-end testing?

I believe there are some specific traits that define what automated end-to-end (e2e) tests are:

  • They test a complete user flow through an application from start to finish (end-to-end)
  • They test how a real user would use a using a fully deployed system
  • They test the happy-path of the most commonly used scenarios, avoiding error validation or edge-cases

End-to-end tests are expensive to maintain and execute so the widely accepted view is to have as few of these as possible for your application, which means avoiding things like negative and error validation testing during end-to-end tests as these things can be tested much more easily and quickly in isolation as other types of automated tests (unit, component or integration).

Is Cypress.io a framework for writing end-to-end tests?

Despite its strongly worded marketing material, I don’t believe Cypress.io has been designed as an end-to-end (e2e) testing framework. I believe this was confirmed by Brian Mann at Assert(js) in the first half of the “I see your point, but…” presentation:

“You should always strive to test pages in total isolation – everything becomes faster, less coupled, and you won’t lose a single point of confidence that it’s all working together correctly. You don’t need to limit yourself trying to act and replicate everything a user would do.”

Screen Shot 2019-07-06 at 2.28.05 pm

I believe what Brian is referring to aren’t end-to-end tests but rather component tests. Brian showed using Cypress.io to test a login page where he wrote 6 isolated test specs to test login validation:

Screen Shot 2019-07-06 at 2.34.36 pm

What makes the question about whether Cypress.io is an e2e testing framework even more confusing is during the second half of the same presentation, Gleb Bahmutov, also from Cypress.io, states:

“Brian showed how we think about end-to-end testing. To us end-to-end should do the same things that a human would do to a fully deployed system. Right, that means real browser, real interactions, no shortcuts…”

Screen Shot 2019-07-06 at 2.45.19 pm

Also, confusingly, he stated that e2e test tools can do a pretty good job in unit testing:

Screen Shot 2019-07-06 at 2.45.38 pm

So what is Cypress.io then?

I now consider Cypress.io to be a strongly opinionated framework suited to writing isolated automated web component tests. I’m not sure why it is marketing as an e2e testing tool, and trying to compare itself to something like Selenium, which isn’t a component testing tool.

If you wanted to write isolated automated web component tests then Cypress.io would be worth a look at, since it offers many features to help you. However for true end-to-end purposes I think the limitations outweigh the benefits. The open source WordPress Gutenberg editor project tried Cypress.io for quite a while but ultimately found it too limiting and switched to Puppeteer.

Some things to considering when trying to use Cypress.io for true end-to-end testing

Even though Cypress.io is demonstrated as a way to write isolated web component tests, if you still want to use it to write true end-to-end tests then there’s some tradeoffs you need to consider.

Screen Shot 2019-07-06 at 2.55.49 pm

Despite the claims that “Cypress works on any front-end framework or website” and “Fast, easy and reliable testing for anything that runs in a browser” there are quite a few scenarios where you can’t use Cypress – for example if your front-end or website uses iFrames you can’t use Cypress.io.

iFrames

Cypress.io itself uses iFrames to inject itself into the browser so supporting iFrames whether on the same domain or cross domain aren’t supported with an open issue since 2016. At WordPress.com we used iFrames for the WordPress site customizer so it wouldn’t be possible to write an end-to-end test for WordPress.com using Cypress.io. In my current role I work on a web application which actually a series of React micro frontends which are rendered in iFrames within a web container, so we also can’t use Cypress.io for end-to-end testing.

Native browser events like file uploads and downloads

Things like uploading and downloading files that are trivial to do in WebDriver are either difficult or not supported in Cypress.io. Even things as simple as using the tab key isn’t supported.

Parallelism

Whilst Cypress is often promoted as a free and open source project there are certain features that are only available when running your tests in “record” mode with the Cypress Dashboard Service, which allows 500 tests (it blocks) to run in parallel per month before needing to pay for it. Parallel execution is one of these features, so you can’t even run tests in parallel locally without recording your results to the dashboard service.

The biggest issue I see with Cypress.io parallelization is that it is machine based, not process based:

parallelization-diagram.4c4cbac6

In this example, the CI container costs triple when each CI machine should be more than capable of running multiple Chromium browser sessions.

At WordPress.com we used CircleCI and we able to have up to 12 headless Chrome browsers using WebDriverJs in parallel on each CircleCI container, and across 3 containers this allowed 36 e2e tests running in parallel. To get the same result using Cypress.io would mean paying for 36 CircleCI containers.

Running e2e tests written in other e2e testing tools in parallel machine processes can be quite easy as I’ve explained previously on this blog.

There are times when running via the command line isn’t the same as running via the GUI

I noticed this when writing a demo cypress spec which would pass when running in the Cypress GUI runner but fail on the command line, which looking at these comments doesn’t seem uncommon.

    it( 'ignores alerts when leaving the page', function() {
        cy.visit('http://webdriverjsdemo.github.io/leave'); 
        cy.get('#homelink').click();
        cy.contains('WebDriverJs Demo Page').should('be.visible');
    } );

GUI Runner:

Screen Shot 2019-07-06 at 4.33.53 pm

Command Line:

Screen Shot 2019-07-06 at 4.37.17 pm

Logging in

One of the key messages of the first video was demonstrating you can log in without using the UI for subsequent tests which speeds things up. This is a good idea, but isn’t unique to Cypress.io: at WordPress.com we re-used a single login cookie across multiple e2e tests using WebDriverJS – the code is here.

Summary

From a distance Cypress looks like a polished tool for automated testing – I just think it’s incorrectly marketed as an end-to-end testing tool when it’s really only good for component testing. There are too many limitations in the tool in acting like a real user to use it to create true end-to-end automated tests.