Automated Testing

My Thoughts on

Run asks…

I’ve just started using As someone who initially learned about testing conventions years ago through your blog, cypress seems to want to burn all old conventions to the ground in a way that immediately turned me off. After playing with it a bit and watching a talk by one of its founders, I’m a little more convinced now. It’s a great tool. Do you have an opinion on Cypress yet, and do you think old testing conventions are becoming obsolete thanks to much better reporting tools around testing?

There certainly seems to be a lot of hype and enthusiasm around I recently saw another (rather evangelical) talk about here in Brisbane so I thought it was time to share my thoughts.

What exactly is

Looking at it is described as a “JavaScript End to End Testing Framework” and “Fast, easy and reliable testing for anything that runs in a browser“. Some other descriptions on include “A complete end-to-end testing experience.” and “Cypress is the new standard in front-end testing that every developer and QA engineer needs.” 

Screen Shot 2019-07-06 at 2.54.48 pm.png

What is end-to-end testing?

I believe there are some specific traits that define what automated end-to-end (e2e) tests are:

  • They test a complete user flow through an application from start to finish (end-to-end)
  • They test how a real user would use a using a fully deployed system
  • They test the happy-path of the most commonly used scenarios, avoiding error validation or edge-cases

End-to-end tests are expensive to maintain and execute so the widely accepted view is to have as few of these as possible for your application, which means avoiding things like negative and error validation testing during end-to-end tests as these things can be tested much more easily and quickly in isolation as other types of automated tests (unit, component or integration).

Is a framework for writing end-to-end tests?

Despite its strongly worded marketing material, I don’t believe has been designed as an end-to-end (e2e) testing framework. I believe this was confirmed by Brian Mann at Assert(js) in the first half of the “I see your point, but…” presentation:

“You should always strive to test pages in total isolation – everything becomes faster, less coupled, and you won’t lose a single point of confidence that it’s all working together correctly. You don’t need to limit yourself trying to act and replicate everything a user would do.”

Screen Shot 2019-07-06 at 2.28.05 pm

I believe what Brian is referring to aren’t end-to-end tests but rather component tests. Brian showed using to test a login page where he wrote 6 isolated test specs to test login validation:

Screen Shot 2019-07-06 at 2.34.36 pm

What makes the question about whether is an e2e testing framework even more confusing is during the second half of the same presentation, Gleb Bahmutov, also from, states:

“Brian showed how we think about end-to-end testing. To us end-to-end should do the same things that a human would do to a fully deployed system. Right, that means real browser, real interactions, no shortcuts…”

Screen Shot 2019-07-06 at 2.45.19 pm

Also, confusingly, he stated that e2e test tools can do a pretty good job in unit testing:

Screen Shot 2019-07-06 at 2.45.38 pm

So what is then?

I now consider to be a strongly opinionated framework suited to writing isolated automated web component tests. I’m not sure why it is marketing as an e2e testing tool, and trying to compare itself to something like Selenium, which isn’t a component testing tool.

If you wanted to write isolated automated web component tests then would be worth a look at, since it offers many features to help you. However for true end-to-end purposes I think the limitations outweigh the benefits. The open source WordPress Gutenberg editor project tried for quite a while but ultimately found it too limiting and switched to Puppeteer.

Some things to considering when trying to use for true end-to-end testing

Even though is demonstrated as a way to write isolated web component tests, if you still want to use it to write true end-to-end tests then there’s some tradeoffs you need to consider.

Screen Shot 2019-07-06 at 2.55.49 pm

Despite the claims that “Cypress works on any front-end framework or website” and “Fast, easy and reliable testing for anything that runs in a browser” there are quite a few scenarios where you can’t use Cypress – for example if your front-end or website uses iFrames you can’t use

iFrames itself uses iFrames to inject itself into the browser so supporting iFrames whether on the same domain or cross domain aren’t supported with an open issue since 2016. At we used iFrames for the WordPress site customizer so it wouldn’t be possible to write an end-to-end test for using In my current role I work on a web application which actually a series of React micro frontends which are rendered in iFrames within a web container, so we also can’t use for end-to-end testing.

Native browser events like file uploads and downloads

Things like uploading and downloading files that are trivial to do in WebDriver are either difficult or not supported in Even things as simple as using the tab key isn’t supported.


Whilst Cypress is often promoted as a free and open source project there are certain features that are only available when running your tests in “record” mode with the Cypress Dashboard Service, which allows 500 tests (it blocks) to run in parallel per month before needing to pay for it. Parallel execution is one of these features, so you can’t even run tests in parallel locally without recording your results to the dashboard service.

The biggest issue I see with parallelization is that it is machine based, not process based:


In this example, the CI container costs triple when each CI machine should be more than capable of running multiple Chromium browser sessions.

At we used CircleCI and we able to have up to 12 headless Chrome browsers using WebDriverJs in parallel on each CircleCI container, and across 3 containers this allowed 36 e2e tests running in parallel. To get the same result using would mean paying for 36 CircleCI containers.

Running e2e tests written in other e2e testing tools in parallel machine processes can be quite easy as I’ve explained previously on this blog.

There are times when running via the command line isn’t the same as running via the GUI

I noticed this when writing a demo cypress spec which would pass when running in the Cypress GUI runner but fail on the command line, which looking at these comments doesn’t seem uncommon.

    it( 'ignores alerts when leaving the page', function() {
        cy.contains('WebDriverJs Demo Page').should('be.visible');
    } );

GUI Runner:

Screen Shot 2019-07-06 at 4.33.53 pm

Command Line:

Screen Shot 2019-07-06 at 4.37.17 pm

Logging in

One of the key messages of the first video was demonstrating you can log in without using the UI for subsequent tests which speeds things up. This is a good idea, but isn’t unique to at we re-used a single login cookie across multiple e2e tests using WebDriverJS – the code is here.


From a distance Cypress looks like a polished tool for automated testing – I just think it’s incorrectly marketed as an end-to-end testing tool when it’s really only good for component testing. There are too many limitations in the tool in acting like a real user to use it to create true end-to-end automated tests.

15 replies on “My Thoughts on”

You can do your positive tests also via fast jsdom/enzyme shallow tests. It’s bad to write a ton of slow integration tests even happy path tests through something like Cypress as most devs try to write every positive outside/acceptance tests that way which is not the ideal way to be going especially for a quick BDD/TDD cycle

I haven’t gone into detail but I’m not a big fan of the chaining based API and since it uses a browser proxy it has similar issues with HTTPS sites to other proxy based tools.

Puppeteer is my favourite Node.js e2e test library at present.

It certainly isn’t an e2e testing tool.
Most of the time when I do e2e automation I seldom have the luxury of web browser as my target platform all the way through the transaction.
Cypress seems to assume it is which basically eliminates it for that very purpose.

Cypress is an interesting development in UI testing, but UI testing is only the tip of an iceberg for most people tasked with automated testing. Things like file system access, external calls to APIs and databases, and remotely executing code on other machines aren’t things you “might need to do”, they are frequent, if not central. Many applications are developed with UI as a frontend for an API. In that case using the API to set state is better than calling internal app functions. As for other Cypress limitations… testing in just one browser type isn’t an option for most people. My last job required automating multiple browsers and and multiple browser types for a tutoring application. I’d love to build a project in this tool to dive in more, but I have no context yet where the benefits outweigh the limitations. Maybe as it matures that will change.

Hi Alister,
Thanks for taking the time to play with Cypress and write this review. As a person who works on Cypress and was referenced in your post, I want to point out that Cypress is for sure end-to-end test runner for anything that runs in the browser. How else would you call a tool that opens a real browser, visits the site and interacts with the page – just like a real human user would? If you want component testing – that is framework-specific mounting of components instead of visiting the website, sure the Test Runner can do it, but the primary focus is to visit the real website and interact with the web applications.

– There are mechanisms to access the operating system if you need to with `cy.exec`, `cy.readFile`, `cy.task` if your tests need to “escape” the browser.
– There are technical limitations to the way Cypress loads and runs the tests, which are covered in Some of the limitations will be removed (iframes, shadow dom support) as we continue, and some limits will probably be there for many years, but we keep our roadmap public at and want everyone’s input of course.

Finally, I would encourage everyone who wants to learn about Cypress to watch this “Cypress in a nutshell” webinar video and flip through slides at – Amir showcases really well what Cypress can and cannot do. Amir will be at ComponentsConf in Melbourne in September, I encourage everyone who can to attend his talk there, see for details.

Thanks for your comment Gleb

Finally, I would encourage everyone who wants to learn about Cypress to watch this “Cypress in a nutshell” webinar video and flip through slides at – Amir showcases really well what Cypress can and cannot do.

I tried to watch this but the audio is poor quality which makes it hard to understand. I did hear this line “if it’s on the web, Cypress can test it” which I still have a problem with as my recent apps I have worked on have both used iFrames can Cypress can’t test them – including which is a super high volume web application.

I agree with some points, but the part of running Cypress tests in parallel can be easily solved by defining different test scripts, that run specific test suites in separate Docker containers, for instance.
I have just experimented running Cypress tests in parallel on GitLab successfully.
And the time to run each test can be improved by caching the dependencies (e.g., Cypress)

My issue is more of their dishonest marketing. “Selenium killer”. “Don’t use page object”. “Use react to automate components”. None of these are true or useful statements but Cypress markets it as such. 1. Selenium isn’t dead or killed by cypress at all. 2. Using react to automate front end? ok, so you need the qa to learn react. and then angular and vue if the front end developers are using that. And then relearn react because hooks were introduced. Then get rid of certain versions of angular and relearn angular. What QA has time for all of this among all the testing (functional, ui, security, performance etc) they are doing? 3. You end up putting logic to automate pages and components SOMEWHERE. It’s page object any way you slice it. Cypress marketing is basically a lie.

Leave a Reply

Your email address will not be published.