Automated Testing Test Automation

Karate: Test Automation Made Simple?

I’ve heard this test automation tool mentioned a few times lately: Karate: Test Automation Made Simple. The idea behind the tool is all one needs to write is gherkin feature files – no step or glue code is required.

I thought I’d give it try to see how it works. My example repository is on GitHub.


Karate allows you to define automated tests in gherkin (Given, When, Then) which are executed directly as either Web Browser or API tests.


A web test looks like:

Feature: Web Browser Automation using Karate

   * configure driver = { type: 'chrome' }

Scenario: Demonstrate webdriver usage
  Given driver ''
  When waitFor('#elementappearschild')
  Then assert exists('#elementappearschild').exists
  Given driver ''
  When click('#homelink')
  And dialog(true)
  And waitFor('#elementappearsparent')
  Then assert exists('#elementappearsparent').exists

and an API test looks like:

Feature: API test example using Karate
    * url ''

  Scenario: get all users should return array of one user
    Given path 'users.json'
    When method get
    Then status 200
    And match response == '#[1]'

  Scenario: get a user should include name and age
    Given path 'user.json'
    When method get
    Then status 200
    And match == '#string'
    And match response.age == '#number'

These tests are fully self-contained, there’s no need to write step definitions or any other code.

Test Execution

There’s a few ways to execute tests. There’s a command line standalone executable you can run passing in details of your tests which are parsed and run. This is a straightforward way to quickly get started, although it seems things like configuring different URLs/environments is more complicated. You can also set it up as a full Java project with IDE and debugging support.

Nice 😇

  • Provided standalone executable makes writing and running your first test super quick – including a VSCode plugin to run/debug
  • In built parallel test execution support (although I couldn’t get it to work)
  • Can easily mix API and web browser automation in same script
  • Don’t need to write “code” just follow an API and write a gherkin steps

Not sure 🤔

  • I couldn’t get parallel execution of tests to work well using the command line execution argument for Chrome browser tests
  • I’m still not sure about using plain gherkin files to define implementation detail such as css selectors and API end points 🤔
  • I’m not sure the best way to get this running in CI: probably using the executable? I didn’t spend time on this.


Whilst Karate is a unique testing tool, I am not sure how well it would scale to a med-large software project considering things like configuration and reusability seem difficult to implement in plain language gherkin files. For a small project or perhaps some throwaway test scripts I think it would be a fun way to write these types of tests.

Automated Acceptance Testing Automated Testing Selenium

Playing with Playwright

Playwright is a new browser automation library from Microsoft:

Playwright is a Node library to automate the Chromium, WebKit and Firefox browsers with a single API. It enables cross-browser web automation that is ever-green, capable, reliable and fast.

I’m a big fan of Puppeteer, so this section in their FAQ stood out to me:

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer project is active and is maintained by Google.

We are the same team that originally built Puppeteer at Google, but has since then moved on. Puppeteer proved that there is a lot of interest in the new generation of ever-green, capable and reliable automation drivers. With Playwright, we’d like to take it one step further and offer the same functionality for all the popular rendering engines. We’d like to see Playwright vendor-neutral and shared governed.

Playwright uses similar concepts to Puppeteer:

“Due to the similarity of the concepts and the APIs, migration between the two should be a mechanical task.”

Luckily I have a demo test suite written in Puppeteer which I have cloned and converted to use Playwright to see how it works, and compares.

Here are my thoughts:

I really, really like the BrowserContext concept

In Puppeteer, and WebDriverJs, you have Browsers and Pages. Each Page in a Browser share the state across the Browser, so to create isolated tests using the same Browser (to avoid the inefficiencies of spawning a Browser per test) you need custom code to delete all cookies and local storage between tests. Playwright solves this with the BrowserContext object which is a new incognito window where its pages are created: each test can use the same browser but a different BrowserContext. Super cool 👌

It automatically waits to click, and supports xpath expressions

Playwright automatically waits for elements to be available and visible before clicking, by default, and also has the same API for xpath expressions, which means this Puppeteer code:

await page.goto( `${ config.get( 'baseURL' )}` );
await page.waitForXPath( '//span[contains(., "Scissors")]' );
const elements = await page.$x( '//span[contains(., "Scissors")]' );
await elements[0].click();
await page.waitForXPath( '//div[contains(., "Scissors clicked!")]' );

becomes a lot cleaner:

await page.goto( `${ config.get( 'baseURL' )}` );
await '//span[contains(., "Scissors")]' );
await page.waitFor( '//div[contains(., "Scissors clicked!")]' );

It supports three “browsers” but not as you know them

Q: Does Playwright support new Microsoft Edge?

The new Microsoft Edge browser is based on Chromium, so Playwright supports it.

Playwright supports three “browsers” but not as you know them. I’d say it supports three rendering engines (Chromium, WebKit & Gecko) rather than Browsers as you can only use the (somewhat modified) browsers that come bundled with Playwright over using an already installed browser on your operating system (like Selenium does). This makes it easier to ensure consistency of test runs since the library is bundled with the browsers, but there are some risks your tests could pass on the bundled browsers but fail on “real” browsers. I would say that the claim it supports running on Microsoft Edge is a little misleading.

I’m unsure of CircleCI Support for WebKit and Firefox

I was able to get my tests running against Chromium on CircleCI using the same configuration as Puppeteer, however I couldn’t get the WebKit or Firefox tests to run on CircleCI even when having the default CircleCI browsers installed. I didn’t want to invest the time, but it is probably due to some headless Linux dependencies missing which could be solved in the project config.


If the only thing Playwright did better than Puppeteer was also supporting WebKit and Gecko then I wouldn’t suggest using it over Puppeteer, since Puppeteer is closely aligned with Chromium, and I’m going to run my tests solely in Chrome/Chromium anyway. I don’t believe in running the same e2e tests in multiple browsers: the maintenance overhead outweighs the benefits in my experience.

However, Playwright offers a much nicer BrowserContext concept, and the xpath support is much nicer (although I rarely use xpath expressions anyway).

If anything I am hoping Puppeteer adds support for BrowserContexts – I’ve raised a feature request here so feel free to comment on it if you think it would be a good idea.

All the sample code is available here:

Automated Acceptance Testing Automated Testing

Identifying elements having different states in automated e2e tests

I was recently writing an automated end-to-end (e2e) test that was expanding a section then taking an action within the expanded content.

A very simplified example is the details html tag that expands to show content:

When do you start on red and stop on green?

When you’re eating a watermelon!

I initially wrote the test so that it clicked the details element each time the test ran, something like:

async expandJoke() {
	return await this.driver.findElement( by.css( '#joke' ) ).click();

The problem with this straightforward approach is that if the element is already open the click will still perform and the test will continue and then subsequently fail when the test tries to access the content within the expanded section which is now collapsed 😾

I wanted to make sure the test was as resilient and consistent as possible, so instead of just assuming the section was already collapsed I then wrote a function like this to expand the element if it wasn’t expanded and then continue the test:

async expandJokeIfNecessary() {
	const open = await this.driver.findElement( by.css( '#joke' ) ).getAttribute( 'open' );
	if (!open) {

The benefits of this are the test is the most resilient, since it caters for whether the UI is open or closed by checking the open attribute and acting accordingly. I realised that the problem with this approach was that our user experience expects this section to be closed on page load (the punchline is hidden on page load in our example) so if we introduced a bug where we immediately displayed the punchline our test would completely miss it since it just skips expanding the section!

Both these approaches use the same selector to refer to a web element which can have two entirely different states: open and not open.

Keeping this in mind the best solution I could come up with was a selector that combines the element and the state, so the test will fail to click the element to expand the section if it can’t be found, including if the element is already expanded. This gives us simple code that is deterministic and fails at the appropriate time:

async assertAndExpandJoke() {
	return await this.driver.findElement( by.css( '#joke:not([open])' ) ).click();

What’s your preferred approach here? How do you identify elements in different states?

All demo code is available here: #

Automated Testing

My Thoughts on

Run asks…

I’ve just started using As someone who initially learned about testing conventions years ago through your blog, cypress seems to want to burn all old conventions to the ground in a way that immediately turned me off. After playing with it a bit and watching a talk by one of its founders, I’m a little more convinced now. It’s a great tool. Do you have an opinion on Cypress yet, and do you think old testing conventions are becoming obsolete thanks to much better reporting tools around testing?

There certainly seems to be a lot of hype and enthusiasm around I recently saw another (rather evangelical) talk about here in Brisbane so I thought it was time to share my thoughts.

What exactly is

Looking at it is described as a “JavaScript End to End Testing Framework” and “Fast, easy and reliable testing for anything that runs in a browser“. Some other descriptions on include “A complete end-to-end testing experience.” and “Cypress is the new standard in front-end testing that every developer and QA engineer needs.” 

Screen Shot 2019-07-06 at 2.54.48 pm.png

What is end-to-end testing?

I believe there are some specific traits that define what automated end-to-end (e2e) tests are:

  • They test a complete user flow through an application from start to finish (end-to-end)
  • They test how a real user would use a using a fully deployed system
  • They test the happy-path of the most commonly used scenarios, avoiding error validation or edge-cases

End-to-end tests are expensive to maintain and execute so the widely accepted view is to have as few of these as possible for your application, which means avoiding things like negative and error validation testing during end-to-end tests as these things can be tested much more easily and quickly in isolation as other types of automated tests (unit, component or integration).

Is a framework for writing end-to-end tests?

Despite its strongly worded marketing material, I don’t believe has been designed as an end-to-end (e2e) testing framework. I believe this was confirmed by Brian Mann at Assert(js) in the first half of the “I see your point, but…” presentation:

“You should always strive to test pages in total isolation – everything becomes faster, less coupled, and you won’t lose a single point of confidence that it’s all working together correctly. You don’t need to limit yourself trying to act and replicate everything a user would do.”

Screen Shot 2019-07-06 at 2.28.05 pm

I believe what Brian is referring to aren’t end-to-end tests but rather component tests. Brian showed using to test a login page where he wrote 6 isolated test specs to test login validation:

Screen Shot 2019-07-06 at 2.34.36 pm

What makes the question about whether is an e2e testing framework even more confusing is during the second half of the same presentation, Gleb Bahmutov, also from, states:

“Brian showed how we think about end-to-end testing. To us end-to-end should do the same things that a human would do to a fully deployed system. Right, that means real browser, real interactions, no shortcuts…”

Screen Shot 2019-07-06 at 2.45.19 pm

Also, confusingly, he stated that e2e test tools can do a pretty good job in unit testing:

Screen Shot 2019-07-06 at 2.45.38 pm

So what is then?

I now consider to be a strongly opinionated framework suited to writing isolated automated web component tests. I’m not sure why it is marketing as an e2e testing tool, and trying to compare itself to something like Selenium, which isn’t a component testing tool.

If you wanted to write isolated automated web component tests then would be worth a look at, since it offers many features to help you. However for true end-to-end purposes I think the limitations outweigh the benefits. The open source WordPress Gutenberg editor project tried for quite a while but ultimately found it too limiting and switched to Puppeteer.

Some things to considering when trying to use for true end-to-end testing

Even though is demonstrated as a way to write isolated web component tests, if you still want to use it to write true end-to-end tests then there’s some tradeoffs you need to consider.

Screen Shot 2019-07-06 at 2.55.49 pm

Despite the claims that “Cypress works on any front-end framework or website” and “Fast, easy and reliable testing for anything that runs in a browser” there are quite a few scenarios where you can’t use Cypress – for example if your front-end or website uses iFrames you can’t use

iFrames itself uses iFrames to inject itself into the browser so supporting iFrames whether on the same domain or cross domain aren’t supported with an open issue since 2016. At we used iFrames for the WordPress site customizer so it wouldn’t be possible to write an end-to-end test for using In my current role I work on a web application which actually a series of React micro frontends which are rendered in iFrames within a web container, so we also can’t use for end-to-end testing.

Native browser events like file uploads and downloads

Things like uploading and downloading files that are trivial to do in WebDriver are either difficult or not supported in Even things as simple as using the tab key isn’t supported.


Whilst Cypress is often promoted as a free and open source project there are certain features that are only available when running your tests in “record” mode with the Cypress Dashboard Service, which allows 500 tests (it blocks) to run in parallel per month before needing to pay for it. Parallel execution is one of these features, so you can’t even run tests in parallel locally without recording your results to the dashboard service.

The biggest issue I see with parallelization is that it is machine based, not process based:


In this example, the CI container costs triple when each CI machine should be more than capable of running multiple Chromium browser sessions.

At we used CircleCI and we able to have up to 12 headless Chrome browsers using WebDriverJs in parallel on each CircleCI container, and across 3 containers this allowed 36 e2e tests running in parallel. To get the same result using would mean paying for 36 CircleCI containers.

Running e2e tests written in other e2e testing tools in parallel machine processes can be quite easy as I’ve explained previously on this blog.

There are times when running via the command line isn’t the same as running via the GUI

I noticed this when writing a demo cypress spec which would pass when running in the Cypress GUI runner but fail on the command line, which looking at these comments doesn’t seem uncommon.

    it( 'ignores alerts when leaving the page', function() {
        cy.contains('WebDriverJs Demo Page').should('be.visible');
    } );

GUI Runner:

Screen Shot 2019-07-06 at 4.33.53 pm

Command Line:

Screen Shot 2019-07-06 at 4.37.17 pm

Logging in

One of the key messages of the first video was demonstrating you can log in without using the UI for subsequent tests which speeds things up. This is a good idea, but isn’t unique to at we re-used a single login cookie across multiple e2e tests using WebDriverJS – the code is here.


From a distance Cypress looks like a polished tool for automated testing – I just think it’s incorrectly marketed as an end-to-end testing tool when it’s really only good for component testing. There are too many limitations in the tool in acting like a real user to use it to create true end-to-end automated tests.

Automated Acceptance Testing Automated Testing Business Analysis Conferences

Adventures of end-to-end automated web testing in Node.js

This is a presentation I gave at ATTAC in Melbourne on Friday 24th May 2019. The full Google Slides are available here.

Good morning everyone and thanks for having me along to speak.
My name is Alister Scott and I’m from Brisbane. I work as a QA generalist at a software company in Brisbane called Console, and I write a blog called WatirMelon. Today I’m going to be sharing my knowledge of end to end automated testing in Node.js. I’ve created very simple but working demos of the main tools I’ll be discussing today and I’ve put these on GitHub as separate repositories you can easily clone and play around with:

Outside of work I enjoy hiking, often to the summits of mountains, and spending time with my wife and our three young kids.
In 2015 I started a paid trial at Automattic. My trial project – on which I would be assessed to gain full time employment – was to establish an automated e2e testing framework for – the first of its kind for Automattic. I quickly spun up a framework using Watir in Ruby – because that’s what I knew – and it worked. However I quickly gained some feedback that of the hundred+ developers at Automattic almost none knew Ruby and creating shared ownership for e2e tests would be a key measure of success so I had to rethink. At the time Automattic was moving from primarily developing in PHP towards Node.js meaning full stack JavaScript and it made sense that the e2e tests were also developed in Node.js.

This of course made my trial project a lot more difficult than I had originally thought as I had to teach myself Node.js and understand what testing tools existed in this space.

Fast forward to 2019 and earlier this year I decided to change jobs as I could no longer travel for work. When I started looking at job advertisements for testing and QE positions in Brisbane I noticed just how many mentioned Node.js as their technology stack of choice: in the 3.5 years at Automattic Node.js had become increasingly popular at other companies as well.
But automated e2e testing in Node.js was and is really hard. Much much harder than I was used to in Ruby and Watir where things just worked.

It’s slightly better in 2019 than 2015, however there are some reasons why it’s hard.
There’s also the paradox of choice when it comes to tooling specifically for e2e web testing. Searching for selenium and WebDriver on NPM provides a list of dozens of libraries – and the official Selenium bindings (WebDriverJs) don’t appear when searching for WebDriver. It’s all very confusing.

My aim today is to create some clarity in this space.
I believe there’s more e2e test tools in Node.js that something like Java or C#. I’ll talk about the four most popular/mature ones. is better suited to component testing than true end-to-end testing. also doesn’t support cross-domain stuff – so beware if you’re doing anything like that.
You’ll need your own test runner and assertion library.
I’ve distilled these four tools down into an easy to read visual.
And an even easier to understand flow chart.
Automated Testing Continuous Delivery Continuous Integration

Scheduling CircleCI Jobs

We use CircleCI to run our automated end-to-end (e2e) tests for

We run our tests pretty frequently – not only against every individual change coming through – but we also run them in Production every time someone deploys (about 30 times per day) – as well as every 6 hours to cover weekends and quiet periods of deployments to make sure our hardware and other changes haven’t impacted our key customer flows.

Originally CircleCI didn’t support scheduled jobs so we set up our own infrastructure to schedule jobs to call the CircleCI API which executed the tests.

Fortunately version 2.0 of the CircleCI config now natively supports scheduling jobs which is exactly what we want to do. Since it also uses cron, the default scheduling format, it was very easy to create our jobs in CircleCI.

This is what our .circleci/config.yml looks like:

Before – no scheduling

version: 2
      - image: circleci/node:10.5.0-browsers
      - checkout
      - restore_cache:
            - v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
            - v1-npmcache-{{ checksum ".nvmrc" }}
            - v1-npmcache
      - run: npm ci
      - save_cache:
          key: v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
            - "~/.npm"
      - run:
          name: Execute the e2e tests
          command: npm test

After – with scheduling

version: 2
      - image: circleci/node:10.5.0-browsers
      - checkout
      - restore_cache:
            - v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
            - v1-npmcache-{{ checksum ".nvmrc" }}
            - v1-npmcache
      - run: npm ci
      - save_cache:
          key: v1-npmcache-{{ checksum ".nvmrc" }}-{{ checksum "package-lock.json" }}
            - "~/.npm"
      - run:
          name: Execute the e2e tests
          command: npm test
  version: 2
      - test
      - schedule:
          cron: "0 0 * * *"
                - master
      - test

The thing I love about CircleCI is that config is checked in as code so it’s super easy to manage and track changes to the config over time. My sample project is available here.

Automated Testing Conferences

→ Scaling Selenium to infinity using AWS Lambda

A great presentation from the recent 2018 Selenium Conference with a few shout-outs to this blog. I really like what Wes and Kurt were able to achieve using AWS Lambda – amazingly fast e2e tests running in parallel.

Ask Me Anything Automated Acceptance Testing Automated Testing JavaScript WebDriver

AMA: Protractor for e2e Testing?

John X asks…

 I have dodged the AngularJS /Protractor bullet for many years now. May last foray, some 5 years ago, was a cluster to put it mildly! Cucumberjs/Angularjs/Protractor/chai/mocha/ the stack was in its infancy and failed miserably!! Many cycles were spent pulling fixes from our own repos instead of waiting for PRs to get done. This was in stark contrast to the ease and stability of the automation I wrote using Watir-Webdriver and eventually Watir.

I am now faced with automating the regression test cases for an Angularjs App.

Question: Do I finally jump back into the using stack that almost caused me to lose my mind, or is it possible to use Watir/Selenium to build out meaningful e2e UI automation for an angularjs app as we dawn on 2019?

My response…

It’s still my opinion in 2018 that writing e2e tests in Node using either Protractor or WebDriverJs is still more difficult than using Watir in Ruby.

Sure using async functions with await commands makes things easier than before (see examples in our project), when you would have first come across Protractor, but I still feel like there’s a lot of catching up to do to get to the stability and ease-of-use of Watir.

My decision would come down to whether others on your project are going to be comfortable maintaining tests in ruby – if they are I’d use Ruby and Watir, otherwise I’d revisit Protractor if they really want to use Node.

Automated Testing Conferences

TestBash Sydney: Automated e2e Testing at

This is an approximate transcript of the talk I delivered at TestBash in Sydney on Friday 19th October 2018.

Alister Scott Test Bash Sydney Oct 2018-1-1

Today I’d like to share my story about how we started with automated end to end testing at since I started at Automattic over 3 years ago.

Alister Scott Test Bash Sydney Oct 2018 2-2

Ask Me Anything Automated Testing Automattic

AMA: Separate Repository for e2e Tests?

Liam asks…

“I did enjoy reading the article about e2e test on wordpress. I noted that e2e test are in a separate repo.

My question will be what is the workflow to make sure new changes does not break the e2e test on pull request ?

For example, if a developer work on some changes, then they need to change the e2e test first and make sure everything pass, however the environment on the pull request might not be stable, developer can overwrite each other changes”

My response…

Thanks for your question Liam.

We have reasons for and benefits in having the e2e tests in a separate repository:

  1. The e2e tests test the entire experience so these test things that happen in different repositories (for example our Calypso user interface or services/API) and having them in the user interface repository isn’t really representative of what the breadth of their scope;
  2. Making changes to the e2e tests are easier in a separate repository since we don’t have to deploy e2e PRs that don’t contain functional changes (we deploy every merge to our master branch immediately dozens of times per day)

The obvious downsides are:

  1. How do we make sure e2e tests know about incoming AB tests?
  2. How do we couple new changes to updates in the e2e test repository?

For incoming AB tests we make sure that our e2e tests know about the change by ensuring we create a matching PR in our e2e tests that override our AB tests during our test runs.

If someone updates the AB tests in Calypso they’re politely reminded to update the e2e tests:

Screen Shot 2018-09-19 at 3.31.47 pm
example prompt

For making sure e2e tests are up to date we automatically run two (of about 40 total) of the most critical e2e tests (in three browsers) when a PR is ready to be reviewed. These can fail and indicate a change is necessary to the e2e tests (or something is broken!)

There’s also a label we can add to any PR that runs the entire set of e2e tests against a PR running live and reports the result back to that PR:

Screen Shot 2018-09-19 at 3.38.29 pm
e2e Test Results against a Calypso PR

If changes are required to the e2e tests someone can create an e2e PR with the exact same branch name which will be used to run against the feature changes before they are merged. This means PRs can be coupled and tested together but merged separately.

To answer the second part of your question I understand it to be about conflicting changes? One of our key philosophies for work is “merge early and merge often” so we make sure that PRs are short-lived and merged quickly to minimize the chance of conflicts. These still do happen occasionally, we just deal with them as they come up.

Whilst there’s been some downsides to the separate repositories all-in-all the benefits continue to outweigh the downsides but we constantly assess and at any point in time we can easily merge them if need be.