Automated Testing Test Automation

Karate: Test Automation Made Simple?

I’ve heard this test automation tool mentioned a few times lately: Karate: Test Automation Made Simple. The idea behind the tool is all one needs to write is gherkin feature files – no step or glue code is required.

I thought I’d give it try to see how it works. My example repository is on GitHub.


Karate allows you to define automated tests in gherkin (Given, When, Then) which are executed directly as either Web Browser or API tests.


A web test looks like:

Feature: Web Browser Automation using Karate

   * configure driver = { type: 'chrome' }

Scenario: Demonstrate webdriver usage
  Given driver ''
  When waitFor('#elementappearschild')
  Then assert exists('#elementappearschild').exists
  Given driver ''
  When click('#homelink')
  And dialog(true)
  And waitFor('#elementappearsparent')
  Then assert exists('#elementappearsparent').exists

and an API test looks like:

Feature: API test example using Karate
    * url ''

  Scenario: get all users should return array of one user
    Given path 'users.json'
    When method get
    Then status 200
    And match response == '#[1]'

  Scenario: get a user should include name and age
    Given path 'user.json'
    When method get
    Then status 200
    And match == '#string'
    And match response.age == '#number'

These tests are fully self-contained, there’s no need to write step definitions or any other code.

Test Execution

There’s a few ways to execute tests. There’s a command line standalone executable you can run passing in details of your tests which are parsed and run. This is a straightforward way to quickly get started, although it seems things like configuring different URLs/environments is more complicated. You can also set it up as a full Java project with IDE and debugging support.

Nice πŸ˜‡

  • Provided standalone executable makes writing and running your first test super quick – including a VSCode plugin to run/debug
  • In built parallel test execution support (although I couldn’t get it to work)
  • Can easily mix API and web browser automation in same script
  • Don’t need to write “code” just follow an API and write a gherkin steps

Not sure πŸ€”

  • I couldn’t get parallel execution of tests to work well using the command line execution argument for Chrome browser tests
  • I’m still not sure about using plain gherkin files to define implementation detail such as css selectors and API end points πŸ€”
  • I’m not sure the best way to get this running in CI: probably using the executable? I didn’t spend time on this.


Whilst Karate is a unique testing tool, I am not sure how well it would scale to a med-large software project considering things like configuration and reusability seem difficult to implement in plain language gherkin files. For a small project or perhaps some throwaway test scripts I think it would be a fun way to write these types of tests.

Meta Wordpress

Why self hosted?

This site still runs the latest version of WordPress but the main difference between this site and my old one is this one is self hosted and not on like my old site was since 2006.

“Why self hosted after 14 years?” I hear you ask. I did work for Automattic who runs for over three years and I am a huge fan of its products and platform but there comes a time when long-standing bugs/missing features weigh you down and make you switch away from something you love.

The first bug was a security limitation on all sites whereby you can’t embed any content from another site.

See I want to embed content from my own site in my knowledge base, but on it isn’t possible – and it just silently fails and falls back to sad looking text hyperlink.

An awesome WordPress embed offers an awesome feature to email new content to subscribers but the inability to change it from anything but immediately was a constant anxiety inducer: if I accidentally hit Publish before being completely ready it would immdiately email 1000+ subscribers with no undo πŸ˜•

I now use MailPoet which enables configuration of when to send emails to subscribers – I send an email once a day at 1pm local time if I have created any new content since it was last sent. This alleviates my publish anxiety as I know I have until 1pm each day to fix it before it’s emailed.

I also really wanted to add an automatically generated table of contents to each page in my knowledge base – there was no way to do this on

One thing I realised by moving to self-hosted is just how much offers without any plugins. I didn’t realise WordPress doesn’t offer in built site stats – that’s a thing.

I try to avoid WordPress plugins since they’re notorious for slowing down your site, however I did end up a small list of handy things that I need:

  1. Akismet Anti-Spam: name says it all 🐷
  2. List Pages Shortcode: to show child pages of a page in a page – just works on
  3. LuckyWP Table of Contents: table of contents
  4. MailPoet 3
  5. Site Kit by Google
  6. SyntaxHighlighter Evolved
  7. Widget Context
  8. Wordfence Security

One thing I can’t do at the moment is automatically publish to Twitter and LinkedIn, but due to my Publish Anxiety anyway I use a manual process for that.

The final question you may ask is why I didn’t use the Business Plan on that allows most plugins and would have solved most of my issues?

This site runs free of ads and creates zero income or revenue for me. The Business Plan would enable some of the features I mentioned above, but it’s AUD$396 per year. My employment income has fallen due to the economic impacts of COVID-19 and I can’t justify that cost when my current managed host costs approx AUD$50 per year to host the same site (8 times less).

Agile Software Dev

Example of lightweight feature documentation πŸ“ƒ

In response to my article about BDD in 2020 I saw this question on Twitter:

I thought it would be easier to write an answer here and provide an example.

I’m not keen on writing specifications in gherkin (Given/When/Then) as I think it’s too generic and frequently makes the feature specifications too verbose – and takes emphasis away from the critical parts.

Here’s a made-up example of some lightweight feature documentation that would be generated during the development process. The important thing to note is the actual documentation artifact generated isn’t as important as the collaborative process that is involved in generating it. This is often about asking the right questions and adding more examples until there’s nothing that is unclear.

The document isn’t a template – the actual documentation varies based on conversation, problem and context.

Example: Royalty Payment Splits


  • Royalties need to be paid in full on disbursement to artists.
  • A royalty payment can be made to an individual artist, or a group of artists.
  • When making payments to an individual artist they get 100% of royalties (no rounding)
  • When making payments to a group of artists this needs to be split by the percentage splits defined in the system which always add to 100%
  • When splitting a payment across a group of artists and it doesn’t split evenly into cents, the system currently randomly splits the cents between artists to balance out the rounding over time
  • This causes issues for both automated tests which need deterministic behaviour, and artists who are confused why they get slightly different amounts if their royalties are the same.

Scenarios / Examples

ScenarioRoyalties OwedCurrent Royalties Paid New Royalties PaidTesting
Single artist gets 100% of whole payments$100.00$100.00$100.00βœ…
Single artist gets 100% of payments including cents$66.67$66.67$66.67βœ…
Two artists with 50% each for a payment that can be split evenly $100.00artist 1: $50.00
artist 2: $50.00
Two artists with 50% each for a payment that can’t be split evenly$100.01$50.01 / $50.00 is randomly assigned to artist1/artist2artist 1 $50.01
artist 2
Two artists with 50% each for a payment that can be rounded to ten cents – no rounding$100.30artist 1: $50.15
artist 2: $50.15
artist 1: $50.15
artist 2: $50.15
Three artists with third splits can’t be split$100.00amounts of $33.33, $33.33 and $33.34 randomly assigned to group members artist 1: $33.34
artist 2: $33.33
artist 3:
Three artists with third splits can’t be split – more than a single cent difference$100.00amounts of $33.33, $33.34 and $33.34 randomly assigned to group membersartist 1: $33.34
artist 2: $33.34
artist 3:

Business Rules

  1. Royalties need to be paid in full on disbursement to artists.
  2. A single artist gets a whole payment.
  3. When payments can be split evenly to a group of artists (to the cent) they are split that way.
  4. When payments can’t be split evenly to a group of artists, the payments are split into an even split and the remaining cents are distributed to the members in whole cents.
  5. The first member in the group – based on earliest date/time added to group – gets the higher amount, followed by the second, third etc. based on earliest date/time added.
  6. Payments aren’t rounded to ten cents or five cent amounts, only whole cents


QuestionDecisionMade By
Do we want to round to five or ten cent distributions?No, we’ll always round to the centProduct Owner via Slack #
How do we distribute based on membership of group?We’ll use the date time added to the group (first gets most)Team during kick off meeting


I think trying to force the above information into gherkin (Given/When/Then) statements would make it less readable and provides no added benefit – whilst Given/When/Then encourages consistency sometimes you just need structured thought that is most relevant to your context. The above document isn’t a template – it varies for the problems we’re trying to solve.

Agile Software Dev

BDD in 2020

It’s 2020: can we officially declare Behaviour Driven Development (BDD) dead?

How did we get here?

β€œThere is no trap so deadly as the trap you set for yourself.”

Raymond Chandler, Long Goodbye

BDD’s lure was the promised land of collaboration between business and technology defining expected behaviour as executable specifications (automated tests). These automated tests defined our software development processes: outside-in software development, behaviour driven development, acceptance driven test development and specifications by example.

BDD’s trap was a bunch of Cucumber specifications written after the fact that create another layer to automate tests in a browser, hidden away in source code which are seldom, if ever, seen by the business, and either too comprehensive, flaky and expensive, or narrowly focused but lacking in enough detail for a complete executable specification of a system.

Sure BDD tests can drive things besides the browser, but the overhead and complexity of maintaining these outweighs the benefits they serve, especially when there is still a need for more expensive integration/e2e tests anyway.

So what actually works in 2020?

I’ve found this process seems to work really well for modern software development in 2020:

  1. Every new piece of work has a kick-off discussion with the product owner, the tester and the developer(s)
  2. The tester usually facilitates this meeting and aims to produce written lightweight documentation during the discussion containing the following information: background info, scenarios, business rules, examples (edge cases), questions and decisions.
  3. This documentation is enough for development work to begin, and the tester to plan how they’ll test it.
  4. The documentation is updated as work progresses: questions arise and decisions are made (and documented).
  5. The developer(s) is/are responsible for making sure there is automated test coverage for these business rules at the cheapest level of testing, these are typically automated unit and/or automated integration tests. These aren’t written in business readable language, they’re test code alongside production code.
  6. The developer also needs to ensure any existing automated end-to-end (e2e) tests are updated for the work that is happening – CI builds highlight these impacts/failures.
  7. When the feature, or a working part of the feature, is ready for testing the developer and tester do a quick handover/walkthrough making sure what was done is what was discussed.
  8. The tester uses the lightweight documentation as a basis for their testing, but adds extra rigour in testing edge cases/unknowns, other impacted areas, different browsers, performance, and devices and anything else they think could be risky.
  9. Bugs that are found and determined by the product owner to be crucial are fixed immediately, some bugs become “fast follows” which are the next thing to ship. Some bugs aren’t fixed.
  10. The tester gives a walkthrough to the product owner and sometimes UX designer to make sure what’s been done is acceptable – if the feature is new/risky it’s launched to a subset of customers and can be disabled at a flick of a toggle.
  11. When an entirely new feature is added to the system and it’s important enough to warrant an (expensive) automated e2e test – the tester works on adding that in amongst their other work. This isn’t a blocker to release and provides a safety blanket for future changes. This isn’t a comprehensive specification of the system – it’s an e2e automated test that covers a business critical flow over multiple moving parts and again doesn’t need to be business readable.
  12. The specifications/documentation lives on in a wiki page and can be referred back to when testing future changes to this feature, or when support tickets or bugs are found in production.

BDD isn’t working for us, what can I do now?

  1. Make sure you commit to having kick-offs and encourage robust discussion creating lightweight documentation as you go – remove uncertainty and create clarity by making and documenting questions and decisions.
  2. Keep your lightweight specifications up to date as you go – don’t worry about making them executable – let people write automated tests in the way that is the most efficient for them.
  3. Only use automated e2e tests to cover key user flows in the most efficient way – don’t use these to create/maintain comprehensive living documentation.

Also See:


Welcome to my new site πŸ™Œ

Welcome to my new site. I thought it was time I gave it a new name: RIP WatirMelon πŸ’€

My focus over coming months alongside my regular blogging will be building up a comprehensive software testing (and more!) knowledge base which will act as a source of information about different aspects of software testing. An example:

If you previously subscribed by email, you’ll need to do that again. See this page for details.

Welcome πŸ‘‹

Agile Software Dev Project Management

The importance of a decision log πŸŽ‹

It’s no secret that I love writing and love documenting things. This is where I’m a bit different to a lot of people I work with who prefer writing code to documentation.

I have a lot of project management responsibilities as part of my current role, and I’m a big fan of “fixed time/variable scope” as a way to get rapidly get important things done. This works in that we can say to our product owner how long do you want to spend on a feature, which then enables us as a development team to do everything we can to get as many things done in that time.

There’s two things I have found that are crucial to succeeding in this approach:

  1. Documenting and maintaining a list of up-to-date things that need doing that is constantly refined and re-prioritized based on discoveries; and
  2. Documenting decisions as they take place in some form of “decision log”

A decision log isn’t for blame. The reason a decision log is crucial is it allows us to decide, document and move on. This enables velocity since we don’t ruminate on decisions. It’s also good during conversations where you feel “dΓ©jΓ  vu” as we often forget what we’ve decided previously, and it’s easy to refer to our log and say “we’ve already talked about this and we decided this for that reason”.

I like the simplest decision log that will possibly work for your context. Ours is a Confluence page with a table in reverse chronological order (so the newest decisions are at the top):

QuestionDecisionMade By / WhereDate
Do we want to limit the max records?We should limit it to 50. That’s a sensible limit and we can adjust based on feedback.Product owner during Slack Conversation (link)28 April 2020
An example decision log

This doesn’t mean we can’t change our mind, flexibility is crucial, we just need another decision log entry to show we changed our minds 😊

How do you make and document decisions? Are you a fan of fixed time/variable scope?

Continuous Delivery Continuous Integration Software

Visualising Technical Debt (with the Debt Boat)

Being successful in software delivery requires a team to constantly balance technical debt and feature delivery. Teams often fall into the trap of delivering features too rapidly at the expense of ever increasing technical debt, or delivering a over-engineered solution at the expense of not delivering things within reasonable time-frames and before any real world usage.

This is one of the key issues facing most software delivery teams I have worked on as there are often different appetites for delivery vs debt throughout an organisation.

I was trying to come up with a way to present technical debt in a visual format that could be easily understood by different people: management, stakeholders and a development team.

One thing we value is shipping things: getting our products in the hands of customers so they can get value from them. Accumulating technical debt, both customer facing and “under the covers” may allow us to rapidly get features to our customers, but eventually the debt grows and affects our ability to deliver.

In the “shipping” theme I came up with the Debt Boat: a way to visually represent things that build up and eventually slow us down. I like splitting technical debt into both feature debt: things our customers want but we chose not to do, and system debt which is technical things our customers (or product owners) can’t see (below the water line) but increasingly slow us down.

We originally had this boat on our team’s physical wall to show there’s only so much technical debt we can take “on board” before we sink.

Now that we’re 100% remote (for now) I’ve created a Jamboard version. I’ve shared the template here – you can easily make a copy if you think your team would find this helpful.

Click to view on Jamboard

Slack tip: closing unnecessary DMs in the sidebar

I like to keep my email inbox as empty as possible: only containing emails that I need to follow up or action which is usually less than 6.

I find Slack can be overwhelming, particuarly Direct Messages (DMs) which build up in the sidebar over time – particularly in a large company. At a previous job I noticed my team lead would use the DMs like an email – open a new DM to someone to ask something, and as soon as there’s nothing more to do close that DM so the sidebar is nice and clean. I didn’t even think to do this but I love doing it now – I only have DMs in the sidebar that require my attention, all other ones aren’t there (you don’t lose history of DMs, clicking the plus βž• and adding a DM to someone shows your previous history).

I recommend giving it a try to see whether it helps you stay on top of your Slack.


Printing an image across multiple pages on macOS

We use our squad wall to display a lot of information and often we’ll want to print out an image across a number of pages so it’s extra large to visualise – for example a screen flow, or a mock up or a chart.

I was trying to work out a way to easily scale an image across numerous pages to stick together on the wall and I found the easiest way is first make sure the image or thing you want to print is saved as a PDF (easy to do in macOS Preview), then open the PDF in the free Adobe Acrobat Reader DC app.

This gives you a poster print option which scales the PDF across numerous pages:

Poster option with scaling and preview

Very handy.

Automated Acceptance Testing Automated Testing Selenium

Playing with Playwright

Playwright is a new browser automation library from Microsoft:

Playwright is a Node library to automate the Chromium, WebKit and Firefox browsers with a single API. It enables cross-browser web automation that is ever-green, capable, reliable and fast.

I’m a big fan of Puppeteer, so this section in their FAQ stood out to me:

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer project is active and is maintained by Google.

We are the same team that originally built Puppeteer at Google, but has since then moved on. Puppeteer proved that there is a lot of interest in the new generation of ever-green, capable and reliable automation drivers. With Playwright, we’d like to take it one step further and offer the same functionality for all the popular rendering engines. We’d like to see Playwright vendor-neutral and shared governed.

Playwright uses similar concepts to Puppeteer:

“Due to the similarity of the concepts and the APIs, migration between the two should be a mechanical task.”

Luckily I have a demo test suite written in Puppeteer which I have cloned and converted to use Playwright to see how it works, and compares.

Here are my thoughts:

I really, really like the BrowserContext concept

In Puppeteer, and WebDriverJs, you have Browsers and Pages. Each Page in a Browser share the state across the Browser, so to create isolated tests using the same Browser (to avoid the inefficiencies of spawning a Browser per test) you need custom code to delete all cookies and local storage between tests. Playwright solves this with the BrowserContext object which is a new incognito window where its pages are created: each test can use the same browser but a different BrowserContext. Super cool πŸ‘Œ

It automatically waits to click, and supports xpath expressions

Playwright automatically waits for elements to be available and visible before clicking, by default, and also has the same API for xpath expressions, which means this Puppeteer code:

await page.goto( `${ config.get( 'baseURL' )}` );
await page.waitForXPath( '//span[contains(., "Scissors")]' );
const elements = await page.$x( '//span[contains(., "Scissors")]' );
await elements[0].click();
await page.waitForXPath( '//div[contains(., "Scissors clicked!")]' );

becomes a lot cleaner:

await page.goto( `${ config.get( 'baseURL' )}` );
await '//span[contains(., "Scissors")]' );
await page.waitFor( '//div[contains(., "Scissors clicked!")]' );

It supports three “browsers” but not as you know them

Q: Does Playwright support new Microsoft Edge?

The new Microsoft Edge browser is based on Chromium, so Playwright supports it.

Playwright supports three “browsers” but not as you know them. I’d say it supports three rendering engines (Chromium, WebKit & Gecko) rather than Browsers as you can only use the (somewhat modified) browsers that come bundled with Playwright over using an already installed browser on your operating system (like Selenium does). This makes it easier to ensure consistency of test runs since the library is bundled with the browsers, but there are some risks your tests could pass on the bundled browsers but fail on “real” browsers. I would say that the claim it supports running on Microsoft Edge is a little misleading.

I’m unsure of CircleCI Support for WebKit and Firefox

I was able to get my tests running against Chromium on CircleCI using the same configuration as Puppeteer, however I couldn’t get the WebKit or Firefox tests to run on CircleCI even when having the default CircleCI browsers installed. I didn’t want to invest the time, but it is probably due to some headless Linux dependencies missing which could be solved in the project config.


If the only thing Playwright did better than Puppeteer was also supporting WebKit and Gecko then I wouldn’t suggest using it over Puppeteer, since Puppeteer is closely aligned with Chromium, and I’m going to run my tests solely in Chrome/Chromium anyway. I don’t believe in running the same e2e tests in multiple browsers: the maintenance overhead outweighs the benefits in my experience.

However, Playwright offers a much nicer BrowserContext concept, and the xpath support is much nicer (although I rarely use xpath expressions anyway).

If anything I am hoping Puppeteer adds support for BrowserContexts – I’ve raised a feature request here so feel free to comment on it if you think it would be a good idea.

All the sample code is available here: