API Testing Automated Testing

API “approval” testing using Jest

I recently added some API integration testing examples to my Playwright/Jest e2e automated testing repository using Supertest.

I wanted to also explore the use of Jest for approval testing of REST JSON APIs.

Jest has in built support for snapshot testing which is most commonly used to test how React components are rendered – however Jest snapshot testing can also be used to test any serializable values.

Snapshot tests are an example of what are are also known as approval tests: whereby the expected output of a test is generated the first time it is run and compared against this on subsequent runs. Changes to the output can be “approved” which become the new baseline.

These are no where near as common as “assertion” based tests whereby an automated test performs some action and asserts some results.

Assertion tests can be used to assist test driven development, whereas approval tests are generated after the fact so they can be useful for outside-in software development, API integration testing of known endpoints, and for legacy code where no assertion based or lower level tests exist.

I wrote the same integration tests in both approval and assertion style to show how these differ.

Say we have an endpoint that returns all the blog posts we have seeded into our system, we could write some assertions on the content:

const response = await request.get('/posts')
expect(response.body[0].title).toBe('Post 1')
expect(response.body[1].title).toBe('Post 2')
expect(response.body[2].title).toBe('Post 3')

Since we’re testing an existing endpoint with known “approved” behaviour we can generate and test a snapshot of it:

  const response = await request.get('/posts')

The first time we run this the following is generated in a separate file:

// Jest Snapshot v1,

exports[`can GET a REST API and check response using approval style 1`] = `
Array [
  Object {
    "id": 1,
    "title": "Post 1",
  Object {
    "id": 2,
    "title": "Post 2",
  Object {
    "id": 3,
    "title": "Post 3",

We commit this to our codebase and if any of the body returned for this endpoint is changed our test fails (with an option to approve the new result as the new baseline).

The benefit of these approval tests is they are quicker to write for an existing endpoint (since they are generated automatically) and are complete in that any change to the body returned will be highlighted – compared to our assertion based tests which only test specific things.

The downside to these tests is they are generated after the fact – so aren’t compatible with test driven development – and need to take into account if there is any dynamic content that changes between test runs – so these generally aren’t good for non-deterministic systems.

Feel free to check out the source code for these API integration tests and let me know what you think and whether you’ve had experience, good or bad, with approval testing.

e2e Testing

Playwright + Jest = 💖

We were looking at a possible replacement for our dated Protractor + Cucumber e2e testing framework. As we move away from Angular to React microapps we have found that Protractor doesn’t work very well/efficiently and cucumber isn’t giving us any benefits.

It was a good opportunity to do some research/tinkering to answer the question lingering in my mind: in 2020 what e2e testing tool would I use by default for a dynamic react based web application?

After some experimentation it came down to Puppeteer + Jest or Playwright + Jest. I’ll compare those in this post.

Why Jest?

As explained previously we don’t need a BDD framework, but we do need something that allows us to specify our tests, create assertions and run these tests (in parallel). Jest, particularly when using the Jest Circus test runner, seems the most mature tool in this regard in 2020. Whilst Jest is often associated with React as it handles the snapshot test results also, it doesn’t need to be used with React and it’s possible and easy to use it as a standalone Node.js testing library.

Parallel Support

Jest by default runs tests across files in parallel and uses the available resources to scale appropriately using processes/threads. I’ve found this particularly good as I write independent e2e tests which can scale through parallelism, and using Puppeteer and Playwright you can spawn new incognito browser contexts to run these.

Auto-Retry Support

Whilst I believe in writing deterministic e2e automated tests, since e2e tests are full-stack there are often external dependencies and services beyond our control (like a third party domain provider) and I’d rather prioritize test reliability over test perfection. With this mind I think it’s important to be able to automatically retry a single failing test scenario before failing a build, and the Jest Circus test runner supports this:


test('can wait for an element to appear', async () => { = await pages.spawnPage()
  await nav.visitHomePage(
  await'#elementappearschild', { visible: true, timeout: 5000 })
}, jestTimeoutMS)

HTML Reports and Screenshots

Nice looking HTML reports are easy to achieve by using jest-stare and screenshots are easy to generate using Jest Circus hooks.

Playwright or Puppeteer?

I’m a fan of Puppeteer however Playwright is a much nicer browser automation library. Whilst it adds support for Firefox and Webkit, even if you’re running your e2e tests in one browser (Chromium) I’d still recommend Playwright over Puppeteer any day of the week. Here’s why:

Automatic Waiting

I’ve found the automatic waiting in Playwright just works™️ Especially when dealing with dynamically rendered react web apps I’ve found my Puppeteer code looks is full of page.waitFor calls to make it run reliably:

await page.waitFor('#loadedchild', { visible: true, timeout: 5000 })

Whilst this is occasionally necessary in Playwright (in particular waiting for an iFrame to switch into), I’ve found it’s almost never required which reminds me of good old Watir days.

Nicer API

Whilst the APIs are similar, Playwright is just nicer to use. Take grabbing some text from a div, in Puppeteer:

await page.goto(`${config.get('baseURL')}`)
await page.waitFor('#loadedchild', { visible: true, timeout: 5000 })
const element = await page.$('#loadedchild')
const text = await (await element.getProperty('textContent')).jsonValue()

In Playwright:

await page.goto(`${config.get('baseURL')}`)
const text = await page.textContent('#loadedchild')


If I was writing an e2e web testing framework from scratch in 2020 I would use Playwright + Jest. Playwright offers automatic waiting and a nice API, whilst Jest offers a solid runner with parallel support, automatic retries and the ability to easily generate HTML reports and capture screenshots.

I’ve created one repository for Playwright + Jest #
And another for Puppeteer + Jest to compare #

As an aside we put changing node libraries on hold for our e2e test framework and will look at some test infrastructure improvements instead.

e2e Testing

Puppeteer Jest Demo

I’m trying to work out what would be the ideal combination for an e2e web testing framework in Node.

I wanted to try Jest with Puppeteer (I’d previously used Mocha) and I also wanted to try avoiding Babel for transpiling and using JavaScript directly so I came up with this example on GitHub.

I think I’ll try using TypeScript for my next tinker project.

Remote Work

Corner Office?

I read and enjoyed this article by Jim Grey: Too soon to declare victory on distributed work

“A group of voices in our industry has said for years that we would all be more productive and have a superior work-life balance if we all worked from home. Now that we’re all working from home and it’s basically working, they are crowing victory.”

Jim Grey

The article linked is to is a NY Times article, ironically printed on paper, of an interview with the CEO of the digital publishing company where I worked remotely for over three years.

“This column is called Corner Office, and most people who choose to have offices are usually the bosses. And I’ve been to the offices of billionaire C.E.O.s that have their own private bathroom, beautiful art and couches. But these are all things that you can have in your house. What I love about distributed organizations is every single employee can have a corner office.”

Matt Mullenweg via the NY Times # (emphasis added)

What Mullenweg doesn’t state is that this is a rich person’s view of distributed work. What isn’t mentioned is the cost of having a dedicated “corner” office in your home, a private bathroom, the cost of beautiful art and couches (and no Automattic wouldn’t reimburse art, nor couches, nor home renovations, only a monitor/desk/chair, at least during my time there).

I wrote about this aspect of remote work in my essay:

“Housing costs are very high in Australia and we’re moving to smaller and more expensive houses so it’s increasingly harder to justify having a dedicated office at home. If you were building a house to live in here in Australia adding a 15-20 square meter dedicated home office for remote work could easily add $30,000-$40,000 to your build costs.”

The future of work? An essay.

As a single-income family-of-five our rented home is much smaller, more humble and basic than the office our company leases. Even as an individual contributor at work I can sit and eat my lunch, or wander around with views like these:

Our office view – accessible to all employees

Another interesting angle on having an office space available to all staff is that it makes a safe, comfortable and productive work environment available to everyone regardless of personal circumstance.

I wrote about this in my essay also:

“If you don’t have a suitable place to work from home, and co-working expenses are too expensive, or not available or too far from home, this means that there’s a whole group of people who can’t consider remote work which reduces the diversity of the workforce.”

The future of work? An essay.

This concept was recently mentioned in an article in Australia about a tragic domestic violence case:

“Ms Foster says for many people experiencing violence their workplace has been a sanctuary — somewhere they can escape “the horrors they’re experiencing at home”.

She says women working from home have been sharing with her how they are being constantly harassed, threatened, and assaulted, only to have to then join a video conference and act professional.”

Sam’s mum was killed while working from home. He hopes his workers compensation win will help others facing family violence #


I still think it’s naive, borderline dangerous, to crow victory on working from home as a one-size-fits-all solution to the world’s problems – and pretend that every person in the world can afford to have their own “corner office” in their home, when for a lot of people, a working environment for work can be a better or only option to work safely and productively.

Agile Software Dev

Rapid feature delivery using “Fast Follows”

I’ve mentioned previously that I like using “Now”, “Next”, “Later”, “Never” as a way to prioritise the system features we’re delivering.

But we often get in a situation where we want to deliver a “now feature” quickly, but we find something that we need to do on it that we don’t want to want to group with the “next” features. To steal a term startups use and reinterpret it: we call these “Fast Follows”.

“Fast Follow” is a startup strategy that aims to build a company by copying an already proven business model- and executing quickly to “follow” the moves of the predecesor.

Fast follows allow us to quickly ship “now” features into the hands of our customers knowing we have a mechanism to follow quickly with an enhancement or bug fix.

Do you do “fast follows”? What do you call them?

Remote Work

Back in the office

I’ve been back in the office for a few weeks now. Only about 10% of our company can work in the office and even then only about that percentage are keen to return (at present).

I find myself enjoying work again and regaining the ability to mentally separate work from home has been energizing me for when I am at work. Also having people around motivates and recharges me at work.

I read an interesting article written by Justin Qin, a year 8 high school student, who reflects on returning to school after many weeks of home-based learning:

“Lockdown and home schooling had been a lonely environment for me — even with two energetic brothers for company, and Instagram on repeat.

And after days at a time like this my friends and I chatted constantly when we saw each other, flushing out the loneliness of isolation and reviving the joy of face-to-face chat even though we had to maintain that all-important distance.

Being in the classroom, and the ease of talking to my teacher and other students, makes school a place where I can actually learn and be engaged.

My school day was spent stuck at a desk at home, with tiny breaks and no time to chat to friends as we transitioned between lessons. I often felt as if I had not stood up for hours at a time.

The result was that learning from home was seriously affecting my physical and mental health.

Mentally, I wrestled with finding the meaning of my life when so much of it was spent alone at a desk.

Physically, I ate significantly less food than normal.

With these struggles I cannot see a strong future for online classes replacing school.”

Justin Qin on returning to Year 8 at School #

When I look at my LinkedIn feed it’s full of articles, comments and observations that many people can’t see themselves returning to the office in the near, or far, future.

I put this down to one of two possible things:

  1. People are still in the “honeymoon” period of full time working from home; and/or
  2. People are confusing working from home with having flexible working conditions

“The first six months were a dreamlike period, where everything you’d ever hated about office life – underlings, overlords, sharing bathrooms with people who you’re thankful aren’t your family – morphed into amusing memories.”

“It’s all fun and games at first, a novelty, the home work thing, and being close to your own fridge is great, but the slow evaporation of the mental distance between your home life and your work one is an insidious one.”

Stephen Corby : You might love working from home now, but give it a year

As I’ve stated previously, I personally don’t find myself healthy, happy or productive working from where I live. I prefer to separate work and home, and to have people I’m working with around in real life.

But I’ve also said we need more flexibility in the way we work, working from home doesn’t have to be the only way we can have flexible working conditions.

I’ve also seen a somewhat disturbing trend to encourage everyone to act like they’re working remotely even if they’re in the office:

“If everyone sits at their desk on an individual video call screen, the playing field is equal. Everyone’s faces are easily distinguishable, it’s easy to know whose turn it is to talk, and it simply puts everyone’s input on equal footing.”

6 Rules To Live By When You Work In An Office But Have Remote Team Members

I’ve tried doing that and the audio feedback from having someone sitting near you talking on the call and also near you is terrible, it’s loud and distracting for other’s in an open plan office, and it feels weird to pretend we’re not in the office when we are.


I still think it’s too early to declare that ever working from an office again is not going to happen. As I have previously stated – it’s been good to show employers that employees can still be productive when given workplace flexibility – however I think there will be recognition that working 100% in physical isolation from colleagues isn’t necessarily a good thing.

“A world in which people who want to can do a bit of both, working home alone some of the week and sharing an office with other humans the rest of the time, might just be the perfect balance, for those lucky enough to swing it”

Stephen Corby : You might love working from home now, but give it a year

Specification by Example

Given/When/Then with Style?

Gojko Adzic has started a series of challenges on the Specflow blog where readers can submit solutions to common problems writing automated tests using gherkin (Given/When/Then).

The first challenge is:

“What’s the best way to describe a Given for a value that’s not supposed to be there? For example, starting the specification from a point of a user who does not yet exist in a database, or a product that’s not been set up. “Given a user that doesn’t exist” sounds vague, but “Given user John that does not exist” sounds silly – how can we talk about John if he doesn’t exist yet?”

Gojko Adzic
Given user John does not exist
When John tries to register
Then the registration is successful

Given user John exists
When John tries to register
Then the registration fails

How would you improve the scenario of our example to solve the challenge of this week?

Gojko Adzic

Interestingly this problem highlights the issues I found using Gherkin to specify business rules and examples I previously wrote about:

“I’m not keen on writing specifications in gherkin (Given/When/Then) as I think it’s too generic and frequently makes the feature specifications too verbose – and takes emphasis away from the critical parts.”

Example of lightweight feature documentation

I prefer bullet points and tables.

I’d specify the above scenarios as follows.

User Registration


  1. A user can only register a single account per unique email address
  2. A user’s email address is their username
  3. A user can register using their Google or Facebook account without having to enter a username or password
  4. A user must enter a password unless they are registering using Google or Facebook
  5. Passwords must contain at least 4 letters and 4 digits
  6. Passwords are maximum length of 10 and can contain any ASCII characters

Registration Scenarios/Examples

ScenarioExample Email AddressResult
New user signs up with a new email✅ Registration Successful
New user signs up with a new email and invalid password❌ Registration Unsuccessful – Validation Error Displayed and no account created
User attempts to sign up with email already registered email for account with password❌ Registration Unsuccessful – Reset password page displayed with email prefilled
User attempts to sign up with email already registered email for account with Google❌ Registration Unsuccessful – Log in with Google page displayed

It’s tempting to put passwords into the above table, which is what I originally did, but I’ve found it’s better to specify those separately (in the same document).

Passwords Scenarios/Examples

Password ScenarioExample PasswordValid/Invalid
Less than 8 characters123
Less than 4 digitsaaaabbbb
Less than 4 letters123456aa
No emoji – ASCII Chars Only1234abcd💀
More than 10 characters123456abcdef
10 Characters with ASCII1234abcd%$
4 digits and 4 lettersabcd1234
Special characters don’t count as lettersabc$1234


Whilst these scenarios aren’t gherkin I don’t think they need to be. They could be taken by developers and written as unit tests alongside the code. I wouldn’t have an e2e automated test that tests this logic at this level – it’s complex and an expensive waste of time.

I’ll be interested to see the follow up posts about the best way to specify these in gherkin, as I can’t see a clearer way than just using bullet points and tables 🙌

Automated Testing Test Automation

Karate: Test Automation Made Simple?

I’ve heard this test automation tool mentioned a few times lately: Karate: Test Automation Made Simple. The idea behind the tool is all one needs to write is gherkin feature files – no step or glue code is required.

I thought I’d give it try to see how it works. My example repository is on GitHub.


Karate allows you to define automated tests in gherkin (Given, When, Then) which are executed directly as either Web Browser or API tests.


A web test looks like:

Feature: Web Browser Automation using Karate

   * configure driver = { type: 'chrome' }

Scenario: Demonstrate webdriver usage
  Given driver ''
  When waitFor('#elementappearschild')
  Then assert exists('#elementappearschild').exists
  Given driver ''
  When click('#homelink')
  And dialog(true)
  And waitFor('#elementappearsparent')
  Then assert exists('#elementappearsparent').exists

and an API test looks like:

Feature: API test example using Karate
    * url ''

  Scenario: get all users should return array of one user
    Given path 'users.json'
    When method get
    Then status 200
    And match response == '#[1]'

  Scenario: get a user should include name and age
    Given path 'user.json'
    When method get
    Then status 200
    And match == '#string'
    And match response.age == '#number'

These tests are fully self-contained, there’s no need to write step definitions or any other code.

Test Execution

There’s a few ways to execute tests. There’s a command line standalone executable you can run passing in details of your tests which are parsed and run. This is a straightforward way to quickly get started, although it seems things like configuring different URLs/environments is more complicated. You can also set it up as a full Java project with IDE and debugging support.

Nice 😇

  • Provided standalone executable makes writing and running your first test super quick – including a VSCode plugin to run/debug
  • In built parallel test execution support (although I couldn’t get it to work)
  • Can easily mix API and web browser automation in same script
  • Don’t need to write “code” just follow an API and write a gherkin steps

Not sure 🤔

  • I couldn’t get parallel execution of tests to work well using the command line execution argument for Chrome browser tests
  • I’m still not sure about using plain gherkin files to define implementation detail such as css selectors and API end points 🤔
  • I’m not sure the best way to get this running in CI: probably using the executable? I didn’t spend time on this.


Whilst Karate is a unique testing tool, I am not sure how well it would scale to a med-large software project considering things like configuration and reusability seem difficult to implement in plain language gherkin files. For a small project or perhaps some throwaway test scripts I think it would be a fun way to write these types of tests.

Meta Wordpress

Why self hosted?

This site still runs the latest version of WordPress but the main difference between this site and my old one is this one is self hosted and not on like my old site was since 2006.

“Why self hosted after 14 years?” I hear you ask. I did work for Automattic who runs for over three years and I am a huge fan of its products and platform but there comes a time when long-standing bugs/missing features weigh you down and make you switch away from something you love.

The first bug was a security limitation on all sites whereby you can’t embed any content from another site.

See I want to embed content from my own site in my knowledge base, but on it isn’t possible – and it just silently fails and falls back to sad looking text hyperlink.

An awesome WordPress embed offers an awesome feature to email new content to subscribers but the inability to change it from anything but immediately was a constant anxiety inducer: if I accidentally hit Publish before being completely ready it would immdiately email 1000+ subscribers with no undo 😕

I now use MailPoet which enables configuration of when to send emails to subscribers – I send an email once a day at 1pm local time if I have created any new content since it was last sent. This alleviates my publish anxiety as I know I have until 1pm each day to fix it before it’s emailed.

I also really wanted to add an automatically generated table of contents to each page in my knowledge base – there was no way to do this on

One thing I realised by moving to self-hosted is just how much offers without any plugins. I didn’t realise WordPress doesn’t offer in built site stats – that’s a thing.

I try to avoid WordPress plugins since they’re notorious for slowing down your site, however I did end up a small list of handy things that I need:

  1. Akismet Anti-Spam: name says it all 🐷
  2. List Pages Shortcode: to show child pages of a page in a page – just works on
  3. LuckyWP Table of Contents: table of contents
  4. MailPoet 3
  5. Site Kit by Google
  6. SyntaxHighlighter Evolved
  7. Widget Context
  8. Wordfence Security

One thing I can’t do at the moment is automatically publish to Twitter and LinkedIn, but due to my Publish Anxiety anyway I use a manual process for that.

The final question you may ask is why I didn’t use the Business Plan on that allows most plugins and would have solved most of my issues?

This site runs free of ads and creates zero income or revenue for me. The Business Plan would enable some of the features I mentioned above, but it’s AUD$396 per year. My employment income has fallen due to the economic impacts of COVID-19 and I can’t justify that cost when my current managed host costs approx AUD$50 per year to host the same site (8 times less).

Agile Software Dev

Example of lightweight feature documentation 📃

In response to my article about BDD in 2020 I saw this question on Twitter:

I thought it would be easier to write an answer here and provide an example.

I’m not keen on writing specifications in gherkin (Given/When/Then) as I think it’s too generic and frequently makes the feature specifications too verbose – and takes emphasis away from the critical parts.

Here’s a made-up example of some lightweight feature documentation that would be generated during the development process. The important thing to note is the actual documentation artifact generated isn’t as important as the collaborative process that is involved in generating it. This is often about asking the right questions and adding more examples until there’s nothing that is unclear.

The document isn’t a template – the actual documentation varies based on conversation, problem and context.

Example: Royalty Payment Splits


  • Royalties need to be paid in full on disbursement to artists.
  • A royalty payment can be made to an individual artist, or a group of artists.
  • When making payments to an individual artist they get 100% of royalties (no rounding)
  • When making payments to a group of artists this needs to be split by the percentage splits defined in the system which always add to 100%
  • When splitting a payment across a group of artists and it doesn’t split evenly into cents, the system currently randomly splits the cents between artists to balance out the rounding over time
  • This causes issues for both automated tests which need deterministic behaviour, and artists who are confused why they get slightly different amounts if their royalties are the same.

Scenarios / Examples

ScenarioRoyalties OwedCurrent Royalties Paid New Royalties PaidTesting
Single artist gets 100% of whole payments$100.00$100.00$100.00
Single artist gets 100% of payments including cents$66.67$66.67$66.67
Two artists with 50% each for a payment that can be split evenly $100.00artist 1: $50.00
artist 2: $50.00
Two artists with 50% each for a payment that can’t be split evenly$100.01$50.01 / $50.00 is randomly assigned to artist1/artist2artist 1 $50.01
artist 2
Two artists with 50% each for a payment that can be rounded to ten cents – no rounding$100.30artist 1: $50.15
artist 2: $50.15
artist 1: $50.15
artist 2: $50.15
Three artists with third splits can’t be split$100.00amounts of $33.33, $33.33 and $33.34 randomly assigned to group members artist 1: $33.34
artist 2: $33.33
artist 3:
Three artists with third splits can’t be split – more than a single cent difference$100.00amounts of $33.33, $33.34 and $33.34 randomly assigned to group membersartist 1: $33.34
artist 2: $33.34
artist 3:

Business Rules

  1. Royalties need to be paid in full on disbursement to artists.
  2. A single artist gets a whole payment.
  3. When payments can be split evenly to a group of artists (to the cent) they are split that way.
  4. When payments can’t be split evenly to a group of artists, the payments are split into an even split and the remaining cents are distributed to the members in whole cents.
  5. The first member in the group – based on earliest date/time added to group – gets the higher amount, followed by the second, third etc. based on earliest date/time added.
  6. Payments aren’t rounded to ten cents or five cent amounts, only whole cents


QuestionDecisionMade By
Do we want to round to five or ten cent distributions?No, we’ll always round to the centProduct Owner via Slack #
How do we distribute based on membership of group?We’ll use the date time added to the group (first gets most)Team during kick off meeting


I think trying to force the above information into gherkin (Given/When/Then) statements would make it less readable and provides no added benefit – whilst Given/When/Then encourages consistency sometimes you just need structured thought that is most relevant to your context. The above document isn’t a template – it varies for the problems we’re trying to solve.