Categories
Automated Testing

“Never Trust an Automated Test You Haven’t Seen Fail”

My number one philosophy when developing automated tests is:

“Never Trust an Automated Test You Haven’t Seen Fail”

I’ve found it can be easy to write a test that always passes, so forcing a test to fail as a starting point gives me confidence that it can fail.

I know of two ways to make a test fail:

  1. Change the implementation to fail and make sure the test fails at first – fixing the implementation fixes the test
  2. Change the assertion to make a test fail, make sure the actual result is correct and then update the assertion to make the test pass

I don’t really care what method I use to know a test fails, but I make sure I always make it fail. Never trust a test that hasn’t failed.

Categories
CSharp e2e Testing

Playwright in C# (.NET Core)

Whilst I was doing some reading of the Playwright docs I noticed they have C# bindings (as well as Python & Java, but not Ruby) – and since it’s been a couple of years since I’ve used C# I thought I’d take a look at how it works – especially considering .NET Core has support for Mac which makes working in C# .NET so much easier for me.

First I downloaded Visual Studio for Mac Community Edition which was pretty easy to install and this included the .NET Core framework which includes the dotnet command line tool.

One thing about .NET Core is there’s a lot more command line options to do things.

Installing Playwright?

dotnet tool install --global Microsoft.Playwright.CLI
playwright install

Creating a new NUnit Project?

dotnet new nunit -n PlaywrightNunitDemo

Adding Playwright, Building and Running Your Tests?

dotnet add package Microsoft.Playwright.NUnit
dotnet build
dotnet test

My tests end up looking like this:

using System.Threading.Tasks;
using Microsoft.Playwright.NUnit;
using NUnit.Framework;
using PlaywrightNunitDemo.lib;

namespace PlaywrightNunitDemo
{
    [Parallelizable(ParallelScope.Self)]
    public class Scenario02 : PageTest
    {
        [Test]
        public async Task CanCheckForErrors()
        { 
            string errors = await AppHelpers.VisitURLGetErrors(Page, "/error");
            Assert.AreEqual(": Purple Monkey Dishwasher Error", errors);
        }

        [Test]
        public async Task CanCheckForNoErrors()
        {
            string errors = await AppHelpers.VisitURLGetErrors(Page);
            Assert.AreEqual(string.Empty, errors);
        }
    }
}

and my reusable Playwright code can live in a class with static methods:

using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
using Microsoft.Playwright;

namespace PlaywrightNunitDemo.lib
{
    public class AppHelpers
    {
        public static async Task<IResponse> VisitURL(IPage page, string path = "/")
        { 
            var config = new ConfigurationBuilder().AddJsonFile("appsettings.json").Build();
            string url = config["BASE_URL"] + path;
            return await page.GotoAsync(url);  
        }

        public static async Task<string> VisitURLGetErrors(IPage page, string path = "/")
        {
            var errors = "";
            page.PageError += (_, exception) => { errors = errors + exception; };
            await VisitURL(page, path);
            return errors;
        }
    }
}

Summary

I quite enjoyed working in C# in previous roles. There’s something nice about typed languages which makes you confident when writing the code that things will do what you expect. Having .NET Core easily accessible on a Mac and being able to use Playwright makes C# even nicer.

Example Code

My example code is: https://github.com/alisterscott/PlaywrightNunitDemo

I’ll see if I can get this running on Linux in CircleCI soon.

Categories
Career

Technical coding interviews for testers?

About a year ago I was asked to apply to work at a startup here in Brisbane. I’ve never really been that keen on the idea of working for a startup as I prefer stability over exhilaration, I’m not looking for rapid promotion and I don’t really care for the speculative wealth that startup employee stock options provide – but I was curious about the job as it was working with a well-respected person I’d known for a while so I went ahead with an interview.

Little did I know that it would be a coding technical interview – where candidates – testers included – connect to a web session where a blank IDE is opened and you’re given a problem verbally and you need to write code to solve it on the spot.

I can’t remember the exact problem(s) but they were something like writing a way to reverse strings, do custom sorts and some fairly complex coding problems.

The reason I don’t remember is I totally blanked. As an anxious person with post traumatic stress disorder, staring at an empty JavaScript IDE whilst the interviewer watches every keystroke you type to solve the problem on the spot felt a lot like this to:

Coding Interview?

The interview was like fuel on my burning imposter syndrome fire and the next six months of my life was spent ruminating and questioning whether I could even do my job as a technical QA.

This was despite the facts I highlighted when applying for the job:

  • I have 16 public GitHub code repositories including 7 in JavaScipt, 4 in Ruby and 1 in C#
  • I’ve been writing this blog since 2006 and have shared numerous code samples across years of blogging.
  • I successfully completed a 3 month trial to work at Automattic/WordPress.com during which I built the foundation for the first ever automated e2e test suite to be successful at WordPress.com

I’d suggest to startups considering to do technical on-the-spot coding examinations whether these are absolutely necessary considering the candidate and knowing that people may not perform well in that situation.

Whilst I raised this feedback after the interview and I could have actually progressed along (despite flunking the coding exercises) I decided I didn’t want to work somewhere that would do that to potential employees. I’ve never had to sit next to someone in my working career and have them watch every keystroke I made writing code from scratch. Instead I’ve worked at great places that have measured me by my great outputs (thoughts/code/prose), and outcomes (software quality) without worrying about how I generate those.

I noticed the person who asked me to apply left the startup a few months after my interview anyway 🙃

Aside: perhaps the diagram above may also explain how some people, including myself, don’t like pair-programming as the main way of delivering code?

Categories
Playwright

Playwright’s “retry with video”

Another feature I love in Playwright Test is the retry with video option.

In your playwright.config.js file you specify:

module.exports = {
  use: {
    headless: true,
    screenshot: 'only-on-failure',
    video: 'retry-with-video'
  },
  retries: 1
}

The video option combined with the retry option means that if a test fails Playwright will re-run it whilst recording a video of that re-run.

If the test fails on the second attempt it will capture the video in the test results (alongside the screenshot if you config it to capture screenshots also).

It’s a webm video that is lightweight, easy to capture during CI, and easy to share:

Categories
Playwright

Ensuring consistent Playwright tests with “repeat each”

My workflow to ensure any new Playwright Test test I write is deterministic and repeatable:

  1. Mark the new test as .only temporarily locally during development
  2. Run npx playwright test --repeat-each 100 where 100 is the number of times you want to run it in parallel locally
  3. Make sure it passes 100% of the times

Here’s an example of how it looks (using 10 times for conciseness):

Playwright Test results showing consistent test runs

I’ve used this to discover a test failed about 5% of the times I ran it which was one of the only Playwright tests I’ve written so far that needed an explicit wait for element.

Categories
e2e Testing Playwright

Playwright Test Runner

I’d previously shared how to set up Playwright with Jest as a test runner which enabled us to do some cool things like:

  • Parallel test execution,
  • Automatic retries,
  • HTML reports (using Jest Stare), and
  • Screenshots when failing

It turns out that Playwright now supports all of the features without having to use Jest! It’s called the Playwright Test Runner.

I went ahead and created a fork of my existing code to show how to set this up with the Playwright Test Runner.

Specifying Tests

Fortunately the syntax to specify tests is almost identical to Jest, but you don’t need to worry about spawning browser contexts yourself 😎

This Jest test:

jest.retryTimes(1)

test('can use xpath selectors to find elements', async () => {
  global.page = await pages.spawnPage()
  await nav.visitHomePage(global.page)
  await home.clickScissors(global.page)
}, jestTimeoutMS)

becomes:

const { test } = require('@playwright/test')

test('can use xpath selectors to find elements', async ({ page }) => {
  await nav.visitHomePage(page)
  await home.clickScissors(page)
})

Running in Parallel

Like Jest, Playwright automatically runs your tests in parallel, sharing isolated browser pages within browsers, and spawning browsers across test files. The time taken was pretty much identical:

Jest

 PASS  scenarios/example4.spec.js
 PASS  scenarios/example3.spec.js
 PASS  scenarios/example5.spec.js
 PASS  scenarios/api.spec.js
 PASS  scenarios/example2.spec.js
 PASS  scenarios/example.spec.js

Test Suites: 6 passed, 6 total
Tests:       10 passed, 10 total
Snapshots:   2 passed, 2 total
Time:        5.514 s

Playwright Test

Running 10 tests using 6 workers

  ✓ scenarios/api.spec.js:5:1 › [chromium] can GET a REST API and check response using approval style (590ms)
  ✓ scenarios/example.spec.js:4:1 › [chromium] can wait for an element to appear (3s)
  ✓ scenarios/example2.spec.js:4:1 › [chromium] can handle alerts (4s)
  ✓ scenarios/api.spec.js:12:1 › [chromium] can GET a REST API and check response using assertion style (587ms)
  ✓ scenarios/example3.spec.js:5:1 › [chromium] can check for errors when there should be none (1s)
  ✓ scenarios/example4.spec.js:5:1 › [chromium] can check for errors when there are present (1s)
  ✓ scenarios/api.spec.js:25:1 › [chromium] can POST a REST API and check response using approval style (536ms)
  ✓ scenarios/example5.spec.js:6:1 › [chromium] can use xpath selectors to find elements (988ms)
  ✓ scenarios/api.spec.js:32:1 › [chromium] can POST a REST API and check response using assertion style (545ms)
  ✓ scenarios/example.spec.js:9:1 › [chromium] can use an element that appears after on page load (220ms)


  10 passed (5s)

Automatic Retries

Jest supports automatic retries in the test itself:

jest.retryTimes(1)

In Playwright Test you can configure it globally or per run:

// playwright.config.js
module.exports = {
  use: {
    retries: 2
  }
}
npx playwright test --retries=3

HTML Reports

I couldn’t see HTML report output for Playwright Test – but there’s heaps of others like JSON and Junit (for CI).

Screenshots

Screenshots are easily configured for Playwright Test: for each test never, always or on failure (my preference).

module.exports = {
  use: {
    screenshot: 'only-on-failure'
  }
}

But wait there’s more

The Playwright Test Runner also has these very cool features:

Visual comparisons (visdifs)

test('can wait for an element to appear', async ({ page }) => {
  await nav.visitHomePage(page)
  await page.waitForSelector('#elementappearschild', { visible: true, timeout: 5000 })
  expect(await page.screenshot()).toMatchSnapshot('element-appears.png')
})

At any point in your tests you can take a screenshot and store this to visually compare with future runs with fine grained tweaking of matches. The best part about this, in my opinion, is these visuals are stored alongside your tests (not in some third party system) so you know exactly what you’re expecting to see 😎

The one downside is the screenshots are platform specific so when I checked in those generated on my Mac then CI failed as it was running on Linux and didn’t have the baseline files to compare to. I just downloaded the captured files from CircleCI and stored them as a baseline.

And it also supports other comparisons – similar to Jest snapshots I have previously demonstrated. You just need to stringify them first:

test('can GET a REST API and check response using approval style', async () => {
  const request = supertest('https://my-json-server.typicode.com/webdriverjsdemo/webdriverjsdemo.github.io')
  const response = await request.get('/posts')
  expect(response.status).toBe(200)
  expect(JSON.stringify(response.body)).toMatchSnapshot('posts.txt')
})

Closing Thoughts

I was pretty blown away by the Playwright Test runner. It offers everything Jest provides with less code I had to write, plus it has in built visual comparison tools that can also be extended to do API approval snapshot testing. Playwright is well becoming my e2e test automation tool of choice.

Show me the Code

Of course: code is here: https://github.com/alisterscott/playwright-test-demo

Passing CI here: https://app.circleci.com/pipelines/github/alisterscott/playwright-test-demo/7/workflows/fe13f579-1f53-4a01-834e-6a4e7388e22c

Categories
Automated Testing e2e Testing

Generating data for e2e automated tests

I generally prefer creating e2e automated tests that generate their own data so that the test is repeatable, more deterministic and less dependent on external data and factors which can change.


When writing automated e2e tests that generate data I’ve found there are two common approaches:

  1. Generate static data: this data remains the same between test runs2.
  2. Generate random data: this data can change each test run

An example of static data for a test would be:

const ContactModel = function () {
  return {
    firstName: 'Sammy',
    lastName: 'Snake',
    phoneType: 'MOBILE',
    phoneNumber: '0422888444',
    emailType: 'PERSONAL',
    email: 'sammy.snake@hotmail.com',
    type: 'INDIVIDUAL'
  }
}

and the same example using randomised data:

function pick (list) {
  return list[Math.floor(Math.random() * list.length)]
}

const ContactModel = function () {
  const firstName = pick(['Aaron', 'Becca', 'Charlie', 'Donna', 'Eckbert', 'Fred', 'Graham', 'Holly', 'Ignatius', 'Josephine'])
  const lastName = pick(['Aardvark', 'Bear', 'Cat', 'Dog', 'Eagle', 'Fox', 'Gorilla', 'Horse', 'Iguana', 'J'])
  return {
    firstName: firstName,
    lastName: lastName,
    phoneType: 'MOBILE',
    phoneNumber: `+${Math.round(Math.random() * 9999999999999)}`,
    emailType: 'PERSONAL',
    email: `${firstName.toLowerCase()}.${lastName.toLowerCase()}@${pick(['hotmail.com', 'yahoo.com', 'gmail.com', 'aol.com'])}`,
    type: 'INDIVIDUAL'
  }
}

Generally speaking I will choose static data when writing automated e2e tests as it’s more repeatable and consistent, however I still commonly see random generated data being used in automated e2e automated tests.

I think the reason is that varying data is a good exploratory testing technique to find bugs, however automated e2e tests aren’t about exploring functionality but rather ensuring the functionality continues to work as expected (regression testing).

They don’t have to be completely separate concepts though, my checked-in automated e2e test can use static data so it’s consistent and repeatable, but I could modify the same automated test to use randomly generated input which I could specifically run looped for a large number of iterations to assist me with my exploratory testing and try to find bugs around input.

I’ve found that great testers uses automated tests as a technique to assist them with their exploratory testing.


What’s your preferred approach? Do you generate static or randomised data or use existing data for your automated e2e tests?

Categories
Remote Work

Implementing a busy light for my office desk

I work full time in the office. Whilst more and more people are coming back to the office, there are still people choosing to work remotely so we still conduct our meetings via Google Meet – even though I am in the office I don’t use meeting rooms: I still dial in from my desk.

Since I use simple Apple earpods on calls it’s not always obvious I am on a call (or just listening to music) and I noticed people would come to discuss something with me whilst I’m on a call which is disruptive so I decided to come up with something that indicates to people I’m busy on a call (at my desk).

Iteration One

My first iteration I wanted a solution that was cheap and didn’t take long to set up – as I wanted to see whether this would reduce my interruptions whilst on calls without a huge investment of time or money.

Solution: a simple battery powered light box above my monitor which indicates I’m on a call with a manual switch on the side to turn it on.

Cost: I found one at a thrift store for 50 cents and wrote “ON AIR” on it with a marker – it took me less than 10 mins total to set up and install. Low cost.

My first “busy” light

Benefits: Low cost, stands out, does its job

Downsides: Have to remember to switch it on/off, not multi-directional, need to replace battery

Iteration Two

Whilst the light in iteration one served its purpose in that it was quick and cheap to implement – and showed it could be effective in reducing interruptions, I found having to remember to turn it on and off meant sometimes it wouldn’t be on when I was in a meeting, or off when I’d finished one.

I started to think of how I could connect this to my meeting schedule to automatically update.

I would firstly need a new light – the 50 cent lightbox had no way of controlling it remotely. Secondly, I would need a datasource to control it – either Google Calendar or Google Meet.

Smart Light

I found a smart light at Kmart for AUD$25 (~$20USD) which seemed like it would be suitable.

The key features were:

  1. USB powered so I could use the USB ports on my desk to power it (no batteries)
  2. Wifi controlled so I could send signals to it to control it
  3. Coloured so I could change the colour displayed to signal I’m busy
  4. Circular so it can be seen in any direction from my desk in the office

Datasource

I was thinking of using Google Meet or Google Calendar to set the busy light. The benefits of Google Meet would be if I was on an ad-hoc call, or finished a call early, the light would reflect that. The downsides to Google Meet was I couldn’t find a way to easily access the Google Meet status.

Google Calendar is an easier approach in that IFTTT supports Google Calendar events which I can wire up to my smart light.

Solution

I set up my smart light using the TUYA app that it came with. This app allows you to control the light using Google Home, Alexa and/or Samsung SmartThings.

I investigated how to connect my Google Calendar to a smart light and came across IFTTT which currently only supports SmartThings out of the list of home automation services my smart light supports – so I added my light to SmartThings, set it to be coloured red, and then set up two “applets” in IFTTT to control my light.

One to turn it on:

And another to turn it off:

IFTTT only provides and ON/OFF switch for SmartThings – which means I set my light to be permanently red and it shows as busy when lit, and not busy it turns off.

I found a way to get it to turn Green by setting up “simulated switches” and “home automation” in SmartThings but the complexity of this outweighed the benefits of having a green light IMO.

My new “smart” busy light

Benefits: Smart, don’t need to remember to switch on/off, multi-directional, don’t need to replace batteries

Downsides: More expensive than previous light, more complicated set up

It took me about 1 hour to get it all working which is much longer than previous iteration but I’m hoping that because it automatically updates this will mean the light is more useful than previously.

How do you manage interruptions in an office environment?

Categories
Career

Software Testing Jobs June 2021

Even though I’m not looking for a new job, I like to browse and read job advertisements as I’ve found it’s a good way to take control of your career as it gives you insights into what future employers are looking for in currently advertised roles.

In this light, I recently did some keyword analysis of the 20 most recent software testing jobs advertised in Brisbane, Australia (the job market seems 🔥 so these were all advertised in the last week or so) attempting to find some insights into what is in demand and what’s happening in the world of software testing (in Brisbane, Australia).

Type of Jobs

Most of the 20 most recent jobs were permanent full time roles. As a part time employee myself I found it interesting none offered or mentioned the flexibility of part time hours.

Salary Advertised

Only 4 of the 20 roles included any information about salary. This saddens me as it pushes the burden of salary negotiation back onto the candidate (I am yet to meet an employer who wants to pay more than bare necessary!)

The four roles that indicated salary were:

  1. $80,000 – $99,999 AUD (Software Quality Assurance Specialist)
  2. $95,000 – $110,000 AUD (Senior Test Analyst)
  3. $80,000 – $109,999 AUD (QA Engineer)
  4. “Up to” $125,000 AUD (Senior QA Engineer)

Location of Work

Despite 2020/2021 being touted as the years the world went fully remote, interestingly a majority of the roles indicated working in the office full time, 6 of 20 offering a “hybrid” model typically being 1-2 days WFH per week (often described as a “work life balance” or “flexibility” perk), and only 1 of 20 was fully remote (no office or expectation to work in the office). What happened?

Job Category

There was a good split between roles advertised as test management, test engineering, and test analysis. However, almost all roles required test automation skills (see next section).

Test Automation Skills Required

This was probably the most and least surprising result to me. Least surprising as there’s been a big shift to technical testing roles in the industry which I’ve been talking about for some time. And most surprising in that even the roles categorized as “test management” and “test analysis” predominantly required test automation skills. Of the four that didn’t mention the requirement for test automation skills three were test analysis and only one was test management.

An example job description:

“As a Test Manager, you will be a hands-on leader with a broad skill-set and knowledge in test methodologies and automation development. “

Qualifications/Certifications

  • 3 of 20 jobs required tertiary IT qualifications
  • 1 of 20 jobs required ISTQB certifications

Automated Testing Tools

  • 6 of 20 jobs mentioned Selenium
  • 1 of 20 jobs mentioned Cypress 😢
  • 1 of 20 mentioned TOSCA 😢😢
  • 0 of 20 mentioned Playwright or Puppeteer 😢😢😢

Buzzword Bingo

  • 9 of the 20 jobs mentioned “flexibility” in their role
  • 11 of 20 mentioned “agile”
  • 7 of 20 mentioned either “high-performing, fast-paced, dynamic or fast-moving” teams and culture

Favourite Description

My favourite description of culture was:

“<employer name> recognises the critically important role of testers in the delivery of top quality applications. Our testers have to be sharp to hunt out issues and work with developers and business stakeholders to deliver truly awesome applications into production, but also mentoring others in the development of their own QA mindset. “

followed by

“We recognise that life is easier when cash flow is easy. That’s why we’ll pay you weekly – not fortnightly or monthly like other companies.”

Least Favourite Descriptions

(in no particular order – verbatim with emphasis added)

  • “You will feel happier & healthier for working at <company name>!
  • “As a Test Manager, you will be a hands-on leader with a broad skill-set and knowledge in test methodologies and automation development.”
  • “Salary and super flexible based on experience
  • Supervise the delivery of high quality production optimised code. “
  • “We are a friendly bunch here at XXX, so we expect you to be a good collaborator, strong communicator and willing to roll up your sleeves and focus on reaching our common goals.”
  • “We are looking for an experienced Automation specialist to join this organisation and help in ensuring thigh quality delivery of their software.” 🍗
  • “You will be working ac across multiple projects within a high paced environment, contributing to team success and having plenty of flexibility within a great company.” 🆙
  • “You’re not afraid of hard work. Curly challenges. Sometimes inconvenient hours.” 😬
  • “During your interview, you’ll be able to tell us the capital city of Bolivia” 🤦‍♂️

Closing Thoughts

I enjoyed this exercise more than I thought. The things that most surprised me was how few jobs were remote in 2021, and also how many roles required test automation skills, particularly jobs at the test management level: is every test manager role “hands on” nowadays?

Categories
Life

Bye Bye Bear 🐻

Science is organized knowledge; wisdom is organized life.

—Immanuel Kant.

I’m a big fan of taking notes to organise my work and life, and I’ve been using Bear as a notetaking app over the last few years* (coming across from Evernote). I like it for its simplicity but it seems like this simplicity has become its Achilles’ heel.

I started using a separate Apple ID for my work and the issue with Bear is it’s macOS and iOS only and it uses iCloud sync so I can’t use it on my work laptop which uses a separate Apple ID – there is a plan to have a web version of Bear at some point but the intention has been talked about for years but nothing has happened.

At some point you give up on what you love and find something that works for you. I’ve been using Zoho Notebook for a few weeks now and I like it. It does tables (which bear doesn’t support) and I like how it arranges notes (or cards as it calls it) into separate notebooks reflecting real life. There’s clients for macOS and iOS (which don’t use iCloud for syncing) and also a web version for use anywhere anytime. It seems to suit my purposes well and it’s free (supported by development of other Zoho products).

My notebooks look like this:

What do you use to organise your work/life/notes?


*We also use a Simplenote account logged into multiple iDevices to share text snippets/to do lists for our household.