Categories
Automated Acceptance Testing Automated Testing Automattic Mocha WebDriver

Feature Toggles for Automated e2e Tests

Feature toggles aren’t just for production code. Feature toggles are also a powerful technique to change the behaviour of your automated end-to-end tests without changing code.

Categories
Automated Testing JavaScript WebDriver

Adding your own WebDriverJs helper methods

Whilst I find the WebDriver API useful, I also find it lacking in certain methods that I wish to do repeatedly throughout my tests.

Categories
Automated Testing Mocha WebDriver

Using WebDriver to automatically check for JavaScript errors on every page (2016 edition)

Back in 2012 I wrote about how to use WebDriver to automatically check for JavaScript console errors on every page. The solution I proposed involved adding some common JavaScript to every page in your app and then checking that errors object when using WebDriver page object classes.

Fortunately since then the WebDriver project now supports checking for these errors without making any changes to your app, so if this has been stopping you doing this you can now do it quite easily.

Categories
Ask Me Anything Automated Testing CSharp SpecDriver Specflow WebDriver

AMA: C# WebDriver Questions

omayer asks…

how to use test data from xlsx in c# webdriver, thank you in advance.

My response…

I haven’t done this as I’ve managed data/scenarios using SpecFlow, but there’s this post that should help you.

omayer also asks…

Handling popup Windows in C# WebDriver – Closing the popup window , finding elements on popup window, locating element inside popup window and frameset

My response…

Stackoverflow is a good resource for these types of questions; there’s some examples on how to do this here.

pallavi asks…

Which tool can we use for report generation in selenium with C#

My response…

This very much depends on which framework you are using to manage your tests. I have used SpecFlow which allows a formatted HTML report of test results.

pallavi also asks…

Can we use selenium with coded ui in visual studio? If I use with coded ui then what template it will follow

My response…

My experience with CodedUI tests in C# has only been negative. When I looked at it, the approach Microsoft used was to generate a huge number of lines of CodedUI test code which I couldn’t understand or make into reusable objects. I much prefer Selenium/WebDriver for any web based tests, and to use White to test any non-web based UIs. I am not familiar with CodedUI templates since it has been a long time since I looked at it.

 

Categories
Ask Me Anything Automated Testing Mocha WebDriver

AMA: reusing the same browser for different automated tests

Ankitha asks…

Let’s say we have two tests files inside specs folder
1) test1.js — has 3 tests
2)test2.js– has 3 tests

When I do npm test, a seperate browser is opened for running test1.js and second browser is opened up for running test2.js.
Whereas I want only one browser to be opened.
Like perform test1.js tests close the browser, For test2.js is now open browser again run test2.js. I would like to see tests running sequentially. How can this be achieved?

My response…

You may have seen that we recently open-sourced our automated e2e tests for WordPress.com.

One the huge number of benefits this brings if you can see how we do things 🙂

If you have a look at driverManager.startBrowser, which is what we call from a before hook in each test, you’ll see how we reuse the browser across different tests.

There’s one additional thing you need to do which is close the browser down at the very end. We use a separate file after.js to do this, which we pass to Mocha when running any test.

Categories
Ask Me Anything Automated Acceptance Testing Automated Testing Watir Watir-WebDriver WebDriver

AMA: JS vs Ruby

Butch Mayhew asks…

I have noticed you blogging more about JS frameworks. How do these compare to Watir/Ruby? Would you recommend one over the other?

My response…

I had a discussion recently with Chuck van der Linden about this same topic as he has a lot of experience with Watir and is now looking at JavaScript testing frameworks like I have done.

Some Background

WordPress.com built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’t make long term sense for WordPress.com where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

Whilst I see merit in both views: I still think having your automated acceptance tests in the same language as your application leads to better maintainability and adoptability.

I still think writing automated acceptance tests in Ruby is much cleaner and nicer than JavaScript Node tests, particularly as Ruby allows meta-programming which means page objects can be implemented really neatly.

The JavaScript/NodeJS landscape is still very immature where people are using various tools/frameworks/compilers and certain patterns or de facto standards haven’t really emerged yet. The whole ES6/ES2015/ES2016 thing is very confusing to newcomers like me, especially on NodeJS where some ES6+ features are supported, but others require something like Babel to compile your code.

But generally with the direction ES is going, writing page objects as classes is much nicer than using functions for everything as in ES5.

Whilst there’s nothing I have found that is better (or even as good) in JavaScript/Mocha/WebDriverJS than Ruby/RSpec/Watir-WebDriver, I still think it’s a better long term decision for WordPress.com to use the JavaScript NodeJS stack for our e2e tests.

Categories
Automated Acceptance Testing Automated Testing Selenium WebDriver

Checking an image is actually visible in WebDriverJs

I recently discovered a gap in one of my e2e automated tests where I was checking the existence of an uploaded image in the DOM, but not that the image was actually displayed.

driver.isElementPresent( By.css( `img[alt='upload.jpg']` ) ).then( function( present ) {
  assert.equal( present, true, 'Image not displayed' );
} );

If the DOM has a reference to the image, but it isn’t actually rendered this test will pass. This isn’t ideal.

I remembered my post about how to check that an image is actually rendered using WebDriver in C# and so I used the same JavaScript script which WebDriverJs sends to the driver:

driver.findElement( By.css( `img[alt='upload.jpg']` ) ).then( function( element ) {
  driver.executeScript( 'return (typeof arguments[0].naturalWidth!=\"undefined\" && arguments[0].naturalWidth>0)', element ).then( function( present ) {
    assert.equal( present, true, 'Image not displayed' );
  } );
} );

This works a treat. I’ve moved it into a helper function so I can use this anywhere without repeating it also.

Categories
Automated Testing JavaScript Mocha Selenium WebDriver

Testing end-to-end with Mocha

As part of my excellent Excellence Wrangler role at Automattic, one of my key tasks has been establishing some end-to-end tests for WordPress.com using Mocha with WebDriverJs. Our testing pyramid doesn’t look much like a pyramid:

wordpress.com test pyramid.png

We’ve got lots of React unit tests at the bottom: these are to speed development.

We’re intentionally missing a middle: the REST API we consume has its own unit tests, we don’t need integration tests for it. We don’t have detailed full stack acceptance tests of our UI: these are too slow and brittle.

We have a handful of e2e flow tests at the top, these are to protect the user experience, we run these on every deployment and frequently in production. These can be brittle on such a fast moving code base, but we limit their number (depth) so they still give us good confidence everything is working well together but limiting our overhead.

So what do these end-to-end tests look like?

I hadn’t used Mocha before and I was used to writing end-to-end tests in Gherkin format in tools like Cucumber and Specflow so I initially began writing end-to-end tests that looked like this:

test.describe('WordPress.com Sign Up', function() {
  test.beforeEach(function() {
    driver.manage().deleteAllCookies();
  });

  test.it('Can Create A Free Blog', function() {
    var signupFlow = new SignUpFlow( driver, 'desktop' );
    signupFlow.createFreeBlog( 'en' );
  });

  test.it('Can Create A New Site With a Paid Domain Upgrade', function() {
    var signupFlow = new SignUpFlow( driver, 'desktop' );
    signupFlow.CreateSiteWithDomainPaidByCreditCard( 'en' );
  });
});

I was pushing the code down into flow classes which I have used before, but the issue with this was the output I was getting from Mocha wasn’t very rich:

WordPress.com Sign Up
      ✓ Can Create A Free Blog
      ✓ Can Create A New Site With a Paid Domain Upgrade

I then realized by looking at an end-to-end test written by another developer that you can nest describe and it statements to give you much more expressive end-to-end tests.

test.describe( `Sign Up (${screenSize})`, function() {

  test.describe( 'Free Site:', function() {
    test.before( 'Delete Cookies and Local Storage', function() {
      driverManager.clearCookiesAndDeleteLocalStorage( driver );
    } );

    test.describe( 'Sign up for a free site', function() {

      test.describe( 'Step One: Themes', function() {
        test.before( 'Can see the choose a theme page as the starting page', function() {
          this.chooseAThemePage = new ChooseAThemePage( driver, { visit: true } );
          return this.chooseAThemePage.displayed().then( ( displayed ) => {
            assert.equal( displayed, true, 'The choose a theme start page is not displayed' );
          } );
        } );

        test.it( 'Can select the first theme', function() {
          return this.chooseAThemePage.selectFirstTheme();
        } );
      } );

      test.describe( 'Step Two: Domains', function() {
        test.before( 'Can then see the domains page ', function() {
          this.findADomainComponent = new FindADomainComponent( driver );
          return this.findADomainComponent.displayed().then( ( displayed ) => {
            assert.equal( displayed, true, 'The choose a domain page is not displayed' );
          } );
        } );

        test.it( 'Can search for a blog name', function() {
          return this.findADomainComponent.searchForBlogNameAndWaitForResults( blogName );
        } );

        test.it( 'Can see a free WordPress.com blog address in results ', function() {
          return this.findADomainComponent.freeBlogAddress().then( ( actualAddress ) => {
            assert.equal( actualAddress, expectedBlogAddress, 'The expected free address is not shown' )
          } );
        } );

        test.it( 'Can select the free address', function() {
          return this.findADomainComponent.selectFreeAddress();
        } );
      } );

This gives us rich feedback:

Sign Up (desktop)
    Free Site:
      Sign up for a free site
        Step One: Themes
          ✓ Can see the choose a theme page as the starting page
          ✓ Can select the first theme
        Step Two: Domains
          ✓ Can then see the domains page
          ✓ Can search for a blog name
          ✓ Can see a free WordPress.com blog address in results
          ✓ Can select the free address

The mistake I had made which I didn’t realize was not creating enough nesting, instead of having Step One, Step Two etc. next to one another, they should be nested within each other. This is because if they’re next to one another, Mocha will run the Step Two, Step Three etc. even if Step One has failed, which is not what we want in an end-to-end scenario where each step is dependent on the previous one.

So, it now looks something like this:

test.describe( 'Free Site:', function() {
    test.before( 'Delete Cookies and Local Storage', function() {
      driverManager.clearCookiesAndDeleteLocalStorage( driver );
    } );

    test.describe( 'Sign up for a free site', function() {
      test.describe( 'Step One: Themes', function() {
        test.before( 'Can see the choose a theme page as the starting page', function() {
          this.chooseAThemePage = new ChooseAThemePage( driver, { visit: true } );
          return this.chooseAThemePage.displayed().then( ( displayed ) => {
            assert.equal( displayed, true, 'The choose a theme start page is not displayed' );
          } );
        } );

        test.it( 'Can select the first theme', function() {
          return this.chooseAThemePage.selectFirstTheme();
        } );

        test.describe( 'Step Two: Domains', function() {
          test.before( 'Can then see the domains page ', function() {
            this.findADomainComponent = new FindADomainComponent( driver );
            return this.findADomainComponent.displayed().then( ( displayed ) => {
              assert.equal( displayed, true, 'The choose a domain page is not displayed' );
            } );
          } );

          test.it( 'Can search for a blog name', function() {
            return this.findADomainComponent.searchForBlogNameAndWaitForResults( blogName );
          } );

          test.it( 'Can see a free WordPress.com blog address in results ', function() {
            return this.findADomainComponent.freeBlogAddress().then( ( actualAddress ) => {
              assert.equal( actualAddress, expectedBlogAddress, 'The expected free address is not shown' )
            } );
          } );

          test.it( 'Can select the free address', function() {
            return this.findADomainComponent.selectFreeAddress();
          } );

          test.describe( 'Step Three: Plans', function() {

which means the output is slightly different but still very useful:

Sign Up (mobile)
  Free Site:
    Sign up for a free site
      Step One: Themes
        ✓ Can select the first theme
        Step Two: Domains
          ✓ Can search for a blog name
          ✓ Can see a free WordPress.com blog address in results
          ✓ Can select the free address
          Step Three: Plans
            ✓ Can select the free plan

These tests are much better written this way. The only issue I am left facing with Mocha is when a before hook fails (such as logging in) the generic afterEach hook we have to take screenshots is not triggered (this is only triggered when an it block is run.

Categories
Automated Acceptance Testing Automated Testing Automattic WebDriver Wordpress

Running Automated Tests with A/B Testing

Like a lot of modern, data driven sites, WordPress.com uses A/B testing extensively to introduce new features. These tests may be as simple as a label change or as complex as changing the entire sign up flow, for example by offering a free trial.

Since I have been working on a set of automated end-to-end tests for WordPress.com, I have found A/B testing to be problematic for automated testing on this very fast moving codebase, namely:

  1. Automated tests need to be deterministic: having a randomised experiment as an A/B test means the first test run may get an entirely different sign up flow than a second test run which is very hard to automate; and
  2. Automated tests need to know which experiments are running otherwise they may encounter unexpected behaviour randomly.

What we need is two methods to deal with A/B tests when running automated tests:

  1. We need to be able to see which A/B tests are active and compare this to a known list of expected A/B tests – so that we don’t suddenly encounter some unexpected/random behaviour for some of our test runs
  2. We need to be able to set the desired behaviour to the control group so that are our tests are deterministic.

Different sites conduct A/B testing using different tools and approaches, WordPress.com uses HTML5 local storage to set which A/B tests are active and which group the user belongs to.

Luckily it’s easy to read and update local storage using WebDriver and JavaScript. This means our approach is to:

  1. Each time a page object is initialised, there is a call on the base page model that checks the A/B tests that are active using something like return window.localStorage.ABTests; and then compares this to the known list of A/B tests which are checked in as a config item. This fails the test if there’s a new A/B test introduced that isn’t in the list of known tests. This is better than not knowing about the A/B test and failing based upon some non-deterministic behaviour.
  2. When a new A/B test is introduced and we wish to ensure our automated tests always use the control group, we can set this using a similar method window.localStorage.setItem('ABTests','{"flow":"default"}'); and refresh the page.

Ideally it would be good to know and plan every A/B test for our automated e2e tests, but since this isn’t possible, checking against known A/B tests and ensuring control groups are set means our automated tests are at least more consistent and deterministic, and fail a lot faster and more consistently when a new A/B test has been introduced.

How do you deal with non-determinism with A/B tests?

Categories
Automated Testing JavaScript Mocha WebDriver

WebDriverJs & Mocha in ES2015

A friend of mine, Mark Ryall, recently created a fork of my WebDriverJs and Mocha example project and updated it to use ES2015. I’ve made some further changes and merged these in, and would like to share these.

Background

JavaScript is an implementation of the ECMAScript scripting language standard.

The latest version of ECMAScript, known as ES2015, ES6, ES6 Harmony, ECMAScript 2015, or ECMAScript 6, has some neat features which are handy to use for our WebDriverJs & Mocha tests I have previously written about.

It seems that there will be yearly releases of the ECMAScript standard from 2015 onwards, and the most common way to refer to these will be as ES2015, ES2016 etc.

Enabling ES2015 Support for our Example Tests

There is a node tool called Babel which is a JavaScript compiler that allows you to use new ECMAScript features and compile these into JavaScript. This requires two node packages which we add to our package.json file:

"babel-core": "^6.3.13",
"babel-preset-es2015": "^6.3.13"

This means we have a babel compiler and a babel library to transform ES2015.

The second thing we need to do is add a plugin to actually tell babel to transform ES2015.

We add a .babelrc file to our project with the following content:

{
"presets": ["es2015"]
}

Running our Specs using Babel

Once we’ve done this, we can use Mocha and WebDriverJs with ES2015. Instead of calling mocha specs we now need to use babel like:
mocha --compilers js:babel-core/register specs.

This isn’t as nice, so we can update our package.json file so our test command is set to the longer babel command, and we just need to call npm test to run our Mocha specs.

Updating our code to use ES2015

The great thing about ES2015 is it is backwards compatible, so we don’t need to update all our code at once, we can made gradual changes to use new features available to us.

Mark made changes to the spec and the page object to use some of the pretty ES2015 features:

Import Statements

This:

var assert = require('assert');
var webdriver = require('selenium-webdriver');
var test = require('selenium-webdriver/testing');
var config = require('config');
var RalphSaysPage = require('../lib/ralph-says-page.js');

Becomes:

import assert from 'assert';
import webdriver from 'selenium-webdriver';
import test from 'selenium-webdriver/testing';
import config from 'config';
import { ralphSays } from '../lib/pages.js';

Using let instead of var

let is block scoped so this is better to use.

This:

var driver;

Becomes:

let driver = null;

Arrow functions

The arrow functions make the clean up hooks simpler to read:

From this:

test.afterEach(function() {
  driver.manage().deleteAllCookies();
});

To this:

test.afterEach(() => driver.manage().deleteAllCookies());

Summary

Moving to use ES2015 wasn’t as daunting as I initially thought as once you add support for it using Babel, you can gradually start using the new features.