Categories
Ask Me Anything Automated Testing Automattic

AMA: Separate Repository for e2e Tests?

Liam asks…

“I did enjoy reading the article about e2e test on wordpress. I noted that e2e test are in a separate repo.

My question will be what is the workflow to make sure new changes does not break the e2e test on pull request ?

For example, if a developer work on some changes, then they need to change the e2e test first and make sure everything pass, however the environment on the pull request might not be stable, developer can overwrite each other changes”

My response…

Thanks for your question Liam.

We have reasons for and benefits in having the WordPress.com e2e tests in a separate repository:

  1. The e2e tests test the entire WordPress.com experience so these test things that happen in different repositories (for example our Calypso user interface or services/API) and having them in the user interface repository isn’t really representative of what the breadth of their scope;
  2. Making changes to the e2e tests are easier in a separate repository since we don’t have to deploy e2e PRs that don’t contain functional changes (we deploy every merge to our master branch immediately dozens of times per day)

The obvious downsides are:

  1. How do we make sure e2e tests know about incoming AB tests?
  2. How do we couple new changes to updates in the e2e test repository?

For incoming AB tests we make sure that our e2e tests know about the change by ensuring we create a matching PR in our e2e tests that override our AB tests during our test runs.

If someone updates the AB tests in Calypso they’re politely reminded to update the e2e tests:

Screen Shot 2018-09-19 at 3.31.47 pm
example prompt

For making sure e2e tests are up to date we automatically run two (of about 40 total) of the most critical e2e tests (in three browsers) when a PR is ready to be reviewed. These can fail and indicate a change is necessary to the e2e tests (or something is broken!)

There’s also a label we can add to any PR that runs the entire set of e2e tests against a PR running live and reports the result back to that PR:

Screen Shot 2018-09-19 at 3.38.29 pm
e2e Test Results against a Calypso PR

If changes are required to the e2e tests someone can create an e2e PR with the exact same branch name which will be used to run against the feature changes before they are merged. This means PRs can be coupled and tested together but merged separately.

To answer the second part of your question I understand it to be about conflicting changes? One of our key philosophies for work is “merge early and merge often” so we make sure that PRs are short-lived and merged quickly to minimize the chance of conflicts. These still do happen occasionally, we just deal with them as they come up.

Whilst there’s been some downsides to the separate repositories all-in-all the benefits continue to outweigh the downsides but we constantly assess and at any point in time we can easily merge them if need be.

Categories
Automattic Conferences

Testbash Australia 2018

I only speak at one conference a year and this year that conference will be the first ever Australian Testbash in Sydney on October 19, 2018:

TestBash_Australia_2018_Adverts_DOJO_EVENT_BANNER.png

My talk:

At WordPress.com we constantly deliver changes to our millions of customers – in the past month alone we released our React web client 563 times; over 18 releases every day. We don’t conduct any manual regression testing, and we only employ 5 software testers in a company of ~680 people with ~230 developers across . So, how do we ensure our customers get a consistently great user experience with all this rapid change?

Our automated end-to-end (e2e) tests give us confidence to release small frequent changes constantly to all our different applications and platforms knowing that our customers can continue to use our products to achieve their desires.

Alister will share how these automated tests are written and work, some of the benefits and challenges we’ve seen in implementing these tests, and what we have planned for future iterations.

Takeaways: How to effectively use automated end-to-end testing to ensure a consistent user experience in high frequency delivery environments

Grab a ticket before they sell out #

Categories
Automated Acceptance Testing Automated Testing Automattic

→ How Canaries Help Us Merge Good Pull Requests

I recently published an article on the WordPress.com Developer’s Blog about how we run automated canary tests on pull requests to give us confidence to release frequent changes without breaking things. Feel free to check it out.

Categories
Ask Me Anything Automated Testing Automattic WordPress.com

A brief update & AMA II

It’s been a while since I wrote something on this blog as you could say my life has been a bit complicated.

Categories
Automattic Conferences

The future of testing is distributed

This are my slides during my presentation on distributed testing at the ANZTB Test 2017 Conference in Wellington, New Zealand last Friday 5th May 2017.

Alister Scott - The Future of Testing is Distributed FINAL(1) 01

Categories
Automattic Wordpress WordPress.com

Finding a home for your content

I’ve been blogging on WordPress.com for over nine years and I’m very happy with my decision to become both a WordPress.com user back then, and a WordPress.com (Automattic) employee in 2015.

Categories
Automated Acceptance Testing Automated Testing Automattic Mocha WebDriver

Feature Toggles for Automated e2e Tests

Feature toggles aren’t just for production code. Feature toggles are also a powerful technique to change the behaviour of your automated end-to-end tests without changing code.

Categories
Automattic

How we Communicate at Automattic

When I talk to people about my job at Automattic, most of them can’t really comprehend how it can possibly work since it’s so different to what they see as a ‘normal job’.

Categories
Automattic WordPress.com

The Importance of Doing Customer Support

One of the many, many things I love about working at Automattic is the being part of a company that sees huge value in everyone spending time doing customer support, and acts on it:

“When you join full-time, you’ll do customer support for WordPress.com for your first three weeks and spend a week in support annually, for evermore, regardless of your position. We believe an early and ongoing connection with the people who use our products is irreplaceable.”

Automattic’s Work With Us Page

Every single bit of that statement is true: I did three full weeks when I joined last September, and I’ve already done an additional week this year.

Categories
Automated Testing Automattic WordPress.com

WordPress.com e2e Tests

WordPress.com Automated Testing Pyramid.PNG

If you would like to read more about our e2e tests for WordPress.com you can do so over on the Automattic Engineering blog where I have recently published an article for the WordPress developer community.