Automated Testing Career Software Testing

Creating a skills-matrix for t-shaped testers

I believe the expression “jack of all trades, master of none” is a misnomer, as I’ve mentioned previously. Being good at two or more complimentary skills is better than being excellent at just one, in my opinion.

But what about being excellent at one skill, and still being good at two or more? Why can’t we be both?

Jason Yip describes a T-shaped person and the benefits that having t-shaped people on teams brings:

A T-shaped person is capable in many things and expert in, at least, one.
As opposed to an expert in one thing (I-shaped) or a “jack of all trades, master of none” generalist, a “t-shaped person” is an expert in at least one thing but also somewhat capable in many other things. An alternate phrase for “t-shaped” is “generalizing specialist”.

jason yip
image by Jason Yip

Ideally we’d like to have a team of t-shaped testers in Flow Patrol at Automattic. But how do we get to this end goal?

I recently embarked on an exercise to measure and benchmark our skills and do just this with our team. Here’s the steps we took.

Step One – Devise Desired Team Skills

The first thing we did was come up with a list of skills that we have in the team and would like to have in the team. These can be ‘hard’ skills like a specific programming languages and ‘soft’ skills like triaging bugs. In a standard co-located team this would be as easy as conducting a brainstorming session and using affinity grouping to discover these skills. In a distributed environment I wrote a blog post to my team’s channel and had individual members comment with a list of skills they thought appropriate, and then I did the grouping and came up with a draft list of skills and groups.

Step Two – Self-assess against a team skills matrix

Once I had a final list of skills and groups (see below for full list), I put together a matrix (in a Google Spreadsheet) that listed team members on the x-axis, and the skills on the y-axis, and came up with a skill level rating. Our internal systems use a three level scale (Newbie, Comfortable, Expert) which we didn’t think was broad enough so we decided upon five levels:

1. Limited
2. Basic
3. Good
4. Strong
5. Expert


Team Skills Matrix

I hadn’t seen Jason yip’s visual representation at that point in time, otherwise I may have used something like that, which has five similar levels:

matrix jason yip
Image by Jason Yip

Step Three – Publish results and cross-skill

Once we had the self assessments done we could then publish the data within our organisation and use the benchmark to cross-skill people in the team. In a co-located environment this could involve pair programming, in a distributed one it could involve mentoring and reviewing other team member’s work.

Have you done a skills matrix for your team? How did you do it? What did you discover?

Full List of Skills and Skill Groups for Flow Patrol at Automattic

Automattic Product Knowledge
WordPress Core Simple Sites Atomic Sites
Mobile Apps
Human Software Testing
Flow Mapping
Bug Triage & Prioritization
Exploratory Testing (pre-release)
Cross-browser Cross-device Testing
Facilitating Beta/Community Testing
Facilitating User Testing
Usability Testing
Accessibility Testing
Automated Testing
Automated End-to-end Browser Testing
Automated API/Integration Testing
Automated Unit Testing
Automated Visual Regression Testing
Android Automated Testing
iOS Automated Testing
Programming Languages
Shell Scripting
Objective C
Testing Tools/Frameworks
Team City (CI)
i18n Testing
Performance Testing
Security Testing
User advocacy – empathy and compassion
Project Management
Product Management
Product Development 
Jetpack API PHP
iOS App
Android App


Ask Me Anything Automated Testing Career Exploratory Testing

AMA: What sets exceptional QA testers apart?

Dayana asks…

I wondered if you could tell me what sets exceptional QA testers apart? Not just personality or work ethic traits, but specific skills and programming knowledge that will be very valuable to a team?

My response…

I think exceptional QA testers, as explained recently, aren’t people who are exceptional at just one thing, eg. testing, but good at lots of things.

So an exceptional QA tester, in my opinion, will typically have (at least good) skills in the following things:

  1. Skills in human exploratory testing: an exceptional QA tester has the ability to effectively find the most important bugs fast. Whilst this skill can be developed, I have found it’s mostly a mindset.
  2. Skills in developing automated tests: an exceptional QA tester will have programming skills needed to develop automated tests and I would recommend these to typically match the programming language(s) that programmers in your organization use. For example, skills in automated testing in .NET if your company primarily uses Microsoft .NET. Although, someone with strong programming skills in one language (eg. ruby) should be able to transfer these skills to another language (eg. python).
  3. Knowledge/Experience in your business domain: an exceptional QA tester will fully understand your business domain and keep this context in mind whilst testing a product and raising issues. An exceptional tester is always testing your system – just as I am testing publishing this post.
  4. An empathetic mindset: we design and develop software for real people and real life. An exceptional QA tester will test with this in mind.
Automated Testing Career Software Testing Test Automation

(Not) Lying about Writing Code

I recently saw this quote in an article by Nikita Hasis on Medium.

“If Your Test Leaders Aren’t Telling You To Write Code, They Are Lying!
Even if it’s by omission.

There’s this argument, almost daily, about whether software testers should learn programming. I’ll jump right in. It is unimaginable that someone would tell you NOT to learn something. That’s the first, and probably shittiest lie that inexperienced testers get fed. It’s further unimaginable, and downright irresponsible to tell people not to learn something that is very clearly where a large, well-paying, and above all interesting part of the industry is heading. Wanna work on innovative, data-driven projects with smart and driven people? You probably need to pull up terminal and at least get your toes wet, y’all.

The worst part of the lie is that it imposes that coding is a difficult grind and will only cause more problems than it solves. I even saw Alister Scott’s blog post referenced as an argument against coding, ironic as it is.”

~ Nikita Hasis (Medium)

Since Medium is a walled garden that doesn’t allow you to leave a comment without creating an account I’ll leave my response here instead (where anyone is free to comment however they like).

Ask Me Anything Automated Testing Career Exploratory Testing

AMA: the future of QA roles

Lroy asks…

How, according to you, is the future of QA roles going to look like. I currently work as a QA in a matured agile team where the devs are responsible enough to practice TDD and write automated tests while I pair with them. I pair with them on test strategy, performance testing, a bit of security testing. It is definitely an interesting shift to see how QA was perceived to how it is now. I understand that it is not like this in every company. But how do you see this role pan to to be? Thanks PS: Thank you for taking time to do AMA, it is really interesting to read your responses. I have enjoyed reading your blog for couple of years now.

Ask Me Anything Career

AMA: describing our work

Chris Stanbridge asks…

Hi Alister, Long time fan of this site and your work – you’ve become the quality go-to source of truth for our scrum team. I think I have a similar role as your day job, but I have a quandary: I’m unable to describe what I do at work. Friends and family lose attention part way through my explanation, and I’ve found it has made me appear to be a crashing bore. I’ve developed a recurring nightmare where I fail to summarise the nature of my work to my demons (game-show hosts or stand-up comedians who pick me out of a crowd) who just don’t get it. I thought I was being clever by saying I use a “robot” to check if computer code works, but the picture painted using that word is considerably off the mark. If only I was a plumber… I’m sure you can provide me with a concise blurb I can memorise and plagiarise for the next neighborhood barbeque that’ll make me appear as interesting as I think I am 🙂 Cheers, Chris PS: Please keep up the excellent and entertaining blog – you’re an inspiration and an education.

My response…

Well, my job title at Automattic is actually Excellence Wrangler which is both awesome (in that no-one has heard of it before), and terrible (in that I have to explain what it is to everyone who asks).

I will often say something quick like “I work with various software teams to create a consistently pleasant user experience on”.

If that’s not sufficient detail I can explain that we continually do lots of testing of various user flows across different devices and browsers trying to find issues and bugs before our users do. We also develop automated scripts that run all the time to help us find these issues quickly since we make changes so frequently.

I’m not sure if that will help you, but since you’re in Brisbane too, let’s catch up and discuss further over lunch and/or tea 😀


Ask Me Anything Career

AMA: QA tools and resources when starting out

Nanthida asks...

Hey Alister!
I’m new to software testing, having come from a Support background, and I’d love to know what tools and/ resources worked best for you when you were starting out? 🙂

My response…

I very recently answered an almost identical question, so I will point you to my response.

If you have any follow-up questions, feel free to leave a comment below.

Ask Me Anything Career Software Testing

AMA: how to cope as a solo tester

Bodda asks…

hey, i like your blog, i’m reading your blog on a daily basis, and i just want to ask you what i can i do to enhance my skills, knowledge and to be a be good in functional testing (manual and automation) IF i’m the ONLY tester in my current company (performing all testing activities), but i feel that i have a mess in my head and lots to learn to be up to date with the last trends.
my question now “What to learn, When(everyday?) and How in case your are the only tester in your company”.

i hope to answer my question soon to make my head calm down

My response…

I have fond memories of a project where I worked as the solo tester on a software delivery team; we had something like 8 developers (including one lead who took on iteration management tasks) and me as a tester. That was it. I loved it because we established really good unhindered rapport with our business stakeholders, and I was always busy!

But getting back to your question: when I was working in that situation it was vital that we had developers working on as much test automation as possible, alongside functional development, since there was just too much functional testing to do for a single person also doing test automation. I found myself spending about 80% of my time just testing new functionality and spot-testing different browsers/devices etc. The remaining 20% I spent ensuring we had good regression test coverage through the automated tests that developers were writing and I was helping maintain. So first and foremost I would strongly encourage you to get the developers you work with to have as much responsibility as possible for automated tests.

If you have this in place you should be able to take a deep breath and spend more time doing quality testing. I have found the best way to learn is on the job, you’ll be in a good position to do this, and often you’ll learn best by making your own mistakes.

learning - 1
via The New Yorker

If you want to know more about how to become better at what you do, I’ve shared some tips in a previous answer. There’s also my Pride & Paradev book which I wrote whilst working as a solo tester on a team.

All the best.

Ask Me Anything Career

AMA: junior QA professional development

Mark asks…

What advice would you give to a talented junior QA who wanted to advance their career in software testing? Assume that they have no formal training in testing and have worked their way up from a junior IT role in an unrelated area.

My response…

I strongly believe you create what you will, so if someone’s willing to take control of their own testing career then they will have a mighty fine one at that.

I find there’s two key components to a career in software testing, and they’re quite opposite roles/skills so it’s worthwhile focusing on both:

  1. Developing skills in human exploratory testing: effectively finding the most important bugs fast. Whilst this skill can developed, I have found it’s mostly a mindset.
  2. Developing skills in test automation: having the ability to collaborate on and implement automated regression tests so a tester can spend more time on point 1.

In terms of resources for professional development for human testing for a tester I would recommend the following:

Books: some books cover the basics: agile testing, specification by example, pride and paradev.

Blogs: I’ve mentioned some blogs I read. There’s also the fantastic 5blogs feed as well which covers some great testing articles each day – so they can follow that.

Conferences: there’s a good site which lists every testing conference, and coming up in Australia is Australian Testing Days 2016 and the ANZTB Conference both next month and both in Melbourne, and CukeUp! Australia later in the year. Internationally I recommend GTAC and the Selenium Conference(s).

In terms of test automation skills, I am finding software developers are much more commonly interested in automated testing than I have found say 5 or 10 years ago. This has two benefits:

  1. Software developers can up-skill testers on their technical programming and automation tasks; and
  2. Software developers can take on some of this responsibility so technical testers can spend more time testing

In terms of building up test automation skills on one’s own, it’s never been easier or more accessible.

Two of the most popular testing tools of all time (Selenium/WebDriver and Cucumber/Specflow) are free and open source and available in all mainstream programming languages and platforms so there really isn’t an excuse not to learn them on one’s own.

If someone was just starting out and platform-agnostic I’d recommend starting with cucumber and watir-webdriver in ruby (because it’s so easy) and buy Cheezy’s book ‘Cucumber & Cheese‘ to learn how.

If you combine all of these together with a willing attitude it’s easy to develop great testing skills.

Finally, Nas has some good career advice:

I know I can
Be what I wanna be
If I work hard at it
I’ll be where I wanna be


Ask Me Anything Career

AMA: hiring technical testers

Mark asks…

Right now the software testing space is a very challenging area to hire in. We see many candidates that lack strong technical skills, in particular strong foundational knowledge in programming and automation. Why do you think this is the case? How can we improve this?

My response…

Great question. I believe there’s a few factors at play here.

Firstly, software testing still doesn’t really get taught at universities. I know of a couple of Australian universities that offer a single software testing course, but from what I hear it’s the same as 10+ years ago when I did my software engineering degree where testing was always just an afterthought rather than taught as a way to build quality self-tested code. So I have found there isn’t technical testers with those skills coming straight out of university – they need to pick up these skills in the industry. I’ve been in some organisations where we’ve put IT graduates into these technical roles to develop these skills but they typically those graduates have wanted to move into software developer roles so this wasn’t successful.

Secondly, in the past, I’ve found few organizations that fully recognise and value technical testers/software test engineers as a profession it it’s own right, so a lot of people who have these skills may rather work as a software engineer/developer where they’ll be recognised and valued more. In the past, places like Facebook had zero testers or people will a sole responsibility for QA, although from what I have heard this may have changed. I have noticed recently there is a general trend to value these skills more highly of late and since there wasn’t enough demand in the past I believe it’s a catch up game of increasing supply.


Finally, technical testing particularly developing test automation for systems that haven’t been built with testability in mind can be a tiring task: especially when dealing with inconsistent or non-deterministic test results which an organization is often relying upon to release software rapidly. This potentially discourages a lot of people from taking on these roles, although as my GTAC talk promoted it doesn’t have to be this way if we build systems with testability in mind as a collaboration between testers and developers.


An Automattician To Be

I am excited to announce I will be starting as a full time Excellence Wrangler for Automattic working on from this coming Monday.

Improve the quality of the experience through testing and triage. Your work will inform product teams to act on the top priority issues facing our users. Tasks include automated UI testing, creating and executing test plans, effective issue tracking and triage, and identifying and monitoring quality metrics.

I’ve dreamed about working for Automattic/ for a long time (I first wrote about working for Automattic in 2008), and with their newly created Excellence Wrangler roles this really is a dream come true.

WordPress is superbly simple yet beautifully powerful software that powers 24% of the Internet (including this blog), not only for blogs like this but sites for businesses, artist’s portfolios, hobbyists and giant media organizations like CNN and TIME.

Some amazing facts about Automattic and how I was hired:

  1. Automattic are 100% distributed with 395 staff across 36 countries all working from home or wherever they choose.
  2. I have already worked for Automattic for almost 3 months on a paid trial, where I was given a real project to work on in my spare time. This is a requirement for all new hires at Automattic. I can’t overstate how great this is, as it gave both Automattic and myself real exposure to each other before committing to a full time job. It now makes taking on a new job without a trial seem too daunting.
  3. Automattic does their entire interviewing/trial/hiring process via asynchronous text chat (Skype/Slack), including the final hiring discussion with Matt, so I have never spoken to a person from Automattic. Whilst this may seem unusual at first, it’s representative of how the company works in such a distributed way, and it’s a great way to eliminate all prejudice/bias from a hiring process as it’s all about what value someone can add, not what they look or sound like.
  4. Everyone who joins Automattic full time spends their first 3 weeks on support, regardless of their position. I am looking forward to this next week as it will give me broad insight into how is used by real customers by working as a ‘Happiness Engineer’: Genchi Genbutsu.

I can’t wait to be a part of the future of, so stay tuned for more updates as I begin this exciting next stage of my career.