Ask Me Anything Automated Testing Career Exploratory Testing

AMA: What sets exceptional QA testers apart?

Dayana asks…

I wondered if you could tell me what sets exceptional QA testers apart? Not just personality or work ethic traits, but specific skills and programming knowledge that will be very valuable to a team?

My response…

I think exceptional QA testers, as explained recently, aren’t people who are exceptional at just one thing, eg. testing, but good at lots of things.

So an exceptional QA tester, in my opinion, will typically have (at least good) skills in the following things:

  1. Skills in human exploratory testing: an exceptional QA tester has the ability to effectively find the most important bugs fast. Whilst this skill can be developed, I have found it’s mostly a mindset.
  2. Skills in developing automated tests: an exceptional QA tester will have programming skills needed to develop automated tests and I would recommend these to typically match the programming language(s) that programmers in your organization use. For example, skills in automated testing in .NET if your company primarily uses Microsoft .NET. Although, someone with strong programming skills in one language (eg. ruby) should be able to transfer these skills to another language (eg. python).
  3. Knowledge/Experience in your business domain: an exceptional QA tester will fully understand your business domain and keep this context in mind whilst testing a product and raising issues. An exceptional tester is always testing your system – just as I am testing publishing this post.
  4. An empathetic mindset: we design and develop software for real people and real life. An exceptional QA tester will test with this in mind.
Ask Me Anything Automated Testing Career Exploratory Testing

AMA: the future of QA roles

Lroy asks…

How, according to you, is the future of QA roles going to look like. I currently work as a QA in a matured agile team where the devs are responsible enough to practice TDD and write automated tests while I pair with them. I pair with them on test strategy, performance testing, a bit of security testing. It is definitely an interesting shift to see how QA was perceived to how it is now. I understand that it is not like this in every company. But how do you see this role pan to to be? Thanks PS: Thank you for taking time to do AMA, it is really interesting to read your responses. I have enjoyed reading your blog for couple of years now.

Ask Me Anything Automated Testing Exploratory Testing

AMA: the test analyst engineer divide

sunjeet asks…

Hi Alister, A mentoring/coaching question…. Keen to know your thoughts on the test “analyst” vs “engineer” trend in the world of software testing. Context –> Having been a practitioner of both “sub disciplines” , I feel that exploratory testing and automated testing require some specific set of skills and mind-set , however deliberately setting this demarcation/partitions, I believe, pigeon-holes testers (especially new graduates ) into identifying themselves as a “manual” or “automated” tester and restricting their development and learning. I believe there are set of overlapping base skills which every tester should have to compliment their core exploratory testing and automated testing roles. So, my questions are – 1. Do you agree or disagree with this dichotomy , and why ? 2. How do you get new testers to grow their exploratory and test engineering skills ? Thanks in advance

Ask Me Anything Automated Testing Exploratory Testing

AMA: balance between testing activities

QA asks…

At my work place I can see testers are caught between scripted testing, exploratory testing and automated testing. Some teams have dedicated resources for each type of tests, whereas others do a bit of everything. When it comes to priority automated testing takes a back seat, scripted testing is rushed through (skipping detailed documentation) and exploratory testing aka “just have a play around to see if you can break” takes a driver seat. What according to you is a right balance? How do you structure your test, so you can effectively do proper testing?

My response…

Finding balance is always a challenge. As I explained recently, I have found more and more software developers are more interested in automated testing these days, so collaborating on these automated tests with developers, or moving (some) responsibility of these tests to developers is a great way to free up some more time for human testing.

As I explained in another response, I don’t believe in scripted manual testing for regression purposes, the automated regression tests should cover these manual scripts, so the only testing I do is of the exploratory kind. I still plan my testing, just before or as I test, to work out the kinds of things I want to explore/cover and what browsers/devices or operating systems I will use. The key to good exploratory testing, I have found, is to be testing small changes of functionality, as there’s much less risk of gradually (continuously) introducing new small pieces of functionality than a single large change.

I don’t think there’s a ‘right’ balance as this will depend on your organization, your resources and how much collaboration you have. I typically spend 40% of my current time writing and maintaining end-to-end automated tests for, and the remaining 60% on human testing: exploratory testing new features, continuous dogfooding on different browsers/devices, catfooding, visually recording flows for reference and historical purposes, triaging existing bugs and raising new enhancement requests. If we had more developers working on our e2e automated tests, which there has been interest in, I imagine my effort could drop to say 20% but I am not really sure until we get there.


Ask Me Anything Exploratory Testing

AMA: are test cases redundant?

Adnan asks…

Hi Alister, I have been reading your blog for quite a while now. I find it quite insightful and engaging.

I wanted to get your opinion on a question (rather observation) I have after doing BDD on few projects as a tester.

If testing is acceptance driven/BDD, then all of the business rules are captured in acceptance criteria in stories and then in feature files when they are automated. Does this makes test cases redundant? Are they actually needed anymore?

In terms of traceability, a feature file clearly maps to a story and hence a business requirement so unless there is a legal requirement to have test cases documented by the client, I feel test cases to be a redundant effort, unless I am missing something. Very interested to hear your experience/thoughts on this.

My response…

My short answer is yes, test cases are redundant.

My long answer is also yes, here’s why:

Ideally I think every software system should aim to have no (zero) manual regression testing, that is you can do a software release and be confident you haven’t introduced any regressions. This isn’t 100% test coverage: this often isn’t practical, zero manual regression is having (just) enough automated tests for you to not have to do any manual regression testing. You still have to manually test new features of course, it’s the existing ones you don’t need to worry about.

When you’re in this situation, there’s no need to have manual test cases, since your automated tests act as living executable specifications of your system, and manual test cases would be redundant.

But what about new functionality? In a continuous delivery model, there’s no time to write manual test cases then execute them, and this is pointless anyway as they won’t live on past that user story which will have automated tests developed alongside it, so I typically test against the acceptance criteria defined for a user story (hopefully in collaboration with you), and add notes to a user story to show what other edge cases I envisioned and what I uncovered during story testing.

I have found having this level of documentation is more than enough even for the audit-heavy environments, as I have found auditors prefer to see automated test results against every build, then a pile of test cases in excel spreadsheets that are out of date.



Ask Me Anything Exploratory Testing Software Testing

AMA: How do you teach someone exploratory testing?

Paul asks…

How do you teach someone exploratory testing?

My response…

For something different, let me start with a quote:

“We shall not cease from exploration, and the end of all our exploring will be to arrive where we started and know the place for the first time.”
~ T. S. Eliot

As a father of three children, I believe humans are innate explorers. So exploring a system should come natural to most people, but I’ve found a lot of people can explore a system and not find any bugs.

Techniques like session-based testing attempt to introduce measurement and control to exploratory testing so people are meant to be more effective, but, like the gorilla basketball video has shown us, introducing a goal for a session can blind us to the things that aren’t specifically in that goal. Much like following a script can blind us to things that aren’t in that script.

So how do we teach people to find bugs by exploration?

wilful-blindnessI believe the biggest thing that stops people finding bugs by exploration is wilful blindness: choosing not to know. The way you can teach someone to be a better exploratory tester therefore is by teaching them to be less blind.

Margaret Heffernan explains this superbly well in her completely non-testing non-technical book that I think every tester should read:

“We make ourselves powerless when we choose not to know. But we give ourselves hope when we insist on looking. The very fact that wilful blindness is willed, that it is a product of a rich mix of experience, knowledge, thinking, neutrons and neuroses, is what gives us the capacity to change it. Like Lear, we can learn to see better, not just because our brain changes but because we do. As all wisdom does, seeing starts with simple questions: what could I know, should I know, that I don’t know? Just what am I missing here?”

I really recommend reading that book.