Agile Software Dev Automated Testing Software Testing

Extensive post release testing is sign of an unhealthy testing process

Does your organization conduct extensive post-release testing in production environments?

If you do, then it shows you probably have an unhealthy testing process, and you’ve fallen into the “let’s just test it in production” trap.

If testing in non-production environments was reflective of production behaviour, there would be no need to do production testing at all. But often testing isn’t reflective of real production behaviour, so we test in production to mitigate the risk of things going wrong.

It’s also the case that often issues are found in a QA environment don’t appear in a local development environment.

But it makes much more sense to test in an environment as close to where the code was written as possible: it’s much cheaper, easier and more efficient to find and fix bugs early.

For example, say you were testing a feature and how it behaves across numerous times of day across numerous time zones. As you progress through different test environments this becomes increasingly difficult to test:

In a local development environment: you could fake the time and timezone to see how your application behaves.
In a CI or QA environment: you could change a single server time and restart your application to see how your application behaves under various time scenarios: not as easy as ‘faking’ the time locally but still fairly easy to do.
In a pre-production environment: you’ll probably have clustered web servers so you’ll be looking at changing something like 6 or 8 server times to test this feature. Plus it will effect anyone else utilizing this system.
In a production environment: you’ll need to wait until the actual time to test the feature as you won’t be able to change the server times in production.

Clearly it’s cheaper, easier and more efficient to test changing times in an environment closer to where the code was written.

You should aim to conduct as much testing as you can in earlier test environments and taper this off so by the time you can a change into production you’ll be confident that it’s been tested comprehensively. This probably requires some change to your testing process though.

Tests Performed per Environment

How to Remedy A ‘Test in Production’ Culture

As soon as you find an issue in a later environment, ask why wasn’t this found in an earlier environment? Ultimately ask: why can’t we reproduce this in a local environment?

Some Hypothetical Examples

Example One: our tests fail in CI because of JavaScript errors that don’t reproduce on a local development environment. Looking into this we realize this is because the JavaScript is minified in CI but not in a local development environment. We make a change to enable local development environments to run tests in minified mode which reproduces these issues.

Example Two: our tests failed in pre-production that didn’t fail in QA because pre-production has a regular back up of the production database whereas QA often gets very out of date. We schedule a task to periodically restore the QA database from a production snapshot to ensure the data is reflective.

Example Three: our tests failed in production that didn’t fail in pre-production as email wasn’t being sent in production and we couldn’t test it in pre-production/QA as we didn’t want to accidentally send real emails. We configure our QA environments to send emails, but only to a white-list of specified email addresses we use for testing to stop accidental emails. We can be confident that changes to emails are tested in QA.


It’s easy to fall into a trap of just testing things in production even though it’s much more difficult and risky: things often go wrong with real data, the consequences are more severe and it’s generally more difficult to comprehensively test in production as you can’t change or fake things as easily.

Instead of just accepting “we’ll test it in production”, try instead to ask, “how can we test this much earlier whilst being confident our changes are reflective of actual behaviour?”

You’ll be much less stressed, your testing will be much more efficient and effective, and you’ll have a healthier testing process.

Agile Software Dev Automated Testing Software Testing

Testing beyond requirements? How much is enough?

At the Brisbane Software Testers Meetup last week there was a group discussion about the requirement to test beyond requirements/acceptance criteria and if you’re doing so, how much is enough? Where do you draw the line? It came from an attendee who had a manager pull him up for a production bug that wasn’t found in testing but wasn’t in the requirements. If it wasn’t in the requirements, how could he test it?

In my opinion, testing purely against requirements or acceptance criteria is never enough. Here’s why.

Imagine you have a set of perfectly formed requirements/acceptance criteria, we’ll represent as this blue blob.


Then you have a perfectly formed software system your team has built represented by this yellow blob


In a perfect, yet non-existent, world, all the requirements/acceptance criteria are covered perfectly by the system, and the system exists of only the requirements/acceptance criteria.

Requirements - System

But in the real world there’s never a perfect overlap. There’s requirements/acceptance criteria that are either missed by the system (part A), or met by the system (part B). These can both be easily verified by requirements or acceptance criteria based testing. But most importantly, there are things in your system that are not specified by any requirements or acceptance criteria (part C).

Requirements - System(1)

These things in part C often exist of requirements that have been made up (assumptions), as well as implicit and unknown requirements.

The biggest flaw about testing against requirements is that you won’t discover these things in part C as they’re not requirements! But, as shown by the example from the tester meetup, even though something may not be specified as a requirement, the business can think they’re a requirement when it effects usage.

Software development should aim to have as few assumptions, implicit and unknown requirements in a system as reasonably possible. Different businesses, systems and software have different tolerances for how much effort is spent on reducing the size of these unknowns, so there’s no one size fits all answer to how much is enough.

But there are two activities that a tester can perform and champion on a team which can drastically reduce the size of these unknown unknowns.

1 – User Story Kick-Offs: I have only worked on agile software development teams over the last number of years so all functionality that I test is developed in the form of a user story. I have found the best way to reduce the number of unknown requirements in a system is to make sure every user story is kicked-off with a BA, tester and developer (often called The Three Amigos) all present and each acceptance criterion is read aloud and understood by the three. At this point, as a tester, I like to raise items that haven’t been thought of so that these can be specified as acceptance criteria and are unlikely to either make it or not make it into the system by other means or assumptions.

2 – Exploratory Testing: As a tester on an agile team I make time to not only test the acceptance criteria and specific user stories, but to explore the system and understand how the stories fit together and to think of scenarios above and beyond what has been specified. Whilst user stories are good at capturing vertical slices of functionality, their weakness, in my opinion, is they are just a ‘slice’ of functionality and often cross-story requirements may be missed or implied. This is where exploratory testing is great for testing these assumptions and raising any issues that may arise across the system.


I don’t believe there’s a clear answer to how much testing above and beyond requirements/acceptance criteria is enough. There will always be things in a system that weren’t in the requirements and as a team we should strive to reduce the things that fall into that category as much as possible given the resources and time available. It isn’t just the testers role to either just test requirements or be solely responsible/accountable for requirements that aren’t specified, the team should own this risk.

Agile Software Dev

Intentionally Disposable Software

Update 11 Feb: Sorry, somehow comments were disabled for this post. This has been resolved now.

There’s a series of code retreats that take place each year where a group of programmers get together to work in groups to solve a problem (kata). They do this in iterations over and over again, and most importantly they delete their entire code at the end of every iteration (typically 45 minutes).

“It’s much easier to walk a new road when the old road is out of sight”

~ Drew Miller

Programmers don’t delete enough production code. Which is funny because I’ve met heaps of programmers, including myself, who love deleting code; it’s a strangely satisfying, cleansing ritual.

What if we could replicate what we do when doing these katas and delete the entire source code of a system every 6 months or a year and start again? Does that sound crazy? Can we do this?

We couldn’t do this with how we currently develop software. It’s because we build software systems that are way too complex, have features that no-one uses and are built to last way too long. We expect our software systems to last 10+ years, and we’re so anxious about long-lasting technology impacts we make overly cautious or conservative decisions that come back to bite us constantly in the years to come. We build technical debt into legacy systems that no one wants to work on. We obsess about re-usability, longevity and salvageability of every piece of code we write. We build massive overly complex regression test suites because we expect the system to work for so long and be maintained for so long we expect it to eventually deteriorate and we want a regression test suite as a ‘safety net’ when it inevitably does.

Maintaining legacy software is like painting the Sydney Harbour Bridge. You start painting it on one side and by the time you get to ‘finish’ it on the other side it’s taken you so long you now need to start repainting the original side again. Wouldn’t it be easier to just build a new bridge?

What we need is Intentionally Disposable Software. We need to build software only designed to last 6 months, a year max. As soon as we deploy it we start immediately on a replacement for it. We put bare minimum effort into maintenance as we’ll just replace what we have in Production as soon as we can: why wash up when you can use fresh paper plates for every meal? As soon as the replacement is ready, we deploy that and completely blow away the old software system. We rinse and repeat.

It’s somewhat similar to planned obsolescence but we don’t do it to annoy our customers and attempt to generate repeat purchases, we do it to refine our software system without any legacy.

We use analytics to tell us exactly what features of the system in production are used and most importantly, what features are little or never used. We don’t build those features into the replacement systems ever again so each system we build is leaner, more focused on the important things it is meant to do and does them better each time. We don’t waste any time on building or supporting unimportant things.

We don’t have time to build up technical debt. We aren’t anxious about choosing a wrong technology. Did we use AngularJS and now hate it? Never fear, we start work immediately on our new system replacement and use ReactJS (or whatever the latest/coolest framework is).

Developer happiness skyrockets! No legacy code! No technical debt! Everyone can work on the latest/best technology to get the job done and want to stick around in our organization to do just that. It’s like being a consultant without being a consultant. Because everyone has already implemented the same thing before, everyone is aware of the gotchas, so whilst people are constantly learning new technology, they’re efficient because they know what they’re actually trying to achieve. And because we’re building such a lean system we’re lean in our approach.

We do need to make sure we use open standards and have an easy way to export/import/migrate data in fast, clean ways – which is good.

The same applies to integration points, we need to be modular enough and use open standards and protocols exclusively to be able to drop out one system and replace it with another that integrates easily.

So what about testing?

If we expect a system to last six months to a year, we need just enough testing. We need just enough testing to make sure the system is built right (doesn’t break), but not too much requirements based testing around building the right system, because we know we won’t build the right system, we’ll be building a ‘more right’ replacement as soon as this one is built.

We need testing that focuses on intention of the system over implementation, because the implementation will constantly change each time we rewrite it. If we write our automated tests in a declarative style devoid of implementation detail we’ll be much better off.


Scenario: Overseas customers must provide ID for expensive purchases
Given I am a non-Australian customer
When I order more than $100 of tea
Then I need to provide additional ID on checkout


Scenario: Overseas customers must provide ID for expensive purchases
Given I go to the Beautiful Tea Home Page
When I click Sign InAnd I enter my username and password
And I click OK

Think of Disposable Software like building a house.

You want somewhere to live so you decide to build a house. You design and build a new ‘dream home’ with the best intentions but soon you realize there’s a big difference between what you actually need and what you thought you needed. There’s also some fundamental design flaws you didn’t even realize until it was built and you’re living in it, like mold in the bathroom as it has not enough air-flow and the bedrooms face the wrong direction and are constantly too hot to sleep at night. Plus life has since thrown a new baby into the mix, so 12 months later you find yourself with a house with a lot of design flaws that doesn’t meet your now much clearer, and since expanded, requirements.

So what do you do? Since you’ve invested a lot (financially and emotionally) into your house and you expected it to last ten years of more, you renovate. But the problem with renovating is that you’ve got to work around all the original design flaws, and since the house is already built it’s much more difficult and expensive to make changes to it, and also since you’re living in it, any renovation comes with risk of disruption/displacement to the occupants including an overly sensitive newborn. You find since it’s mainly custom work that the renovations you’re planning will cost nearly as much as the original house.

This is like legacy software. You can’t renovate it easily as it’s already in use, you’re constantly working around its design flaws so it’s much more difficult and costly to make changes to it. Plus it’s really hard to remove the unnecessary parts of your house by renovation.

But what’s the alternative? What if you built the house knowing that come 12 months time you could knock it down, recycle it, and build a new house knowing exactly what you want this time around. You’ll know not to face the bedrooms West. You’ll know the make the bathroom has plenty of air-flow. You’ll even include a bedroom suitable for the baby. But you don’t get too caught up in getting this house ‘perfect’ because come 12 months time you can do it all again. The house could be prefabricated so it’s much cheaper to build off-site in a construction friendly environment, and the migration involves temporarily moving some furniture from the old structure, placing the new house in place with the furniture and recycling the old. You own less objects (clutter) as you know this happens and are prepared for it. As you kids grow up their needs change so instead of doing ‘extensions’ and ‘renovations’ you simply redesign your new house which will be delivered in 12 months time.

This is like Intentionally Disposable Software.

As Alex Bundardzic wrote almost ten years ago in his fantastic Disposable Software post:

“In reality, a good software developer should be able to… deliver a disposable, short lived product that addresses only the pressing needs of the moment, and then simply disappears. No maintenance, no enhancements, no song-and-dance, nothing.”

What do you think?

Agile Software Dev

Never compare your organization’s insides with another organization’s outsides

I once heard the brilliant suggestion that you should never compare your insides with another person’s outsides; because they’re not the same thing. For example, just because someone may seem happy and drive an expensive car, that’s only the outside view of that person and it doesn’t paint the full picture of that person’s insides, which you’re dangerously trying to compare with your own inner thoughts/status.

The same applies with comparing things at your organization to things you’ve heard about other organizations. Countless times, including just this week, have I heard managers and colleagues say things they’ve heard like: “Facebook don’t have testers”, “Google has 10,000+ engineers in 40 offices working on trunk” and “Flickr deploys to production 10 times a day so we can too”. These are examples of comparing our insides to other’s outsides, again.

Yes, Google may have 10,000+ engineers committing to one branch but having spoken to people who work at Google it’s not quite as amazing as it seems. For example, firstly the code-base is broken down into projects (imagine the checkout time without this), each and every change set must be code reviewed, have automated and manual tests performed against it (which can take hours/days) before it is even committed to the trunk, even before it can even be considered for a production release.

I didn’t realize it at the time but the keynote at GTAC last year captured this phenomenon perfectly:

Google from the outside (like a jet plane)Google from the inside (lots of people pushing a broken down car)


It can not only be really annoying/unhealthy for staff to constantly hear such comparisons, it can also be dangerous because doing something just because Google/Facebook/Twitter/Flickr does it without knowing the inner workings of their organizations will inevitably lead to failure when you try to do it without that context and experience.

So next time you are tempted to drop something you’ve heard from a conference or a blog post about how another company does something better than yours, or to justify that we can/should do it this way, remember, never compare your organization’s insides with another organization’s outsides.

Agile Software Dev

Iterative vs Incremental Software Development

What’s the difference between ‘iterative’ and ‘incremental’ software development?

I know a lot of agile software development teams call their blocks of development time ‘iterations’ instead of ‘sprints’. Does that mean they’re doing iterative software development?

You’ve probably seen the Mona Lisa analogy by Jeff Paton that visually tries to show the difference between the two development approaches:

Incremental Development:


Iterative Development:


But which is better?

Well, if for some (very likely) reason (lack of money, changed business conditions, change in management) we had to stop after iteration/increment one or two, which approach would yield a better outcome?

Mona Lisa

Incremental development gives us a painting of half a lady whereas iterative development gives us an outline of a lady, but both paintings really wouldn’t belong in The Louvre. Perhaps we could have just painted a smaller painting?

This is where I think the Mona Lisa art analogy falls apart. A work of art, like a book, but unlike a piece of software, has a pretty clear definition of done. An artist knows when their piece of art is done: not a single stroke more, not a single stroke less.

But I’ve never worked on a piece of software that was considered done: there’s always more functionality to add/remove/fix.

If we can recognize that software is never done, all we need to do it work out how to get it to where we want it to be (for now).

“We shall not cease from exploration, and the end of all our exploring will be to arrive where we started and know the place for the first time.”
~ T.S. Eliot

If we are driven by time to market we should internally iterate just enough so we can  release ‘increments’ fast and often, and iterate/release again and again.

If we are driven by user experience, we should internally iterate a lot to get things right, release increments only when necessary, and iterate again.

Both approaches are about iterating. Both are also about incrementing. The difference is how soon we release after how many times we iterate.

Compare the beginnings of the two dominant mobile operating systems. Google went for time to market with Android, they released an unpolished, yet feature rich, operating system quickly and made it better by iterating/incrementing again and again over time. Apple took the opposite approach: they released iOS with highly polished features relatively slowly (it took three major iOS releases to get MMS and copy & paste!) but focused on getting things right from the start.

Both approaches are different but neither are wrong: they highlight the differences between Apple and Google and their approach to developing software.


We can’t build anything without iterating to some degree: no code is written perfectly the second that it is typed or committed. Even if it looks like a company is incrementally building their software: they’re iteratively building it inside.

We can’t release anything without incrementing to some degree: no matter how small a release is, it’s still an incremental change over the last release. Some increments are bigger because they’ve already been internally iterated upon more, some are smaller as they’re less developed and will evolve over time.

So, we develop software iteratively and release incrementally in various sizes over time.

Agile Software Dev Software Testing

What is a good ratio of software developers to testers on an agile team?

The developer:tester ratio question comes up a lot and I find most, if not all, answers are “it depends”.

I won’t say “it depends” (it’s annoying). I will tell you what works for me given my extensive experience, but will provide some caveats.

I’ve worked on different agile software development teams as a tester for a number of years and I personally find a ratio of 8:1 developers to tester(s) (me) works well (that’s 4 dev-pairs if pair programming). Any less developers and I am bored; any more and I have too must to test and cycle time is in jeopardy.

Some caveats:

  • I’m an efficient tester and the 8:1 ratio works well when there’s 8 equally efficient programmers on the team – if the devs are too slow, or the user stories are too big, I get bored;
  • Everyone in the team is responsible for quality; I have to make sure that happens;
  • A story must be kicked off with the tester (me) present so I can question any assumptions/anomalies in the acceptance criteria before any code is written;
  • A story is only ready for test if the developer has demonstrated the functionality to me at their workstation (bonus points in an integrated environment) – we call this a ‘shoulder check’ – much the same way as monkeys check each others shoulders for lice;
  • A story is also only ready for test if the developer has created sufficient and passing automated test coverage including unit tests, integration tests (if appropriate) and some acceptance tests; and
  • Bug fixes take priority over new development to ensure flow.

What ratio do you find works for you?

Agile Software Dev user stories

Why to avoid t-shirt sizes for user story estimation

The more I work on agile software development teams who use t-shirt sizes (S,M,L,XL etc.) to estimate user stories the more I dislike this approach. Here’s why:

  • In my opinion, the most import thing about user story sizing is relativity, and t-shirt sizes are a subjective measure of relativity: someone in the team might think a large is two times as big as a small, whereas another person might think it’s three times as big. This isn’t helped by the t-shirt analogy where it’s actually hard to determine how much bigger is a large t-shirt than a small one?
  • You can’t create a single measure of team velocity unless you define a scale that converts t-shirt sizes into a numeric size so you can measure t-shirt size relativity and velocity.
  • As soon as create a scale to convert t-shirt sizes into a numeric size you’ve essentially started using story points (in a convoluted way).

TL;DR: Using t-shirt sizes for user story estimation is confusing and ultimately leads the team to using story points so just skip t-shirt sizes and use relative story points instead.

Agile Software Dev

There’s no such thing as doing Agile

I am amused by this advertisement released by Nintendo in 1990.

There's no such thing as a Nintendo

which inspired me to do something similar for the word ‘agile’.

There's no such thing as doing Agile

Here’s a link to Prag Dave’s advice.

Do me a favour and print my poster out in A3 and put it on your office wall. Here’s a pdf version.

Agile Software Dev Software

Waterfall, Agile Development & Hyperbole

Hyperbole. Love it or hate it, it’s been around for centuries and is here to stay. And, as someone pointed out this week, I’m guilty as charged of using (abusing?) it on this blog. You just need to quickly flick through my recent posts to find such melodramatic titles such as ‘Do you REALLY need to run your WebDriver tests in IE?‘, ‘UI automation of vendor delivered products always leads to trouble‘, and  ‘Five signs you’re not agile; you’re actually mini-waterfall‘. Hyperbole supports my motto for this blog and my life: strong opinions, weakly held.

But it’s not just me who likes hyperbole mixed into their blog posts. Only this morning did I read the catchy titled ‘Waterfall Is Never the Right Approach‘ followed quickly with a similarly catchy titled rebuttal: ‘Why waterfall kicks ass‘ (I personally would have capitalized ‘NEVER’ and ‘ASS’).

While I found both of articles interesting, I think they both missed the key difference between waterfall and agile software development (and why waterfall rarely works in these fickle times): waterfall is sequential whereas agile development is (at least meant to be) iterative.

I personally don’t care whether you do SCRUM or XP, whether you write your requirements in Word™ or on the back of an index card, or even if you stand around in a circle talking about what card you’re working on.

What I do care about is whether you’re delivering business value frequently and adjusting to the feedback you get.

Sequential ‘big bang’ development such as waterfall, by its nature, delivers business value less frequently, and chances are when that value is realized the original problem has changed (depending on how long ago that was), because as I stated and believe, we live in fickle times.

Iterative development addresses this by developing/releasing small fully functional pieces of business value iteratively and adjusting to feedback/circumstance.

Just because an organization practices what they call ‘agile’, doesn’t mean they’re delivering business value iteratively. I’ve seen plenty of ‘agile’ projects deliver business value very non-frequently, they’re putting a sequential process into agile ‘sprints’ followed by a large period of end to end, business and user acceptance testing, with a ‘big bang’ go live.

Whilst I believe iterative development is the best way to work; I’m not dogmatic (enough) to believe it’s the only way to work. Whilst I believe you could build and tests parts of say an aeroplane iteratively, I still hope there’s it’s a sequential process with a whole heap of testing at the end on a fully complete aeroplane before I take my next flight in it.

Agile Software Dev

Tips for great brown bag lunches

I’m a big fan of brown bag seminars also called brown bag lunches or just brown bags. I’ve seen them used very successfully to share knowledge and increase team bonding. Here’s some tips to make them successful for you.

Commit to a date and lock in a topic and presenter

Since a brown bag lunch is just as much about discussion as content, I find it’s good to commit to a date and lock in a topic and presenter. This puts pressure on the presenter to make time to get their content ready, and also not worry about having it ‘perfect’.

Give everyone an opportunity to present: try to avoid having the same person presenting over and over again. A good way to harvest ideas is to have spot near your team wall (or a trello board) where people can suggest topics they would like to hear or present.

Don’t limit the audience

Resist the temptation to make a brown bag lunch only for programmers, or only for business analysts etc. Even if the topic is aimed at programmers or testers, it’s good to have a goal to make your content interesting enough that it’ll appeal to the programmer or tester in anybody.

Don’t limit yourself to content that is directly aligned with your current work

Whilst content that is directly aligned to work is good as it’s a good way to get buy in, it’s also good to present content loosely related to what people are working on. For example, you could present a brown bag on distributed version control systems (such as git) to a team purely used to working with centralized version control (such as Subversion or TFS).

If you have a couple of short presentations during a single brown bag lunch you could possibly even have one that isn’t related to work. This is a little risky of course, but it can also be fun (I’m sure that everyone would love to hear about arid plants!). It’s also a good way to break any information filters we have.

Provide lunch

When I first started organizing brown bags, I couldn’t work out whether the term brown bag seminars came from people bringing along their own lunch in a brown bag or being provided lunch in a brown bag. But through experience I have found providing a good lunch is a key contributor to a successful brown bag seminar: ‘chimpanzees who share are chimpanzees who care‘. It also provides a good motivator for people to give up their lunch break and come along because who can resist a free lunch, right?

Make sure everyone knows each other

If you’ve got a new team, or people from different areas who don’t know each other, start with a quick icebreaker where you go around the room and get everyone to introduce themselves. I usually follow the format of ‘name’, ‘role’, ‘a fun fact’ and another random tidbit such as ‘my biggest fear’ or ‘what I’m looking forward to’.

Make sure everyone takes something away

I follow the icebreaker with a question to the audience: ‘what do you expect to get out of today’s session?’ I bring a bunch of Post-it notes and sharpies along and get each person to write a few things they want to get out of the session and stick them to the wall. Ten minutes before the end of the session the presenter reads out each objective and confirms each one has been met with whomever wrote it. If there’s something that wasn’t covered, it can be discussed, or it could even become the topic of a future brown bag.

I’ve seen lots of great objectives written from things like “learn more about automated mobile testing” to “have a nice lunch with my colleagues”.

Always leave plenty of time for discussion

The discussion generated by a brown bag seminar is as important as the content. Make sure you leave plenty of time to discuss what is being presented.


I thoroughly recommend brown bag lunches as an effective information sharing and team bonding technique, and if you get them right people can really enjoy them and look forward to them.

What’s your experience been with brown bag lunches? Good? Bad? Do you have any tips yourself?