Categories
Business Driven Specification by Example

The color of acceptance is gray

James Shore recently wrote some brillant words about acceptance testing:

I think “acceptance” is actually a nuanced problem that is fuzzy, social, and negotiable. Using tests to mediate this problem is a bad idea, in my opinion. I’d rather see “acceptance” be done through face-to-face conversations before, after, and during development of code, centering around whiteboard sketches (earlier) and manual demonstrations (later) rather than automated tests.

To rephrase: “acceptance” should be a conversation, and it’s one that we should allow to grow and change as the customer sees the software and refines her understanding of what she wants. Testing is too limited, and too rigid. Asking customers to read and write acceptance tests is a poor use of their time, skill, and inclinations.

This is pretty much where my head is at right now around automating acceptance tests. Automated tests are black and white, acceptance is gray.

“The color of truth is gray.”
~ André Gide

I prefer to have a handful of end to end automated functional tests that cover the typical journey of a user than a large set of acceptance tests constantly in a state of flux as the system is being developed and acceptance is being defined and changed.

We need to take feedback from the customer that we are building the right thing and ensure our automated tests model this, not make them responsible for specifying the actual tests.

Categories
Automated Testing Test Automation

Mobile apps still need automated tests

Jonathan Rasmusson recently wrote what I consider to be quite a contentious blog post about iOS application development titled “It’s not about the unit tests”.

“…imagine my surprise when I entered a community responsible for some of the worlds most loved mobile applications, only to discover they don’t unit test. Even more disturbing, they seem to be getting away with it!”

Whilst I agree with the general theme of the blog post which is change your mind, challenge assumptions:

“All I can say is to keep growing sometimes we need to challenge our most cherished assumptions. It doesn’t always feel good, but that’s how we grow, gain experience, and turn knowledge into wisdom.”

“The second you think you’ve got it all figured out you’ve stopped living.”

I don’t agree with the content.

Jonathan’s basic premise is that you can get away with little or no unit testing for your iOS application for a number of reasons including developing for a smaller screen size, no legacy, one language, visual development and developing on a mature platform. But the real reason that iOS get away with it is by caring.

“These people simply cared more about their craft, and what they were doing, than their contemporaries. They ‘out cared’ the competition. And that is what I see in the iOS community.”

But in writing this post, I believe he missed two critical factors when deciding whether to have automated tests for your iOS app.

iOS users are unforgiving

If you accidentally release an app with a bug, see how quickly you’ll start getting one star reviews and nasty comments in the App Store. See how quickly new users will uninstall your app and never use it again.

The App Store approval process is not capable of supporting quick bug fixes

Releasing a new version of your app that fixes a critical bug may take you 2 minutes (you don’t even need to fix a broken test or write a new test for it!) but it then takes Apple 5-10 business days to release it to your users. This doesn’t stop the one star reviews and comments destroying your reputation in the meantime.

Case in Point: Lasoo iPhone app

I love the Lasoo iPhone app, because it allows me to read store catalogs on my phone (I live in an apartment block and we don’t get them delivered). Recently I upgraded the app and then tried to use it but it wouldn’t even start. I tried the usual close/reopen, delete/reinstall but still nothing. I then checked the app store:

Lasoo iPhone app reviews
Lasoo iPhone app reviews

Oh boy, hundreds of one star reviews within a couple of days: the app is stuffed! I then checked twitter to make sure they knew it was broken, and to my surprise they’d fixed it immediately but were waiting for Apple to approve the fix.

I can’t speculate on whether Lasoo care or not about their app, but imagine for a second if they had just one automated test, one automated test that launched the app to make sure it worked, and it was run every time a change, no matter how small, was made. That one automated test would have saved them from hundreds of one star reviews and having to apologize to customers on twitter whilst they waited for Apple to approve the fix.

Which raises another point:

“[Apple] curate and block apps that don’t meet certain quality or standards.”

The Lasoo app was so broken it wouldn’t even start, so how did it get through Apple’s approval process for certain quality or standards?

Just caring isn’t enough to protect you from introducing bugs

We all make mistakes, even if we care. That’s why we have automated tests, to catch those mistakes.

Not having automated tests is a bit like having unprotected sex. You can probably get away with it forever if you’re careful, but the consequences of getting it wrong are pretty dramatic. And just because you can get away with it doesn’t mean that other people will be able to.

Categories
Random

Why hot-desking is a bad idea

Hot-desking (aka hotelling) in open plan offices seems to be growing in popularity, and why not, since it seems to make sense from an financial and collaborative viewpoint. But I am of the belief that it’s actually a bad idea. Here’s why:

It’s unhygienic: Most hot-desk arrangements I have seen involve a thin client PC (eg. Windows Thin PC) on each hot-desk which is used to provide a way for a current user to access their computer session wherever they log in. Since the average keyboard has sixty times more germs than a toilet seat, I actually feel disgusted every time I sit down at a hot-desk and am expecting to use the filthy keyboard all day (much like if someone asked me to set up my computer on a toilet seat).

It’s confusing: Not knowing where someone is each day is particularly confusing, especially for new starters who are getting to know people. Sure, IM solves this situation to some degree, but I’ve spent time roaming the office floors looking for people who I don’t know where they are sitting today.

It doesn’t actually work: Even though organizations have hot-desking so they can cut down on the number of desks and get people to sit together, I have found that people still get established in certain desks as they know they’ll be working there for some time, and they can’t be bothered packing their things up and setting up each day in a new desk. The only time these people seem to get displaced is when they go on leave and someone else has to sit at their ‘hot desk’ amongst the stuff they have left behind. The lack of dedicated space has actually been shown to make employees feel isolated and teamless, amongst other things:

A study released by the University of Sheffield in the UK shows it diminishes the connection between colleagues, and the scattered locations make it difficult for people to communicate with each other.

It continues the obsession with open plan: I seriously don’t like open plan offices and don’t understand why software development workplaces continue to foster them. They encourage constant interruption and distraction which inhibits productivity. Paul Graham explained it best, back in 2004:

After software, the most important tool to a hacker is probably his office. Big companies think the function of office space is to express rank. But hackers use their offices for more than that: they use their office as a place to think in. And if you’re a technology company, their thoughts are your product. So making hackers work in a noisy, distracting environment is like having a paint factory where the air is full of soot.

So what is the answer?

Progressive software companies like Campaign Monitor provide dedicated offices to each team member and a large kitchen table to ensure everyone eats lunch together every day.

I like the idea of providing a dedicated office to each team member to work quietly without disruption, and separate (sound proofed) open areas for team collaboration/socializing. The open areas must be easily booked or can used for an impromptu discussion, and must be clean and connected, to maximise productivity.

As Kelly Executive Recruitment GM Ray Fleming says:

Productivity and motivation are maximised when employees have their own workspace. It helps them to feel part of the organisation and solidifies their position in the team, and businesses need to keep this in mind. Businesses also need to be aware that shifting to hot-desking just to save money may drive some employees to look elsewhere for employment.

Categories
Random

The sliding scale of client, stakeholder and management engagement

It’s a beautiful colour, grey, and as soon as you put it into black and white it gets lost.
~ Ian Thorpe

I’ve been thinking about client engagement a lot lately: how much engagement should you aim for when working for someone? Whether that someone be a client, a sponsor of your project, or your boss.

I’ve come to realize that engagement isn’t black and white, engaged or not. It’s a sliding scale, and like most things in life, the sweet spot is somewhere in the middle, not at the outer ends.

On the far right you have a client who is too engaged, to the point of being a micro-manager and a hindrance on your progress. This isn’t ideal, but you can manage the micro-management and it still is possible to get a good deal done in this circumstance by using the high level of engagement to your advantage (to clear roadblocks for example).

On the far left is what I consider to be the most dangerous spot: a client who isn’t engaged whatsoever. The benefit is that you will have freedom to do whatever you want, the obvious downside is you’re inevitably going to fail because you won’t know what your client actually wants, and it’ll eventually unravel that you haven’t delivered. Whilst it’s easy to become complacent in this situation, it should be avoided at all costs: raise the red flag early and often, and if you can’t engage your client, get a new client, or leave!

I also believe this same scale applies to child development. As the father of two young boys I spend a fair amount of my leisure time in playgrounds, where I observe other parenting styles and interactions. On the far right of this scale fit the parents who hold their kids (or make them wear helmets) as they try to climb the jungle gym. On the far left are the parents who read the paper or play with their latest iPhone 5, oblivious to the amazing physical and mental development they are missing meters away. In the sweet spot are those amazing parents whose children are confident to independently learn, whilst knowing their parents are always there if they need help or something goes wrong.

Summary

If you can aim for somewhere in the middle of my scale, where your client is engaged enough to know broadly what they want, but not engaged too much for them to tell you what to do and how to do it, that’s the sweet spot: that’s where you should be.

What are your thoughts? Do you like one particular end or like living in the grey spots like me?

Categories
Ruby on Rails

RailsMelon: how to set up Rails 3 Reference Data using Seeds

Preamble

I’ve started learning Ruby on Rails by building a web application for my own startup (sorry but it’s still too early to release full details of what it is).

Like all the things that I do, I like to share my learnings. As I will be doing more Rails stuff than Watir stuff, I was thinking of starting a new blog called RailsMelon, but I changed my mind and have decided to blog about Rails things here along side Watir and Testing things. I will prefix any Rails related posts with RailsMelon so those of you who aren’t interested in such things can safely ignore them. If the Rails content becomes too strong or I receive too many complaints, I will move the content to a separate RailsMelon blog.

Blog All The Things

Rails 3 Reference Data using Seeds

I’ve been trying to work out the best way to load reference data in a Rails 3 application. The inbuilt seeding mechanism (seeds.rb run via rake db:seed or db:setup) seemed the obvious choice to begin with but then I started seeing examples of people using seeds.rb to also load test data which confused me. When I refer to ‘reference data’ I am referring to data that needs to exist for your application to work, whether in development, test and then ultimately production. Test data is different in that I would only use it for testing, and I would never want to load it into production.

I decided upon using seeds.rb (and rake db:seed) but only to specify reference data I will use in production. Test data will be loaded in other ways (which I will discuss in a future post).

I was confused initially because the Agile Development with Rails book actually says you should delete all the data before then creating it when using seeds.

Product.delete_all
Product.create(title: 'Programming Ruby 1.9')

If I do this for reference data, it will continually delete/create the reference data and cause problems with other models that have related data.

To avoid this, I thought I could check to see if the object already exists, and only create if it didn’t.

Product.create(title: 'Programming Ruby 1.9') unless Product.find_by_title('Programming Ruby 1.9')

This is slightly better, but I then realized that if you have a uniqueness constraint on a property of your object, then you don’t even need to check that the object exists, because it will only create it if it’s not already there (but if you don’t have a uniqueness constraint it will create your object every time).

Product.create(title: 'Programming Ruby 1.9')

If you ran this on a blank database ten times, you would only get one ‘Programming Ruby 1.9’ product because title is unique, and each run of rake db:seed would be successful, which is exactly what we want.

Making it prettier for multiples

With reference data I find there are often lots of items, so instead of calling .create methods line after line, you can store the items in an array or hash and iterate over them:

A simple list of colors:

colors = %w(blue red green orange brown)
colors.each { |color| Color.create name: color}

A more detailed list of users:

users =
    [
      { given_name: 'Admin', surname: 'User', email: 'admin@test.com', password: 'password', password_confirmation: 'password', admin: true },
      { given_name: 'Standard', surname: 'User', email: 'user@test.com', password: 'password', password_confirmation: 'password', admin: false }
    ]
users.each { |user| User.create user }

Summary

I have found that Rails 3 seeds provides an easy way to manage reference data, but it shouldn’t be used to manage test data as well. As reference data is needed in all environments, it should be repeatable and therefore shouldn’t be destructive to existing data, so avoid using statements such as delete_all in your seeds.rb file.

Categories
ruby Selenium Watir-WebDriver WebDriver

The webdriver-user-agent gem now supports random user agents

My webdriver-user-agent gem now supports random user agents. This idea belonged to Christoph Pilka who released the webdriver-user-agent-randomizer gem and suggested that we merge this feature back into the orginal gem.

Well, I have done it and now you can access this functionality like so:

require 'selenium-webdriver'
require 'webdriver-user-agent'
driver = UserAgent.driver(:agent => :random)
driver.execute_script('return navigator.userAgent')
# random agent like "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1"

See README for full details.

Categories
Presentations Random

I used to believe in the tooth fairy

I gave this presentation about changing your mind as a lightning talk in Austin in March.

[slideshow]

If you can do one thing this year, I’d encourage you to change your mind on something and share it.

Categories
Automated Testing Page Objects Watir-WebDriver WebDriver

Roll your own page objects

There seems to be a lot of focus being put into page object ruby gems at the moment. Cheezy has done a fantastic job of the aptly named page-object that supports Watir-Webdriver and Selenium-Webdriver, and then there’s the more recent site_prism (also fantastic) by Nat Ritmeyer that works with Capybara. Before these two came along, I even wrote my own; the now retired watir-page-helper gem.

The premise of these gems is they make it super easy to create page objects for your ruby automated testing projects. But today I want to discuss another crazy idea with you: do you even need a gem at all to do page objects?

Background

I recently refactored some automated tests that Chris McMahon wrote as a potential framework for Wikimedia Foundation (creators of Wikipedia). Chris’s code used Cheezy’s excellent page-object gem so I happily went about my refactoring his code using that gem. Suddenly… I found instead of helping me it started to hinder me. I kept having to refer to page-object user guide I got from Cheezy in Austin to work out how things work. Namely:

  1. How to define elements: as they are different from watir-webdriver (eg. list_item vs li, cell vs td etc.)
  2. How to identify elements: as they are limited to certain supported attributes by element type, unlike watir-webdriver which supports every attribute for all elements
  3. What methods each element provides and what each does: as different elements create different methods with different behaviours, so calling a link element clicks it, whilst a span element returns its text.

The main problem I personally found was that Page-object has essentially created its own DSL to describe elements in page objects, and this DSL is subtly and not so subtly different from the Watir-Webdriver API, so the API I know and love doesn’t work in a lot of places.

An example. There’s a common menu bar on all the Wikimedia sites that displays the logged in user as a link (to the user’s page).

The link is only recognizable by its title attribute, and whilst this is supported by watir-webdriver (it supports any attribute), it is not supported by page-object. The source html looks like:

<a class="new" accesskey="." title="Your user page [ctrl-.]" href="/wiki/User:Alister.scott">Alister.scott</a>

What I would have liked to do was:

  link :logged_in_as, :title => 'Your user page [ctrl-.]'

But, instead, I had to do this (which isn’t very DRY):

  def logged_in_as
    @browser.link(:title => 'Your user page [ctrl-.]').text
  end

I believe essentially what has happened is the page-object, in its neutrality between selenium-webdriver and watir-webdriver, has created its own DSL that is somewhat of a halfway point between the two. This is probably fine for most people starting out, it’s just an API to learn, but for someone like me who has extensive experience with the watir-webdriver API (and loves the power of it), I find it limiting. This is particularly evident when I write a majority of my code using the watir-webdriver API under IRB.

So, I had to take a re-think. Why not roll my own page objects for Wikimedia Foundation?

Roll your own page objects

I recently had a discussion with a colleague/good friend about page objects which went along the lines of “I don’t understand those page object gems because you end up writing a custom page object pattern for each project anyway, as every project/application you work on is different in its own way”. It was one of those aha moments for me.

What I needed was to roll my own Wikimedia page objects.

Taking it back to basics, essentially there are three functions I see a page object pattern provides:

  1. Ability to easily instantiate pages in a consistent manner;
  2. Ability to concisely describe elements on a page, keep it DRY by avoiding repetition of element identifiers (using the underlying driver’s API); and
  3. Ability to provide higher level methods that use the elements to perform user oriented functions.

You can probably notice the helper methods – the magic – that gems like page-object and site_prism provide are missing from my list. This is on purpose, and is because, after lots of thought, I actually don’t find these useful, as they encourage specifications/steps to be too lower level. I would rather a high level method on the page (eg. login) than exposing my username and password fields and a login button.

A Wikimedia Page Model

Taking those things into consideration: this is the page model I came up with for Wikimedia.

Generic Base Page Class

The generic base page class is what everything else extends. It contains the instantiation code common to all pages, and the class methods needed to define elements and methods (more on these later).

Wikimedia Base Page Class

This page class contains elements and methods that are common to all Wikimedia pages. The ‘logged in user’ example above is a good example of something that is the same on every Wikimedia page, whether you’re on Wikipedia, or Wikimedia Commons etc.

Commons & Wikipedia Base Page Classes

These two classes are placeholders for elements and methods are common to a particular site. At the moment with my limited examples, these don’t contain content.

Commons & Wikipedia Page Classes

These are the actual pages that are representations of pages in Wikimedia. These are in separate modules so they are in different namespaces (you can have a Wikipedia::LogonPage and a Commons::LogonPage).

Some example pages:

class Wikipedia::LogoutPage < Wikipedia::BasePage
  page_url "#{Wikipedia::BASE_URL}/w/index.php?title=Special:UserLogout"
  expected_title "Log out - #{Wikipedia::TITLE}"

  element(:main_content_div) { |b| b.div(id: 'mw-content-text' ) }
  value(:main_content) { |p| p.main_content_div.text }
end

Here we can see we define a page_url and expected_title, which are used to instantiate the page.

Next we define an element passing in a block of watir-webdriver code for it, and a value by referencing the element we defined before it. Since these element and value methods execute blocks against self, and the class delegates missing methods to our browser, we can refer to either the browser (shown as b) or the page class (shown as p) in our blocks.

class Commons::LoginPage < Commons::BasePage
  page_url "#{Commons::BASE_URL}/w/index.php?title=Special:UserLogin"
  expected_title "Log in / create account - #{Commons::TITLE}"

  login_elements
  value(:logged_in?) { |p| p.logged_in.exists? }

  def login_with username, password
    username_field.set username
    password_field.set password
    login_button.click
  end
end

In this example, we again define the page_url and expected_title, but we have stored the login_elements with the WikimediaBasePage (as they are the same across all the sites) so we include them by specifying login_elements. We have also defined a login_with method that performs actions on our elements.

There are three available methods to define page elements, values and actions, and these all follow the same format of specifying the method name, and passing in a block of watir-webdriver code.

Calling the page objects from Cucumber step definitions

I chose to use Cucumber for the Wikimedia Foundation framework over Chris’s choice of RSpec as I find it easier to specify end-to-end tests in this way. I find the Cucumber step definitions encourage reuse of steps typically used to set up a test (that are often duplicated in RSpec).

I try to stick to calling the exposed methods, values or actions instead of the elements themselves from my Cucumber steps to ensure I am writing them at a high level. An example step using the page above looks like:

Given /^I am logged into Commons$/ do
  visit Commons::LoginPage do |page|
    page.login_with Commons::USERNAME, Commons::PASSWORD
    page.should be_logged_in
  end
end

The visit and on methods are defined in a Pages module that is mixed into the Cucumber World so these available on all step definitions. As named, the visit method instantiates and visits the page, whereas the on just instantiates it.

module Pages
  def visit page_class, &block
    on page_class, true, &block
  end

  def on page_class, visit=false, &block
    page = page_class.new @browser, visit
    block.call page if block
    page
  end
end

Summary

That’s all there really is the rolling your own page objects. I found this excercise useful as it gives me maximum flexibility and allows me to clearly define pages how I want to define them. I appreciate all the great work that Cheezy and Nat have done on their page object gems, if anything these contain great inspirations on how to roll your own custom page objects most suited to your environment and applications.

You can check out my full code here on Github.

Categories
Automated Testing

Software Testing Ice-Cream Cone

If you came here via a link and you’re looking for this blog post, please check out the Testing Pyramids and Ice-Cream Cones page.