Categories
Ask Me Anything ATDD Business Driven Cucumber Specflow Specification by Example

AMA: adopting a TDD/BDD approach

João Monteiro asks…

I recently joined a small company where I am the only QA and the test automation suite was already written in a given/when/then style but rarely (if not at all) gets read by the rest of the team (product or developers). Any tips on how to mentor the team to adopt a BDD approach? Do you recommend any tools / framework to share the features in a centralised place easily accessible by the rest of the team an not just on the tests repository?

Categories
Android Automated Acceptance Testing Automated Testing Cucumber iOS OSX

The current state of iOS automated functional testing

I’ve been working on an iOS project and have been looking for a suitable automated functional test library to drive the iOS GUI. I must say that automated iOS testing feels like automated web testing did about 10 years ago: lots of different tools taking different approaches, and all approaches quite flaky!

Through research I came across quite a few different tools that support iOS automated functional testing but need to work out which one is best.

Whilst I often advocate writing functional tests in the same language as your codebase, in the case of iOS and Objective C, I think using a more lightweight language like Ruby has its own advantages (I don’t really like Objective C).

The one thing I really dislike with a lot of these iOS functional test tools is how they are married to Cucumber and encourage users to write tests like “I tap button x” and “I scroll through list y”. These types of tests are much harder to read as they express implementation over intention and should be avoided.

I will also be writing tests for an Android app so being able to use the same tool-set is a great advantage to me.

I also personally prefer an approach where I don’t need to modify the core behavior of my app to run tests against it (designing for testability is completely different and vital). Some approaches embed some sort of server to receive automation commands, which the user must then remove later because it uses undocumented Apple APIs which will lead to app store rejection. Being able to run tests against the app that you submit to the app store means you can be more confident you have tested it right.

Finally, the iOS app I am working on using embedded web views to display dynamic content from the server, which can be interacted with, and therefore it is vital that these can be interacted with. This feature is actually very rare in an iOS automation framework.

Here’s the list of iOS functional automation tools I am aware of and how they stack up:

  • Tool: Frank
    • Language: Ruby
    • Test Framework: Also supports OSX apps
    • Supports other mobile plaforms: Also supports OSX apps
    • Approach: Requires you to embed a symbiote server in your app and uses undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO (only via JavaScript calls)
  • Tool: KIF
    • Language: Objective C
    • Test Framework: OCUnit/SenTest
    • Supports other mobile plaforms: NO
    • Approach: Modifies your app to use undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO
  • Tool: Subliminal
    • Language: Objective C
    • Test Framework: OCUnit/SenTest
    • Supports other mobile plaforms: NO
    • Approach: Embeds into your project but uses UIAutomation instead of undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO
  • Tool: Zucchini
    • Language: Custom DSL (CoffeeScript/Ruby)
    • Test Framework: Custom
    • Supports other mobile plaforms:
    • Approach: Generates UIAutomation JavaScript that is executed against your app
    • Deploy same app to store: YES
    • Supports testing web-views?: NO
  • Tool: Calabash
    • Language: Ruby
    • Test Framework: Cucumber
    • Supports other mobile plaforms: Also supports Android Apps
    • Approach: Requires you to embed a server in your app to control it
    • Deploy same app to store: NO
    • Supports testing web-views?: YES
  • Tool: Appium
    • Language: Ruby, C#, Java, JavaScript, Objective C, PHP, Python, Perl, Clojure
    • Test Framework: Agnostic
    • Supports other mobile plaforms: Also supports Android
    • Approach: Uses Instruments to control app using the standard WebDriver WIRE protocol
    • Deploy same app to store: YES
    • Supports testing web-views?: YES
  • Tool: ios-driver
    • Language: Ruby, C#, Java, JavaScript, Objective C, PHP, Python, Perl, Clojure
    • Test Framework: Agnostic
    • Supports other mobile plaforms: NO
    • Approach: Uses Instruments to control app using the standard WebDriver WIRE protocol
    • Deploy same app to store: YES
    • Supports testing web-views?: YES

I’ll let you guess which tool I selected until I write my next blog post about how to get started in that tool.

Categories
ATDD Automated Acceptance Testing Automated Testing Cucumber Specflow

Avoid using case statements in your cucumber/specflow/jbehave step definitions

I quite frequently come across a scenario that looks something like this:

Scenario: Create some animals
  Given I am a zoo keeper
  When I create a giraffe
  And I create a lion
  And I create a pony
  And I create a unicorn
  Then I should have a zoo

and step definitions that implement the When steps with a single step definition:

When /^I create a (\D+)$/ do |animal|
  case animal
    when 'lion'
      create_a_lion()
    when 'giraffe'
      create_a_giraffe()
    when 'pony'
      create_a_pony()
    else
      raise 'Unknown animal'
  end
end

I don’t like having case statements in steps for a number of reasons:

  • For readability and maintainability reasons I try to keep my step definitions as short as possible (usually a couple of lines), and using a case statement violates this principle;
  • Raising an exception to catch invalid usage of the step (in an else clause) replicates what these BDD frameworks already do, provide feedback about unimplemented steps;
  • IDEs that support step auto completion (such as RubyMine & Visual Studio) will not suggest valid steps as they don’t understand how you’ve implemented a case statement; and
  • If used inappropriately (such as our unicorn step), the test will only fail at run-time whereas most IDEs will highlight non-matching steps as you’re writing.

For example, you could change our steps to look something like this:

When /^I create a lion$/ do
  create_a_lion()
end

When /^I create a giraffe$/ do
  create_a_giraffe()
end

When /^I create a pony/ do
  create_a_pony()
end

Even though this is three times as many step definitions, it is actually less code (9 lines compared to 12).

By using this approach it is obvious we can’t currently create a unicorn as RubyMine tells us before we even run our tests. And we don’t need to raise any exceptions myself.

rubymine highlights unimplemented steps

Whilst a lot of people use case statements in steps to reduce the number of steps, it is actually counter intuitive as it means you have more code to do so, and the outcome is less usable when writing scenarios. So, please avoid putting case statements in your step definitions.