Automated Testing Test Automation

Mobile apps still need automated tests

Jonathan Rasmusson recently wrote what I consider to be quite a contentious blog post about iOS application development titled “It’s not about the unit tests”.

“…imagine my surprise when I entered a community responsible for some of the worlds most loved mobile applications, only to discover they don’t unit test. Even more disturbing, they seem to be getting away with it!”

Whilst I agree with the general theme of the blog post which is change your mind, challenge assumptions:

“All I can say is to keep growing sometimes we need to challenge our most cherished assumptions. It doesn’t always feel good, but that’s how we grow, gain experience, and turn knowledge into wisdom.”

“The second you think you’ve got it all figured out you’ve stopped living.”

I don’t agree with the content.

Jonathan’s basic premise is that you can get away with little or no unit testing for your iOS application for a number of reasons including developing for a smaller screen size, no legacy, one language, visual development and developing on a mature platform. But the real reason that iOS get away with it is by caring.

“These people simply cared more about their craft, and what they were doing, than their contemporaries. They ‘out cared’ the competition. And that is what I see in the iOS community.”

But in writing this post, I believe he missed two critical factors when deciding whether to have automated tests for your iOS app.

iOS users are unforgiving

If you accidentally release an app with a bug, see how quickly you’ll start getting one star reviews and nasty comments in the App Store. See how quickly new users will uninstall your app and never use it again.

The App Store approval process is not capable of supporting quick bug fixes

Releasing a new version of your app that fixes a critical bug may take you 2 minutes (you don’t even need to fix a broken test or write a new test for it!) but it then takes Apple 5-10 business days to release it to your users. This doesn’t stop the one star reviews and comments destroying your reputation in the meantime.

Case in Point: Lasoo iPhone app

I love the Lasoo iPhone app, because it allows me to read store catalogs on my phone (I live in an apartment block and we don’t get them delivered). Recently I upgraded the app and then tried to use it but it wouldn’t even start. I tried the usual close/reopen, delete/reinstall but still nothing. I then checked the app store:

Lasoo iPhone app reviews
Lasoo iPhone app reviews

Oh boy, hundreds of one star reviews within a couple of days: the app is stuffed! I then checked twitter to make sure they knew it was broken, and to my surprise they’d fixed it immediately but were waiting for Apple to approve the fix.

I can’t speculate on whether Lasoo care or not about their app, but imagine for a second if they had just one automated test, one automated test that launched the app to make sure it worked, and it was run every time a change, no matter how small, was made. That one automated test would have saved them from hundreds of one star reviews and having to apologize to customers on twitter whilst they waited for Apple to approve the fix.

Which raises another point:

“[Apple] curate and block apps that don’t meet certain quality or standards.”

The Lasoo app was so broken it wouldn’t even start, so how did it get through Apple’s approval process for certain quality or standards?

Just caring isn’t enough to protect you from introducing bugs

We all make mistakes, even if we care. That’s why we have automated tests, to catch those mistakes.

Not having automated tests is a bit like having unprotected sex. You can probably get away with it forever if you’re careful, but the consequences of getting it wrong are pretty dramatic. And just because you can get away with it doesn’t mean that other people will be able to.

8 replies on “Mobile apps still need automated tests”

The main point I took away from Jonathan’s article was that caring about quality is more valuable than just writing unit tests. I think he’s right about that, but more to the point, if you do really care about the quality of your product, then you’ll probably *want* to automate some tests to help you detect when something’s gone wrong.

Caring about quality isn’t just about getting up in the middle of the night to tweak a colour scheme. It’s not just about getting excited about the latest groundbreaking amazeballs superwidgets you’re adding to your product. It’s also about making sure you’re not wrecking up the place whenever you roll out a shiny new feature. You can have all the pretty buttons and delicate UX touches in the world but it won’t help your app store rating if your users can’t login anymore.

I disagree. The reason people “get away with it” — not testing mobile apps is because the users are very forgiving. Every single mobile app is for casual use. They pay (at most) $2 per app — and most are free. Almost all are for amusement only, and those that have a real purpose are simply supplemental views to web apps.

You might get a 1 star rating and watch your iFart revenue plummet in a single week because of a bug. But you might get the same result from a thousand other factors, including a negative rating from a user who doesn’t yet understand how to use their portable electronic toy.

When casual amusement is the name of the game, popularity is an ephemeral trait, and frankly, one that is hardly rewarded by quality. That’s why mobile apps can get away without testing.

I’m not sure if you’re joking or not Aaron but I seriously hope you are not serious.

Mobile apps are an increasingly important channel for organizations to interact with their customers – done poorly can be disastrous to their reputation.

I don’t use a single type of app you mentioned but use many corporate apps as a way to do existing activities. Think mobile banking, ordering pizza, booking hotels etc. If one of these apps introduces bugs they seriously affect my, and others, abilities to use that company.

Instagram was an iPhone app that recently sold to Facebook for one billion dollars. Could they afford to have regression defects with millions of daily users?

Hi Alister,

Thank you for engaging in this conversation. I feel it’s an important one, and I think the work you and others do on tools like Watin/Watir are invaluable contributions to the community. Thank you for that.

Regarding the article you are referring to, my intention wasn’t to say don’t unit test, or that automated tests aren’t a good idea for iOS projects. They clearly are for those areas that lend themselves to that form of testing.

It was more that by not being able to unit test ‘everything’ like I could in Ruby/.NET/Java I had to do something else (like use automated acceptance testing frameworks like Frank).

It also made me realize that there was a lot more to building a quality app that just the unit tests. And I did want to have a conversation about ‘caring’ (I feel it gets over looked).

In someways it feels like Javascript 5-7 years ago when nobody unit tested Javascript. Now more and more people are with tools like Jasmine. I see iOS going through a similar transformation.

So thank you for your article. I am glad there is push back from good folks like you saying: “Hold on – what is this guy saying!”.

Automated testing is important.

Cheers – Jonathan

From the sound of it, it seems like the makers of Lasoo weren’t running any tests at all on their code before they released it, automated or manual. Did not a single developer think to run the dev code on their own iOS device (e.g. via TestFlight) before submitting it to Apple?

Also, do Apple not run any automation tests before releasing it into the wild? One of the points in the Apple App Store submission guidelines states that apps “that exhibit bugs will be rejected.” Well, the app not starting up at all is a pretty damn big bug all on its own!

You imply that Lasoo’s bug could have been caught with a unit test, but you don’t know that. It could have been something with integration. It could be a case that they hadn’t come across in their test environment that a unit test would not have caught because unit tests can only test for situations that you know about.

Having written several apps, I’ve seen the majority of bugs are with integration with a back-end service and in unforseen edge cases (which are many times hard to debug because it’s hard to reproduce them, let alone unit test them).

I’m not saying don’t unit test, but I am saying there is something about ObjC/UIKit development that lets developers “get away” with fewer tests, even if it’s because the majority of tests that you have to do is in the Simulator playing with the app.

Here’s an oldie-but-goodie by Mr. Will Shipley on how much he hates unit testing:

Unit testing is a technique; a tool in your toolbox. That’s how you should think of it. Telling people that they should unit test without any other context turns unit testing into dogma, and you should always be immediately suspicious of dogma. You can write quality software without unit tests. And you can also write crappy software with unit tests. It’s the craftsman, not the tool.

Leave a Reply

Your email address will not be published.