Fixing bugs in production: is it that expensive any more?

You’ve most likely seen a variant of this chart before:

bug fix costs

I hadn’t seen it for a while, until yesterday, but it’s an old favourite of test managers/test consultants to justify a lot of testing before releasing to production.

But I question whether it’s that accurate anymore.

Sure, in the good old days of having a production release once or twice a year it cost a large order of magnitude more to fix a bug in production, but does it really cost that much more in the present age of continuous delivery/continuous deployment where we release into production every fortnight/week/day?

If the timeline on the chart above is a year then of course bugs will cost more to fix, because presumably, if the project took a year to start with, you don’t have a very rapid software development process. And there’s more likely to be requirements ‘bugs’ in production because an awful lot happened in the year that the requirement was being developed. Hence along came agile with its smaller iterations and frequent releases.

Mission critical systems aside, most web or other software applications we build today can be easily repaired.

Big waterfall projects, like building a plane, are bound to fail. The Boeing 787 Dreamliner was an epic fail. Not only was it five delays and many years late, it had two major lithium ion battery faults in its first 52,000 hours of flying which caused months of grounding and has no doubt affected future sales, causing millions of dollars in damages. But it seems to have been well tested:

“To evaluate the effect of cell venting resulting from an internal short circuit, Boeing performed testing that involved puncturing a cell with a nail to induce an internal short circuit. This test resulted in cell venting with smoke but no fire. In addition, to assess the likelihood of occurrence of cell venting, Boeing acquired information from other companies about their experience using similar lithium-ion battery cells. On the basis of this information, Boeing assessed that the likelihood of occurrence of cell venting would be about one in 10 million flight hours.”

NTSB Interim Report DCA13IA037 pp.32-33

After months of grounding, retesting, and completely redesigning the battery system, the cause of the original battery failures are still unknown. If they can’t work out what the problem is after it has occured twice in production, it’s not likely it could have been found or resolved in initial testing.

But most of us don’t work on such mission critical systems anyway.

And production fixes can be very easy.

Take this very different example; I provide support for a production script that uploads a bunch of files to a server. There was a recent issue where a file-name had an apostrophe in it which meant this file was skipped when it should have been uploaded.

Upon finding out about the problem I immediately looked at my unit tests. Did I have a unit test with a file name with an apostrophe? No I didn’t. I wrote a quick unit test – it failed: as expected. I made a quick change to the regular expression constant that matches file names to include an apostrophe, I reran the unit test which passed. Yippee. I quickly reran all the other unit and integration tests and all passed, meaning I could confidently package and release the script. All of this was done in a few minutes.

I could have possibly prevented this happening by doing more thorough testing to begin with, but I am pretty sure that would have taken more effort than it did for me to fix the production bug, by writing a test for it and repackaging it. So for me it wasn’t an increase in cost whatsoever to find that bug ‘late’.

Unless you’re working on mission critical software, shipping some bugs into production is almost always better than shipping no software at all. If you work on very small, frequent deployments into production, the cost of fixing bugs once they have gone live will only be marginally greater than trying to find every bug before you ship. The longer your spend making sure your requirements are 100% correct and everything is 100% tested, ironically, your software is more likely to be out of date, and hence incorrect, once you finally go live.

Internet Other

Leanpub pricing analysis

I am nearly at the point of publishing my eBook on Leanpub which has raised the question about how much I will price it at (it will be available free online in HTML form). There are two price points on Leanpub (minimum and suggested) and since I don’t go into these decisions lightly I thought I would do some analysis around existing titles to see what they set the price-points at.

One caveat is that I completely understand that it’s just as important to have a good product as worrying about its pricing as pricing alone doesn’t make a book good.

I wrote a quick watir-webdriver script (at the end of this post and online) to scrape the top 100 all time grossing and best-selling (volume) books off Leanpub and some data about each. One data point I would have loved to have seen on each book page was page count so I could correlate page count to price, but sadly it’s missing.

I put this into a google spreadsheet and did some analysis

Leanpub pricing analysis

What I can read about Leanpub pricing is:

  • Obviously, higher selling books are more likely to be available for free
  • The number of books with the same minimum and suggested prices are the roughly the same across the lists at about one quarter
  • The average non-free minimum price is slightly less for higher selling books, but roughly $12-$14
  • The average non-free suggested price is much higher for higher grossing books at roughly $19-$22, compared to $12-$14 for higher selling books

I did some further analysis of the raw data and a particular title stood out to me: ‘How to Do What You Love & Earn What You’re Worth as a Programmer‘. What made this stand out was it’s ranking at number 2 on number of copies sold (8191 copies), but just number 28 on gross revenue. Based upon the minimum price of $9.99, this translates into minimum revenue of $81,828.09. But this book is at number 28 on the earnings list, meaning that possibly 27 other books each earned more than $81,828.09. But there’s no a single book that has earned more than this book that has a minimum price times copies (or suggested price times copies for that matter), that is higher than this book. Which is very odd.

I know of two possible explanations, but I don’t know if either is true.

One is that this book once was free, or a lot cheaper than it’s current minimum price, which explains why it has sold many copies but (comparatively) grossed so little.

The second is that people who buy books on Leanpub pay a lot more than minimum or even suggested prices.

I think the first explanation is true but hope that it’s the second.

Here’s my script:

$: << File.dirname(__FILE__)+'/lib'

require 'book_list'
require 'watir-webdriver'

profile =
profile['permissions.default.image'] = 2 #no images
browser = :firefox, :profile => profile

book_list = 'bestsellers', ''

browser.goto book_list.url
browser.links(:class => 'book-link').each_with_index do |link, index|
  book_list.add_book link.text, link.href, index + 1

book_list.books.each do |book|
  browser.goto book.url
  book.min_price = browser.element(:css => 'span[itemprop="lowPrice"]').text
  book.suggested_price =  browser.element(:css => 'span[itemprop="highPrice"]').text
  if browser.strong(:text => /^This book has (\d+) readers!$/).exists?
    book.copies_sold = browser.strong(:text => /^This book has (\d+) readers!$/).text.match(/^This book has (\d+) readers!$/).captures.first

book_list.to_csv 'copies.csv'

require 'book'

class BookList

  attr_reader :name, :books, :url

  def initialize name, url
    @name = name
    @url = url
    @books = []

  def add_book name, url, rank
    @books <<, url, rank)

  def to_csv filename, 'w') do |file|
      @books.each do |book|
        file.puts book.to_csv_string
class Book

  attr_reader :name, :url, :rank
  attr_accessor :min_price, :suggested_price, :copies_sold

  def initialize name, url, rank
    @name = name
    @url = url
    @rank = rank

  def to_csv_string