Page 1 of 1

The Cost of Bugs

Posted: Thu Jun 11, 2009 3:35 pm
by McDanielCA
The Cost of Bugs was originally posted on the main page of LDSTech. It was written by Christian Hargraves.


-------------------------------------

Two things contribute to unhappy customers: bugs and late delivery. A bug is generally referred to as a feature in the application that does not work according to the customer’s expectations. This can be due to an unspecified or misunderstood requirement or just a mistake in the development of the software. Either way, bugs not only frustrate the customer, they considerably expand the project’s cost and timeline.
Making an effort to catch bugs at the earliest point in the life cycle will result in a higher return on investment (ROI). The cost of fixing a bug differs depending on the stage of development it is caught in.
  • Requirements Stage: Bugs caught in the requirements writing stage simply cost the time to rewrite the requirement. Time spent in this stage is usually constant.
  • Coding Stage: Bugs caught here require developer hours. Time varies but is considerably less than fixing a bug found by someone else. When a bug is found during this stage, the developer discovers it, already understands the problem, and often knows how to fix it.
  • Integration Stage: Bugs caught here require developer and other system engineer hours. Time is usually at least twice as much, since the problem occurs at a higher level and there is a need to figure out which exact code or configuration is wrong.
  • Testing Stage: Bugs caught here require developer, system engineer, PM, and QA hours. The process is much larger than before. Now things need to be tracked and prioritized. This now requires finding reproduction steps, submitting a bug, prioritizing the bug, meeting with developers, fixing the bug, pushing the fix to the test environment, verifying the fix, and tracking the changes of the bug in the system.
  • Production Stage: Bugs caught here require developer, support, system engineer, PM, customer, and QA hours. This process always involves all of the roles. It requires more planning and more prioritizing than in the testing stage. Usually a phone call comes to support, and they decide if it's a bug or if it’s working as designed. The customer is notified, the PM is contacted, and then the process in the testing stage is followed.
As you can see, finding a bug later in the life cycle of the software costs more. The greatest ROI is to find bugs early. Doing all that can be done to collaborate with the customer will greatly help reduce development against buggy requirements. We've all played the operator game, and requirements can easily be watered down or even misunderstood once the project gets to the tester and developer.

Because the majority of bugs, by far, are introduced in the code, developers should place very high priority on testing the code themselves. Most of the bugs found in the coding and requirements gathering phases can be substantially reduced by performing careful agile methods such as paper-prototyping, defining acceptance criteria, and doing test-driven development.

Requirements Gathering

Producing, on paper, what the customer envisions can often prove difficult. There are several ways of doing this, but my favorite is to give the customer something rough that looks similar to the desired product and then ask for feedback (see paper prototyping). Use a rough-looking sample so the customer feels more open to give input and worries less about the time already spent. Ask for a list of criteria that will help define what the customer sees as “done.” This is known as acceptance criteria. It can greatly help a developer write more focused code and a tester better understand how the application is supposed to work.

Unit Testing

Unit testing is basically a developer test that proves the unit—the smallest testable part of the code—of code behaves as expected. This includes testing the code against unexpected conditions as well as “pass or fail” conditions. (Unit testing can be accomplished by manually stepping through the code via a debugger, but is generally referred to as writing an automated test.) One of the challenges in writing good unit tests is that it often requires a different mindset from developing. The goal of writing code is to automate something, but the goal of testing code is to prove that something actually works.

Test-Driven Development

Test-Driven Development (TDD) is a technique in which code is not written without first setting expectations in an automated unit test. Often unit tests are written to prove that existing code works as expected. A problem with this approach is that the code being tested is read by the person writing the unit test. This is usually done by assuming that the code being read behaves appropriately. In other words, some of the successful conditions are tested while other conditions are usually assumed. Another problem is that code is usually not written to be testable.

With TDD, the only assumptions generally made are that the acceptance criteria are correct. It is also easier to put the testing hat on and then replace it with the development hat, thus producing higher quality tests. The code ends up being very easy to test since the tests are written first. There are several studies that suggest that TDD can reduce bug count from 40% to 90% compared to unit tests without TDD. For more information read a recent report about TDD by IBM and Microsoft and The Cost of Test Driven Development.

Summary

It’s easy to forget the original goal of delivering something the customer wants when going through all the processes that are often included in software development. I have found that embracing full collaboration with the customer and practicing the above discussed methods are great steps towards reducing the overall cost and increasing the customer’s satisfaction with a product.


Christian Hargraves is a principal engineer for the Church.