Evolving Beyond the Code Coverage Percentage Debate: Part 1 of 2

Magnifying glass and binary code. Conceptual photo - analyzing code.

What is Code Coverage?

According to Wikipedia, code coverage (a.k.a. test coverage), is defined as a measure used to describe the degree to which the source code of a program is tested by a particular test suite”. A less precise explanation in layman’s terms would be to say it’s the percentage of source code exercised by tests. In my experience, it’s a concept that the majority of seasoned programmers have come to agree upon as a tried and true practice in developing quality software.

What’s the debate?

As certain as we programmers are about the necessity to measure code coverage, we seem equally uncertain in our ability to agree upon a standard as to what percentage constitutes adequate code coverage. For two examples, see the vast array of answers to these Stack Overflow and Stack Exchange questions.

  1. What is a reasonable code coverage % for unit tests (and why)?
  2. How much Code Coverage is “enough”?

While at first it may seem odd to be so united in the understood need for the practice, and at the same time divided on its application, it begins to make sense as you come to understand some of the experiences that have fueled the debate. Consider a team that is new to code coverage metrics that establishes a minimum code coverage percentage of 80%. It is all but certain that after some arbitrarily short period of time a bug will surface, in frustration to the team manager. The manager, while logical (though unrealistic) and irrational (though realistic), will object to the occurrence of the bug, questioning the team’s choice of the code coverage percentage.

In extreme cases this invariably results in a modification of the goal from 80 to 100%. However, as this adjustment plays out, it’s observed to cause a significant drop in productivity as the programmers struggle to exhaustively test very-difficult-to-test code, which ultimately results in further management frustration. Once again the coverage limit is modified and the cycle continues perpetually, like the search for the Holy Grail, destined never to end and continually fueling the great code coverage percentage debate.

It’s pretty easy to see why all this matters. When the code coverage expectations are too low, the outcomes are meaningless, and when they are too high the tests become difficult to write and achieve. All the while, the tail (code coverage percentage) is wagging the dog (quality software). Everyone is fixated on code coverage instead of writing quality tests.

Thankfully all is not lost. While this debate has long existed, it has begun to show signs of change in recent years. For example, both Main Fowler and Mark Seemann have written the following respective articles, calling out the great debate as an inappropriate focus on a means as if it were an end.

Or, take for example the highest voted answer on the aforementioned Stack Overflow post, which eloquently concludes with the point that one should not focus on the coverage percentage per se”. It seems to be sinking in that we should not be consumed by this debate. A summation of these well-articulated points might be to say that code coverage percentage is irrelevant compared to the relevance of test quality. For all intents and purposes these authors have effectively settled the code coverage debate for me. At least mostly.

The problem I’m left with is how to apply this truism, which I have come to believe and cannot deny. The allure to the code coverage percentage is its quantifiable and measurable nature. Engineers are, after all, naturally predisposed to such things. Yet the aspiration to write genuine quality tests tends to require a more intuitive-driven, artistic approach than one based on science. These are two seemingly juxtaposed ideals and yet good teams find a way to balance them.

In my career, the majority of successful teams I’ve come across, in this regard, have more or less stumbled into a proficiency of balancing these objectives based on trial and error, or accumulated experience (if there is even a difference). While that approach has certain merits, it can leave one wanting. Part of my responsibility as a Lead Engineer at Saturn Systems is to create processes and systems that intentionally achieve this balance. To this end I’ve spent a good portion of the last two years refining a process to achieve this result. And, while like any process, its application to other scenarios will vary based on circumstances, it is governed by a single widely applicable core principal called code coverage accountability. Which I propose is a healthy evolutionary step forward, above and away from the code coverage percentage debate.

To Be Continued…

In my follow-up post, I’ll define code coverage accountability and illustrate how you can use it. Whether you’re an executive, manager or a programmer, I’ll share how you can leave the code coverage percentage debate behind and turn your team’s attention where it needs to be, on writing worthwhile tests and ultimately producing quality software.

About Aaron Osterwyk

2 comments on “Evolving Beyond the Code Coverage Percentage Debate: Part 1 of 2

  1. Avatar
    Bob Buske on

    I look forward to part II

  2. Avatar
    Anonymous on

    This really is the fitting weblog for anybody who desires to locate out about this subject. You notice a whole lot its almost onerous to argue with you (not that I truly would want to, HaHa). You undoubtedly put a brand new spin on a topic that’s been written about for years. Good stuff, just excellent!


Leave a Reply

Your email address will not be published.