Coverage in Scope

In a software project, you can come across quite a few kinds of coverage. There’s Line coverage, Decision coverage, Use Case coverage, Branch coverage,… It makes sense to talk in terms of percentages when addressing these.

The above examples are descriptions of exhaustive, finite lists. The lists may not be complete, but somehow, by someone, somewhere these lists are conceived and decided upon.

They will grow in time, as new information becomes apparent and new important issues are uncovered.

This is important: the changes to these lists are manageable, even if they don’t seem to be at first. You can prioritize, as some of the items are more important than others. You start from what is apparent and deal with what comes your way. Use cases are added, lines are added, decisions are clarified, branches are checked,… It’s a work in progress and eventually the project will reach a point where change isn’t relevant anymore. Coping with that constant change is being agile.

Coverage Estimation

Suppose you have 1.000.000 lines of code. There are already tool-enabled checks in place that support 90% line coverage. Could you estimate how much time it would take to reach 95% line coverage?

It’s a straightforward question but a herculean task to gather information to offer as precisely as possible an estimate. Even then the solution is ambiguous  and the result uncertain. But you can give an informed estimate, if you desire. An educated guess, calculating in some margin for error and unforeseen difficulties.

Possible parameters  for line coverage estimation:

• # of lines
• Complexity of lines
• Disparity of lines

In order to agree upon an estimation, the influencing parameters must be agreed upon. To agree upon a parameter, it must be known. Estimations are more than guesses, they are an agreement between two parties. One side calculates in the time for his or her planning, the other wants to get results within the borders of the estimate.

Test Coverage

By definition, testing is infinite. You have knowledge of only so many influencing parameters and there are many more in hiding.
Test Coverage, in percentages, is an absurd concept.

Consider the following proposition:

Will you work for me? It involves pushing buttons.

I won’t tell you how long, how many times, what happens when buttons are pushed, where you’ll be doing this, in which environment or whether it involves trained wild-life.

I’m guessing you find this absurd. Estimating Test Coverage is just as preposterous as you can’t put your finger on the infinite parameters flying around.
People, by nature, are unwilling to agree to a set of uncertainties.

However, if you, or anyone on your team is still hell-bent on getting estimates for testing, there are possibilities.

• You both could agree on a set of known parameters. Keeping in mind that new ways will open up and temper with your estimates.
• Or both parties could outline a model of the feature and work from there.
• You could agree on spending X time on the feature and nothing more.
• An initial hour or two could be estimated for a first probing session on which a clearer ‘estimate’ could be given.
• …

All of the above are severely lacking in either meaningful coverage or estimation. There are many workarounds and different possibilities, but they are usually artificial and bound to escalate.

Estimates seem harmless at first, but:
They have an impact on people’s planning, frustrating them as estimates are shattered. They can hit a dent in your credibility as a tester.
They can severely implicate your freedom as a tester.

You need to be able to push where it hurts, being told where and how long you can test restricts this.