Unit test code coverage resolution

  • 2020-06-19 11:03:54
  • OfStack

preface

When doing unit tests, code coverage is often used as a measure of how well a test is being tested, or even as a measure of how well a test is being done, such as code coverage must be 80% or 90%. As a result, testers work hard to design case coverage code. In terms of code coverage, there are pros and cons. In this article, we discuss the code coverage and welcome your comments.

First, let's look at what we call "code coverage." I found the definition:

Code coverage = code coverage, 1 measure.

The short, concise text above describes exactly what code coverage means. There are many ways to measure code coverage. Here are the most common ones:

1. Statement overwrite (StatementCoverage)

Also known as line overwrite (LineCoverage), segment overwrite (SegmentCoverage), and base block overwrite (BasicBlockCoverage), this is the most common and common method of overwrite, measuring whether each executable statement in the code under test has been executed. This is an "executable statement", so it doesn't include header statements like C++, code comments, blank lines, etc. It makes perfect sense to count only the number of lines of code that can be executed.

Note that curly braces {} for a single line are also often counted. Statement overlays are often criticized as "the weakest overlays," which simply overwrite the executed statements in the code without considering the various combinations of branches, and so on.

If your boss only asks you to achieve statement coverage, you can save a lot of effort, but the actual test results are not obvious, and it is difficult to find more problems in the code.

To give an example that could not be simpler, let's look at the following code under test:


int foo(int a, int b)
{
 return a / b;
}

Suppose our testers wrote the following test case:


TeseCase: a = 10, b = 5

The tester's test results will tell you that his code coverage is 100% and that all his test cases pass. Unfortunately, our statement coverage reached the so-called 100%, but we didn't find the simplest Bug. For example, when I set b=0, I threw a dividing by zero exception.

Because of this, if the above requirement is only how much statement coverage the tester should achieve, the tester can easily achieve the requirement by writing test cases specifically on how to cover the lines of code. This, of course, suggests several things:

1. There is a problem with supervisors only using statement coverage to assess testers.

2. The tester's purpose is to test the code, and it is unethical to exploit such a loophole.

3. Should there be a better way to evaluate testers?

In order to find better metrics, we must first understand what the code coverage is. If your manager only knows statement coverage, line coverage, then you should proactively introduce him to more coverage. Such as:

2. Decision Coverage (DecisionCoverage)

Also known as branch coverage (BranchCoverage), all boundary coverage (ES50en-ES51en), basic path coverage (BasicPathCoverage), decision path coverage (ES53en-ES54en-ES55en). It measures whether every determined branch in the program has been tested. This is a step forward and should be very easy to confuse with conditional overrides. So let's go straight to the third type of coverage, and then compare it with the decision to cover 1 to see what's going on.

3. Conditional Coverage (ConditionCoverage)

It measures whether true and false are tested for each of the subexpression results in the determination. To illustrate the difference between decision coverage and conditional coverage, let's take an example. Suppose our code under test is as follows:


int foo(int a, int b)
{
 if (a < 10 || b < 10) //  determine 
 {
 return 0; //  branch 1
 }
 else
 {
 return 1; //  branch 2
 }
}

When designing the coverage case, we only need to consider the two cases of true and false, so we can design the following case to achieve the coverage rate of 100% :


TestCaes1: a = 5, b  =   Any Numbers   Covered with branches 1
TestCaes2: a = 15, b = 15  Covered with branches 2

When designing conditions to cover the case, we need to consider the results of each condition expression in the decision. In order to reach 100% coverage, we design the following case:


TestCase1: a = 5, b = 5 true, true
TestCase4: a = 15, b = 15 false, false

From the above example, it should be clear that there is a difference between determining coverage and conditional coverage. It is important to note that conditional overrides are not permutations and combinations of the results of each conditional expression in the decision, but OK as long as the results of each conditional expression true and false are tested. Therefore, it can be inferred that complete conditional coverage does not guarantee complete decision coverage. For example, suppose I design a case as follows:


TestCase1: a = 5, b = 15 true, false  branch 1
TestCase1: a = 15, b = 5 false, true  branch 1

We see that although we have complete conditional coverage, we don't have complete decision coverage, we only cover branch 1. As you can see from the above example, neither of these approaches seems to work. Let's look at the fourth mode of coverage.

4. Path coverage (PathCoverage)

Also known as assertion overwrite (PredicateCoverage). It measures whether every branch of the function has been executed. All possible branches are executed once. If there are multiple branches nested, multiple branches need to be arranged and combined, so it is conceivable that the test path increases exponentially with the number of branches. For example, the following test code has two decision branches:


int foo(int a, int b)
{
 int nReturn = 0;
 if (a < 10)
 {//  branch 1
 nReturn += 1;
 }
 if (b < 10)
 {//  branch 2
 nReturn += 10;
 }
 return nReturn;
}

For the above code, we designed test cases for the first three coverage methods:

a. Statement overwrite


TestCase a = 5, b = 5 nReturn = 11

Statement coverage 100%

b. Determine coverage


TestCase1 a = 5, b = 5  nReturn = 11
TestCase2 a = 15, b = 15 nReturn = 0

Decision coverage rate: 100%

c. Conditional coverage


TestCase1 a = 5, b = 15 nReturn = 1
TestCase2 a = 15, b = 5  nReturn = 10

Conditional coverage 100%

As we can see, the above three coverage results all look pretty cool! Both reached 100%! Head can be very happy, but, let's go to have a closer look at, measured in the code above, nReturn results a total of 4 kinds of possible return values: 1 0,1,10,11, while our coverage of for each of the above design test cases cover only a part of the return value, therefore, can use the above 1 covered way, although the coverage reached 100%, but there is no test completely. Let's take a look at a test case designed for path coverage:


TeseCase: a = 10, b = 5
0

Path coverage 100%

That's great! Path overrides test all possible return values. This is why it is considered by many to be the "strongest coverage".

There are other overrides, such as loop overrides (LoopCoverage), which measures whether zero, one, or more cycles have been performed on the body of the loop. The remaining 1 other coverage methods will not be introduced.

conclusion

With that in mind, let's go back and think about what coverage data really means. I have summarized the following points and welcome your discussion.

Coverage data can only represent what code you have tested, not whether you have tested it well. (for example, Bug, the first dividing by zero, above)

b. Don't put too much faith in coverage data.

Do not measure your testers only by statement coverage (line coverage).

d. Path coverage > Determine coverage > Statement coverage

Instead of blindly pursuing code coverage, testers should try to design more and better cases, even if the extra cases have no impact on coverage.


Related articles: