Why Code Coverage is not Enough

code, tdd 0 comments suggest edit

One of the holy grails for unit testing is to get 100% code coverage from your tests. However, you can’t sit back and smoke a cigar when you reach that point and assume your code is invulnerable. Code coverage just is not enough.

One obvious reason is that Code Coverage cannot help you find errors of omission. That is, even if you had 100% code coverage from your tests, if you forget to implement a feature (and a test for that feature), then you’re shit out of luck.

However, apart from errors of omission, there’s the case presented here. Imagine you have the following simple class (I’m sure your real world class is much more complicated and interesting, but bear with me).

using System;
using System.Collections;

public class MyClass
{
    Dictionary<string, int> _values = new Dictionary<string, int>();

    public MyClass()
    {
        _values.Add("keyOne", "1");
        _values.Add("keyTwo", "7");
        _values.Add("keyThree", "10");

        // ...
    }

    public int SumIt(string[] keys)
    {
        int total = 0;
        
		foreach(string key in keys)
        {
            total += _values[key];
            _values[key] = total;

            //Maybe we do some other
            //interesting things here.
        }

        return total;
    }
}

Now imagine you test this class with the following NUnit fixture.

using System;
using XUnit;

public class MyClassTest
{
    [Fact]
    public void TestSumIt()
    {
        var mine = new MyClass();
        string[] keys = {"keyOne", "keyTwo"};
        Assert.Equal(8, mine.SumIt(keys));
    }
}

Voila! 100% code coverage. But does this satisfy the little QA tester inside? I would hope not and suggest that it shouldn’t. Code coverage is worthy goal, but often unnattainable in large systems (hence the need for prioritization) and doesn’t provide all the benefits it would seem.

To handle situations like this, unit tests need to go beyond concentrating on code coverage and also consider data coverage. Of course, that’s not always practical. In the above example, if I only have 10 keys, testing the possible permutations of SumIt becomes a huge burden. Often the best you can do is to test a small sample and the boundary cases.

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.

Comments

avatar

3 responses

  1. Avatar for Jonathan de Halleux
    Jonathan de Halleux November 9th, 2004

    A valuable technique for lowering the number of enumeration is pairwize testing... and it is supported by MbUnit. Let me illustrate that by writing "combinatorial unit test for you example (note that I added a GetValue method to MyClass).



    The CombinatorialTest attributes tells MbUnit to do a pairwize enumeration over the different data sample of each parameter. The Using*** atttibutes specify where the data should be get from. In this case, a method returning an array of keys is sufficient.



    [TestFixture]

    public class MyClassFixture

    {

    [Factory]

    public string[] GetKeys()

    {

    string[] keys = new string[3];

    keys[0] = "keyOne";

    keys[1] = "keyTwo";

    keys[2] = "keyThree";



    return keys;

    }



    [CombinatorialTest]

    public void Sumit(

    [UsingFactories("GetKeys")] string key1,

    [UsingFactories("GetKeys")] string key2

    )

    {

    MyClass mine = new MyClass();

    string[] keys = { key1, key2 };

    int sum = mine.GetValue(key1) + mine.GetValue(key2);

    Assert.AreEqual(sum, mine.SumIt(keys));

    }

    }





    And here's the result in the output window using TD.NET:



    Info: Test Execution

    Info: Exploring MbUnit.Demo, Version=2.22.1.0, Culture=neutral, PublicKeyToken=null

    Info: MbUnit 2.22.0.0 Addin

    Info: Found 9 tests

    Info: [assembly-setup] success

    Info: [success] MyClassFixture.Sumit(GetKeys(keyOne),GetKeys(keyOne))

    Info: [success] MyClassFixture.Sumit(GetKeys(keyOne),GetKeys(keyTwo))

    Info: [success] MyClassFixture.Sumit(GetKeys(keyOne),GetKeys(keyThree))

    Info: [success] MyClassFixture.Sumit(GetKeys(keyTwo),GetKeys(keyOne))

    Info: [success] MyClassFixture.Sumit(GetKeys(keyTwo),GetKeys(keyTwo))

    Info: [success] MyClassFixture.Sumit(GetKeys(keyTwo),GetKeys(keyThree))

    Info: [success] MyClassFixture.Sumit(GetKeys(keyThree),GetKeys(keyOne))

    Info: [success] MyClassFixture.Sumit(GetKeys(keyThree),GetKeys(keyTwo))

    Info: [success] MyClassFixture.Sumit(GetKeys(keyThree),GetKeys(keyThree))

    Info: [assembly-teardown] success

    Info: [reports] generating HTML report





    Humm, no bug... Ok, let's go for 3 arguments. This makes 15 tests using my pairwize algo:



    Info: Test Execution

    Info: Exploring MbUnit.Demo, Version=2.22.1.0, Culture=neutral, PublicKeyToken=null

    Info: MbUnit 2.22.0.0 Addin

    Info: Found 15 tests

    Info: [assembly-setup] success

    Info: [failure] MyClassFixture.Sumit(GetKeys(keyOne),GetKeys(keyOne),GetKeys(keyOne))

    TestCase 'MyClassFixture.Sumit(GetKeys(keyOne),GetKeys(keyOne),GetKeys(keyOne))' failed: Equal assertion failed: [[3]]!=[[4]]

    MbUnit.Core.Exceptions.NotEqualAssertionException

    Message: Equal assertion failed: [[3]]!=[[4]]

    ...

    Info: [success] MyClassFixture.Sumit(GetKeys(keyOne),GetKeys(keyOne),GetKeys(keyTwo))

    ...



    That's much better because we hit the test. So what do you think ?

  2. Avatar for Haacked
    Haacked November 9th, 2004

    Very cool. Very cool!

  3. Avatar for buy viagra
    buy viagra October 26th, 2006

    gazes conceived Bavarian intoxicated followings biplanes photocopier <A HREF="http://www.fleetairarmarchi... http://www.fleetairarmarchi...