Structuring Unit Tests

code, tdd 0 comments suggest edit

In the past, I’ve tried various schemes to structure my unit tests but never fell into a consistent approach. Pretty much the only rule I had (which I broke all the time) was to write a test class for each class I tested. I would then fill that class with a ton of haphazard test methods.

That was until I saw the approach that Drew Miller took with NuGet.org. The way he structured the unit tests struck me as odd at first, but quickly won me over. Drew tells me he can’t take all the credit for this approach. This approach came from when he worked at CodePlex, and builds upon practices he learned from Brad Wilson and Jim Newkirk. That’s the thing I like about Drew, he won’t take credit for other people’s work. Unlike me, of course.

The structure has a test class per class being tested. That’s not so unusual. But what was unusual to me was that he had a nested class for each method being tested.

I’ll provide a simple code example to illustrate this approach and then highlight some of the benefits. The following has two methods for embellishing names with more interesting titles. What it does isn’t really that important for this discussion.

using System;

public class Titleizer
{
    public string Titleize(string name)
    {
        if (String.IsNullOrEmpty(name))
            return "Your name is now Phil the Foolish";
        return name + " the awesome hearted";
    }

    public string Knightify(string name, bool male)
    {
        if (String.IsNullOrEmpty(name))
            return "Your name is now Sir Jester";
        return (male ? "Sir" : "Dame") + " " + name;
    }
}

Under Drew’s system, I’ll have a corresponding top level class, with two embedded classes, one for each method. In each class, I’ll have a series of tests for that method.

Let’s look at a set of potential tests for this class. I wrote xUnit.NET tests for this, but you could apply the same approach with NUnit, mbUnit, or whatever you use.

using Xunit;

public class TitleizerFacts
{
    public class TheTitleizerMethod
    {
        [Fact]
        public void ReturnsDefaultTitleForNullName()
        {
            // Test code
        }

        [Fact]
        public void AppendsTitleToName()
        {
            // Test code
        }
    }

    public class TheKnightifyMethod
    {
        [Fact]
        public void ReturnsDefaultTitleForNullName()
        {
            // Test code
        }

        [Fact]
        public void AppendsSirToMaleNames()
        {
            // Test code
        }

        [Fact]
        public void AppendsDameToFemaleNames()
        {
            // Test code
        }
    }
}

Pretty simple, right? If you want to see a real-world example, look at these tests of the user service within NuGet.org.

So why do this at all? Why not stick with the old way I’ve done in the past?

Well for one thing, it’s a nice way to keep tests organized. All the tests (or facts) for a method are grouped together. For example, if you use the CTRL+M, CTRL+O shortcut to collapse method bodies, you can easily scan your tests and read them like a spec for your code.

unittests-spec

You also get the same effect if you run your tests in a test runner such as the xUnit test runner:

unittests-testrunner

When the test class file is open in Visual Studio, the class drop down provides a quick way to see a list of the methods you have tests for.

unittests-method-list

This makes it easy to then see all the tests for a given method by using the drop down on the right.

unittests-test-list

It’s a minor change to my existing practices, but one that I’ve grown to like a lot and hope to apply in all my projects in the future.

Update: Several folks asked about how to have common setup code for all tests. ZenDeveloper has a simple solution in which the nested child classes simply inherit the outer parent class. Thus they’ll all share the same setup code.

Tags: unit testing, tdd, xunit

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.

Comments

avatar

61 responses

  1. Avatar for Matt Wrock
    Matt Wrock January 2nd, 2012

    I work with some of the folks that Drew, Brad and Jim Newkirk used to work with over at MSDN. We use this style of organizing tests and I really like it. It provides an organized way to quickly find a set of tests I'm looking for. Its far too easy for a single test class to fill up with a hundred tests and then lose track of any kind of order. This organization strategy goes a long way to solve this.
    We have further refined the single "Testable" class using auto mocking. I have found that auto mocking makes the testable class pattern much more DRY. See my blog post Unit Testing ASP.Net Http Handlers and a discussion of Auto Mocking and the Testable pattern for details.

  2. Avatar for Drew Miller
    Drew Miller January 2nd, 2012

    Glad to hear it Matt! I'll have to look into what y'all have been doing with the testable pattern. I've actually started to move away from it completely, favoring action method injection instead. I need to write about that sometime soon.

  3. Avatar for D3thm0nk3y
    D3thm0nk3y January 2nd, 2012

    Me likey.

  4. Avatar for Bil Simser
    Bil Simser January 2nd, 2012

    Interesting. I've been looking for something different and this might be it. I've tried various namespace approaches and naming strategies (at one point putting the test cases in folders underneath the main classes). Nothing has really clicked. I never thought of nested classes so it seems pretty elegant. Going to try this on a new project and see if I get more traction than before. Thanks guys!

  5. Avatar for Tamizhvendan S
    Tamizhvendan S January 2nd, 2012

    Nice write up Haack.
    An another way of structuring unit tests which I personally prefer is having separate directory in the test project for each class that we wanted to test and having separate test class for each methods of that class underneath that directory.
    For Example,
    TitlelizerTests
    --> Titlelize
    --> ReturnDefaultTitleForNullName
    --> AppendTitleToName
    --> Knightify
    --> ReturnDefaultTitleForNullName
    --> AppendSirForMaleNames
    --> AppendDameForFemaleNames
    I feel, its hard to generalize and follow the same structure across all the projects.

  6. Avatar for Mihai Lazar
    Mihai Lazar January 2nd, 2012

    If only you would remove duplication with it.
    At least for the example of UsersService I find that there is
    1. Too much code duplication for the setup of the CryptoService
    2. The method Create for example, has a lot of responsibilities beyond its intended purpose (validation, hashing, token creation)
    Other than that, it will allow for a bit better code organization in some cases, so thanks for it.

  7. Avatar for Daniel marbach
    Daniel marbach January 2nd, 2012

    Nice idea! I generally avoid terms like "method", "call" etc in my test names. You should name the feature and not the method of the class. When I do refactorings I often give other names to methods than I quckly added during the "make the tests green" cycle. But the feature I work towards remains the same.
    Daniel

  8. Avatar for acl
    acl January 2nd, 2012

    I've been doing this for a while after seeing it in an article about BDD.
    Unfortunately Resharper has been late to the party: youtrack.jetbrains.net/.../RSRP-199079

  9. Avatar for Roger Lipscombe
    Roger Lipscombe January 2nd, 2012

    This is something we've been thinking about for our unit tests. In particular, we're using MSpec, which calls for a class for each test.


    Obeying the one-class-per-file guideline quickly makes the project hard to navigate (though arguably still easier than finding all of those classes in the same file).


    Nesting them like this -- and with MSpec, it'll probably be nested 3 deep (class -- method -- test) -- might be what we need.


  10. Avatar for Fabian Schmied
    Fabian Schmied January 2nd, 2012

    How do you work with variables initialized by SetUp methods in such a scenario?
    AFAICS, you'd have to repeat the SetUp (ctor) initializations for every nested class (unless they only apply to a single tested method). Or is there another, better practice?

  11. Avatar for Fabian Schmied
    Fabian Schmied January 2nd, 2012

    Hmm, of course, you could derive the nested classes from the outer class; that way, they'd inherit all initialization logic put into the outer class.
    Any other good ideas for this?

  12. Avatar for Thomas Eyde
    Thomas Eyde January 2nd, 2012

    Do you really name the classes after methods, or is it for illustration only? Don't you have issues when the method under test is renamed? Or even moved to a different class?

  13. Avatar for Chris Airey
    Chris Airey January 2nd, 2012

    BDD has been applying this approach already. Seperate test classes for each concern. Look at some BDD frameworks like Machine Specifications.

  14. Avatar for Seth Petry-Johnson
    Seth Petry-Johnson January 2nd, 2012

    I do something very similar, except instead of nested classes I create a namespace for the class under test and then create top level classes in that namespace for each method or scenario. I think this works a little nicer when using a test runner because it avoids the "ParentClass+Whatever" display, which I find noisy and hard to read.

  15. Avatar for Matt Honeycutt
    Matt Honeycutt January 2nd, 2012

    This is similar to the preferred way of organizing tests with SpecsFor, except instead of a nested class per method, you use a nested class per scenario. Not everyone likes this approach though because it can become a bit crowded when you have a lot of scenarios for one class.

  16. Avatar for haacked
    haacked January 2nd, 2012

    @Thomas I do name the classes after the method. I use unit tests to drive the design of my code (ala TDD). In most cases, when I needed to rename something, it's due to a change in the design of my code. I rarely rename in pure isolation. In those cases, I use tests to drive the change in code.
    Even in cases where I really am just renaming, I just rename the tests first, then rename the code. The tests always lead the code. Except when I get sloppy. :)
    @Fabian I think we've written helper methods for that setup in nuget.org. In many cases, each method is isolated and doesn't have the same setup code anyways.

  17. Avatar for Karep
    Karep January 2nd, 2012

    Ok but instead of this you could do
    [Fact]
    public void TitleizerMethod_ReturnsDefaultTitleForNullName()
    {
    // Test code
    }
    [Fact]
    public void TitleizerMethod_AppendsTitleToName()
    {
    // Test code
    }
    Not so much as in your solution and still you can use dropdowns to find tests for methods.
    And how do you setup your tests? Duplicate code in constructors of each class? Create Helper class that is called from ctor of each class (XUnit uses ctor's for setup right?)

  18. Avatar for twk
    twk January 2nd, 2012

    Just starting my journey with tests and this is really good example for me, now testing becomes fun :)
    Thanks for sharing!

  19. Avatar for Jim Cooper
    Jim Cooper January 2nd, 2012

    "Prepends", not "appends"!
    I'm a bit surprised you're using xunit.net - I've used both it and NUnit, and XUnit is much clumsier and old fashioned. And it bugs me they appropriated the generic term for unit testing frameworks.
    I used to name my tests like that, but I've moved in a more BDD direction in recent years, and have much more descriptive names.
    Something to consider: suppose the test "ReturnDefaultTitleForNullName" fails for someone in a year's time. Is the problem in the code under test, or is the test no longer correct? This is a trivial example, but sometimes it isn't so clear.

  20. Avatar for Russ Painter
    Russ Painter January 3rd, 2012

    I have previously been naming my test functions with [FUNCTIONNAME]_[TESTNAME]. But I've tried this method and I'm liking it so far. It forces you to keep related tests together.
    In the VS2010 Test Results view I found it useful to add a column for "Class Name" which when combined with the "Test Name" column gives you all the info. Very nice.

  21. Avatar for Alex T
    Alex T January 3rd, 2012

    This is an interesting way to organize the unit tests indeed.
    Yet, I'm afraid I'll have to second the critics here ;-)
    The problem I see with such approach is that it "perpetuates" the methods structure. Which is not good if you intend to continuously refactor your code (that inevitably results in methods extraction, collapsing, etc.).
    Having a flat, single test class per single class under test structure makes it easier to keep the test suite in sync with the code. But I can see how laziness (among other factors ;-)) would prevent developers from reorganizing the UT classes in parallel with reorganizing the source code.
    After all, a significant portion of refactoring is driven by code improvement rationale, not by adding new tests...
    And notice that there's no automatic way to migrate and rename the corresponding tests when the business logic is moved to a different method.
    So, my fear is that we'll end up (rather sooner than later) with the UT structure that is more confusing than helping, because it's out of sync with the real code structure.

  22. Avatar for Craig
    Craig January 3rd, 2012

    We use the same naming convention that Russ Painter mentions. It makes the tests easy to find and lets us know immediately what method is being tested just by looking at the name. I'm not knocking the idea mentioned in this post, but I honestly don't see that it buys me a lot over a good naming convention.

  23. Avatar for Justin
    Justin January 3rd, 2012

    Interesting approach. I always have a hard time making sure I properly test my code. This looks like it will help me to easily see areas of code where I have missed writing tests for.

  24. Avatar for Karep
    Karep January 3rd, 2012

    If anyone of you unit testers uses NUnit maybe you will find helpfull Nunit.Snippets Nuget Package.

  25. Avatar for Matthew Manela
    Matthew Manela January 3rd, 2012

    @Drew and @Matt,
    Method argument injection sounds really interesting. In one sense I can see it makes a lot of sense in a large class where methods are using different dependencies. But I wonder how often the public methods in a class really have very different dependencies. If two methods in one class depends on many different interfaces maybe they don't belong in the same class?
    Thoughts?

  26. Avatar for Scott Koon
    Scott Koon January 3rd, 2012

    This is a very neat approach and I'd think it makes refactoring methods a bit easier. I've been kind of bastardizing BDD style syntax by just naming my classes and facts in a BDD style manner.
    github.com/...

  27. Avatar for SandRock
    SandRock January 4th, 2012

    A great thing BTW is when you are searching for unit tests concerning a method. If you Ctrl+, (Navigate to...) on a method name, you will clearly see the test class in the list :)

  28. Avatar for Tom Cabanski
    Tom Cabanski January 4th, 2012

    Sorry. Not digging it. I would say more, but Ayende said it better: ayende.com/.../structuring-your-unit-tests-why

  29. Avatar for dotnetchris
    dotnetchris January 4th, 2012

    I write all my unit tests in MSpec with Machine.Fakes.Moq using Machine.Fakes allow you to share test configuration through composition instead of inheritance. It's a rather godly pattern.
    I'm still not set on how to organize tests using MSpec and opened a question on stackoverflow about this very topic a few weeks ago: stackoverflow.com/...

  30. Avatar for Khuzema
    Khuzema January 4th, 2012

    Question about Nuget website (nuget.org) related, is there is some documentation (or blog post), which can explain about the website design, as it is so generously been open sourced and so soundly structured. (A really real world Asp.Net MVC uses example).
    Thanks

  31. Avatar for Paul
    Paul January 5th, 2012

    I really like this. I have been using a convention of starting my test names with the function name i.e. Knightify_ShouldAppendSir_WhenArgMailIsTrue. This new structure would set the context making my test names shorter. Thanks for sharing this tip!

  32. Avatar for Robert Penner
    Robert Penner January 6th, 2012

    Test features, i.e. use cases, not methods.

  33. Avatar for William Hoge
    William Hoge January 6th, 2012

    Just wanted to say I'm new here and love the blog. Great conversation and many good laughs.

  34. Avatar for Gary McLean Hall
    Gary McLean Hall January 10th, 2012

    I blogged about unit test organization here:
    garymcleanhall.wordpress.com/...
    ...and I came up with Behavioral, here:
    http://behavioral.codeplex.com/
    Not sure what direction it's going to go in, and it needs work, but it was intended to create more readable unit tests, as well as to allow reuse.

  35. Avatar for Julien Bérubé
    Julien Bérubé January 12th, 2012

    @Karep : You'd lose one big advantage, which is maintainability. If you change the name of one of your method under test, you end up having to rename every test method manually.
    Renaming one class name is faster than renaming 27 method names. Even by using replace all, that's risky.
    Plus, if you do peer review, you have less lines of code to review, and if you use source control, less risk of conflict.

  36. Avatar for Zak David
    Zak David January 15th, 2012

    Interesting idea but I'm concerned that tying your tests and your code so closely will prove to be a pain when you refactor (e.g. rename a production method and you need to rename tests).
    I prefer to be a bit looser with this: name my tests to describe what they test instead of tying them to a method name and use the test group feature offered by TestNG (Java) to group my tests logically:
    @Test(groups = { "front-end", "slow", "servlet" }
    public void loginShouldWork() { ... }
    This method will be run if I run the group "front-end" but it will be skipped if I only want to run fast tests.
    TestNG: http://testng.org

  37. Avatar for JF
    JF January 23rd, 2012

    I understand that when wrapping tests around existing, untested, legacy code, these are fine guidelines. For new development, though, Titleizer violates SRP - it isn't single responsibility. That class does two distinct operations.
    Once have Titlizer.Titleize() and Knightifier.Knightify(), I've got Single Responsibility and Naming down, and the tests now organize themselves. I don't need to nest anything into child classes.
    As a bonus, now when I add the functionality to prepend "Lord" or "Lady" to Titleize(), I don't need to modify the Knightifier class or its tests.

  38. Avatar for ZenDeveloper
    ZenDeveloper January 26th, 2012

    Phil,
    Thanks for a great article. I have take the examples here and adapted them to an MS Test environment with a few issues resolved (namely DRY). Check it out http://bit.ly/yE7GbZ

  39. Avatar for haacked
    haacked January 26th, 2012

    @ZenDeveloper I like the little touch where you have the child classes inherit from the parent for common setup. Clever. :)

  40. Avatar for Shane
    Shane March 7th, 2012

    Certainly beats my naming convention for test methods where I start with a common prefix and then add the specific test case description.
    Grouping by nested classes is much more sensible. Thanks for the info!

  41. Avatar for Alimentacao
    Alimentacao March 10th, 2012

    Good Work ! NUnit maybe you will find helpfull Nunit.Thanks !

  42. Avatar for Ramsay
    Ramsay March 30th, 2012

    I've always wondered how other c# programmers would structure their unit tests. I sorta structure mines like yours, but I rather not use keywords such as 'method' for the test method names - I know by your programming skills that you won't do that in RL applications -, althought I think it fits very well in your example.
    Thanks for the tips, really insightful!

  43. Avatar for Radenko Zec
    Radenko Zec September 9th, 2012

    This does not work anymore in VS2012 even in Resharper test runner.
    Do you have some alternative ?

  44. Avatar for Philip Panyukov
    Philip Panyukov September 23rd, 2012

    Great technique. I have taken this further and modified it slightly, and now I can test internal and private stuff without reflection or any other dirty and ugly hacks.
    Blogged about it here

  45. Avatar for Chris F Carroll
    Chris F Carroll October 30th, 2012

    Humbly disagree.
    As at 2012, BDD and the Given When Then pattern looks like the better way forward for TDD.
    That steers you towards aligning your tests not to classes and methods, but to requirements.
    Given_a_male_knight_Titleizer_should_prefix_Sir_to_name
    Given_a_female_knight_Titleizer_should_prefix_Dame_to_name
    Which generates tests that can be shown to, and corrected by, and written in collaboration with, non-technical users.
    And gives you traceability from requirements to code -- suddenly it's much easier to understand 2 years later why the code does X and not Y.
    Which is not to disagree with the nested classes idea.

  46. Avatar for Moshfegh Hamedani
    Moshfegh Hamedani January 15th, 2013

    That was pretty awesome!! Loved it!!

  47. Avatar for ansciath
    ansciath March 5th, 2013

    I've recently adopted this structure but for a reason that is more practical than mere organizaiton. Arranging unit tests in this way makes it easier to run all of the unit tests for a specific unit, without running any other tests. This prevents extraneous coverage due to non-relevant tests when white box testing.

  48. Avatar for Ovidiu Caba
    Ovidiu Caba April 9th, 2013

    Hi Phil,

    I wonder how would you test an interface. Let's propose the following sample scenario:

    interface ISimpleInterface
    {
    //This function always returns 1
    int GetX();
    }

    class C1 : ISimpleInterface
    {
    int GetX() {return 1;}
    }

    class C2 : ISimpleInterface
    {
    int GetX() {return 2;} //intentional wrong implementation
    }

    For a few days I tried a lot of approaches, but none of them work for me. The last one is this:

    public abstract class ISimpleInterfaceUnitTest
    {

    public ISimpleInterface Instance;

    public abstract class GetX()
    {

    [TestMethod]
    GetXShouldReturn1()
    {
    Assert.AreEqual(1, Instance.GetX());
    }
    }
    }

    public class C1UnitTest : ISimpleInterfaceUnitTest
    {

    [ClassInitialize]
    public void InitializeClass()
    {
    ISimpleInterface Instance = new C1();
    }

    public class GetXUnitTest : GetX
    {
    }
    }

    public class C2UnitTest : ISimpleInterfaceUnitTest
    {

    [ClassInitialize]
    public void InitializeClass()
    {
    ISimpleInterface Instance = new C2();
    }

    public class GetXUnitTest : GetX
    {
    }
    }

    The problem is that the methods InitializeClass from C1UnitTest and C2UnitTest are never called. If I want this tests to work correctly I have to copy the InitializeClass method in every subclass of C1UnitTest and C2UnitTest, but I don't want to do that, I don't want to duplicate the code.

    What approach would you suggest for such scenario?

    Thank you in advance,
    Ovidiu

  49. Avatar for haacked
    haacked April 15th, 2013

    This is one reason I don't use MSTest. If you used XUnit you could use the [Theory] attribute with [PropertyData] and pass in each instance of your interface into your unit test. That way the test is written once.

  50. Avatar for Carter
    Carter April 24th, 2013

    The link "look at these tests of the user service within NuGet.org." is broken - 404 error.

  51. Avatar for Guest
    Guest October 23rd, 2013

    this is obviously not test driven.

  52. Avatar for Guest
    Guest October 23rd, 2013

    I still like Roy O's convention [UnitOfWork_StateUnderTest_ExpectedBehavior]

  53. Avatar for Mathieu
    Mathieu February 5th, 2014

    I found the new link: https://github.com/NuGet/Nu...

  54. Avatar for Steven Cramer
    Steven Cramer April 25th, 2014

    So if your class under test has a constructor, where do you test that? If you create a nested class for the Constructor, what do you name it?

  55. Avatar for haacked
    haacked April 25th, 2014

    ```csharp
    public class TheCtor
    {
    [Fact]
    public void DoesWhatItShould() {...}
    }
    ```

  56. Avatar for Steven Cramer
    Steven Cramer April 25th, 2014

    Simple enough. Thanks.

  57. Avatar for Matt DiTrolio
    Matt DiTrolio December 12th, 2014

    I like this approach. I'll try using it with the methods I'm working on right now. Thanks for the great explanation!

  58. Avatar for matty
    matty March 8th, 2015

    YES! Optional dependencies makes so much sense. Not only for tooling from other package managers, but for other NuGet packages as well.

  59. Avatar for John Callaway
    John Callaway July 24th, 2017

    Very nice approach. I've been doing similar for a while, except that instead I have a test folder per class and inside that folder a test class per method. I think I may be missing some of the benefits you describe here. I'll have to try it and see which I like better.

  60. Avatar for Lachlan Miller
    Lachlan Miller October 26th, 2017

    This is handy, thanks. I was looking for something similar to describe and it in rails, this can help me replicate a similar way to organise my tests.

  61. Avatar for Zoran
    Zoran August 14th, 2018

    Well, if you use nested classes for collapsing, why not use regions instead?