Structuring Unit Tests

In the past, I’ve tried various schemes to structure my unit tests but never fell into a consistent approach. Pretty much the only rule I had (which I broke all the time) was to write a test class for each class I tested. I would then fill that class with a ton of haphazard test methods.

That was until I saw the approach that Drew Miller took with NuGet.org. The way he structured the unit tests struck me as odd at first, but quickly won me over. Drew tells me he can’t take all the credit for this approach. This approach came from when he worked at CodePlex, and builds upon practices he learned from Brad Wilson and Jim Newkirk. That’s the thing I like about Drew, he won’t take credit for other people’s work. Unlike me, of course.

The structure has a test class per class being tested. That’s not so unusual. But what was unusual to me was that he had a nested class for each method being tested.

I’ll provide a simple code example to illustrate this approach and then highlight some of the benefits. The following has two methods for embellishing names with more interesting titles. What it does isn’t really that important for this discussion.

using System;

public class Titleizer
{
    public string Titleize(string name)
    {
        if (String.IsNullOrEmpty(name))
            return "Your name is now Phil the Foolish";
        return name + " the awesome hearted";
    }

    public string Knightify(string name, bool male)
    {
        if (String.IsNullOrEmpty(name))
            return "Your name is now Sir Jester";
        return (male ? "Sir" : "Dame") + " " + name;
    }
}

Under Drew’s system, I’ll have a corresponding top level class, with two embedded classes, one for each method. In each class, I’ll have a series of tests for that method.

Let’s look at a set of potential tests for this class. I wrote xUnit.NET tests for this, but you could apply the same approach with NUnit, mbUnit, or whatever you use.

using Xunit;

public class TitleizerFacts
{
    public class TheTitleizerMethod
    {
        [Fact]
        public void ReturnsDefaultTitleForNullName()
        {
            // Test code
        }

        [Fact]
        public void AppendsTitleToName()
        {
            // Test code
        }
    }

    public class TheKnightifyMethod
    {
        [Fact]
        public void ReturnsDefaultTitleForNullName()
        {
            // Test code
        }

        [Fact]
        public void AppendsSirToMaleNames()
        {
            // Test code
        }

        [Fact]
        public void AppendsDameToFemaleNames()
        {
            // Test code
        }
    }
}

Pretty simple, right? If you want to see a real-world example, look at these tests of the user service within NuGet.org.

So why do this at all? Why not stick with the old way I’ve done in the past?

Well for one thing, it’s a nice way to keep tests organized. All the tests (or facts) for a method are grouped together. For example, if you use the CTRL+M, CTRL+O shortcut to collapse method bodies, you can easily scan your tests and read them like a spec for your code.

unittests-spec

You also get the same effect if you run your tests in a test runner such as the xUnit test runner:

unittests-testrunner

When the test class file is open in Visual Studio, the class drop down provides a quick way to see a list of the methods you have tests for.

unittests-method-list

This makes it easy to then see all the tests for a given method by using the drop down on the right.

unittests-test-list

It’s a minor change to my existing practices, but one that I’ve grown to like a lot and hope to apply in all my projects in the future.

Update: Several folks asked about how to have common setup code for all tests. ZenDeveloper has a simple solution in which the nested child classes simply inherit the outer parent class. Thus they’ll all share the same setup code.

What others have said

Requesting Gravatar... Matt Wrock Jan 02, 2012 6:31 AM
# re: Structuring Unit Tests
I work with some of the folks that Drew, Brad and Jim Newkirk used to work with over at MSDN. We use this style of organizing tests and I really like it. It provides an organized way to quickly find a set of tests I'm looking for. Its far too easy for a single test class to fill up with a hundred tests and then lose track of any kind of order. This organization strategy goes a long way to solve this.

We have further refined the single "Testable" class using auto mocking. I have found that auto mocking makes the testable class pattern much more DRY. See my blog post Unit Testing ASP.Net Http Handlers and a discussion of Auto Mocking and the Testable pattern for details.
Requesting Gravatar... Drew Miller Jan 02, 2012 6:41 AM
# re: Structuring Unit Tests
Glad to hear it Matt! I'll have to look into what y'all have been doing with the testable pattern. I've actually started to move away from it completely, favoring action method injection instead. I need to write about that sometime soon.
Requesting Gravatar... D3thm0nk3y Jan 02, 2012 6:44 AM
# re: Structuring Unit Tests
Me likey.
Requesting Gravatar... Bil Simser Jan 02, 2012 6:52 AM
# re: Structuring Unit Tests
Interesting. I've been looking for something different and this might be it. I've tried various namespace approaches and naming strategies (at one point putting the test cases in folders underneath the main classes). Nothing has really clicked. I never thought of nested classes so it seems pretty elegant. Going to try this on a new project and see if I get more traction than before. Thanks guys!
Requesting Gravatar... Tamizhvendan S Jan 02, 2012 7:23 AM
# re: Structuring Unit Tests
Nice write up Haack.

An another way of structuring unit tests which I personally prefer is having separate directory in the test project for each class that we wanted to test and having separate test class for each methods of that class underneath that directory.

For Example,
TitlelizerTests
--> Titlelize
--> ReturnDefaultTitleForNullName
--> AppendTitleToName
--> Knightify
--> ReturnDefaultTitleForNullName
--> AppendSirForMaleNames
--> AppendDameForFemaleNames

I feel, its hard to generalize and follow the same structure across all the projects.
Requesting Gravatar... Mihai Lazar Jan 02, 2012 7:58 AM
# re: Structuring Unit Tests
If only you would remove duplication with it.
At least for the example of UsersService I find that there is
1. Too much code duplication for the setup of the CryptoService
2. The method Create for example, has a lot of responsibilities beyond its intended purpose (validation, hashing, token creation)

Other than that, it will allow for a bit better code organization in some cases, so thanks for it.
Requesting Gravatar... Daniel marbach Jan 02, 2012 8:13 AM
# re: Structuring Unit Tests
Nice idea! I generally avoid terms like "method", "call" etc in my test names. You should name the feature and not the method of the class. When I do refactorings I often give other names to methods than I quckly added during the "make the tests green" cycle. But the feature I work towards remains the same.

Daniel
Requesting Gravatar... acl Jan 02, 2012 8:15 AM
# re: Structuring Unit Tests
I've been doing this for a while after seeing it in an article about BDD.
Unfortunately Resharper has been late to the party: youtrack.jetbrains.net/.../RSRP-199079
Requesting Gravatar... Roger Lipscombe Jan 02, 2012 8:17 AM
# re: Structuring Unit Tests

This is something we've been thinking about for our unit tests. In particular, we're using MSpec, which calls for a class for each test.



Obeying the one-class-per-file guideline quickly makes the project hard to navigate (though arguably still easier than finding all of those classes in the same file).



Nesting them like this -- and with MSpec, it'll probably be nested 3 deep (class -- method -- test) -- might be what we need.


Requesting Gravatar... Fabian Schmied Jan 02, 2012 10:33 AM
# re: Structuring Unit Tests
How do you work with variables initialized by SetUp methods in such a scenario?

AFAICS, you'd have to repeat the SetUp (ctor) initializations for every nested class (unless they only apply to a single tested method). Or is there another, better practice?
Requesting Gravatar... Fabian Schmied Jan 02, 2012 10:37 AM
# re: Structuring Unit Tests
Hmm, of course, you could derive the nested classes from the outer class; that way, they'd inherit all initialization logic put into the outer class.

Any other good ideas for this?
Requesting Gravatar... Thomas Eyde Jan 02, 2012 12:31 PM
# re: Structuring Unit Tests
Do you really name the classes after methods, or is it for illustration only? Don't you have issues when the method under test is renamed? Or even moved to a different class?
Requesting Gravatar... Chris Airey Jan 02, 2012 1:22 PM
# re: Structuring Unit Tests
BDD has been applying this approach already. Seperate test classes for each concern. Look at some BDD frameworks like Machine Specifications.
Requesting Gravatar... Seth Petry-Johnson Jan 02, 2012 1:34 PM
# re: Structuring Unit Tests
I do something very similar, except instead of nested classes I create a namespace for the class under test and then create top level classes in that namespace for each method or scenario. I think this works a little nicer when using a test runner because it avoids the "ParentClass+Whatever" display, which I find noisy and hard to read.
Requesting Gravatar... Matt Honeycutt Jan 02, 2012 2:10 PM
# re: Structuring Unit Tests
This is similar to the preferred way of organizing tests with SpecsFor, except instead of a nested class per method, you use a nested class per scenario. Not everyone likes this approach though because it can become a bit crowded when you have a lot of scenarios for one class.
Requesting Gravatar... haacked Jan 02, 2012 6:16 PM
# re: Structuring Unit Tests
@Thomas I do name the classes after the method. I use unit tests to drive the design of my code (ala TDD). In most cases, when I needed to rename something, it's due to a change in the design of my code. I rarely rename in pure isolation. In those cases, I use tests to drive the change in code.

Even in cases where I really am just renaming, I just rename the tests first, then rename the code. The tests always lead the code. Except when I get sloppy. :)

@Fabian I think we've written helper methods for that setup in nuget.org. In many cases, each method is isolated and doesn't have the same setup code anyways.
Requesting Gravatar... Karep Jan 02, 2012 10:04 PM
# re: Structuring Unit Tests
Ok but instead of this you could do

[Fact]
public void TitleizerMethod_ReturnsDefaultTitleForNullName()
{
// Test code
}

[Fact]
public void TitleizerMethod_AppendsTitleToName()
{
// Test code
}

Not so much as in your solution and still you can use dropdowns to find tests for methods.

And how do you setup your tests? Duplicate code in constructors of each class? Create Helper class that is called from ctor of each class (XUnit uses ctor's for setup right?)
Requesting Gravatar... twk Jan 02, 2012 11:12 PM
# re: Structuring Unit Tests
Just starting my journey with tests and this is really good example for me, now testing becomes fun :)
Thanks for sharing!
Requesting Gravatar... Jim Cooper Jan 02, 2012 11:50 PM
# re: Structuring Unit Tests
"Prepends", not "appends"!

I'm a bit surprised you're using xunit.net - I've used both it and NUnit, and XUnit is much clumsier and old fashioned. And it bugs me they appropriated the generic term for unit testing frameworks.

I used to name my tests like that, but I've moved in a more BDD direction in recent years, and have much more descriptive names.

Something to consider: suppose the test "ReturnDefaultTitleForNullName" fails for someone in a year's time. Is the problem in the code under test, or is the test no longer correct? This is a trivial example, but sometimes it isn't so clear.



Requesting Gravatar... Russ Painter Jan 03, 2012 11:11 AM
# re: Structuring Unit Tests
I have previously been naming my test functions with [FUNCTIONNAME]_[TESTNAME]. But I've tried this method and I'm liking it so far. It forces you to keep related tests together.

In the VS2010 Test Results view I found it useful to add a column for "Class Name" which when combined with the "Test Name" column gives you all the info. Very nice.
Requesting Gravatar... Alex T Jan 03, 2012 4:12 PM
# re: Structuring Unit Tests
This is an interesting way to organize the unit tests indeed.
Yet, I'm afraid I'll have to second the critics here ;-)

The problem I see with such approach is that it "perpetuates" the methods structure. Which is not good if you intend to continuously refactor your code (that inevitably results in methods extraction, collapsing, etc.).
Having a flat, single test class per single class under test structure makes it easier to keep the test suite in sync with the code. But I can see how laziness (among other factors ;-)) would prevent developers from reorganizing the UT classes in parallel with reorganizing the source code.
After all, a significant portion of refactoring is driven by code improvement rationale, not by adding new tests...

And notice that there's no automatic way to migrate and rename the corresponding tests when the business logic is moved to a different method.

So, my fear is that we'll end up (rather sooner than later) with the UT structure that is more confusing than helping, because it's out of sync with the real code structure.
Requesting Gravatar... Craig Jan 03, 2012 4:39 PM
# re: Structuring Unit Tests
We use the same naming convention that Russ Painter mentions. It makes the tests easy to find and lets us know immediately what method is being tested just by looking at the name. I'm not knocking the idea mentioned in this post, but I honestly don't see that it buys me a lot over a good naming convention.
Requesting Gravatar... Justin Jan 03, 2012 5:04 PM
# re: Structuring Unit Tests
Interesting approach. I always have a hard time making sure I properly test my code. This looks like it will help me to easily see areas of code where I have missed writing tests for.
Requesting Gravatar... Karep Jan 03, 2012 5:09 PM
# re: Structuring Unit Tests
If anyone of you unit testers uses NUnit maybe you will find helpfull Nunit.Snippets Nuget Package.
Requesting Gravatar... Matthew Manela Jan 03, 2012 5:59 PM
# re: Structuring Unit Tests
@Drew and @Matt,

Method argument injection sounds really interesting. In one sense I can see it makes a lot of sense in a large class where methods are using different dependencies. But I wonder how often the public methods in a class really have very different dependencies. If two methods in one class depends on many different interfaces maybe they don't belong in the same class?

Thoughts?
Requesting Gravatar... Scott Koon Jan 03, 2012 6:53 PM
# re: Structuring Unit Tests
This is a very neat approach and I'd think it makes refactoring methods a bit easier. I've been kind of bastardizing BDD style syntax by just naming my classes and facts in a BDD style manner.

github.com/...
Requesting Gravatar... SandRock Jan 04, 2012 2:05 PM
# re: Structuring Unit Tests
A great thing BTW is when you are searching for unit tests concerning a method. If you Ctrl+, (Navigate to...) on a method name, you will clearly see the test class in the list :)
Requesting Gravatar... Tom Cabanski Jan 04, 2012 4:19 PM
# re: Structuring Unit Tests
Sorry. Not digging it. I would say more, but Ayende said it better: ayende.com/.../structuring-your-unit-tests-why
Requesting Gravatar... Chris Marisic Jan 04, 2012 4:34 PM
# re: Structuring Unit Tests
I write all my unit tests in MSpec with Machine.Fakes.Moq using Machine.Fakes allow you to share test configuration through composition instead of inheritance. It's a rather godly pattern.

I'm still not set on how to organize tests using MSpec and opened a question on stackoverflow about this very topic a few weeks ago: stackoverflow.com/...
Requesting Gravatar... Khuzema Jan 04, 2012 9:22 PM
# re: Structuring Unit Tests
Question about Nuget website (nuget.org) related, is there is some documentation (or blog post), which can explain about the website design, as it is so generously been open sourced and so soundly structured. (A really real world Asp.Net MVC uses example).

Thanks
Requesting Gravatar... Paul Jan 05, 2012 7:56 PM
# re: Structuring Unit Tests
I really like this. I have been using a convention of starting my test names with the function name i.e. Knightify_ShouldAppendSir_WhenArgMailIsTrue. This new structure would set the context making my test names shorter. Thanks for sharing this tip!
Requesting Gravatar... Robert Penner Jan 06, 2012 7:46 PM
# re: Structuring Unit Tests
Test features, i.e. use cases, not methods.
Requesting Gravatar... William Hoge Jan 07, 2012 1:40 AM
# re: Structuring Unit Tests
Just wanted to say I'm new here and love the blog. Great conversation and many good laughs.
Requesting Gravatar... Gary McLean Hall Jan 10, 2012 9:37 PM
# re: Structuring Unit Tests
I blogged about unit test organization here:

garymcleanhall.wordpress.com/...

...and I came up with Behavioral, here:

http://behavioral.codeplex.com/

Not sure what direction it's going to go in, and it needs work, but it was intended to create more readable unit tests, as well as to allow reuse.
Requesting Gravatar... Julien Bérubé Jan 12, 2012 7:18 PM
# re: Structuring Unit Tests
@Karep : You'd lose one big advantage, which is maintainability. If you change the name of one of your method under test, you end up having to rename every test method manually.
Renaming one class name is faster than renaming 27 method names. Even by using replace all, that's risky.
Plus, if you do peer review, you have less lines of code to review, and if you use source control, less risk of conflict.
Requesting Gravatar... Zak David Jan 15, 2012 6:42 PM
# re: Structuring Unit Tests
Interesting idea but I'm concerned that tying your tests and your code so closely will prove to be a pain when you refactor (e.g. rename a production method and you need to rename tests).

I prefer to be a bit looser with this: name my tests to describe what they test instead of tying them to a method name and use the test group feature offered by TestNG (Java) to group my tests logically:

@Test(groups = { "front-end", "slow", "servlet" }
public void loginShouldWork() { ... }

This method will be run if I run the group "front-end" but it will be skipped if I only want to run fast tests.

TestNG: http://testng.org
Requesting Gravatar... JF Jan 23, 2012 6:58 PM
# re: Structuring Unit Tests
I understand that when wrapping tests around existing, untested, legacy code, these are fine guidelines. For new development, though, Titleizer violates SRP - it isn't single responsibility. That class does two distinct operations.
Once have Titlizer.Titleize() and Knightifier.Knightify(), I've got Single Responsibility and Naming down, and the tests now organize themselves. I don't need to nest anything into child classes.
As a bonus, now when I add the functionality to prepend "Lord" or "Lady" to Titleize(), I don't need to modify the Knightifier class or its tests.
Requesting Gravatar... ZenDeveloper Jan 26, 2012 4:47 PM
# re: Structuring Unit Tests
Phil,

Thanks for a great article. I have take the examples here and adapted them to an MS Test environment with a few issues resolved (namely DRY). Check it out http://bit.ly/yE7GbZ
Requesting Gravatar... haacked Jan 26, 2012 8:10 PM
# re: Structuring Unit Tests
@ZenDeveloper I like the little touch where you have the child classes inherit from the parent for common setup. Clever. :)

What do you have to say?

(will show your gravatar)
Please add 2 and 8 and type the answer here: