Structuring Unit Test Code

Jeremy Miller asks the question, “How do you organize your NUnit test code?”.  My answer? I don’t, I organize my MbUnit test code.

Bad jokes aside, I do understand that his question is more focused on the structure of unit testing code and not the structure of any particular unit testing framework.

I pretty much follow the same structure that Jeremy does in that I have a test fixture per class (sometimes more than one per class for special cases).  I experimented with having a test fixture per method, but gave up on that as it became a maintenance headache.  Too many files!

One convention I use is to prefix my unit test projects with "UnitTest".  Thus the unit tests for Subtext are in the project UnitTests.Subtext.dll.  The main reason for this, besides the obvious fact that it’s a sensible name for a project that contains unit tests, is that for most projects, the unit test assembly would show up on bottom in the Solution Explorer because of alphabetic ordering.

So then I co-found a company whose name starts with the letter V.  Doh!

UPDATE: I neglected to point out (as David Hayden did) that with VS.NET 2005 I can use Solution Folders to group tests. We actually use Solution Folders within Subtext. Unfortunately, many of my company work is still using VS.NET 2003, which does not boast such a nice feature.

One thing I don’t do is separate my unit tests and integration tests into two separate assemblies.  Currently I don’t separate those tests at all, though I have plans to start. 

Even when I do start separating tests, one issue with having unit tests in two separate assemblies is that I don’t know how to produce NCover reports that merge the results of coverage from two separate assemblies.

One solution I proposed in the comments to Jeremy’s post is to use a single assembly for tests, but have UnitTests and Integration Tests live in two separate top level namespaces.  Thus in MbUnit or in TD.NET, you can simply run the tests for one namespace or another.

Example Namespaces: Tests.Unit and Tests.Integration

In the root of a unit test project, I tend to have a few helper classes such as UnitTestHelper, which contains static methods useful for unit tests. I also have a ReflectionHelper class, just in case I need to “cheat” a little. Any other classes I might find useful typically go in the root, such as my SimulatedHttpRequest class as well.

What others have said

Requesting Gravatar... Jamie Cansdale Oct 11, 2006 12:30 AM
# re: Structuring Unit Test Code
"Even when I do start separating tests, one issue with having unit tests in two separate assemblies is that I don’t know how to produce NCover reports that merge the results of coverage from two separate assemblies."

If you are using TD.NET you can select all of the test assemblies you wish to merge. You then 'Test With... Coverage' and all of the coverage results will be merged.

If you're using NCover in a build script, ask Grant Drake about using NCoverExplorer reporting to merge the coverage file. I think there's a way, but I couldn't find it after doing a quick search.
Requesting Gravatar... Haacked Oct 11, 2006 8:36 AM
# re: Structuring Unit Test Code
Thanks Jamie! Yes, my concern is with our CruiseControl.NET build script.
Requesting Gravatar... Ben Scheirman Oct 11, 2006 9:19 AM
# re: Structuring Unit Test Code
Jamie: That's awesome. TDD.NET never ceases to amaze me.

Phil: You should REALLY separate your unit tests from integration tests, because as your projects grow you'll have a very large number of tests.

Unit tests run very quickly, because they are usually testing in-memory behavior, however integration tests likely talk to other systems (like a database) and are inherently slow.

You want your developers to run their tests regularly, but this will be less likely if it takes 45 seconds (or more) to run the test suite (and this number is likely to skyrocket as the project gets more complex). Unit tests should be run regularly (like with every build on a dev box) Integration tests should be automated on a build machine, and run upon-need on a dev box.

My 2nd comment is the naming. Your recommendation above breaks good namespacing guidelines.

You should structure it like SubText.UnitTests.dll, so that it is obvious that "SubText" owns that namespace.

I've seen NAnt scripts that build app code into /src/app and test code into /src/test, so you'd still achieve separation.

just a though...
Requesting Gravatar... Haacked Oct 11, 2006 10:11 AM
# re: Structuring Unit Test Code
You should REALLY separate your unit tests from integration tests, because as your projects grow you'll have a very large number of tests.


Hmm I have to respectfully disagree. The count of unit tests does not necessarily require a separate assembly. Even if I did separate them, the unit test assembly could still grow very large. Would I then separate my unit tests into multiple assemblies?

As I pointed out, TDD.NET and MbUnit allow you to run tests in a namespace. I believe that solves this problem. It's a matter of convention. If my team learns to run the unit test namespace while developing, and then run all tests (unit + integration) before commiting, that's satisfactory.

Your recommendation above breaks good namespacing guidelines.


The namespace guidelines you speak of are for public APIs, code that is referenced by other assemblies. For example, if you look at all other subtext assemblies, they follow the conventional namespace conventions.

The unit test assembly is never deployed and never referenced by another assembly. It's namespace is pretty much irrelevant. It's essentially a "private" assembly.

Right now, I do happen to include "subtext" in the unit test namespace like so:

UnitTests.Subtext

But in proposing something new, I wanted to avoid unnecessary namespacing, hence the new proposal to make it very easy to only run unit tests or only run integration tests via the namespace.
Requesting Gravatar... Ben Scheirman Oct 11, 2006 6:59 PM
# re: Structuring Unit Test Code
Hmm I have to respectfully disagree. The count of unit tests does not necessarily require a separate assembly. Even if I did separate them, the unit test assembly could still grow very large. Would I then separate my unit tests into multiple assemblies?


What I mean is not just the sheer number of tests, just how long it takes to run the entire suite.

Your solution of having developers test by namespace is perfectly fine, in fact I didn't think of that! :)

The unit test assembly is never deployed and never referenced by another assembly.


I would still suggest the other strategy, but it is just a matter of preference.

to make it very easy to only run unit tests or only run integration tests via the namespace.


Indeed. I fully agree with you.
Requesting Gravatar... Haacked Oct 11, 2006 7:05 PM
# re: Structuring Unit Test Code
Regarding the namespaces, it definitely is a matter of preference. I like having my unit tests sort to the bottom, but that's just an idiosyncracy. I guess I could break with conventions and use standard namespaces, but name the assembly in a different manner.

Though that also rubs me the wrong way. ;)
Requesting Gravatar... Igor Brejc Oct 12, 2006 5:39 AM
# Merging NCover results
About merging coverage results: you can use NCoverExplorer cmd line tool to do that, here's an example of NAnt command:

<exec
program="lib\NCoverExplorer\NCoverExplorer.Console.exe"
workingdir=".">
<arg value="${NUnitTestResultsDir}\*-coverage.xml"/>
<arg value="/x:${CruiseControlDir}\CoverMerge.xml"/>
<arg value="/r:5"/>
<arg value="/p:${ProjectId}"/>
</exec>

You should change the names of NCover coverage XML reports to something like '<assembly name>-coverage.xml' and store them into a single directory. I hope this helps ;)
Requesting Gravatar... jokiz Oct 12, 2006 7:25 AM
# re: Structuring Unit Test Code
ncover has assemblylist configuration and you just have to configure mbunit console to run the tests on two assemblies (space delimited). re: separation of integration and unit tests, i believe the category attribute of mbunit fit for this, there is also a blog post somewhere on how to configure TD.NET to just run a particular category (unittests in most scenarios) on right click run tests.
Requesting Gravatar... Haacked Oct 12, 2006 9:43 AM
# re: Structuring Unit Test Code
Thanks Igor and Jokis, now I have two different things to try. I should be able to integrate this into CCNET if we decide to use two assemblies.
Requesting Gravatar... Igor Brejc Oct 16, 2006 7:37 AM
# re: Structuring Unit Test Code
Check out new post from the autor of NCoverExpolorer, explaining various ways to merge NCover results
http://www.kiwidude.com/blog/2006/10/ncoverexplorer-merging-ncover-reports.html

What do you have to say?

(will show your gravatar)
Please add 8 and 3 and type the answer here: