Unit Test Project Structure Poll

Apr 9, 2008 code tdd suggest edit

When I build applications, I personally like to have my unit tests in a separate class library project than the application I am testing. That’s just how I roll.

I just assumed this is how everyone structures their unit tests, but I’ve talked with some people who have a deep history in TDD who put their code in the same project.

So I wanted to create a simple poll to find out how people are actually structuring their unit tests. I’ve been involved in various internal discussions looking at how Microsoft design, tools, code can better support unit testing across the board.

Note that this is not an opinion question. I don’t think one way is “more right” than another. I just want to make sure we have an accurate view of what the real practice is out there.

So please do answer this poll and encourage others to do so. I’d hate for internal teams to make choices based on a wrong assumption. Thanks!

How do you structure your unit tests?
(polls)

Found a typo or mistake in the post? suggest edit

Comments

33 responses

Jon Limjap • April 9th, 2008
If you're really really doing TDD with near 100% coverage, your tests will only bloat your assembly if included in the same project/assembly.
So my answer above was to put unit tests in a separate project, though I dunno if there's a way to split assemblies within a project. Point is your tests shouldn't be in the same assembly.
Derik Whittaker • April 9th, 2008
his is debate that I have had many, many times over the years. I too like the in separate projects as it allows me to 'black box' test my code.
Both ways have trade offs though, and to say one way is 'better' is not valid. Both are good, but just form different agles..
Haacked • April 9th, 2008
@Jon Limjab true, but it's possible to have a build step that strips the tests from your application code when building a production build, for example.
Currently, VS doesn't have an easy way to do that. But if it did, would it change your mind about where you put your tests?
Owen Evans • April 9th, 2008
I post in a separate assembly for one reason, to remove reliance on Nunit/mbunit/nspecify assemblies, which has bitten me before when it came to deployment.
Also i will generally add an assembly:InternalsVisibleTo so that i can test internal methods, which is one of the reasons I used to have tests in the same assembly.
Cheers
O
Derik Whittaker • April 9th, 2008
@Phil,
You can use #if to remove your tests in release code.
Jeff Gonzalez • April 9th, 2008
I think I started out putting them in a seperate project just to help me get my head around testing. I think an upside is that there is some physical seperation and it keeps me from doing too many clever things in my tests. I mean, it is possible, I, uh, hear, to have an app that won't run without the presence of the tests. You'd be a bad programmer to do that, but, I, uh, hear it could happen.
David R. Longnecker • April 9th, 2008
I keep my tests in a separate assembly just for cleanliness of the production projects and convenience of seeing my tests (and only my tests) in one location.
These advantages may simply be in my mind's eye, but it just "feels right".
Scott • April 9th, 2008
@Jon LimJap

though I dunno if there's a way to split assemblies within a project. Point is your tests shouldn't be in the same assembly.

Yeah, you can if you use the command line compiler or some kind of build tool with a compiler task. In a nutshell, you build modules, then combine the modules you want into a single assembly. So your makefile could pass a list of all the code files except the ones containing your tests, to the compiler, even across projects, and output a single assembly. AFAIK, there's no way to do this in the Visual Studio IDE.
Mike Thomas • April 9th, 2008
I put tests in a separate console application project and use MbUnit so that I can run that project and execute my tests without needing a test runner. I do use TestDriven.NET but it is also nice to be able to run the tests in an environment that doesn't have that installed.
Chris Brandsma • April 9th, 2008
Further, my typical solution has multiple projects. I like having a multiple test projects (sometimes one test project for each production project, but not always), plus I will have another test project if I have integration tests (which take longer to run).
What I find more interesting right now is the idea of turning your use cases into test names, and your test classes center around use case features. I've seen Jean-Paul Boodhoo do that.
Steve Campbell • April 9th, 2008
The reason I put tests in a different assembly is that they can have different dependencies (references) than the project being tested. If I were to mingle in the same project then I could easily become confused as to the real dependencies.
Ed McCaffrey • April 9th, 2008
Different assemblies for the application and tests. Sometimes different assemblies for the test of each project, if there are a lot of projects with a lot of tests.
Bryan Watts • April 9th, 2008
Separation of tests from their target is essential for separation of responsibility - to employ a slippery slope, why not embed deployment code too?
I prefer a single test assembly per project assembly, with the same root namespace and structure, leading to natural scoping.
S M Sohan • April 9th, 2008
I have done it the in both ways. Before recent days, I kept my test codes in the same assembly and manually edited the csproj files to conditionally include the files containing test codes. This way it worked for simple assemblies.
However, Presently I favor the separate assembly approach, since it makes it easy to deploy and separates the two concerns.
Rasmus Grouleff • April 9th, 2008
I don't know if my comment belong here, since you're all talking about .Net.
I'm a Java developer, and Eclipse is my IDE of choice - Eclipse has in my eyes an excellent middleway with the possibility to create several source folders in a single project.
The brilliant thing about source folders is that they map to different paths on the disk. It's therefore really easy to filter unit tests away in the build tool, when releasing a new version, but you still get the advantages of having unit tests within the same project.
Colin • April 9th, 2008
In most cases I'll use a separate assembly. Sometimes if there is a non-public class / method that needs testing I'll link all the classes of the target assembly into the test assembly and access the internal classes that way.
Torkel • April 9th, 2008
In a separate assembly.
It would be nice if the CLR or C# provided a way complete expose private and protected classes and methods to a testing assembly. As far as I know this can only be done for internals?
Ritchie Swann • April 9th, 2008
I always put the tests in a separate assembly. My coding conventions dictate this will always be the name of the assembly it's testing with an addition "Test" wart.
At post-build time, it's then easy to not ship the unit test assemblies - simply don't publish anything that doesn't end in ".Test.dll".
"It would be nice if the CLR or C# provided a way complete expose private and protected classes and methods to a testing assembly."
You can do this using reflection, by using the GetMethod() method on a type, passing in the NonPublic flag. Google for "GetMethod method". I've got some sample code that does this if you want it.
David Arno • April 9th, 2008
Ritchie,
C# does provide a very nice way of exposing protected and internal members.
Protected members are exposed to derived classes, thus can be easily accessed via a derived test class.
Internal members are exposed to friend assemblies, thus can be easily access via test classes in a friend assembly.
I've never found the need to access private types and would be concerned with any mechanism that allowed it to occur. The implication that reflection lets you bypass the security of a private member and access it outside the scope of the class is disturbing. A far safer, and more controlled, solution would be to make use of #define, #if etc to expose private data as internal within the context of a test build.
David Arno • April 9th, 2008
Sorry the previous post should have addressed Torkel, not Ritchie. Apologies for the confusion.
I've written a couple of tutorials on using friend assemblies and #define etc, that may be of use here.
Freek Leemhuis • April 9th, 2008
I'm assuming most of you use NUnit or some alternative, but if you're using the build-in test framework in Visual Studio, you are actually required to place your unit test in a separate assembly ('test project'), so there's no way you'd mix in your unit tests with your code.
Joe Brinkman • April 9th, 2008
Another Microsoftie spouting off on TDD when they clearly don't.... Oh, wait... sorry, this was supposed to go on Rob's blog ;)
I'm with you Phil. The last thing I want on my webserver is a bunch of code that will never execute (and should never execute in production). On the web, every byte counts, whether it is traveling across the wire or consuming server memory.
Ritchie Swann • April 9th, 2008
David,
That's okay. What you said about private types should be taken on board by anybody trying to hack around it in reflection. Just because you can, doesn't mean you should.
Bil Simser • April 9th, 2008
Thanks to Phil for yet another great discussion. We were actually having this discussion on our day jobs and are trying the experiment of keeping test assemblies in our domain code (check out my blog for a long description and comments from others on the approach).
I'm leaning on the side of:
physical structure != domain structure != test structure != deployment
We'll see how far I fall from the horse.
Shawn Oster • April 11th, 2008
I agree with others here, test in a seperate assembly project. I do it mostly for cleanliness and keeping references down in the main project, plus it really makes me think about my external API, my future-self and other developers will also be using my objects "from the outside" so I want to test from the outside.
Also, TDD isn't *just* about testing, it's about using your framework before it's built to help you hammer out real-life usages, naming conventions and object interactions. It's about getting a feel for how others will interact with your classes and testing from the inside blurs that line.
Fervent Coder • April 13th, 2008
I am all about having the tests in a different project with one caveat. When I start developing a class I have the tests for it in the same file to speed up TDD. This quickly moves out as soon as I feel comfortable with what I am working with. This allows quick development of the item and R# really helps with the moving of the item out on it's own.
Actually, we use several test libraries. We also use several assemblies for the solution that tie down to...function (or some natural point of separation). Each project gets it's own test project. So if we have Organization.Solution.Project, we have Organization.Solution.Project.Tests.
What goes in these projects are logical tests.
Then, with anything that touches external things to the code goes to a project that is named in this format: Organization.Solution.Tests.Integration.
These are what we define as integration external to the system. Do I have to read from a file? Touch a database? Call a Web Service? We test that we can actually do this here.
Fervent Coder • April 13th, 2008
One follow up...I didn't really explain why I would want to keep things in one file for a short amount of time. This is so I keep all of my focus right there and not have to go back and forth between files. Plus I know I am done working with the class (in the short interim) when I have moved the tests out to there own file, copied that file down to it's test project and removed the newly created file.
David Fauber • April 13th, 2008
I keep them in a separate project. I think I probably started doing this because I originally started unit testing with the Unit Test project templates which came with Visual Studio, so that may invalidate my response to some degree.
jeff jarrell • April 13th, 2008
Separate DLL. I started out with 1 test assembly per release assembly but that proved too tedious to maintain with the continuous integration build. Now I have 1 assembly for unit tests (across the solution) and 1 assembly for integration tests).
I do leave some types public that should really be internal but I try and hide these with a more detailed namespace.
evan • April 23rd, 2008
I remember before all these xunit programs were well known having a static void main in each java class that tested that class and were executed in batch from the command line.
Things sure have changed though and these xunit frameworks are great. Putting all that test code in the main project has a bad smell to me. Looks like lots of people smell that too :)
Jason Truesdell • April 28th, 2008
I like keeping unit tests in the same project, but I think that's due to my exposure to Java and Rails where it's painless to strip them out of any production packages (assemblies in the dotnet context).
Maven also enables test-scoped dependencies, so adding a reference to a unit testing framework doesn't cause any deployment-time problems.
However, for "integration" tests, I put them in their own project.
In the current ASP.net MVC experimental project I'm doing, I've followed the Visual Studio convention of having a separate project for unit tests, though it messes with my head a bit.
Generally, I just try to follow the conventions of the particular environment I'm using, unless those conventions suck. For example, in ruby/rails I use this_naming_convention_for_methods but I use TheDotNetStyle when writing stuff in C#, and theCamelCaseStyle for methods in Java, just because it's idiomatic in those worlds and looks wrong when I break the rules. The same with unit testing: If the way it's usually done in VS doesn't cause me trouble, I'll do it that way.
Lance Kind • January 26th, 2009
I name my tests using the following pattern: <unit under test>Test.cs|java. Because of this, I can use an alphabetized view to quickly assess what has unit tests and what does not.
I actually don't care where my unit tests are physically located as long as the IDE gives me a view that lets me see a list of units under test and their corresponding unit test right next to each other. I want to easily identify if someone dropped a unit in that needs testing (or split a class into two).
This seems to mean putting the unit tests in the same assembly when I'm using Visual Studio. Assembly bloat has never been even close enough to be a concern. Besides, you want to be careful about optimizing on the wrong thing. The most expensive thing about software is the developer (way more expensive than disk space), so you want them to be working as effectively as possible.
In Eclipse, I use the package view so I can see my units and their corresponding tests side by side as long as they are declared in the same namespace/package. If you're building under Maven, it will automatically strip out the test classes so if you were to worry about the drop-in-the-bucket issue of code bloat, it will be gone.
sly • April 18th, 2010
I 've tried successfully this idea:
1) package all your code-to-be-tested under a root package
2) Create a "Test solution" in which you create a " test project"
3) Add the root package between under the test solution ("Add existing folder")
By this way, root package is shared between the actual solution and the test solution.
In the test solution, code-to-be-tested and test are in the SAME assembly.
There no deployment problem ,as tests even do no exist in the actual solution.
This is maybe a draback of this methodology. You have to deal with two solutions, and it is not as comfortable as to have only one with testing projects.