Tell Me Your Unit Testing Pains

If you know me, you know I go through great pains to write automated unit tests for my code. Some might even call me anal about it. Those people know me too well.

For example, in the active branch of Subtext, we have 882 unit tests, of which I estimate I wrote around 800 of those. Yep, if you’re browsing the Subtext unit test code and something smells bad, chances are I probably dealt it.

IMG_1117 Unfortunately, by most definitions of Unit Test, most of these tests are really integration tests. Partly because I was testing legacy code that and partly because I was blocked by the framework, not every method under test could be easily tested in isolation.

Whether you’re a TDD fan or not, I think most of us can agree that unit testing your own code is a necessary practice, whether it is manually or automated.

I still think it’s worthwhile to take that one step further and automate your unit tests whenever possible and where it makes most sense (for large values of make sense).

When writing an automated unit test, the key is to try and isolate the unit under test (typically a method) by both controlling the external dependencies for the method and being able to capture any side-effects of the method.

Sometimes though, external code that your code makes calls into can sometimes be written in such a way that makes it challenging to test your own code. Often, you have to resort to building all sorts of Bridge or Adapter classes to abstract away the thing you’re calling. Sometimes you are plain stuck.

What “external code” might exhibit this characteristic of making your own code hard to test? I don’t have anything in particular in mind but...oh...off the top of my head if you made me pick one totally spontaneously, I might mention my way of example one little piece of code called the .NET Framework.

For the most part, I’ve had very few problems with the Base Class Libraries or other parts of the Framework. Most of my testing woes came when writing code against ASP.NET. The ASP.NET MVC framework hopes to help address some of that.

I’ve been in a lot of internal discussions recently talking with various people and teams about testable code. In order to contribute more value to these discussions, I am trying to gather specific cases and scenarios where testing your code is really painful.

What I am not looking for is feedback such as

It’s hard to write unit tests when writing a Foo application/control/part”.

or

Class Queezle should really make method Bloozle public.

Perhaps Bloozle should be public, but I am not interested in theoretical pains. If the inaccessibility of Bloozle caused a real problem in a real unit test, that's what I want to hear.

What I am looking for is specifics! Concrete scenarios that are blocked or extremely painful.Including the actual unit test is even better! For example...

When writing ASP.NET code, I want to have helper methods that accept an HttpContext instance and get some values from its various properties and then perform a calculation. Because HttpContext is sealed and is tightly coupled to the ASP.NET stack, I can't mock it out or replace it with a test specific subclass. This means I always have to write some sort of bridge or adapter that wraps the context which gets tedious.

Only, you can’t use that one, because I already wrote it. Ideally, I’d love to hear feedback from across the board, not just ASP.NET. Got issues with WPF, WCF, Sharepoint Web Parts, etc... tell us about it. Please post them in my comments or on your blog and link to this post. Your input is very valuable and could help shape the future of the Framework, or at least help me to sound like I am clued into customer needs next time I talk to someone internal about this. ;)

Technorati Tags: ,

What others have said

Requesting Gravatar... DotNetKicks.com Feb 06, 2008 8:36 AM
# Tell Me Your Unit Testing Pains
You've been kicked (a good thing) - Trackback from DotNetKicks.com
Requesting Gravatar... Chris Brandsma Feb 06, 2008 8:52 AM
# re: Tell Me Your Unit Testing Pains
For me, testing is all about my ability to fake, mock, or stub other code. When I test a method, all I want to test is that method -- nothing else. The problem occurs when that method calls anther method in the same class -- I can't fake out that call, not easily anyway. So now I have to account for the method I'm calling, the method it is calling, and so on and so forth.

Constructors can also be a problem. You try to keep your constructors as simple as possible, just pass in any dependencies and let the work happen. It isn't always possible. A typical issue is you have to hook up an event in the constructor. It just make setting up a mock object...interesting.

Next, static singletons. These are often used in the place of globals for an application, holding special values (connection strings, etc). A static singleton can't be mocked (not by Rhino Mocks anyway -- my favorite tool so far).

Finally: trying to explain to another developer what a stub, fake, and a mock are. I've had many occasions where just pounding my head against the wall would have been more fruitful.
Requesting Gravatar... Scott Bellware Feb 06, 2008 9:28 AM
# re: Tell Me Your Unit Testing Pains
Phil,

It's great that you're asking the community for this kind of input and experience report. Kudos to you for the initiative.

However, if this input is needed to help guide the testability aspects of the MVC framework for ASP .NET, then it's a cause for dismay in the case that you wouldn't already be in possession of the relevant, tangible, hard-won experience in testability design and TDD.

I hope you can excuse my cynicism on this and the obvious and inevitable slight to you, but my experience with Microsoft over the past few years in regards to lip service paid to these issues informs my caution quite strongly.
Requesting Gravatar... Thomas Eyde Feb 06, 2008 9:59 AM
# re: Tell Me Your Unit Testing Pains
Not entirely on topic, but the MailAddress object validates the email address in its constructor and fails with an exception when the address is not properly formatted. That caused a bug in my application, and I want to write a unit test to verify that. The test is easy to write, but the fix is not. I would like to use the very same validation code as the constructor, but I can't. It's internal.
Requesting Gravatar... Jason Feb 06, 2008 10:10 AM
# re: Tell Me Your Unit Testing Pains
I'm yet new to the unit testing world so I might be doing a lot of things wrong, but I'd like to chime in as a newbie.

I think Chris has one point that I agree with. If a class (or a presenter) has a method A, B, and C, and both A and B call method C, how can i avoid writing the unit test for C twice? So I have a unit test for C, a unit test for B, and a unit test for A. Do I just call unit test C from Unit Test A and B? or do I repeat it in each unit test? It doesn't seem like a pure "unit" test if I call unit test C from A and B.

Another pain point I have is testing the Data Access objects. We're not using any database mock objects (mainly due to the fact we started the project using an OR mapping framework, which is gone now). We were actually putting all the DB calls in a transaction so that everything gets rolled back after the test. The bad side is I'd have to write sql scripts to do unit tests. For example, if I need to test a Delete or Update, I need to first do an Insert, which may not exist at the time of testing.

Another struggle I've had with using dynamic mock views on the UI in a MVP pattern, is that a lot of my unit tests seemed meaningless. For example, certain methods just are there to set the properties of my view just to populate a form with a lot of fields. So I end up writing meaningless Expect's just to say a bunch of Expect.MyView.FirstName.To(anyhting); Expect.MyView.LastName.To(anything); ... etc

If I add another field to the form, the unit test fails because I'm missing that extra Expect call for that added field.
Requesting Gravatar... Vish Feb 06, 2008 10:12 AM
# re: Tell Me Your Unit Testing Pains
Hi Phil,

Here are my thoughts on unit tests

1) They are cumbersome
2) Takes a whole bunch of time and typing
3) Currently almost no reuse
4) There is no challenge to it (once you figure out how to do testing); so doesn't keep the developers interest

What i would love to see in the future is tools that spit out tests by almost recording a macro of what i do on the screen. Think recording macros in like the 'object test bench' (remember that... but on class diagrams??). Others may have differing opinions on it but that is only way it will be widely adopted by Joe Developer as a part of everyday development tasks.

Thank You,
Vish
Requesting Gravatar... David Fauber Feb 06, 2008 10:26 AM
# re: Tell Me Your Unit Testing Pains
Its very difficult to test anything in the template area of a WebForms page or user control. Its possible to do by calling on Execute on the Server object (which requires HttpContext) and caching the output,but still very painful. (although you could easily just say "use mvc instead", webforms is going to stick around)
Requesting Gravatar... mike h Feb 06, 2008 10:43 AM
# re: Tell Me Your Unit Testing Pains
On the SharePoint Search Bench project (http://codeplex.com/spsearchbench), I'm specifically struggling with unit testing methods that execute searches with the SharePoint API. Like HttpContext, The SharePoint search API classes are sealed and do not implement an interface. Mocking isn't an option and multiple times I've decided that creating a bridge or adapter is too much work for the value that my wrapper API provides. I've settled for at least creating a "test" context that can be swapped in/out with the "real" context that uses the real SharePoint API. The test context is great for unit testing the UI, but not the API itself. This is the kind of stuff that really makes unit testing a turn off for me. In the end, I just have to keep writing layer upon layer of buffer code that adds little value to the real functionality I'm trying to develop.
Requesting Gravatar... Haacked Feb 06, 2008 11:49 AM
# re: Tell Me Your Unit Testing Pains
@Scott I'm involved in Framework-wide discussions, not just ASP.NET MVC. Hence the request for feedback.

Even if this were limited to ASP.NET, I don't see how asking for more feedback is a cause of dismay. I've been doing TDD and ASP.NET for years and have tons of battle scars in doing so and the "hard-won experience" you mentioned. That doesn't mean I've run into *every* possible testability problem.

My problems are not necessarily your problems. etc... etc...

So I welcome feedback across the board. :)

Phil
Requesting Gravatar... Dave S. Feb 06, 2008 1:07 PM
# re: Tell Me Your Unit Testing Pains
My pain is I cannot get said team members to even grasp the concept. *yikes*
Requesting Gravatar... Wendy Feb 06, 2008 1:22 PM
# re: Tell Me Your Unit Testing Pains
I find its easier to write testable classes with WPF. However, testing classes that use ObservableCollections can be difficult. The setup can be very large and the tests difficult to follow. Also, the event for the collection change is annoying to work with (why aren't the item lists generic??)...

One helpful pattern is to create a single class that observes the collection and forwards changes to an (or many) INotifyObservableCollectionChanged interface. That way you only have one class you need to test with an ObservableCollection and everyone else who wants to watch implements the interface, which is easy to test.

public interface INotifyObservableCollectionChanged<TypeOfObjectsInCollection>
void ObjectsAdded(IList<TypeOfObjectsInCollection> addedObjects)
void ObjectsRemved(IList<TypeOfObjectsInCollection> removedObjects)
...

I hope that made some sense :P
Requesting Gravatar... Pawel Pabich Feb 06, 2008 2:37 PM
# re: Tell Me Your Unit Testing Pains
A few examples from the back-end.
1. OperationContex in WCF is sealed and does not expose its functionality via interface(s).
2. WorkflowInstance in WF. The same as above.
Requesting Gravatar... Daniel Feb 06, 2008 3:08 PM
# re: Tell Me Your Unit Testing Pains
For me, it's testing database-related code. I'm quite sure the pain is from something I just don't "get", but here's a scenario:

I create a LINQ-to-SQL model for Northwind, and some classes around the model. To test properly I need to restore the db from a known-good state before each test, or else use a transaction-based approach that always rolls back the db after each test. Either way, the tests get pretty long-running pretty quickly. Also, I may need to test against large datasets to be sure my app works with 1000s of records. But a database backup that large doesn't fit in source control well.
Requesting Gravatar... Vijay Santhanam Feb 06, 2008 4:20 PM
# re: Tell Me Your Unit Testing Pains
Howdy Phil,

Like Daniel, I'm having trouble mocking the System.Data.Linq.DataContext and all the LINQ-to-SQL ORM classes I've created.

I wish there was an easy way to mock or set up a low-footprint in-assembly database that doesn't require installing a new SQL server instance.

Maybe embedding the desktop engine into the test assembly will work. Does that support LINQ ORM?

I don't think this testing scenario is easy enough, or at least I don't know a good solution yet.



Requesting Gravatar... seerox Feb 06, 2008 10:50 PM
# re: Tell Me Your Unit Testing Pains
Database for sure. I can't find any good examples of how to do it. People talk about distributed transactions etc but noone has an end to end. There is a ton of info out there and many pitfalls. I used datafresh a bit and I liked that but you have to have xp_cmdshell privileges etc. I don't mind an integration tests either for the db layer.

How about a team with many products and source control...How do you version the db? My situation here may be somewhat unique since we have hundreds of apps that hit the same db structure but keeping unit tests synced with db structure etc is a real killer.
Requesting Gravatar... Anand Feb 07, 2008 1:29 AM
# re: Tell Me Your Unit Testing Pains
I follow Test Last Development, where I design the object model and write unit tests then. Recently I had a problem unit testing singleton class. It used eager instantiation and had only one constructor, which used to get data from two dependent systems. The singleton class depends upon two gateway classes which interacted with the external dependent system. I could not introduce the constructor injection since it followed singleton pattern. I did some workaround and was able to carry out unit testing but still feel that the workaround is poor.

I would like to hear from you the solution for unit testing singleton classes.
Requesting Gravatar... Mister P Feb 07, 2008 8:00 AM
# re: Tell Me Your Unit Testing Pains
Phil, great writing as usual. I'm trying to catch up on all your latest writings on TDD, is there any book and/or online-resource that you could recommend for TDD (preferable with C# and .NET)?

Thank you
Requesting Gravatar... Jimmy Bogard Feb 07, 2008 8:20 AM
# re: Tell Me Your Unit Testing Pains
@Anand

My suggestion is to check out Michael Feather's Legacy Code book for suggestions. Singletons are notoriously difficult to test, and you'll usually find that you want to move away from them.

If we want singletons, we configure our IoC container to only construct one object when asked.

@Vish

All I can say is I hope I never inherit a codebase from you. Code without tests is legacy code, period. Instead of saying "Joe Developer can't grasp this or gets bored", let's raise the bar. MVC is doing just that.

What you're asking for is the IDE to read your mind on the intentions of you WANT the code to be, not even it what it is in reality.

What's more cumbersome than unit tests? Fielding support calls for months after a release because the software doesn't do what you wanted it to do.
Requesting Gravatar... Justice~! Feb 07, 2008 10:09 AM
# re: Tell Me Your Unit Testing Pains
Vish, in my typical unbiased way I must say that this commment was LEEEWWWSER.

Phil - my big difficulty right now (aside from the ones we've already talked about) is the inability to properly override RedirectToAction(string, string) for inheritance-based tests. Even for mocking it is a bit odd given that it boils down (eventually) to a Response.Redirect call which, if I am not mistaken, requires the full path with the http://. I might be full of crap on this last part though so I'm going to investigate further.
Requesting Gravatar... The Other Steve Feb 07, 2008 11:38 AM
# re: Tell Me Your Unit Testing Pains
Ok, my concern is it almost seems like I'm writing more code in tests and stubs and such, than actual logic. The more code I write the more bugs I introduce.

So who will tests the tests?

This is the problem I have with writing layers of abstraction and such just to make my code easier. By introducing all that other stuff, I risk crapifying my codebase.
Requesting Gravatar... Bradley Landis Feb 07, 2008 1:23 PM
# re: Tell Me Your Unit Testing Pains
I could be doing something wrong but I always have to take extra steps when unit testing code using System.Net.Mail. The Send method of SmtpClient is not marked as virtual.
Requesting Gravatar... Bradley Landis Feb 07, 2008 2:25 PM
# re: Tell Me Your Unit Testing Pains
I don't mean to sound harsh or disrespect anyone, but to the people are stating that unit testing is hard:

The difficulty associated with testing your code is usually directly proportional to the design and quality of your code.

In other words, poorly designed, poorly written code IS hard test. That is the reasoning for Test Driven Development. If you write your code so that it is testable, you will almost certainly write better designed, better written code.
Requesting Gravatar... Bradley Landis Feb 07, 2008 2:36 PM
# re: Tell Me Your Unit Testing Pains
To "The Other Steve":

I believe I read that a typical TDD application has a 2-1 ratio of lines of test code to lines of system code. I believe that is accurate in my experience.

Think of it like this, if you were to write a document and then write a document about how your wrote the first document, the second document would certainly be twice as long as the first.

As for your issue with the layers of abstraction; if you have to add the layers of abstraction to test, those layers almost certainly should have been there in the first place. Your tests just forced you to write better code.
Requesting Gravatar... Haacked Feb 07, 2008 11:53 PM
# re: Tell Me Your Unit Testing Pains
@The Other Steve - I actually wrote on this topic in my post, Who Tests the Tests?

The more code I write the more bugs I introduce.


Perhaps, but the extra code is mostly in your unit tests. You don't deploy your unit tests, so even if the occasional unit test has a bug, overall, it's still helped the code you will deploy.
Requesting Gravatar... Jeffrey Palermo Feb 08, 2008 8:33 AM
# re: Tell Me Your Unit Testing Pains
I'm confused. You told me a lot of issues are fixed for the next release and that is why current feedback on the Dec CTP might already be outdated. I have plenty of feedback, much too much to write here. Give me a call, and we'll talk about it.
Requesting Gravatar... Will Shaver Feb 08, 2008 10:08 AM
# re: Tell Me Your Unit Testing Pains
Just got finished writing a framework to assist with unit testing the MVC. (Currently on my blog, I'll probably be submitting it to MVC Contrib.) There were a couple of spots that were particularly annoying.

Mocking the session is perhaps the most difficult aspect of unit testing controllers. HttpSessionStateWrapper is of course marked internal, and HttpSessionState (contained by the wrapper) is public sealed with an internal constructor. Specifically mocking the saving of variables and retrieving of variables is difficult.


MyController mc = new MyController(); // or use a factory
mockedSession["variable"] = "value";
mc.ActionWhichUsesSession();
Assert.AreEqual(mc.HttpContext.Session["variable"],"value")


Part of the difficulty comes in the fact that HttpContext exposes a property indexer to save and retrieve items.

Also annoying was RedirectData and RedirectDataWithController being marked as internal. These are used in the RedirectToAction function internally, and when passing strings for the action and controller to RedirectToAction(string,string) it creates these classes. When overriding the virtual RedirectToAction one must iterate over PropertyInfo objects instead of just doing an "is" comparison on the type.

Requesting Gravatar... Ngu Soon Hui Feb 08, 2008 11:28 PM
# re: Tell Me Your Unit Testing Pains
I have documented my experience here
Requesting Gravatar... Slevdi Feb 09, 2008 2:12 AM
# re: Tell Me Your Unit Testing Pains
TDD best practice examples. I learn nothing from books, articles etc unless I can fire up a relevant example. Thanks to Phil H here for some seminal articles.

Webcasts seem to have the highest ratio of time spent to knowledge gained (I particularly recommend the awesome 1 hour long webcast on RhinoMocks using TDD in VS2005 featuring Resharper from Ayende - it was worth a month of reading books to see an expert at work)
Requesting Gravatar... SeeR Feb 09, 2008 11:59 AM
# re: Tell Me Your Unit Testing Pains
Testing in WPF is hard for me because of following things:

1) There can be only one Application class instance
2) Window with XAML can't be inherited
3) Again (like in Windows.Forms) OpenFileDialog and SaveFileDialog are sealed
4) Microsoft should use interfaces instead of class dependencies in freamework or should remove from CLI keywords sealed and internal
Requesting Gravatar... Will Shaver Feb 11, 2008 11:14 AM
# re: Tell Me Your Unit Testing Pains
Still trying to mock the session... this time by hand. Turns out that one of the interface members is:

public NameObjectCollectionBase.KeysCollection Keys { get; }

Doesn't look so bad at first, but it turns out that while the KeysCollection is public, the constructor is marked as internal. This makes it practically impossible to mock this interface.

I think a general practice for framework design is to either:

a) Create a test-specific side of the framework that has members mocked out for users.

or

b) All interface members should return either interfaces or classes that are not defined as internal.
Requesting Gravatar... Alex G Feb 14, 2008 7:53 AM
# re: Tell Me Your Unit Testing Pains
Testing ASP.NET pipe line is somewhat similiar in experience to using a sandpaper for whiping oneself. It's doable, but the process is mildly pleasing.

Half of the public classes have internal constructors and half of the overloads are internal as well. The novelty of things like AppDomain.CurrentDomain.SetData(".appDomain", "*") never wears off.

Requesting Gravatar... Justice~! Feb 21, 2008 2:30 PM
# re: Tell Me Your Unit Testing Pains
Hey Phil, while still on this discussion it would be nice to have IHttpContext (sorry, HttpContextBase) implement either an interface or wrapper around Cache, so that I don't have to write my own wrapper to test some of those implementations...
Requesting Gravatar... Jay Jul 28, 2008 12:24 AM
# re: Tell Me Your Unit Testing Pains
How can I Test silverlight controls in .net?
I have to create every time new project for testing it
Requesting Gravatar... Mudit Vajpayee Mar 27, 2009 5:54 AM
# re: Tell Me Your Unit Testing Pains
Hey..We have developed a portal where we kept our bussiness logic into .XSD File and for creating the Unit Test Case we placed the file in APP_Code Folder.But while creating unit test case for any method defined in a class which in turn accesing that .XSD file an exception occured.So we place that file into APP_Data Folder but after that we can't access the logic written in that .XSD file while creating test cases.

Please suggest..

What do you have to say?

(will show your gravatar)
Please add 6 and 3 and type the answer here: