Tell Me Your Unit Testing Pains

Feb 5, 2008 tdd aspnet code suggest edit

If you know me, you know I go through great pains to write automated unit tests for my code. Some might even call me anal about it. Those people know me too well.

For example, in the active branch of Subtext, we have 882 unit tests, of which I estimate I wrote around 800 of those. Yep, if you’re browsing the Subtext unit test code and something smells bad, chances are I probably dealt it.

Unfortunately, by most definitions of Unit Test, most of these tests are really integration tests. Partly because I was testing legacy code that and partly because I was blocked by the framework, not every method under test could be easily tested in isolation.

Whether you’re a TDD fan or not, I think most of us can agree that unit testing your own code is a necessary practice, whether it is manually or automated.

I still think it’s worthwhile to take that one step further and automate your unit tests whenever possible and where it makes most sense (for large values of make sense).

When writing an automated unit test, the key is to try and isolate the unit under test (typically a method) by both controlling the external dependencies for the method and being able to capture any side-effects of the method.

Sometimes though, external code that your code makes calls into can sometimes be written in such a way that makes it challenging to test your own code. Often, you have to resort to building all sorts of Bridge or Adapter classes to abstract away the thing you’re calling. Sometimes you are plain stuck.

What “external code” might exhibit this characteristic of making your own code hard to test? I don’t have anything in particular in mind but…oh…off the top of my head if you made me pick one totally spontaneously, I might mention my way of example one little piece of code called the .NET Framework.

For the most part, I’ve had very few problems with the Base Class Libraries or other parts of the Framework. Most of my testing woes came when writing code against ASP.NET. The ASP.NET MVC framework hopes to help address some of that.

I’ve been in a lot of internal discussions recently talking with various people and teams about testable code. In order to contribute more value to these discussions, I am trying to gather specific cases and scenarios where testing your code is really painful.

What I am not looking for is feedback such as

“It’s hard to write unit tests when writing a Foo application/control/part”.

“Class Queezle should really make method Bloozle public.”

Perhaps Bloozle should be public, but I am not interested in theoretical pains. If the inaccessibility of Bloozle caused a real problem in a real unit test, that’s what I want to hear.

What I am looking for is specifics! Concrete scenarios that are blocked or extremely painful.Including the actual unit test is even better! For example…

“When writing ASP.NET code, I want to have helper methods that accept an HttpContext instance and get some values from its various properties and then perform a calculation. Because HttpContext is sealed and is tightly coupled to the ASP.NET stack, I can’t mock it out or replace it with a test specific subclass. This means I always have to write some sort of bridge or adapter that wraps the context which gets tedious.”

Only, you can’t use that one, because I already wrote it. Ideally, I’d love to hear feedback from across the board, not just ASP.NET. Got issues with WPF, WCF, Sharepoint Web Parts, etc… tell us about it. Please post them in my comments or on your blog and link to this post. Your input is very valuable and could help shape the future of the Framework, or at least help me to sound like I am clued into customer needs next time I talk to someone internal about this. ;)

Found a typo or mistake in the post? suggest edit

Comments

36 responses

DotNetKicks.com • February 5th, 2008
You've been kicked (a good thing) - Trackback from DotNetKicks.com
Chris Brandsma • February 5th, 2008
For me, testing is all about my ability to fake, mock, or stub other code. When I test a method, all I want to test is that method -- nothing else. The problem occurs when that method calls anther method in the same class -- I can't fake out that call, not easily anyway. So now I have to account for the method I'm calling, the method it is calling, and so on and so forth.
Constructors can also be a problem. You try to keep your constructors as simple as possible, just pass in any dependencies and let the work happen. It isn't always possible. A typical issue is you have to hook up an event in the constructor. It just make setting up a mock object...interesting.
Next, static singletons. These are often used in the place of globals for an application, holding special values (connection strings, etc). A static singleton can't be mocked (not by Rhino Mocks anyway -- my favorite tool so far).
Finally: trying to explain to another developer what a stub, fake, and a mock are. I've had many occasions where just pounding my head against the wall would have been more fruitful.
Scott Bellware • February 5th, 2008
Phil,
It's great that you're asking the community for this kind of input and experience report. Kudos to you for the initiative.
However, if this input is needed to help guide the testability aspects of the MVC framework for ASP .NET, then it's a cause for dismay in the case that you wouldn't already be in possession of the relevant, tangible, hard-won experience in testability design and TDD.
I hope you can excuse my cynicism on this and the obvious and inevitable slight to you, but my experience with Microsoft over the past few years in regards to lip service paid to these issues informs my caution quite strongly.
Thomas Eyde • February 5th, 2008
Not entirely on topic, but the MailAddress object validates the email address in its constructor and fails with an exception when the address is not properly formatted. That caused a bug in my application, and I want to write a unit test to verify that. The test is easy to write, but the fix is not. I would like to use the very same validation code as the constructor, but I can't. It's internal.
Jason • February 5th, 2008
I'm yet new to the unit testing world so I might be doing a lot of things wrong, but I'd like to chime in as a newbie.
I think Chris has one point that I agree with. If a class (or a presenter) has a method A, B, and C, and both A and B call method C, how can i avoid writing the unit test for C twice? So I have a unit test for C, a unit test for B, and a unit test for A. Do I just call unit test C from Unit Test A and B? or do I repeat it in each unit test? It doesn't seem like a pure "unit" test if I call unit test C from A and B.
Another pain point I have is testing the Data Access objects. We're not using any database mock objects (mainly due to the fact we started the project using an OR mapping framework, which is gone now). We were actually putting all the DB calls in a transaction so that everything gets rolled back after the test. The bad side is I'd have to write sql scripts to do unit tests. For example, if I need to test a Delete or Update, I need to first do an Insert, which may not exist at the time of testing.
Another struggle I've had with using dynamic mock views on the UI in a MVP pattern, is that a lot of my unit tests seemed meaningless. For example, certain methods just are there to set the properties of my view just to populate a form with a lot of fields. So I end up writing meaningless Expect's just to say a bunch of Expect.MyView.FirstName.To(anyhting); Expect.MyView.LastName.To(anything); ... etc
If I add another field to the form, the unit test fails because I'm missing that extra Expect call for that added field.
Vish • February 5th, 2008
Hi Phil,
Here are my thoughts on unit tests
1) They are cumbersome
2) Takes a whole bunch of time and typing
3) Currently almost no reuse
4) There is no challenge to it (once you figure out how to do testing); so doesn't keep the developers interest
What i would love to see in the future is tools that spit out tests by almost recording a macro of what i do on the screen. Think recording macros in like the 'object test bench' (remember that... but on class diagrams??). Others may have differing opinions on it but that is only way it will be widely adopted by Joe Developer as a part of everyday development tasks.
Thank You,
Vish
David Fauber • February 5th, 2008
Its very difficult to test anything in the template area of a WebForms page or user control. Its possible to do by calling on Execute on the Server object (which requires HttpContext) and caching the output,but still very painful. (although you could easily just say "use mvc instead", webforms is going to stick around)
mike h • February 5th, 2008
On the SharePoint Search Bench project (http://codeplex.com/spsearc..., I'm specifically struggling with unit testing methods that execute searches with the SharePoint API. Like HttpContext, The SharePoint search API classes are sealed and do not implement an interface. Mocking isn't an option and multiple times I've decided that creating a bridge or adapter is too much work for the value that my wrapper API provides. I've settled for at least creating a "test" context that can be swapped in/out with the "real" context that uses the real SharePoint API. The test context is great for unit testing the UI, but not the API itself. This is the kind of stuff that really makes unit testing a turn off for me. In the end, I just have to keep writing layer upon layer of buffer code that adds little value to the real functionality I'm trying to develop.
Haacked • February 6th, 2008
@Scott I'm involved in Framework-wide discussions, not just ASP.NET MVC. Hence the request for feedback.
Even if this were limited to ASP.NET, I don't see how asking for more feedback is a cause of dismay. I've been doing TDD and ASP.NET for years and have tons of battle scars in doing so and the "hard-won experience" you mentioned. That doesn't mean I've run into *every* possible testability problem.
My problems are not necessarily your problems. etc... etc...
So I welcome feedback across the board. :)
Phil
Dave S. • February 6th, 2008
My pain is I cannot get said team members to even grasp the concept. *yikes*
Wendy • February 6th, 2008
I find its easier to write testable classes with WPF. However, testing classes that use ObservableCollections can be difficult. The setup can be very large and the tests difficult to follow. Also, the event for the collection change is annoying to work with (why aren't the item lists generic??)...
One helpful pattern is to create a single class that observes the collection and forwards changes to an (or many) INotifyObservableCollectionChanged interface. That way you only have one class you need to test with an ObservableCollection and everyone else who wants to watch implements the interface, which is easy to test.
public interface INotifyObservableCollectionChanged<TypeOfObjectsInCollection>
void ObjectsAdded(IList<TypeOfObjectsInCollection> addedObjects)
void ObjectsRemved(IList<TypeOfObjectsInCollection> removedObjects)
...
I hope that made some sense :P
Pawel Pabich • February 6th, 2008
A few examples from the back-end.
1. OperationContex in WCF is sealed and does not expose its functionality via interface(s).
2. WorkflowInstance in WF. The same as above.
Daniel • February 6th, 2008
For me, it's testing database-related code. I'm quite sure the pain is from something I just don't "get", but here's a scenario:
I create a LINQ-to-SQL model for Northwind, and some classes around the model. To test properly I need to restore the db from a known-good state before each test, or else use a transaction-based approach that always rolls back the db after each test. Either way, the tests get pretty long-running pretty quickly. Also, I may need to test against large datasets to be sure my app works with 1000s of records. But a database backup that large doesn't fit in source control well.
Vijay Santhanam • February 6th, 2008
Howdy Phil,
Like Daniel, I'm having trouble mocking the System.Data.Linq.DataContext and all the LINQ-to-SQL ORM classes I've created.
I wish there was an easy way to mock or set up a low-footprint in-assembly database that doesn't require installing a new SQL server instance.
Maybe embedding the desktop engine into the test assembly will work. Does that support LINQ ORM?
I don't think this testing scenario is easy enough, or at least I don't know a good solution yet.
seerox • February 6th, 2008
Database for sure. I can't find any good examples of how to do it. People talk about distributed transactions etc but noone has an end to end. There is a ton of info out there and many pitfalls. I used datafresh a bit and I liked that but you have to have xp_cmdshell privileges etc. I don't mind an integration tests either for the db layer.
How about a team with many products and source control...How do you version the db? My situation here may be somewhat unique since we have hundreds of apps that hit the same db structure but keeping unit tests synced with db structure etc is a real killer.
Anand • February 6th, 2008
I follow Test Last Development, where I design the object model and write unit tests then. Recently I had a problem unit testing singleton class. It used eager instantiation and had only one constructor, which used to get data from two dependent systems. The singleton class depends upon two gateway classes which interacted with the external dependent system. I could not introduce the constructor injection since it followed singleton pattern. I did some workaround and was able to carry out unit testing but still feel that the workaround is poor.
I would like to hear from you the solution for unit testing singleton classes.
Mister P • February 6th, 2008
Phil, great writing as usual. I'm trying to catch up on all your latest writings on TDD, is there any book and/or online-resource that you could recommend for TDD (preferable with C# and .NET)?
Thank you
jbogard • February 6th, 2008
@Anand
My suggestion is to check out Michael Feather's Legacy Code book for suggestions. Singletons are notoriously difficult to test, and you'll usually find that you want to move away from them.
If we want singletons, we configure our IoC container to only construct one object when asked.
@Vish
All I can say is I hope I never inherit a codebase from you. Code without tests is legacy code, period. Instead of saying "Joe Developer can't grasp this or gets bored", let's raise the bar. MVC is doing just that.
What you're asking for is the IDE to read your mind on the intentions of you WANT the code to be, not even it what it is in reality.
What's more cumbersome than unit tests? Fielding support calls for months after a release because the software doesn't do what you wanted it to do.
Justice~! • February 6th, 2008
Vish, in my typical unbiased way I must say that this commment was LEEEWWWSER.
Phil - my big difficulty right now (aside from the ones we've already talked about) is the inability to properly override RedirectToAction(string, string) for inheritance-based tests. Even for mocking it is a bit odd given that it boils down (eventually) to a Response.Redirect call which, if I am not mistaken, requires the full path with the http://. I might be full of crap on this last part though so I'm going to investigate further.
The Other Steve • February 7th, 2008
Ok, my concern is it almost seems like I'm writing more code in tests and stubs and such, than actual logic. The more code I write the more bugs I introduce.
So who will tests the tests?
This is the problem I have with writing layers of abstraction and such just to make my code easier. By introducing all that other stuff, I risk crapifying my codebase.
Bradley Landis • February 7th, 2008
I could be doing something wrong but I always have to take extra steps when unit testing code using System.Net.Mail. The Send method of SmtpClient is not marked as virtual.
Bradley Landis • February 7th, 2008
I don't mean to sound harsh or disrespect anyone, but to the people are stating that unit testing is hard:
The difficulty associated with testing your code is usually directly proportional to the design and quality of your code.
In other words, poorly designed, poorly written code IS hard test. That is the reasoning for Test Driven Development. If you write your code so that it is testable, you will almost certainly write better designed, better written code.
Bradley Landis • February 7th, 2008
To "The Other Steve":
I believe I read that a typical TDD application has a 2-1 ratio of lines of test code to lines of system code. I believe that is accurate in my experience.
Think of it like this, if you were to write a document and then write a document about how your wrote the first document, the second document would certainly be twice as long as the first.
As for your issue with the layers of abstraction; if you have to add the layers of abstraction to test, those layers almost certainly should have been there in the first place. Your tests just forced you to write better code.
Haacked • February 7th, 2008
@The Other Steve - I actually wrote on this topic in my post, Who Tests the Tests?

The more code I write the more bugs I introduce.

Perhaps, but the extra code is mostly in your unit tests. You don't deploy your unit tests, so even if the occasional unit test has a bug, overall, it's still helped the code you will deploy.
Jeffrey Palermo • February 7th, 2008
I'm confused. You told me a lot of issues are fixed for the next release and that is why current feedback on the Dec CTP might already be outdated. I have plenty of feedback, much too much to write here. Give me a call, and we'll talk about it.
Will Shaver • February 7th, 2008
Just got finished writing a framework to assist with unit testing the MVC. (Currently on my blog, I'll probably be submitting it to MVC Contrib.) There were a couple of spots that were particularly annoying.
Mocking the session is perhaps the most difficult aspect of unit testing controllers. HttpSessionStateWrapper is of course marked internal, and HttpSessionState (contained by the wrapper) is public sealed with an internal constructor. Specifically mocking the saving of variables and retrieving of variables is difficult.
MyController mc = new MyController(); // or use a factory mockedSession["variable"] = "value"; mc.ActionWhichUsesSession(); Assert.AreEqual(mc.HttpContext.Session["variable"],"value")
Part of the difficulty comes in the fact that HttpContext exposes a property indexer to save and retrieve items.
Also annoying was RedirectData and RedirectDataWithController being marked as internal. These are used in the RedirectToAction function internally, and when passing strings for the action and controller to RedirectToAction(string,string) it creates these classes. When overriding the virtual RedirectToAction one must iterate over PropertyInfo objects instead of just doing an "is" comparison on the type.
nsoonhui • February 8th, 2008
I have documented my experience here
Slevdi • February 8th, 2008
TDD best practice examples. I learn nothing from books, articles etc unless I can fire up a relevant example. Thanks to Phil H here for some seminal articles.
Webcasts seem to have the highest ratio of time spent to knowledge gained (I particularly recommend the awesome 1 hour long webcast on RhinoMocks using TDD in VS2005 featuring Resharper from Ayende - it was worth a month of reading books to see an expert at work)
SeeR • February 9th, 2008
Testing in WPF is hard for me because of following things:
1) There can be only one Application class instance
2) Window with XAML can't be inherited
3) Again (like in Windows.Forms) OpenFileDialog and SaveFileDialog are sealed
4) Microsoft should use interfaces instead of class dependencies in freamework or should remove from CLI keywords sealed and internal
Will Shaver • February 11th, 2008
Still trying to mock the session... this time by hand. Turns out that one of the interface members is:
public NameObjectCollectionBase.KeysCollection Keys { get; }
Doesn't look so bad at first, but it turns out that while the KeysCollection is public, the constructor is marked as internal. This makes it practically impossible to mock this interface.
I think a general practice for framework design is to either:
a) Create a test-specific side of the framework that has members mocked out for users.
or
b) All interface members should return either interfaces or classes that are not defined as internal.
Alex Gorbatchev • February 13th, 2008
Testing ASP.NET pipe line is somewhat similiar in experience to using a sandpaper for whiping oneself. It's doable, but the process is mildly pleasing.
Half of the public classes have internal constructors and half of the overloads are internal as well. The novelty of things like AppDomain.CurrentDomain.SetData(".appDomain", "*") never wears off.
Justice~! • February 21st, 2008
Hey Phil, while still on this discussion it would be nice to have IHttpContext (sorry, HttpContextBase) implement either an interface or wrapper around Cache, so that I don't have to write my own wrapper to test some of those implementations...
Jay • July 27th, 2008
How can I Test silverlight controls in .net?
I have to create every time new project for testing it
Mudit Vajpayee • March 26th, 2009
Hey..We have developed a portal where we kept our bussiness logic into .XSD File and for creating the Unit Test Case we placed the file in APP_Code Folder.But while creating unit test case for any method defined in a class which in turn accesing that .XSD file an exception occured.So we place that file into APP_Data Folder but after that we can't access the logic written in that .XSD file while creating test cases.
Please suggest..
dimple • November 18th, 2010
jeux de casino
By way of illustration, I found 5 bugs (admittedly somewhat obscure) on the first page I used this technique on, a login page that had been in production for some months, purely because it was easy to setup a number of different scenarios in the mock objects that the login code used.
Reuben Helms • October 10th, 2012
It's 2012, but the ASP.NET application I've been working on recently has been in development since 2004. For the last few years I've been using PHP, so I don't even know what the code I have is called.
There's a .aspx, and a .aspx.cs and it's definitely not done with the newer, spankier ASP.NET MVC. Somewhere, deep behind a custom UI library, there's a reference to a System.Web.UI.Page.
I'd love to write tests that just deal with the backing class. No browser, just mocked http contexts and away we go. But this is 2012. NUnitASP might have come close, but it's defunct.