Unit Testing Loves Beta Testing And Vice Versa

0 comments suggest edit

Jeff links to a post by Wil Shipley criticizing unit testing. You knew I had to chime in on this… ;)

I won’t rehash what I’ve already written on the subject, but will merely try to add a couple key points.

There are a couple of misconceptions I want to clear up.

First, the proponents of unit testing are not promoting it as the end all and be all of software development. However, I would say that unit tests are very important when done right, much in the same way version control is important when done right. Unit tests should be applied using a cost benefit analysis just as you’d do anything else. For example, problematic or tricky code should have more unit tests. Important code (such as the code in a banking system that performs a calculation) should have more unit tests. Simple getters and setters on the other hand can do without.

But I’ve NEVER, EVER seen a structured test program that (a) didn’t take like 100 man-hours of setup time, (b) didn’t suck down a ton of engineering resources, and (c) actually found any particularly relevant bugs.

Then perhaps you haven’t seen unit testing done right. My setup time for unit testing is about as long as it takes to set up a class library and run NUnit or MBUnit. Marginal.

Most unit tests are written as code is developed, not tacked on after the fact. The design aspect of writing unit tests cannot be overstated. Especially in teams where one person writes a piece of code that another person is going to call. It’s very easy to create a really awful API to a class library that then costs other developers who have to use the API extra time to fiddle around with it and understand it. With unit tests, at least the author has had to “dogfood” his own medicine and the API is more likely to be usable. If it’s still confusing, well the unit test can serve as a code sample. I tend to learn better from code samples than Intellisense.

​1) When you modify your program, test it yourself. Your goal should be to break it, NOT to verify your code.

Yes, unit testing does not take away the need to test your own code. However, pages and pages of studies show how easy it is for even very talented developers to develop blind spots when testing their own code. That’s why you still have a QA team dedicated to the task without the baggage that the developer carries.

However, testing the feature you changed isn’t necessarily good enough. For example, suppose you make a slight schema change. Are you sure you haven’t broken a feature developed by another developer? Are you prepared to test the entire system for every change you make? With unit tests you have some degree of regression testing at your fingertips.

While it is true that unit testing can some additional upfront time, in my experience, especially if you work in a team, it always produces a cost savings overall. The time and cost savings of unit tests cannot be overstated. Yes. Savings!

One TDD practice I am a firm proponent of is to make sure that when a bug is discovered in the code, before you fix the bug, you write a unit test that exposes the bug (if possible and cost effective). By “exposing” the bug I mean you write a test that would pass if the code was working properly, but fails because of the bug. Afterwards you fix the bug, make sure the test passes and then check in the code and the unit test. Now you have a fair degree of certainty that particular bug won’t crop up again. By the very existence of the bug in the first place, you know that area of code is troublesome and deserves to have unit tests testing it.

These sort of unit tests address issues that unit tests are too soft on the code as they are effectively generated as a result of human interaction with the system.

On a recent project, a developer checked in a schema change and tested the system and it seemed to work just fine. Meanwhile, I had gotten latest and noticed several of my unit tests were suddenly failing. After a few minutes of digging, I called the other developer and confirmed the schema change. It required a small change in my code and everything was running smoothly. Without my suite of unit tests, I would have no easy way to judge the true impact of that schema change. In fact, it may have taken hours for me to even notice the problem, as things “seemed” to be working fine from a UI perspective. It was the underlying calculations that were broken.

Real testers hate your code. A unit test simply verifies that something works. This makes it far, far too easy on the code. Real testers hate your code and will do whatever it takes to break it– feed it garbage, send absurdly large inputs, enter unicode values, double-click every button in your app, etcetera.

Yes they do! And when they throw in garbage that breaks the code, I make sure to codify that as a unit test so it doesn’t break again. In fact, over time as I gain experience with unit testing, I realize I can just as easily throw garbage at my code as a human tester can. So I write my unit tests to be harsh. To be mean angry bad mofos. I make sure the tests probe the limits of my code. It takes me a bit of upfront time, but you know what? My automated unit test can throw garbage at my code faster than a human can. Plus, don’ forget, it is a human who is writing the test.

Testing is hugely important. Much too important to trust to machines. Test your program with actual users who have actual data, and you’ll get actual results.

Certainly true, but regression testing is much too boring to be left to humans. Humans make mistakes, especially when performing boring repetitive tasks. You would never tell a human manually sum up a long row of numbers, that’s what computers are for. You let the machine do the tasks its well suited for, and the humans can do what they are well suited for. It all works hand in hand. Unit testing is no substitute for Beta testing. But Beta testing is certainly no substitute for unit testing.

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.



5 responses

  1. Avatar for Jeff Atwood
    Jeff Atwood October 18th, 2005

    I've read your posts on TDD and they seemed rational enough to me.

    It's my fault for not highlighting this more, but Wil's post was very specificially referring to GUI-heavy applications. Not web apps, console apps, etc. Have you *seen* Delicious Monster? It's all GUI. Good luck TDD-ing that beast.

    What I mainly object to is the hard-core religious test-only development folks.

    I think spending time testing your app with real users is more important than pounding away on thousands of lines of testing code. You might be writing a stack of unit tests for a feature users end up never touching in your app.

    That's also something Wil (and Spolsky) seems to be saying: at a small company, do you want 100% test code coverage, or five new features that users actually asked for? Because you can't have both.

    I know which one will result in more sales...

  2. Avatar for Haacked
    Haacked October 18th, 2005

    Ah yeah, even in ASP.NET projects, I don't apply TDD to the UI. The UI level has too much churn and unit tests are less effective there.

    But a well factored application has all its guts separated from the UI. So I can test the guts.

    And 100% code coverage is not the goal of TDD. As I wrote before http://haacked.com/archive/2004/11/03/1559.aspx even if you had it, you have a false sense of security.

    Even for GUI apps, I'd say the right blend of TDD is useful. I don't think it's an either or proposition.

    I'd tell Joel and Wil, you can have decent and sufficient test coverage (not 100%) AND deliver SIX new features. ;)

    TDD is a lot like documentation. You could wait till the end of the project to write up end-user documentation. BUt it's so much more effective if you do it as you go along, and it ends up taking less time overall.

  3. Avatar for Jeremy Miller
    Jeremy Miller October 20th, 2005

    Nice post. There's also the issue that whatever TDD you *can* do on a GUI stops some fraction of bugs from ever getting to beta users or testers in the first place. That's a huge chunk of efficiency for the team as a whole. And I really don't think you want beta users to see a lot of defects in a GUI at any stage.

  4. Avatar for Brian White
    Brian White November 26th, 2006

    The benefits of automated unit testing are not limited to TDD. I don't personally write my tests first, but I now write tests for all my code at some point during the development of it. I find this "test during" approach works better for me. Still, I can't say enough good stuff about writing built-in tests, no matter when they're written!

  5. Avatar for Community Blogs
    Community Blogs January 17th, 2008

    After joining Microsoft and drinking from the firehose a bit, I’m happy to report that I am still alive