Research Supports The Effectiveness of TDD

code, tdd 0 comments suggest edit

In a recent post, Frans Bouma laments the lack of computer science or reference to research papers by agile pundits in various mailing lists (I bet this really applies to nearly all mailing lists, not just the ones he mentions).

What surprised me to no end was the total lack of any reference/debate about computer science research, papers etc. … in the form of CS research. Almost all the debates focused on tools and their direct techniques, not the computer science behind them. In general asking ’Why’ wasn’t answered with: “research has shown that…” but with replies which were pointing at techniques, tools and patterns, not the reasoning behind these tools, techniques and patterns.

science-in-actionAs a fan of the scientific method, I understand the difference between a hypothesis and a theory/law. Experience and anecdotal evidence do not a theory make, as anyone who’s participated in a memory experiment will learn that memory itself is easily manipulated. At the same time though, if a hypothesis works for you every time you’ve tried it, you start to have confidence that the hypothesis holds weight.

So while the lack of research was a slight itch sitting there in the back of my mind, I’ve never been overly concerned because I’ve always felt that the efficacy of TDD would hold weight when put to the test (get it?). It is simply a young hypothesis and it was only a matter of time before experimentation would add evidence to bolster the many claims I’ve made on my blog.

I’ve been having a lot of interesting discussions around TDD internally here lately and wanted to brush up on the key points when I happened upon this paper published in the Proceedings of the IEEE Transactions on Software Engineering entitled On the Effectiveness of Test-first Approach to Programming.

The researchers put together an experiment in which an experiment group and a control group implemented a set of features in an incremental fashion. The experiment group wrote tests first, the control group applied a more conventional approach of writing the tests after the fact. As a result, they found…

We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.

The interesting result here is that quality is more a factor of the number of tests you write, and not whether you write them first. TDD just happens to be a technique in which the natural inclination (path of least resistance) is to write more tests than less in the same time span. The lesson here is even if you don’t buy TDD, you should still be writing automated unit tests for your code.

Obviously such a controlled experiment on undergraduate students leaves one wondering if the conclusions drawn can really be applied to the workplace. The researches do address this question of validity…

The external validity of the results could be limited since the subjects were students. Runeson [21] compared freshmen, graduate, and professional developers and concluded that similar improvement trends persisted among the three groups. Replicated experiments by Porter \ and Votta [22] and Höst et al. [23] suggest that students may provide an adequate model of the professional population.

Other evidence they refer to even suggests that in the case of advance techniques, the benefit that professional developers gain outweighs that of students, which could bolster the evidence they present.

My favorite part of the paper is the section in which they offer their explanations for why they believe that Test-First programming might offer productivity benefits. I won’t dock them for using the word synergistic.

We believe that the observed productivity advantage of Test-First subjects is due to a number of synergistic effects:

  • Better task understanding. Writing a test before implementing the underlying functionality requires \ the programmer to express the functionality unambiguously.
  • Better task focus. Test-First advances one test case at a time. A single test case has a limited scope. Thus, the programmer is engaged in a decomposition process in which larger pieces of functionality are broken down to smaller, more manageable chunks. While developing the functionality for a single test, the cognitive load of the programmer is lower.
  • Faster learning. Less productive and coarser decomposition strategies are quickly abandoned in favor of more productive, finer ones.
  • Lower rework effort. Since the scope of a single test is limited, when the test fails, rework is easier. When rework immediately follows a short burst of testing and implementation activity, the problem context is still fresh in the programmer’s mind. With a high number of focused tests, when a test fails, the root cause is more easily pinpointed. In addition, more frequent regression testing shortens the feedback cycle. When new functionality interferes with old functionality, this situation is revealed faster. Small problems are detected before they become serious and costly.

Test-First also tends to increase the variation in productivity. This effect is attributed to the relative difficulty of the technique, which is supported by the subjects’ responses to the post-questionnaire and by the observation that higher skill subjects were able to achieve more significant productivity benefits.

So while I don’t expect that those who are resistant or disparaging of TDD will suddenly change their minds on TDD, I am encouraged by this result as it jives with my own experience. The authors do cite several other studies (future reading material, woohoo!) that for the most part support the benefits of TDD.

So while I’m personally convinced of the benefits of TDD and feel the evidence thus far supports that, I do agree that the evidence is not yet overwhelming. More research is required.

I prefer to take a provisional approach to theories, ready to change my mind if the evidence supports it. Though in this case, I find TDD a rather pleasant fun enjoyable method for writing code. There would have to be a massive amount solid evidence that TDD is a colossal waste of time for me to drop it.

Technorati Tags: TDD,Unit Tests,Research

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.

Comments

avatar

50 responses

  1. Avatar for Tom Opgenorth
    Tom Opgenorth January 21st, 2008

    I don't know how to implement it, but I think it would be good to do a study of TDD vs. no form of automated testing (as opposed to the test last approach used in the study). Quite often you run into this scenario too, and it gets tiresome have to explain/argue that it really *isn't* a waste of time to write the extra code via TDD.

  2. Avatar for Karthik Hariharan
    Karthik Hariharan January 21st, 2008

    A lot of the former TDD pundits are now emphasizing BDD and DDD as being the next logical step in the Agile progression. I wonder when we'll see more research in this realm.
    It seems ironic that Agile development is one area where the industry seems to be driving innovation much more so than academia.

  3. Avatar for Haacked
    Haacked January 21st, 2008

    @Karthik When you look at the jump from non-TDD to TDD, that's a huge conceptual and practical leap. But the leap from TDD to BDD is more of a little hop in comparison.
    For example, if you work with a framework that is not written with testability in mind, testing your own code written on that framework using TDD or BDD is challenging. But when the framework itself is designed for testability (perhaps itself written in a testable manner), then the framework looks very different and it supports *both* TDD and BDD.
    My hypothesis is that BDD won't exhibit much improvement over TDD. I think it's more of a conceptual style difference. In other words, the difference between code without TDD and code with TDD is a huge difference. The code written using TDD vs BDD is a much smaller difference. I'm happy to be proven wrong.

  4. Avatar for John
    John January 21st, 2008

    In the study, a story is broken down into tasks A and B. Then their methodology is to classify Test-First like so:
    1) Write Test A
    2) Implement Task A (until test A passes)
    3) Write Test B
    4) Implement Task B (until tests A & B pass)
    Then Test-Last is classified thusly:
    1) Implement the whole story
    2) Write tests A and B (until they pass)
    I think that could be considered a strawman version of Test-Last. Why not implement and test the tasks separately, just like Test-First does? The benefits of breaking down a large task into smaller ones are plain, and I don't think the study proves anything else, since it doesn't control for task subdivision.

  5. Avatar for Martin C. Nagel
    Martin C. Nagel January 21st, 2008

    I would agree with the original author's use of the term synergistic, especially when referring to the part about reducing rework. When things work together better than they do separately, it is *indeed* synergistic. So, the way those effects all combine to improve productivity is the very thing that businesses so want--increased overall productivity.

  6. Avatar for Christopher Bennage
    Christopher Bennage January 21st, 2008

    Phil Haack has an interesting post on some research regarding the effectiveness of Test Driven Development

  7. Avatar for alberto
    alberto January 21st, 2008

    I don't know if it's an idiomatic difference, but to me, hypothesis and a theory are pretty much the same, a statement about how we believe something works/behaves but not proven right. For example, the String Theory.
    Those are opposed to/different than a theorem(which I think is what you meant) or law.
    Technicalities apart, it's nice going from being a TDD-believer/TDD-ist to a TDD-ician.

  8. Avatar for John
    John January 21st, 2008

    Scientifically speaking, hypothesis and theory are VERY different. A hypothesis is just a guess, usually used as a point of departure for experimentation. A hypothesis only becomes a theory when experimentation proves it out. Hypotheses do not have to be backed up by any evidence; theories MUST be.

  9. Avatar for Keith
    Keith January 22nd, 2008

    @John: I wholeheartedly agree. Anyone who sat down and actually looked through the linked PDF would've seen the flaw in their study, and I think Phil Haack should've perhaps mentioned the fuzzy details about it...
    I wonder how big each story actually was. Were the stories already pretty broken down? They don't really say. The only thing it really mentions was what the task was (section 3.5, page 6), and that it was separated into several 'small' tasks. I suppose that means that they're already broken down? But that's not very definitive... Wish we could see all the stories and make the call for ourselves.
    I think that yes, it should have been more of a control and allowed the Test-Last developers to break down their stories even more as well, but it would've been a much larger straw man if they had, say, made the Test-Last devs code the entire project THEN make tests... Or if their stories were more or less something like, "Implement a search engine." or something :) From the task size, though, I think they really were pretty small.

  10. Avatar for Jacob Proffitt
    Jacob Proffitt January 22nd, 2008

    One of the most useful classes I took in college was an introductory statistics class that was intended

  11. Avatar for Scruffy-Looking Cat Herder
    Scruffy-Looking Cat Herder January 22nd, 2008

    TDD Proven Effective! Or is it?

  12. Avatar for Haacked
    Haacked January 22nd, 2008

    @Keith Well to be fair, I did mention some of the fuzzy details. I point out that quality was actually correllated with the number of tests written, and not necessarily whether it was TDD.
    While we're on the topic of "Straw Men" I like how Jacob's post changes my point to "TDD Proven Effective!" and then sets the lighter to that straw-man. I point out that this merely adds evidence.
    Social experimentation is a tough business. All they've really proven is that a group of undergrads in a controlled experiment exhibit certain quality levels. However, until there are many experiments that provide evidence to the contrary, this still does provide experimental evidence towards something many developers experience day to day.
    The paper also cites multiple other studies which are worth reading that support the TDD claims. So the question is, if you've never really given it a try, but you've heard lots of anecdotal evidence from other devs in favor of it, and research papers are starting to come out in support of it, on what basis are you denying that TDD is effective?

  13. Avatar for DevKm
    DevKm January 22nd, 2008

    The point with TDD is not that it is the best approach. In fact it isn't about TDD at all. Success rate achieved by traditional models are so abysmal, alternative approaches have to be considered. TDD/BDD is definitely one such alternative.

  14. Avatar for Jens
    Jens January 22nd, 2008

    At University of Karlsruhe (Germany) there was an experiment that compared the way professional software developers apply TDD to the way students do. The full report can be found here:
    http://wwwipd.ira.uka.de/Ti...
    The results show quite clearly that students (despite having been trained in using TDD and claiming to have applied this process strictly) in general have a completely different way of TDD. This does not say anything about TDD being effective or not. But it is quite obvious that you cannot generalize results from academic experiments to the "real world".

  15. Avatar for Dev Theo (Jay Kimble)
    Dev Theo (Jay Kimble) January 22nd, 2008

    Phil,
    I have great respect for you. BUT I think there is a great quote from Jacob's response that sort of gets cast aside in the discussion:
    "Having a big clump in that upper left quadrant is troubling enough but then having the "Test Last" group almost double your "Test First" group in the over 90% quality range is something that should be noticed and highlighted."
    I haven't done the analysis on the article. But I have read both your analyses and I have come to a conclusion. It's the same one that both of you come to. More tests = better quality.
    But to extrapolate that conclusion to say that TDD is more effective seems a little dubious to me which is what both the authors of the study and you have asserted. The problem for me is that the "quality" product was actually produced by the "Test Last" team. Like Jacob I would have liked to have seen a third group in here... one that didn't use any automated testing tools at all.
    Also, the fact that the experience level of the programmers is not revealed anywhere also puts the value of the study in question. It says grad students which could mean people who learned to program in the last 5 years or people who have programming professionally for 10 years, or people who started programming as a hobby 15 years ago (all of which would have a slightly different skill set).
    I have said for years and I'll say it here. It's the Man/Woman not the method that brings success in the programming field.
    Jay

  16. Avatar for Amund
    Amund January 22nd, 2008

    TDD is still a very manual (tedious) process, believe it could gain from increased automation, ref: http://amundblog.blogspot.c...

  17. Avatar for Will Sargent
    Will Sargent January 22nd, 2008

    If you're interested in statistics and the testing of various processes, you should check out 'empirical software engineering' and Victor Basili in particular.

  18. Avatar for Isaac Gouy
    Isaac Gouy January 22nd, 2008
    My favorite part of the paper is the section in which they offer their explanations for why they believe that Test-First programming might offer productivity benefits.


    I guess it's your favourite part because you wish those things were true. It's no more than speculation; there's no attempt to test that they actually did have better task understanding, or that they did have better task focus, or ...

  19. Avatar for Haacked
    Haacked January 22nd, 2008

    @Jay

    But to extrapolate that conclusion to say that TDD is more effective seems a little dubious to me.


    I never said this study demonstrated TDD was *more* effective. Just that it was effective. You are correct that I probably could have downplayed the results more. To me, the results show that automated testing might be the effective part of TDD. But even that could be disputed because there's no control group doing zero testing.

    It's the Man/Woman not the method that brings success in the programming field.


    I agree with that, but I think of it like the 7 Habits of Effective People. You don't need to do those habits to be effective. But they are common among effective people. I think automated unit testing is a habit that can make even effective people *more* effective.
    For me, it's like the whole placing the keys in the same place. Even smart people misplace keys. But if you adopt good habits, you make those mistakes less.
    I realize now that I haven't communicated the message much. I now strongly agree that this particular study is not conclusive (I said it all along actually). However, I think it would be nice to look at the other studies this study references.
    My goal is not that pointing to these studies suddenly convinces people that TDD or Automated Unit Testing is effective, but that it convinces people it's worth giving it a try for themselves rather than dismissing it outright. Capiche? :)

  20. Avatar for Jacob
    Jacob January 22nd, 2008
    I realize now that I haven't communicated the message much. I now strongly agree that this particular study is not conclusive (I said it all along actually). However, I think it would be nice to look at the other studies this study references.


    I agree that you were pretty careful to keep your language in line with what you saw in the study. You didn't say the study proves anything, merely "support". But then, while I never said you made claims of proof it's the conclusion you jumped to based on the title of my post and its conjunction to what you did say.
    In other words, you jumped to a conclusion about what I said just as many others jumped to a conclusion about what you've been saying. Frankly, both leaps aren't terribly unreasonable based on how things were laid out. You're a TDD fan so it's easy to make assumptions about why you are linking the article. I'm a TDD skeptic so it's easy to make assumptions about why I'm linking to your post. Neither one of us did so in bad faith, I don't think, and depicting it as such is a disservice to the discussion.

    My goal is not that pointing to these studies suddenly convinces people that TDD or Automated Unit Testing is effective, but that it convinces people it's worth giving it a try for themselves rather than dismissing it outright. Capiche? :)


    I never dismiss things outright. But this study didn't make me want to try out TDD, either. Personally, I tend not to try things out until I understand how they're supposed to be better than my current processes. For me, TDD hasn't made that case yet.
    I'm not ruling it out. I'm not denying that others have had good results using TDD. I'm not even saying that my way is for sure better. All I'm saying is that the case hasn't been made well enough that I want to spend the resources to give it a fair shot. It's a pretty intrusive change to how I do things now. That gives it a pretty high hurdle to clear.

  21. Avatar for Isaac Gouy
    Isaac Gouy January 23rd, 2008

    Haacked wrote

    "if a hypothesis works for you every time you’ve tried it, you start to have confidence that the hypothesis holds weight"


    No. When a hypothesis is tested repeatedly and does not fail then you start to have confidence. Doing TDD without a comparison test gives no confidence that it is more effective or no different than some other technique - it might be less effective.

    Haacked wrote

    "The authors do cite several other studies (future reading material, woohoo!) that for the most part support the benefits of TDD."


    If you haven't read them [future reading material] how do you know that they do in fact support the benefits of TDD?

    Look at the 2007 "A Longitudinal Study of the Use of a Test-Driven Development Practice in Industry" and you might be surprised to see it's not really about TDD, it's about developer testing. (Maybe it's just easier to get papers accepted if you put TDD in the title!)
    They moved from "... the unit test process was not disciplined and was done as an afterthought ..." to "For each important class, we enforced complete unit testing".
    They moved from Untested Development to Tested Development:
    (Test-Driven Development) Tested Development | Untested Development

    Here's a more recent source of papers to read - Guest Editors' Introduction: TDD--The Art of Fearless Programming, IEEE Software, May/June 2007 (Vol. 24, No. 3) pp. 24-30 (pdf available for download after guest-login).

  22. Avatar for Isaac Gouy
    Isaac Gouy January 23rd, 2008

    Haacked wrote

    "There would have to be a massive amount solid evidence that TDD is a colossal waste of time for me to drop it."


    afaict you have become convinced about TDD without "a massive amount of solid evidence" but now you would require "a massive amount of solid evidence" to become unconvinced.
    You may be "a fan of the scientific method" but in this case you don't seem to be following "the scientific method" :-)

    Haacked wrote

    "...on what basis are you denying that TDD is effective?"


    On what basis would we deny that the moon is made of green cheese? We wouldn't try - the burden of proof lies on those who contend that the moon is made of green cheese.

    Incidentally, for correlations about the software development process:
    MacCormack, A., Kemerer, C.F., Cusumano, M., and B. Crandall, "Trade-offs between Productivity and Quality in Selecting Software Development Practices." IEEE Software 20(5) 2003 78-85

  23. Avatar for Haacked
    Haacked January 23rd, 2008
    afaict you have become convinced about TDD without "a massive amount of solid evidence" but now you would require "a massive amount of solid evidence" to become unconvinced.
    You may be "a fan of the scientific method" but in this case you don't seem to be following "the scientific method" :-)


    @Isaac I'm also a big fan of engineering. When you try something and it works for you repeatedly, you don't need to know the science behind it to know that it works.
    For example, for centuries people have been able to brew beer while having no freaking clue what was really going on. They also realized that drinking beer made them feel better than drinking the polluted water. They didn't need science to tell them that beer contains alcohol which kills microbes to suddenly gain the benefit.
    And if science came along and said that brewing beer is impossible and isn't fun to drink, I'm sorry, but it would take overwhelming evidence to make all those happy drunks think twice. ;)
    In this case, writing automated unit tests is an engineering practice that has proven itself effective to me. It's worked for me time and time again. Thus if the science came out and said it's not effective, it would take a lot of evidence to make that claim to me. It's quite possible that research might refine our understanding of why it is effective, or that it is something else that is actually effective.
    For example, I've assumed that TDD is effective. But this study seems to point to writing automated unit tests is the actual source of benefit. Fine. It just may be that I don't have to write tests first to gain the benefit. However I personally find it's a great way to help me focus on a problem.
    Like the way an athlete does visualization exercises before a game (which is demonstrated scientifically to be effective by the way).

  24. Avatar for Haacked
    Haacked January 23rd, 2008

    @Jacob When you wrote that post, it put a trackback in my comments, "TDD is proven, or is it" which creates a direct link between that statement and my blog post. I thought this would lead people to think you're saying I'm making this claim.
    However, it occurs to me now based on your comment that you may have been referring to the study itself as claiming TDD is proven. Either way, it doesn't matter. I still think that's a strawman argument. ;) But I also don't think you did it in bad faith and I apologize for characterizing it as such. Your analysis of the study was illuminating and certainly opened my eyes to some facts I glossed over. Confirmation bias indeed!
    One question for you, I can understand if you don't feel that TDD is a practice that might provide you (or your team) personal benefit. But setting aside TDD, what are your thoughts on having developers writing automated unit tests iteratively throughout the development cycle? Can you see how that might be a good process?
    It seems to me it's hard to argue against the benefit of even giving that a try because the side-effect is so valuable. Especially when you try to change code later on that is called by code you didn't write. Perhaps not every shop experiences the pain of dealing with large brittle code bases as I've had. ;)

  25. Avatar for Christopher Steen
    Christopher Steen January 23rd, 2008

    Misc Items Microformats: A DoD Use Case [Via: Jeff Black ] Misc Items NET Foundations - Memory model...

  26. Avatar for Kevin G.
    Kevin G. January 23rd, 2008

    What I find interesting about the NCR study is at the very end, where it says that "Test-First programmers did not achieve better quality on average, although they achieved more consistent quality results."
    Personally, I had been under the impression that better consistency *was* better quality. So what this it is really saying to me is that TDD won't make you a better programmer, just a more organized one.
    You have to admit, a lot of developers lack organization.
    As a programmer, I'd rather work on a well organized piece of code with clear divisions (knows what its talking about!) than a phenomenally clever piece of code that looks like a jungle.
    I always think of [jungle code]: "This code is so cool, but I could *use it* if it was organized"

  27. Avatar for Isaac Gouy
    Isaac Gouy January 23rd, 2008

    haacked wrote: "I'm also a big fan of engineering.
    I'm also big fan of dictionaries ;-)

    Engineering :- "The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or ..."



    haacked wrote: "proven itself effective to me"
    What does that really mean?
    Yesterday you wrote "I never said this study demonstrated TDD was *more* effective. Just that it was effective" - which really makes me wonder what you mean by effective?
    The authors of "On the Effectiveness of Test-first Approach to Programming" seem to mean effective in enhancing both product quality and programmer productivity - the whole point is *more* quality and *more* productivity.
    So when you write "proven itself effective to me" do you mean the same thing; and if so how do you know that product quality and productivity improved?

  28. Avatar for Loosely Coupled Human Code Fac
    Loosely Coupled Human Code Fac January 24th, 2008

    I've been working toward a more and more proactive testing, or test driven development, approach to my...

  29. Avatar for Jay Kimble
    Jay Kimble January 24th, 2008

    There's still love here for you... Sorry if I jumped into fight... I guess I spent too much time in that environment on another blog site.
    We are in ABSOLUTE agreement that automated testing tools really CAN increase the quality of a project...
    The concern was just that some would read this and think "aha... TDD is proven effect. Why won't those dummies do it?"
    As always we're cool (that is if you're still cool with me)... sorry if I came on strong. I have a new blog community and I didn't want to see one my co-bloggers get beat up...
    Jay Kimble

  30. Avatar for Troy DeMonbreun
    Troy DeMonbreun January 24th, 2008

    @Phil
    I think the value of Unit Testing/Automated Testing sometimes gets
    lost in the arguments against TDD, so I appreciate you giving a "shout out" to the core value of unit tests themselves.
    It ties into to a recent comment of mine (on a previous article of yours) regarding "Test-Supported Development", TSD, at http://haacked.com/archive/2008/01/14/tdd-is-also-an-organizational-process.aspx#61768

  31. Avatar for Jacob
    Jacob January 24th, 2008
    One question for you, I can understand if you don't feel that TDD is a practice that might provide you (or your team) personal benefit. But setting aside TDD, what are your thoughts on having developers writing automated unit tests iteratively throughout the development cycle? Can you see how that might be a good process?


    Good question. I'm personally a fan of unit tests and think they should be written as proximately in both time and place as possible with the functionality tested. I'm skeptical of writing the tests first as is required by TDD is all. Ideally both tests and functionality will be written in any given iteration, I think.
    BTW, I'm also skeptical of striving for 100% code coverage, though I haven't fully fleshed out why yet. I think it's likely that the 80/20 rule applies though I'm not sure on what dimensions.

  32. Avatar for Haacked
    Haacked January 24th, 2008

    @Jacob Even if you had 100% test coverage, it wouldn't be enough. For library code or code with few "side effects" it's nice to strive for something close to total coverage, but for most code, 100% starts to get asymptotic and not worth the benefit.
    @Jay as always, we're cool. :)
    @Isaac Scientific principles doesn't mean you necesarily understand all the science. For example, people built steam engines before understanding the physics behind the engines. Once they did, engine quality improved immensely. To me, it was still engineering before they understood the science. They experimented, figured out what worked for them and what didn't, and continued to make improvemens. Those are the scientific principles.
    For TDD, it solved a problem I ran into. Large code bases that were hard to change and hard to vouch for the quality. Once I got my team (back in the day) writing tests, the quality improved. I didn't keep stats, but I don't need a luminosity meter to know when it's sunnier one day more than another. Our clients were happier, the number of bugs in production declined to nearly zero, and we were able to make changes to the system much easier.
    It's like beer. Back in the day, people got sick drinking tainted water. But they didn't get sick drinking beer. So they brewed and drank beer, with no understanding or confirmation of why it worked. The science came later, but really, who waited around for that to get the benefit of beer!

  33. Avatar for Isaac Gouy
    Isaac Gouy January 25th, 2008

    haacked wrote "... people built steam engines ..."
    You seem to be stuck back in the late 18th century, first Industrial Revolution. The 19th century second Industrial Revolution was built on science, and now in the 21st century ...
    haacked wrote "Back in the day, people got sick drinking tainted water. But they didn't get sick drinking beer."
    If you want to tell yourself you know what happened 7000 years ago that's fine, but I'm just trying to understand what you've said this week.
    haacked wrote "I didn't keep stats, but I don't need a luminosity meter to know when it's sunnier one day more than another."
    Was it sunnier on 12 March 2005 than on 12 April 2005?
    If there was a clear sky on 9th April, overcast on 12th April, and clear sky on 15th April - would you know whether the 9th or 15th was sunnier, or would it just be a guess?

    haacked wrote "Once I got my team ... writing tests ..."
    Automated unit testing goes back before TDD and before Agile -
    "Simple Smalltalk testing" Kent Beck, Smalltalk Report 4(2) Oct 1994 - and iirc Kent Beck clearly states TDD is a development technique not a testing technique, so let's not confuse writing tests with TDD.
    haacked wrote "It just may be that I don't have to write tests first to gain the benefit."
    The issue is whether writing tests is a trade-off between productivity and quality (and note that simply doing a daily build correlates with increased productivity):
    MacCormack, A., Kemerer, C.F., Cusumano, M., and B. Crandall, "Trade-offs between Productivity and Quality in Selecting Software Development Practices." IEEE Software 20(5) 2003 78-85

  34. Avatar for ISerializable - Roy Osherove's
    ISerializable - Roy Osherove's January 25th, 2008

    Thanks to Haacked for posting this one .  I haven't yet read it fully, but this IEEE paper shows

  35. Avatar for ISerializable
    ISerializable January 25th, 2008

    Thanks to Haacked for posting this one .  I haven't yet read it fully, but this IEEE paper shows

  36. Avatar for Frans Bouma
    Frans Bouma January 28th, 2008

    "So while I’m personally convinced of the benefits of TDD and feel the evidence thus far supports that, I do agree that the evidence is not yet overwhelming. More research is required."
    Indeed. One word which covers it is: 'context'. As you're inside MS walls, do you see the large(r) teams around you use TDD to develop their work? Why not?
    Btw, it's good that TDD is researched if it is effective and if so, when and how, what can be improved, and what makes it really effective?
    Especially that last part is essential and this isn't dug up by this research: if two truly equal teams start a series of different projects, one using their own homegrown method, the other pure TDD, which group produces in all cases the best results? To me it's still unclear.
    I'm still convinced that simply the best professionals create the best stuff, regardless what kind of methodology they use, because if you suck in software, your software will suck, no XP/TDD/Agile can help you make it better (and for the record: neither waterfall or other methodology).
    What I find a little disturbing though is that reading the comments here, some people jump to the conclusion that it is now proven that TDD assures better productivity/better quality etc.

  37. Avatar for Jacob Proffitt
    Jacob Proffitt January 29th, 2008

    The discussion spawned from my TDD post has been interesting to me. I've enjoyed the comments by Phil

  38. Avatar for Scruffy-Looking Cat Herder
    Scruffy-Looking Cat Herder January 29th, 2008

    Post Comments

  39. Avatar for Haacked
    Haacked January 29th, 2008
    I'm still convinced that simply the best professionals create the best stuff, regardless what kind of methodology they use...


    Well that goes without saying, right Frans? ;)
    I think of it like the book, the 7 Habits of Highly Effective People. I'm sure there are truly effective people who don't exhibit all of these habits. Perhaps even the rare few who don't have any of these habits. But I bet overall, the habits are quite common among highly effective people.
    However, I bet that even a very effective person could be even more effective by adopting these habits if they haven't done so already.
    So the question becomes, is writing automated unit tests (separate from TDD) a habit that can help good developers become even better? I think there's strong evidence for that, but again, more research is needed.
    The next question is whether TDD specifically is a habit that can help effective developers become even more effective? It's worked for me and I strongly believe it can work for others in general, but I think the evidence is less conclusive.
    It may well be that for me personally, TDD is a habit that instills the discipline I lacked before. In that case, it greatly improved my software development skills. It's quite possible there may be devs who naturally have that discipline and TDD would not provide that much benefit. My intuition is that the number of devs in that camp is very small.
    Frans, you are probably in that camp, so perhaps you don't see the benefit of TDD because it wouldn't benefit you (just guessing there). However it may well benefit a great majority of developers simply for the reason that it's a habit that requires discipline. Kind of like how it's easier to do a long run when wearing headphones playing music with a good beat. TDD provides that beat for me.

  40. Avatar for Jeremy D. Miller -- The Shade
    Jeremy D. Miller -- The Shade February 2nd, 2008

    I really don't have time for this, but I'm going to do it anyway. I'm starting a new blog

  41. Avatar for Community Blogs
    Community Blogs February 4th, 2008

    Phil Haack has an interesting post on some research regarding the effectiveness of Test Driven Development

  42. Avatar for Slevdi
    Slevdi February 4th, 2008

    Just my thoughts on TDD in the modern world:
    Whilst I have always created automated tests (well, often), I am new to TDD. I love it, however it has proven conclusively to me that I need to know lots about what I need to achieve (the problem) and how to achieve it, as well as lots more about the technology (the tools) I am using before I can make any acceptable progress.
    This is very different to the (my) old approach of having a general idea of the problem then developing a solution incrementally as one's understanding of the capabilities of the technology increases. I did this with Perl and VBScript apps and webapps throughout the 90s, developing tests where I could along the way, using more effective techniques to improve the delivered application's functionality and maintainability as I learned more.
    C#, MVP, object orientation, patterns, asp.net framework(s), mocking packages, Http simulators (quick ad for Phil:-), Resharper, VS and so on, now potentially can make me amazingly productive once I have become totally familiar with them. The learning curve is very substantial to achieve this, and it is currently a very frustrating if intellectually stimulating time for me.

  43. Avatar for Troy DeMonbreun
    Troy DeMonbreun February 7th, 2008

    I finally posted my thoughts on what TSD (Test Supported Development) would look like:
    http://blog.troyd.net/Test+...

  44. Avatar for Troy DeMonbreun
    Troy DeMonbreun February 7th, 2008

    Sorry, that looks ugly in IE7. :/
    I'll try that again.
    I finally posted my thoughts on what TSD (Test Supported Development) would look like:
    Test Supported Development (TSD) is not Test Driven Development (TDD)

  45. Avatar for Scott
    Scott February 10th, 2009

    I'm working on two projects simultaneously. On one, I'm mandated to use TDD. On another, I am not. In fact, I'm using no testing at all. Both are commercial projects that need to be bug free.
    I hear some of you screaming! No testing? At all? Arrest that man!
    The thing is, I am VASTLY more efficient working without writing tests. It isn't simply laziness: I get more done in a day if I don't have to write tests, and I don't think my code is any worse for it.
    How can that be? Well, I've been doing this for twenty years. I'm good at what I do, and I test my code as I go - usually by setting a breakpoint and stepping through it. Very occasionally I'll write a test for a particular algorithm. Often I'll either remove it when I'm done, or I'll leave the test in the working code to be run only in debug mode.
    Instead of using "mock data" - which I have to create myself, and which may not be representative of the real data the program will use - I test against the real deal.
    In my experience:
    - It's very difficult to write tests that catch the hard stuff: memory leaks, memory corruption, thread blocking, API bugs, race conditions, etc.
    - It's hard to write tests that exhaustively test functions.
    - Tests, particularly exhaustive tests, are difficult to read and so don't serve to document code.
    - "Mock data" only represent a set of assumptions, not real world data.
    - An incorrect test is worse than one at all.
    For me, programming is both a creative act and an act of discovery. I often come to one solution by trying a different one first and then having an insight that leads me to something better. I cannot design this way through writing tests.
    And finally, I've yet to see our tests catch a single bug, and I've only very rarely gotten any value out of them as "documentation".

  46. Avatar for Chris
    Chris February 10th, 2009

    Like other people I am still hesitant to apply TDD. Here are some doubts in my mind:
    1. Those TDD survey results that companies published are bias in my opinion. I have never come across an article about companies that used TDD and failed and I don't think it's possible that there weren't any.
    2. I haven't yet read articles or blogs that mentioned about the benefits of TDD like:
    a. Code quality - nobody will argue with this so let's skip this *
    b. Overall cost of the project - is it expensive?
    c. Overall development time - was it shortened?
    d. Maintenance cost - ROI benefits?
    3. Part of software development is about negotiations and estimates. If your estimate is too expensive then the client will look for an alternative solution or better yet look for cheaper companies that will still produce high quality software. So the phrase "in time cost will be forgotten because of quality", mentioned by TDD followers, as their justification to cost, will not be applicable, correct?
    4. TDD is about design so it encourages developers to design their classes well. So I'd assumed that in TDD, developers == architects? If the answer is no then isn't there a possibility that there will be duplicate classes everywhere if 10 developers start writing their own classes and/or interfaces as they see fit?
    5. Two set of codes to maintain. Change request means two sets of code to fix. Is this something like a trade off when using TDD? Or, is this the price that we have to pay for the sake of "quality"? How do you justify the maintenance/development cost to your client given that you also have a QA department to test the application?
    6. If developers writes tests then who tests their tests? An endless loop? A developer that writes crappy code will get away with this practice if no one validates his work, right?

  47. Avatar for downloadable files
    downloadable files June 24th, 2010

    I find that full-on TDD, especially when coupled with 100% code coverage, can be sluggish and awkward. I feel something in the spirit of the 80/20 rule is more apt to produce a better ROI if time to market is limited (given available resources). For instance, hit the most complex/critical code the hardest, with _some_ tests driving/exploring the interface design, but with other interfaces just following good design idioms.

  48. Avatar for Steve Fenton
    Steve Fenton November 23rd, 2011

    When you work in software, one of the valuable things you pick up over time is experience.
    I have worked in teams that don't test, teams that write tests after they code and teams that use TDD.
    If I measured each of these teams against the others, it would definitely show that having tests is better than having no tests and it would also show that TDD has additional benefits over and above simply writing automated tests.
    One important caveat to these (very personal) results is that I find that the the quality of the developers was massively different between these teams, so even if we removed the variability of tests the teams would still rank the same - so the question is about order, does TDD (combined with other methods) make great developers, or do great developers find something of value in TDD? The great developers were also inclined towards regular releases, lean process and paired programming.
    I don't know what the cause/effect cycle is, but TDD is definitely related to great developers in some way.

  49. Avatar for bobfoster
    bobfoster December 3rd, 2011

    That link to the paper doesn't work, leaving me with the strong impression that the research doesn't support the conclusions you and the researchers have drawn and no way to be convinced otherwise.

  50. Avatar for Saranarunmallur
    Saranarunmallur February 20th, 2013

    hi,
     Is it able to prove TDD is best compared to code inspection.suggest some idea and also related paper to prove