TDD Is Also An Organizational Process

0 comments suggest edit

After joining Microsoft and drinking from the firehose a bit, I’m happy to report that I am still alive and well and now residing in the Seattle area along with my family. In meeting with various groups, I’ve been very excited by how much various teams here are embracing Test Driven Development (and its close cousin, TAD aka Test After Development). We’ve had great discussions in which we really looked at a design from a TDD perspective and discussed ways to make it more testable. Teams are also starting to really apply TDD in their development process as a team effort, and not just sporadic individuals.

BugsTDD doesn’t work well in a vacuum. I mean, it can work to adopt TDD as an individual developer, but the effectiveness of TDD is lessened if you don’t consider the organizational changes that should occur in tandem when adopting TDD.

One obvious example is the fact that TDD works much better when everyone on your team applies it. If only one developer applies TDD, the developer loses the regression tests benefit of having a unit test suite when making changes that might affect code written by a coworker who doesn’t have code coverage via unit tests.

Another example that was less obvious to me until now was how the role of a dedicated QA (Quality Assurance) team changes subtly when part of a team that does TDD. This wasn’t obvious to me because I’ve rarely been on a team with such talented, technical, and dedicated QA team members.

QA teams at Microsoft invest heavily in test automation from the UI level down to the code level. Typically, the QA team goes through an exploratory testing phase in which they try to break the app and suss out bugs. This is pretty common across the board.

The exploratory testing provides data for the QA team to use in determining what automation tests are necessary and provide the most bang for the buck. Engineering is all about balancing constraints so we can’t write every possible automated test. We want to prioritize them. So if exploratory testing reveals a bug, that will be a high priority area for test automation to help prevent regressions.

However, this appears to have some overlap with TDD. Normally, when someone reports a bug to me, the test driven developer, I will write a unit test (or tests) that should pass but fails because of that bug. I then fix the bug and ensure my test (or tests) now pass because the bug is fixed. This prevents regressions.

In order to reduce duplication of efforts, we’re looking at ways of reducing this overlap and adjusting the QA role slightly in light of TDD to maximize the effectiveness of QA working in tandem with TDD.

In talking with our QA team, here’s what we came up with that seems to be a good guideline for QA in a TDD environment:

  • QA is responsible for exploratory testing along with non-unit tests (system testing, UI testing, integration testing, etc…). TDD often focuses on the task at hand so it may be easy for a developer to miss obscure test cases.
  • Upon finding bugs in exploratory testing, assuming the test case doesn’t require external systems (aka it isn’t a system or integration test), the developer would be responsible for test automation via writing unit tests that expose the bug. Once the test is in place, the developer fixes the bug.
  • In the case where the test requires integrating with external resources and is thus outside the domain of a unit test, the QA team is responsible for test automation.
  • QA takes the unit tests and run them on all platforms.
  • QA ensures API integrity by testing for API changes.
  • QA might also review unit tests which could provide ideas for where to focus exploratory testing.
  • Much of this is flexible and can be negotiated between Dev and QA as give and take based on project needs.

Again, it is a mistake to assume that TDD should be done in a vacuum or that it negates the need for QA. TDD is only one small part of the testing story and if you don’t have testers, shame on you ;) . One anecdote a QA tester shared with me was a case where the developer had nice test coverage of exception cases. In these cases, the code threw a nice informative exception. Unfortunately a component higher up the stack swallowed the exception, resulting in a very unintuitive error message for the end user. This might rightly be considered an integration test, but the point is that relying soley on the unit tests caused an error to go unnoticed.

TDD creates a very integrated and cooperative relationship between developers and QA staff. It puts the two on the same team, rather than simply focusing on the adversarial relationship.

Technorati Tags: TDD,Microsoft

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.



14 responses

  1. Avatar for Chris Ortman
    Chris Ortman January 14th, 2008

    What do the QA teams use to automate their UI testing?

  2. Avatar for Ian Suttle
    Ian Suttle January 15th, 2008

    Thanks for this Phil. Does Microsoft have any sort of official response to productivity gained vs. lost with TDD?

  3. Avatar for Trumpi's blog
    Trumpi's blog January 15th, 2008

    I'm quietly spending my last week of leave at home... Agile and Project Management Why do Some Projects

  4. Avatar for Haacked
    Haacked January 15th, 2008

    @Chris Ortman I believe it's some internal tool.
    @Ian No official response that I know of.

  5. Avatar for Troy DeMonbreun
    Troy DeMonbreun January 15th, 2008

    I would also propose the moniker of "Test-Supported Development", a.k.a. "TSD" to indicate the use of tests during development, that is, when most tests are not necessarily created first, but also not necessarily created as an afterthought, either.
    So far what I have personally adopted as a testing strategy looks more like Test-Supported Development (TSD).
    I find that full-on TDD, especially when coupled with 100% code coverage, can be sluggish and awkward. I feel something in the spirit of the 80/20 rule is more apt to produce a better ROI if time to market is limited (given available resources). For instance, hit the most complex/critical code the hardest, with _some_ tests driving/exploring the interface design, but with other interfaces just following good design idioms.
    Unless, of course, that's what you meant by Test After Development, which, at face value sounds like testing after development is completed.

  6. Avatar for Haacked
    Haacked January 15th, 2008

    @Troy, yeah, TAD doesn't capture what I mean. TSD is a much better moniker. Or perhaps TFD (Test Focused Development) or WTD (Well Tested Development. ;)

  7. Avatar for Christopher Steen
    Christopher Steen January 15th, 2008

    ASP.NET More ASP.NET debugging with SOS [Via: Tess ] Sharepoint Sahil Malik Shares the Point!! [Via:...

  8. Avatar for Christopher Steen
    Christopher Steen January 15th, 2008

    Link Listing - January 15, 2008

  9. Avatar for Craig Beck
    Craig Beck January 31st, 2008

    TAD doesn't have quite the same ring (or symmetry) as DDT -- Development Driven Testing...
    TDD <=> DDT
    not to mention that DDT is poisonous n a TDD environment...

  10. Avatar for Troy DeMonbreun
    Troy DeMonbreun February 7th, 2008

    I finally posted my thoughts on what TSD (Test Supported Development) would look like:
    Test Supported Development (TSD) is not Test Driven Development (TDD)

  11. Avatar for Edwin Pajemola
    Edwin Pajemola February 17th, 2009

    TDD == DDT assuming you know the end results before hand. My response to Craig is; Its not poisonous if software patterns repeat and if you know the end result, then why not take the shortcut. Time to completion is always a big factor.
    Great blog by the way!

  12. Avatar for June 24th, 2010

    For me, programming is both a creative act and an act of discovery. I often come to one solution by trying a different one first and then having an insight that leads me to something better. I cannot design this way through writing tests.

  13. Avatar for Miles
    Miles March 27th, 2012

    We have tried TDD in our organization. It is really tricky...It's good approach but at the same time Exploratory testing is Equally important. Not the case false confidence may lead to disasters after delivery. Because there is every possibility we will loose some cases which we cannot predict and integration of components also reveal new things.

  14. Avatar for Tddpro123
    Tddpro123 February 16th, 2013