Test Better

code, tdd, github 0 comments suggest edit

Developers take pride in speaking their mind and not shying away from touchy subjects. Yet there is one subject makes many developers uncomfortable.


I’m not talking about drug testing, unit testing, or any form of automated testing. After all, while there are still some holdouts, at least these types of tests involve writing code. And we know how much developers love to write code (even though that’s not what we’re really paid to do).

No, I’m talking about the kind of testing where you get your hands dirty actually trying the application. Where you attempt to break the beautifully factored code you may have just written. At the end of this post, I’ll provide a tip using GitHub that’s helped me with this.

TDD isn’t enough

I’m a huge fan of Test Driven Development. I know, I know. TDD isn’t about testing as Uncle Bob sayeth from on high in his book, Agile Software Development, Principles, Patterns, and Practices,

The act of writing a unit test is more an act of design than of verification.

And I agree! TDD is primarily about the design of your code. But notice that Bob doesn’t omit the verification part. He simply provides more emphasis to the act of design.

In my mind it’s like wrapping a steak in bacon. The steak is the primary focus of the meal, but I sure as hell am not going to throw away the bacon! I know, half of you are hitting the reply button to suggest you prefer the bacon. Me too but allow me this analogy.

bacon-wrapped-steakMMMM, gimme dat! Credit: Jason Lam CC-BY-SA-2.0

The problem I’ve found myself running into, despite my own advice to the contrary, is that I start to trust too much in my unit tests. Several times I’ve made changes to my code, crafted beautiful unit tests that provide 100% assurance that the code is correct, only to have customers run into bugs with the code. Apparently my 100% correct code has a margin of error. Perhaps Donald Knuth said it best,

Beware of bugs in the above code; I have only proved it correct, not tried it.

It’s surprisingly easy for this to happen. In one case, we had a UI gesture bound to a method that was very well tested. Our UI was bound to this method. All tests pass. Ship it!

Except when you actually execute the code, you find that there’s a certain situation where an exception might occur that causes the code to attempt to modify the UI on a thread other than the UI thread #sadtrombone. That’s tricky to catch in a unit test.

Getting Serious about Testing

When I joined the GitHub for Windows (GHfW) team, we were still in the spiking phase, constantly experimenting with the UI and code. We had very little in the way of proper unit tests. Which worked fine for two people working in the same code in the same room in San Francisco. But here I was, the new guy hundreds of miles away in Bellevue, WA without any of the context they had. So I started to institute more rigor in our unit and integration tests as the product transitioned to a focus on engineering.

But we still lacked rigor in regular non-automated testing. Then along comes my compatriot, Drew Miller. If you recall, he’s the one I cribbed my approach structuring unit tests from.

Drew really gets testing in all its forms. I first started working with him on the ASP.NET MVC team when he joined as a test lead. He switched disciplines from a developer to become a QA person because he wanted a venue to test this theories on testing and eventually show the world that we don’t need separate QA person. Yes, he became a tester so he could destroy the role, in order to save the practice.

In fact, he hates the term QA (which stands for Quality Assurance):

The only assurance you will ever have is that code has bugs. Testing is about confidence. It’s about generating confidence that the user’s experience is good enough. And it’s about feedback. It’s about providing feedback to the developer in lieu of a user in the room. Be a tester, don’t be QA.

On the GitHub for Windows team, we don’t have a tester. We’re all responsible for testing. With Drew on board, we’re also getting much better at it.

Testing Your Own Code and Cognitive Bias

There’s this common belief that developers shouldn’t test their own code. Or maybe they should test it, but you absolutely need independent testers to also test it as well. I used to fully subscribe to this idea. But Drew has convinced me it’s hogwash.

It’s strange to me how developers will claim they can absolutely architect systems, provide insights into business decisions, write code, and do all sorts of things better than the suits and other co-workers, but when it comes to testing. Oh no no no, I can’t do that!

I think it’s a myth we perpetuate because we don’t like it! Of course we can do it, we’re smart and can do most anything we put our minds to. We just don’t want to so we perpetuate this myth.

There is some truth that developers tend to be bad at testing their own code. For example, the goal of a developer is to write software as bug free as possible. The presence of a bug is a negative. And it’s human nature to try to avoid things that make us sad. It’s very easy to unconsciously ignore code paths we’re unsure of while doing our testing.

While a tester’s job is to find bugs. A bug is a good thing to these folks. Thus they’re well suited to testing software.

But this oversimplifies our real goals as developers and testers. To ship quality software. Our goals are not at odds. This is the mental switch we must make.

And We Can Do It!

After all, you’ve probably heard it said a million times, when you look back on code written several months ago, you tend to cringe. You might not even recognize it. Code in the brain has a short half-life. For me, it only takes a day before code starts to slip my mind. In many respects, when I approach code I wrote yesterday, it’s almost as if I’m someone else approaching the code.

And that’s great for testing it.

When I think I’m done with a feature or a block of code, I pull a mental trick. I mentally envision myself as a tester. My goal now is to find bugs in this code. After all, if I find them and fix them first, nobody else has to know. Whenever a customer finds a bug caused by me, I feel horrible. So I have every incentive to try and break this code.

And I’m not afraid to ask for help when I need it. Sometimes it’s as simple as brainstorming ideas on what to test.

One trick that my team has started doing that I really love is when a feature is about done, we update the Pull Request (remember, a pull request is a conversation about some code and you don’t have to wait for the code to be ready to merge to create a PR) with a test plan using the new Task Lists feature via GitHub Flavored Markdown.

This puts me in a mindset to think about all the possible ways to break the code. Some of these items might get pulled from our master test plan or get added to it.

Here’s an example of a portion of a recent test plan for a major bug fix I worked on (click on it to see it larger).


The act of writing the test plan really helps me think hard about what could go wrong with the code. Then running through it just requires following the plan and checking off boxes. Sometimes as I’m testing, I’ll think of new cases and I’ll just edit the plan accordingly.

Also, the test plan can serve as an indicator to others that the PR is ready to be merged. When you see everything checked off, then it should be good to go! Or if you want to be more explicit about it, add a “sign-off” checkbox item. Whatever works best for you.

The Case for Testers

Please don’t use this post to justify firing your test team. The point I’m trying to make is that developers are capable of and should test their own (and each others) code. It should be a badge of pride that testers cannot find bugs in your code. But until you reach that point, you’re probably going to need your test team to stick around.

While my team does not have dedicated testers, we consider each of us to be testers. It’s a frame of mind we can put our minds into when we need to.

But we’re also not building software for the Space Shuttle so maybe we can get away with this.

I’m still of the mind that many teams can benefit from a dedicated tester. But the role this person has is different from the traditional rote mechanical testing you often find testers lumped into. This person would mentor developers in the testing part of building software. Help them get into that mindset. This person might also work to streamline whatever crap gets in the way so that developers can better test their code. For example, building automation that sets up test labs for various configuration in a moment’s notice. Or helping to verify incoming bug reports from customers.

Related Posts

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.



24 responses

  1. Avatar for TristanIrwin
    TristanIrwin March 4th, 2013

    I find myself having difficulty sometimes testing UI code, because I tend to do things "right" (as in, not in a way that would make it break). Because I wrote it, and know exactly what you are supposed to do to achieve a specific result, I often miss the "what if the end user is stupid?" bugs. In the business end of the code, I can imagine myriad possibilities where some dumb programmer misused my code and I need to defend myself from doing something wrong, but I can't always dream of the ways that end users have managed to mess things up.

  2. Avatar for Grahame SD
    Grahame SD March 4th, 2013

    I agree with much of what you say, Phil, but I can't agree that a developer who tests is a workable replacement for a dedicated tester (or QA or whatever you want to call it). 
    Sure, I hate having bugs found in my software, and I am capable of testing and I test the heck out of my code before I hand it over to QA, but they still find stuff.
    Over the years I have always seen quality decrease (and therefore always seen confidence decrease) when the team has lost a dedicated QA.
    So, what you say is nice in theory, and maybe it can work with exceptional individuals on a small team such as you have, but I've just never seen it work in practice and especially not with average developers.

  3. Avatar for Vitoc
    Vitoc March 4th, 2013

    Good senior software engineers aren't cheap.  There has to be a balance.  For example, if I spent an hour testing the code I just wrote and found an obscure bug that a tester making less than half would have found in that same hour, was that really a win for the company?  I can tell you what my boss and CEO would say.

    I'm in a SaaS situation.  It might be a different situation for independent contractors and consultants, when their customer really is another company and they can't rely on a dedicated team of testers to put their work through the ringer before hitting end users in production.

  4. Avatar for Mike
    Mike March 4th, 2013

    I guess developers denying their ability to test is a form of self preservation as well. It obviously it depends on the company you work for, but us developers can be portrayed as incapable for doing something as horrible as creating a bug.  When you say that developers can't test their own code you just shift the responsibility :)

  5. Avatar for Nick
    Nick March 4th, 2013

    I think one aspect is motivation. Some people have the desire and hunger to do a sincere job. Part of them going to work is about doing the best job possible.

    Others are developer prima-donnas and quite arrogant, and just don't have that enthusiasm to do a thorough job of testing it manually.

    There's a lot more to in that, but I think motivation is a big factor. Glad you raised the topic Phil.

  6. Avatar for Luke McGregor
    Luke McGregor March 4th, 2013

    Off topic i know but just out of interest whats with the 9% anyway, (it seems like this is the bit where it actually connects up and starts trying to sync the repository)

  7. Avatar for Steve Lydford
    Steve Lydford March 5th, 2013

    "But we’re also not building software for the Space Shuttle so maybe we can get away with this."

    No, but you might be building software for people who are building software for the Space Shuttle! https://github.com/nasa
    Great post, really enjoyed it. Many thanks

  8. Avatar for Eli Weinstock-Herman
    Eli Weinstock-Herman March 5th, 2013

    In counterpoint, the feature that your expensive software developers just spent 2 days finishing wasn't cheap either. Separating the testing duties tends to make for a longer delivery pipeline, so that expensive feature sits on the shelf longer and it takes longer to get anything from the start point to the customer (the faster you can do this, the more responsive the company feels, the more you can experiment w/ your customers to figure out what they want, etc). Plus, every time the responsibility for that code changes hands, it becomes more expensive for the original developer to fix anything, as they have switched gears to something else and will require that much longer to get back into the details (while putting whatever they moved on to the side on hold). It also requirs that you have much more detailed requirements or handoff processes (the tester now needs to understand all of the requirements and ancillary impacts well enough to test it now).

    To me, the trade off is not worth it. The flexibility and speed to market has a value that is more then the difference of one tester (and that's assuming I hire one tester per developer), especially when you factor in the costs of working on bugs in software you had handed off (2x-10x, depending on where you are when you find it), additional processes/requirements documentation/whatever, and the additional detailed communications overhead.

  9. Avatar for firoz khan
    firoz khan March 5th, 2013


  10. Avatar for Vitoc
    Vitoc March 5th, 2013

    I've got another example, and this one is slightly less business oriented.  I've created a free online multiplayer text-based RPG as a personal hobby.  It's currently in beta testing where we have 100+ players willing to play the game knowing full well there could potentially be bugs (and with the understanding they should be willing to report those bugs when they do find them).  Sure, I could try to set aside more of my spare time and painstakingly go through every new section of code I write, trying to account for every possible bug condition possible.  Or... I could step through the code a couple times, be fairly confident it meets the requirement, and count on a form of crowd-sourcing to help me find the more complex multi-condition bugs that potentially could have taken me weeks to find on my own as the primary developer.  Many of the players have many more hours in their days to play through and find bugs while I'm at work (this is a side project), whereas I can only allocate my spare time.

    I'm not advocating developers shouldn't test their code.  I'm simply making the point sometimes it does make more sense for developers to offload some of the more involved testing onto QA and other resources.  Costs and time-availability are two factors (among others I'm sure) to consider.

  11. Avatar for Uri Nativ
    Uri Nativ March 5th, 2013

    Very good article and is pretty much aligned with my thinking. I also gave a talk in a developer conference on how we work without testers - "QA without QA" - http://www.slideshare.net/u... 

  12. Avatar for softwaresprint
    softwaresprint March 6th, 2013

    Absolute agree. Good post.

    I believe testing starts with the developer and not the testing team. However, if we look at the testing hiearchy we find that above unit testing is integration testing and then about that UI testing and finally, manual testing. The manual testing just adds the human touch to testing.

    This blog post supports very much what you are saying


  13. Avatar for Brian Tabios
    Brian Tabios March 6th, 2013

    The best developers I've worked with have also been great testers. Thanks for this post, it's a good reminder for having the right mindset.

  14. Avatar for jimbarter
    jimbarter March 8th, 2013

    I'm a tester. Not a developer. There, said it.

    Quality Assurance is NOT testing code, QA is testing and *assuring* process. Whilst the developer may write 100% test covered code, if the test process is wrong or if the tests themselves  do not meet real world requirements then your 100% test covered code is worthless - the code works, the tests pass - it doesn't do what the client requires.

    A dedicated QA tester will test methodology, requirements, UX and more. A developer will test code.

  15. Avatar for Paweł
    Paweł March 8th, 2013

    "when I approach code I wrote yesterday, it’s almost as if I’m someone else approaching the code."

    That's true for most developers including me except when I write code for UI and I tend to be more emotionally engaged so I remember "paths" or "user stories" for which I made that UI code. It must take years before I can sit and see this UI as fresh user and discover bugs. Yes, testing is definitely our job as developers but in some cases it's good to know about limitation. Okay, maybe it's all about experience. Later you tend to think about more and more those "unexpected paths". But still, there is always something on the corner, I think :-)

  16. Avatar for CoffeeAddict
    CoffeeAddict March 8th, 2013

    This cracks me up.  I am totally in agreement with Jim on the QA deal.  QA is also to verify that features meet the needs of the original story.  While users can verify it in Quick Agile meetings going over stories and showing progress, it's still good to have QA go over functionality as it relates to the story to make sure it meets requirements.  I've seen this in action, you definitely need QA but you need GOOD QA who know what they should be doing, not finding bugs but testing functionality, usability, etc. which is more important which I think you need both UX (a good UX person who focuses on usability not just creating graphics) and QA working together on that. If TDD is done well bugs are at a bare minimum and then QA should be more focused on story requirements and circling around to double check them.

    I"m sorry I'm a developer and I can say not only I, but all developers don't know shit about a good UX and it's not our job anyway, I don't care what each developer may claim.  You need to have someone who knows UX in terms of "experience", "flow", "whitespace", etc. to make sure the experience makes sense and there are no part of the app that seem usable to you the developer but there might be flaws that even the business overlooked or improvements to the UX that might make a case to move x Story to a new iteration for the better.   

    But in the end I wish our Trade would freaking move forward for Christ sakes and do what you should be doing TDD and I even say XP; we're only at about 10% in reality overall in this world which is just pathetic at this point.  Tired of those shops not doing TDD yet, there should be no tolerance for such amateur ways of engineering...that being without TDD.  Take a look at Cengage Learning's XP teams, and you'll see why XP and TDD is the only way to go.

  17. Avatar for Pavel Gritsai
    Pavel Gritsai March 12th, 2013


  18. Avatar for Pavel Gritsai
    Pavel Gritsai March 12th, 2013

     Nice post! I have one note though, while your PR idea seems useful, in my opinion it can be better substituted with acceptance criteria (AC). There is no particular reason to wait until code is written to list down the AC (even using the format you used for PR) - in my team we are trying to do them in a "grooming" stage for each user story. Having all scenarios to be supported by a work item is helpful to have when developing (it makes sure dev doesn't forget about the specific thing, which may not align well with a chosen design - and result in a better design choice at the early stage).
    Also AC are helpful to improve an estimation quality.

  19. Avatar for Pepe
    Pepe April 5th, 2013

    That image is disgusting!

  20. Avatar for Dan Friedman
    Dan Friedman April 9th, 2013

    Yea, and I had a manager who believed that we didn't need QA or PMs because Devs could do everything. Think about that, if we got rid of tester and PMs, we would need to hire a lot more Devs. But just like a Ford assembly line, people will work better and faster at the jobs they are good at. Can I test? Yep. Can I manage? Sure. But do I do those as well as I develop? Nope. Could I learn to do them as well? Maybe. Are there people who are better testers and PMs than devs? Of course! So why would I make devs do jobs that they are not as good at, when there are people who are actually good at those jobs? Why would I not hire people for their strengths where those jobs are needed instead of making a group of people learn something that they will be OK at?

  21. Avatar for Henry Aung
    Henry Aung April 22nd, 2013

    Three words; Separation of Concerns. Developers should develop. Testers should test. Developer machine should be setup to develop applications productively. It is not a good test environment. Tester machine must be setup for testing and smoking out bugs productively. The keyword here is "productiveness". If you are busy switching role between being a developer and a tester, you waste a lot of valuable resources.

  22. Avatar for jjxtra
    jjxtra April 25th, 2013

    Developers should do unit tests, integration tests and probably a few sanity tests, but should not be doing much manual testing other than the basics. Paying developer salary for manual testing is probably not a good idea.

  23. Avatar for zersys
    zersys May 29th, 2013

    I am new to this software field.So,I used to read many blog like yours to get knowledge ...Thanks for sharing a really valuable information..

  24. Avatar for OLDetc
    OLDetc June 18th, 2013

    How about "What if the end user is smart and sees a new use?"