Not Really Interested In Lean

code, nuget 0 comments suggest edit

We could have done better. That’s the thought that goes through my mind when looking back on this past year and reflecting on NuGet.

Overall, I think we did pretty well with the product. Nobody died from using it, we received a lot of positive feedback, and users seem genuinely happy to use the product. So why start off with a negative review?

It’s just my way. If you can’t look back on every project you do and say to yourself “I could have done better”, then you weren’t paying attention and you weren’t learning. For example, why stop at double rainbows when we could have gone for triple?


When starting out on NuGet, we hoped to accomplish even more in our first full release. Like many projects, we have iteration milestones which each culminate in a public release. Ours tended to be around two months in duration, though our last one was one month.

Because we were a bit short staffed in the QA department, at the end of each milestone our one lone QA person, Drew Miller, would work like, well, a mad Ninja on fire trying to verify all the fixed bugs and test implemented features. Keep in mind that the developers do test out their own code and write unit tests before checking the code in, but it’s still important to manually test code with an eye towards thinking like a user of the software.

This my friends, does not scale.

When we look back on this past year, we came to the conclusion that our current model was not working out as well as it could. We weren’t achieving the level of quality that we would have liked and continuing in this fashion would burn out Drew.

I came to the realization that we need to assume we’ll never be fully staffed on the QA side. Given this, it became obvious that we need a new approach.

This was apparent to the developers too. David Fowler noted to me that we needed to have features tested closer to when they were implemented. As we discussed this, I remember a radical notion that Drew told me about when he first joined our QA team. He told me that he wants to eliminate dedicated testers. Not actually kill them mind you, just get rid of the position.

An odd stance for someone who is joining the QA profession. But as he explained it to me in more detail over time, it started to make more sense. In the most effective place he worked, every developer was responsible for testing. After implementing a feature and unit testing it (both manually and via automated tests), the developer would call over another developer and fully test the feature end-to-end as a pair. So it wasn’t that there was no QA there, it was that QA was merely a role that every developer would pitch in to help out with. In other words, everyone on the team is responsible for QA.

So as we were discussing these concepts recently, something clicked in my thick skull. They echoed some of the concepts I learned attending a fantastic presentation back in 2009 at the Norwegian Developer’s Conference by Mary Poppendieck. Her set of talks focused on the concept of a problem solving organization and the principles of Lean. She gave a fascinating account of how the Empire State Building finished in around a year and under budget by employing principles that became known as Lean. I remember thinking to myself that I would love to learn more about this and how to apply it at work.

Well fast forward a year and I think the time is right. Earlier in the year, I had discussed much more conservative changes we could make. But in many ways, by being an external open source project with a team open to trying new ideas out, the NuGet team is well positioned to try out something different than we’ve done before as an experiment. We gave ourselves around two months starting in January with this new approach and we’ll evaluate it at the end of those two months to see how it’s working for us.

We decided to go with an approach where each feature was itself a micro-iteration. In other words, a feature was not considered “done” until it was fully done and able to be shipped.

So if I am a developer working on a feature, I don’t get to write a few tests, implement the feature, try it out a bit, check it in, and move on to the next feature. Instead, developers need to call over Drew or another available developer and pair test the feature end-to-end. Once the feature is fully tested, only then does it get checked into the main branch of our main fork.

Note that under this model, every developer also wears the QA hat throughout the development cycle. This allows us to scale out testing whether we have two dedicated QA, one dedicated QA, or even zero. You’ll notice we’re still planning to keep Drew as a dedicated QA person while we experiment with this new approach so that he can help guide the overall QA process and look at system level testing that might slip by the pair testing. Over time, we really want to get to a point where most of our QA effort is spent in preventing defects in the first place, not just finding them.

Once a feature has been pair tested, that feature should be in a state that it can be shipped publicly, if we so choose.

We’re also planning to have a team iteration meeting every two weeks where we demonstrate the features that we implemented in the past two weeks. This serves both to review the overall user experience of the features as well as to ensure that everyone on the team is aware of what we implemented.

You’ll note that I’m careful not to call what we’re doing “Lean” with a capital “L”. Drew cautioned me to user lower-case “lean” as opposed to capital “Lean” because he wasn’t familiar with Lean when he worked this model at his previous company. We wouldn’t want to tarnish the good name of Lean with our own misconceptions about what it is.

This is where I have to confess something to you. Honestly, I’m not really that interested in Lean. What I’m really interested in is getting better results. It just seems to me that the principles of Lean are a very good approach to achieving those results in our situation.

I’m not one to believe in one true method of software development that works for all situations. What works for the start-up doesn’t work for the Space Shuttle and vice versa. But from what I understand, NuGet seems to be a great candidate for gaining benefits from applying lean principles.

So when I said I’m not interested in Lean, yeah, that was a bit of a fib. I definitely am interested in learning more about Lean (and I imagine I’ll learn a lot from many of you). But I am so much more interested on the better results we hope to achieve by applying lean principles.

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.



21 responses

  1. Avatar for Emmanuel Morales
    Emmanuel Morales December 20th, 2010

    How will this work in the context of community contributions? Or does it not apply?

  2. Avatar for fschwiet
    fschwiet December 20th, 2010

    Are your developers only writing unit tests, or are they also writing acceptance tests?

  3. Avatar for Bill Christie
    Bill Christie December 20th, 2010

    I'm really curious, Phil. What results have you seen so far? Was the process effective? What are the pain points?

  4. Avatar for Phill
    Phill December 20th, 2010

    NuGet will continue to fail until it becomes up to date with it's packages. Everything is far too out of date and it's a burden relying on other people to submit stuff. OpenWrap fails at being useable.

  5. Avatar for Adam Caviness
    Adam Caviness December 20th, 2010

    Regardless of what comments you may receive for this, and I expect some. I think you've made your point clear and I think you're making the right choice for NuGet. Impressed you're willing to do it.

  6. Avatar for hcoverlambda
    hcoverlambda December 20th, 2010

    I couldn't quite tell from your post, but is your team practicing TDD?

  7. Avatar for Adron
    Adron December 20th, 2010

    I sure hope you're willing to pair & TDD too. If so than lean will be freaking awesome for you guys.
    Lean as a principle really helps, and I was about to fly off the handle when I read your title, but glad I read on without being too hot headed. ;)
    I wish you the best in moving forward with NuGet. I imagine I'll be using it a bit in the coming year! Hats off and have fun!

  8. Avatar for Slava
    Slava December 20th, 2010

    Phill, I think you need stop and think a little bit. You need to fix all current issues and bugs without implementing new staff. After that you need to think about most important functionality for users and make reverse testing one more time. Only after that you can continue development of your new features. IMHO.

  9. Avatar for T_Zano
    T_Zano December 20th, 2010

    I do agree Slava , We haven't to re-buikd the wheel every time , try to find the interest and the more interest fonctionality for the user , after that you can fire your creativity to add other feautures .

  10. Avatar for Adam
    Adam December 20th, 2010

    Well we've stopped feeding our employees, that's how we keep them 'Lean.' Insted we spend the money on power Yoga to make them more 'Agile.' We also introduced a team 'Rugby' team. Scrum didn't go down well. Next year I'll find a new way of treating everyone like they've got sh*t for brains. I'm watching the engineering industry very closely for ideas.

  11. Avatar for Ryan Cromwell
    Ryan Cromwell December 20th, 2010

    It's going to hurt at first. It will feel like your velocity fell through the floor and people will be frustrated with the feature "completion" rate. I quote that, because a buggy feature isn't complete for the person experiencing the bug or the person who's feature isn't being developed while buggy feature is being triaged and fixed.
    In the end having automated regression tests is the key to raising and maintaining your velocity over time. Nuget is lacking that in many areas, especially around the packages themselves.
    My #1 recommendation on this particular project would be to optimize the mechanism test packages. A-la, Ayende's NHProf

  12. Avatar for Keith Kmett
    Keith Kmett December 20th, 2010

    So what I'm reading in this article is this:
    TDD (Unit Tests) ≠ QA
    The team I work on has known this for years. It doesn't matter how many green lights your unit tests have, it doesn't account for people. Our team has decided to focus less on unit tests and more on QA and usability test (yes, the developers do all this) - which has raised our quality level to a very high standard.
    I was once told, "You need to lower your quality standard for other developers." My response was, "Lowering my standards would mean that I can get any trained chimp to code. I want our entire team to be stronger and smarter than that!"

  13. Avatar for BorisCallens
    BorisCallens December 20th, 2010

    At our company we evolved our development process down the same road during the course of the last few years and I must say that the last two projects are going rather well.
    You might want to google Definition Of Done ("DOD"). I was sceptic at first, but it has proven very useful and I changed my opinion.
    A demo every sprint (we have sprints of about 3 weeks, depending on the project) for our stakeholders has also proven positive. It increased interest in our project development, gave our stakeholders an idea of the scale of the project and has created useful feedback sooner in the project cycle.
    We also do (almost) everything in pair, from creating sprint items, to devving and testing (QA)

  14. Avatar for John Stovin
    John Stovin December 21st, 2010

    I was working on a project a couple of years ago where we tried exactly this approach. We were a Scrum team with a perpetual problem of 'not enough QA', so we agreed that developers would not start a new feature while any user stories were still waiting for testing, but should test instead. It actually increased our velocity because we got all our stories finished every sprint.
    Having one QA person as a fallback/oracle/adviser was vital though - they always have a different take from developers.

  15. Avatar for BorisCallens
    BorisCallens December 21st, 2010

    @John: We tried sitting with our end user from time to time and watch how they used our application. Although it didn't bring us the revelation we had hoped for, it did show us some new views on how our application would be used so we could tune our testing focusses differently.
    Go sit next to your user and ask them to do their day-to day job whilst talking out loud about what they are doing. The issue we had was it felt awkward and users would behave somewhat unnatural. We haven't found a way to improve the process yet, but it did help us some

  16. Avatar for haacked
    haacked December 21st, 2010

    @Emmanuel I plan to blog about that very topic.

  17. Avatar for smnbss
    smnbss December 21st, 2010

    @Emmanuel you can always use skype to share the desktop while testing or if (as you should) have well documented steps to reproduce the bug or a well defined scenario for a feature, do peer testing. You test the features of somebody else, and somebody else test your features. the task is not completed until someone test the feature.
    In a community this would also help to speed up the development cycle of the top priority features/bugs

  18. Avatar for haxen
    haxen December 21st, 2010

    "This is where I have to confess something to you. Honestly, I’m not really that interested in Lean. What I’m really interested in is getting better results. It just seems to me that the principles of Lean are a very good approach to achieving those results in our situation."
    I would say that this is the core of lean. Reflecting of how we can get better results, remove unnecessary stuff, add stuff needed to improve and above all responsibility for everyone to be part of delivering functionality to the end user.

  19. Avatar for Federico Silva
    Federico Silva December 22nd, 2010

    Hi Phil. Knowing a bit about the team, I really like how your experiment sounds :)
    In my new job this is what we do as well. You kinda have to when there's 6 "dedicated" testers to 100 devs :P
    Happy holidays!

  20. Avatar for Joe Chung
    Joe Chung December 25th, 2010

    "I came to the realization that we need to assume we’ll never be fully staffed on the QA side." Well, that's a shame.

  21. Avatar for Ellen Williams
    Ellen Williams December 28th, 2010

    I really like the idea of shared ownership of the "big P" product, not just delivery of individual features. The pair-testing approach should foster a better understanding on all sides.
    I'd be interested in knowing whether you fix bugs as you find them during pair-test or take notes, address, then restart the pair-testing cycle?