Quotas, What Are They Good For?

Sep 3, 2012 code suggest edit

If you look hard enough at our industry (really at all industries), you’ll find many implicit quotas in play. For example, some companies demand a minimum set of hours worked per week.

This reminds me of an apocryphal story of the “know where man”. Here’s one variant of this famous legend as described on snopes:

Nikola Tesla visited Henry Ford at his factory, which was having some kind of difficulty. Ford asked Tesla if he could help identify the problem area. Tesla walked up to a wall of boilerplate and made a small X in chalk on one of the plates. Ford was thrilled, and told him to send an invoice.

The bill arrived, for $10,000. Ford asked for a breakdown. Tesla sent another invoice, indicating a $1 charge for marking the wall with an X, and $9,999 for knowing where to put it.

In this variant, Ford is surprised by the price because $10,000 is a lot to pay for a few minutes of work. But as Tesla points out, he’s not paying for Tesla’s time, he’s paying for a solution to an expensive problem.

Another example is the idea of measuring a developer’s productivity by lines of code. Unless you sell code by the line, this is also pointless as Bill Gates once pointed out:

Measuring programming progress by lines of code is like measuring aircraft building progress by weight

Set working hours is another example of a poor quota. Developers aren’t paid for lines of code, number of hours in the office, or being in the office at certain hours. They’re paid to create value!

I got to thinking about this after reading an article completely unrelated to software - this heart wrenching and infuriating account of young offenders being enlisted as confidential informants and placed in extremely dangerous situations that far outweigh the gravity of their alleged crime.

One thing in particular caught my attention:

Mitchell McLean has come to see his son’s death as the result of an equally cynical and utilitarian calculation. “The cops, they get federal funding by the number of arrests they make—to get the money, you need the numbers,” he explained, alluding to, among other things, asset-forfeiture laws that allow police departments to keep a hefty portion of cash and other resources seized during drug busts. \

Notice the incentive here. The focus is on number of arrests. This focuses on a symptom, but not on the actual desired outcome.

That’s the problem with quotas. They rarely lead to the actual outcome you want. They simply reward gaming the quota by any means necessary.

This is not to say that all quotas are useless. Perhaps there are cases where they are called for. But they have to overcome the dreaded law of unintended consequences.

The law of unintended consequences, often cited but rarely defined, is that actions of people—and especially of government—always have effects that are unanticipated or unintended.

I imagine a good quota would be one in which it brings the system closer to the desired outcome and manages to avoid unintended consequences that would set the overall system back in worst shape than before. For example, perhaps if the gulf between your current state and the desired outcome is huge, a quota might help make small gains.

If you have examples where you think quotas produce the desired outcome with negligible unintended consequences, please do comment.

Found a typo or mistake in the post? suggest edit

Comments

20 responses

feng qijun • September 3rd, 2012
I love Bill's quota
Bertrand Le Roy • September 3rd, 2012
The Microsoft review system comes to mind. When you decide ahead of time that no matter what, 30% of your employees are going to underperform, what could possibly go wrong?
Apostol • September 3rd, 2012
An example of a place where quotas are misused is also one of the great problems of our society - medical services. Everywhere in the medical industry the doctors and medical staff and hospitals a paid for treatment of diseases, not cure. So more or less the system suggests if there are more treated people(sick people) - you get more funding.
You could reverse that and make it so if you cure more people from a disease - you get more funding. Or if you get funding based on the healthy people in the hospital's area instead of the 'treated' ones.
Imagine how much more motivated would the medical industry to produce healthier people and nation instead of a sick one.
GC • September 3rd, 2012
The classic quota causing unintended consequences is the European Union fishing quota. They restrict the amount of fish that can be landed to protect fish stocks, but what actually happens is tons of fish get thrown back dead...
http://www.fishfight.net/
GC
@ipartington • September 3rd, 2012
Other Examples we of numbers like this
Test code Coverage, just because we have code coverage is 100% dosnt mean we are testing how the SUT should work.
"Defects in you work" is another quota i dont like, taken out of context, eg complexity of work, number of interuptions, existing test coverage, could make a good developer look like a bad one.
Petar Repac • September 3rd, 2012
IMO 8h work days are there to protect workers, Eight-hour_day. But to be effective in today's global economy it should be enforced worldwide. Otherwise I'm afraid that competition will force us to work 12-15h per day.
Chris • September 3rd, 2012
I would have thought that stuff like "10% time" was a good example - you've already agreed upfront that the employee can spend their time on whatever takes their fancy so there isn't really a negative unintended consequence - any risk associated with abusing the time falls entirely on the employee (and in any case, its likely that same abuse would be present in the other 90% time anyway...)
John Atten • September 3rd, 2012
Don't get me wrong in the following - I totally agree with the reasoning in your post and the references cited. It's only a small stretch to apply the Heisenberg uncertainty principle to any performance measurement system, especially in the HR world (the mechanism of measuring and the choice of what is measured will tend to skewer the output, as employees "work to the test").
However, the desirability of the outcome you ask about is a perspective-based quantity as well. Think sales quota's. From the Sales managere's standpoint, imposition of sales quotas may well have the "desired" effect(s) of increasing overall sales, while also identifying sup-par performers. However, from the perspective of good customer service, the impact may be less fruitful, as sale people focus no selling to meet the quota, without regard for the actual needs of the customer.
I don't believe imposing a quota can be accomplished without causing some unintended consequence. A more effective strategy (which will *still* suffer from the impact of the measurement mechanism) is to establish targets which mitigate the most visible areas of undesirable consequence e.g:
"The Sales Team will achieve a 10% quarterly increase in sales revenue, while maintaining a 98% customer satisfaction rating and 80% customer retention."
Or however you want to express it (note I made up nonsense percentages). This strategy at least attempts to impose constraints upon those areas which would likely suffer if only the first metric ("Increased Sales") were in place.
Great post - thought-provoking. No doubt, as soon as I hit the submit button, I will think of some candidate answr to your original question!
John Ludlow • September 3rd, 2012
@Apostol A few years back there was a similar thing in schools in the UK. League tables were produced, with the intention of identifying schools that were struggling. Money would then be plumbed into those schools to help them do better.
But of course, some of the schools on the borderline purposely made their stats a little worse to get extra funding.
haacked • September 3rd, 2012
One of the better candidates I've heard for a good quota is minimum vacation time (thanks @rioter!). It's good for workers as they need to replenish. Good for employers because burnout is costly to the bottom line.
@Bertrand nothing could go wrong! ;p
Bertrand Le Roy • September 3rd, 2012
Thanks for reassuring me, Phil, for a moment I had a fleeting suspicion that the system maybe wasn't perfect :D
Fickle Panther • September 3rd, 2012
I can't imagine any quota that would ever be a decent measure of a good developer. It's not the lines of code, amount of comments, or hours worked. It's more abstract like how much effort was put in, how much was learned, how happy was the client, how much value was provided to the employer, etc.
I think every developer on every project is at a different level of understanding. So one developer may be able to do a project on time and make the client happy while another one who put in more effort may have some weakness in an area that caused the project to be late or make a client angry. Which one reached the quota? The one that put in the effort, or the one with more experience who made the client happy?
It's like trying to measure a student's success in school by some standardized method. Every student has talents in some areas and problems in other areas. Trying to compare one student with another is like comparing apples to oranges.
So, I think it's too abstract to measure by one specific quantitative value. You almost have to measure the developer to his own effort and ability.
Chris Rogers • September 3rd, 2012
While I do agree with the sentiments expressed, it is difficult to imagine a different world to the typical "set working hours" regime that we live with currently. Perhaps I'm just not imaginative enough.
But it is understandable for employers (and employees!) to want to put in place some protections against being taken advantage of. No employer will give you a set salary without putting in place some parameters for what they expect in return... and hours worked is at least an unambigious metric.
It would take a very brave business owner to allow staff to work whatever hours they think are appropriate for delivering value.
On some level, they *ARE* paying for your time. How much your time is worth is typically what varies.
From the other comments here, I do note how easy it is to find other examples of bad metrics ... but it is clearly more difficult to find workable suggestions which would be an improvement.
Robin Massart • September 4th, 2012
re: Apostol's comment above and slightly off topic: the health industry isn't interested in healthy people or dead people. Only in people who are ill. This leads to almost all of us being diagnosed with some problem or other so they can sell us their services and drugs.
As for quotas. I think this is a really good discussion to have. I suppose from a freelancer point of view, the best is to charge per hour and set your hourly rate as high as you feel is appropriate for your ability and experience. The problem is companies don't like this and often want fixed price quotes. In my opinion this results in more costs to them as a freelancer has to budget for unexpected stuff and invariably doubles or tripples his/her quote. This can lead to situations where a company ends up paying $1000s for a few hours work because they wanted a fixed price quote. Of course the reverse is also true.
Kris Vandermotten • September 4th, 2012
"Work is something you do, not a place you go to"
I guess it depends on your contract and legal system. But to me it boils down to this: if your employer is happy with your results, they should try to keep you on board. If they're not happy with them, they should let you go.
What makes them happy depends on the job, but surely the time you needed to achieve those results should not count. And if it did, it should count in a negative way. It's not the quantity of time needed, but the quantity AND quality of actual results that matter.
Take an example: your house needs a paint job. How will you choose your contracter? Will you prefer the one that will need the most time to do it? Of course not. In fact, when you have a choice of two contracters with equal reputation and total price, you will use the one that get's the job done in the smallest amount of time. Otherwise, you'll look at the value for money ratio, within contraints (minimum quality wanted, maximum amount of money to spend).
Now the question really is: can you quantify results?
Measuring square meters of paint on a wall is easy, but how do you measure the quality?
It turns out to be a very difficult thing to do, if you want absolute numbers. However, ranking employees based on relative performance seems easier, especially if you use peer review or a 360° evaluation scheme.
Employers should try to keep the employees they want to keep. Quantifying why they want to keep them is actually less important. Employees should try to get or keep the job they want. Again, quantifying why is not important.
Jeff Putz • September 4th, 2012
Ha! Bertrand beat me to it regarding the Microsoft review system. Put another way, you've got a team full of losers, and you only have to designate a few at the bottom. What could possibly go wrong, indeed!
Sadiq • September 4th, 2012
Overall very nice post
samy • September 5th, 2012
A fun example of quotas going wrong was in a company i interned at. Bonuses were awarded to developers according to the number of bugs they fixed. The system was promptly gamed by devs leaving bogus bugs in their code, in order to fix them very easily before review time.
var stuff = theCorrectValue;
// DELETE NEXT LINE TO CORRECT BUG
stuff += 1;
return stuff;
George Mauer • September 8th, 2012
A rarely (purposefully) used but sometimes effective strategy is to frequently change quotas. Since each individual quota does presumably work in the direction of optimizing some desirable aspect (eg minimum hours in the office might help get people working together more), by switching the main measurement somewhat frequently and in an unpredictable manner you can achieve some useful results.
OST • September 28th, 2012
we tell some of our developers that they need to take time off- We know they do quality work and work hard for it, so the rest is welcomed.