Can You Really Measure Customer Satisfaction?

One question that came up recently during my mid-year review is how am I measuring customer satisfaction with the products that I work on? For example, how can I measurably demonstrate that customers are happy with the work we are doing on ASP.NET MVC and that my team is responding to customer feedback?

Umm, I can’t?

At least not right now in a measurable manner. I don’t have any such metric and I’m not sure how reliable any metric I might come up with will be.

rulerBut perhaps that’s simply due to a lack of imagination on my part.

In the past, we’ve tried various unscientific online twitter polls. One thing we did was ask the world for a list of ASP.NET pain points and compile them into one big list. Then at the end of the product cycle, we could show that we were able to address 10% of those pain points.

Personally, I feel that’s a bit contrived, but if it satisfies the muckety mucks, it works for me. ;)

Well actually, I’m not satisfied with that. At least not until I’ve tried to come up with something better. So I started thinking about what are ways to measure this and apart from simply using SurveyMonkey, I’ve sort of hit a wall.

This is where I ask you the question. If you were in my shoes what would you do? How have you quantitatively measured your effectiveness in responding to customer input? I’d love to hear your ideas.

UPDATE 2/4 1:06 AM By the way, I didn’t mean to give the impression that this is the only metric I need to collect. This is just the one I need help with. I think the number of downloads, installations, and clients is a much more important metric. However, some of those numbers are hard to collect with any accuracy. That’s why I think some of these other metrics might be useful as corroborating evidence. At least that’s my theory.

What others have said

Requesting Gravatar... Joshua Freislich Feb 03, 2010 4:58 PM
# re: Can You Really Measure Customer Satisfaction?
what about getsatisfaction.com?
Requesting Gravatar... Galilyou Feb 03, 2010 4:58 PM
# Success is the metric
The most honest way to tell if a product is successful is the number of clients using it. Checking out how many sites are built with MVC is an obvious way to count the number of users. Afterwards, support tickets, polls, even posts on the blog-sphere can provide some help.

For a personal feedback, though, as a client who built a commercial site with MVC 1.0, i'm pretty satisfied.
Requesting Gravatar... haacked Feb 03, 2010 5:05 PM
# re: Can You Really Measure Customer Satisfaction?
@Galilyou the number of clients using it is a difficult number to measure with any accuracy. I think the point of these other metrics (this blog post only addresses one metric, but there are others) is to provide extra corroborating evidence.
Requesting Gravatar... Chris Hardy Feb 03, 2010 5:07 PM
# re: Can You Really Measure Customer Satisfaction?
I would go with getsatisfaction.com - from what I remember it seemed to work pretty well for Silverlight 3/4.
Requesting Gravatar... haacked Feb 03, 2010 5:12 PM
# re: Can You Really Measure Customer Satisfaction?
Perhaps one metric to focus on is am I and my team responding to feedback? If I show how many bugs were accepted as a result of customer feedback, that would indicate we're effectively collecting customer feedback.
Requesting Gravatar... Louis DeJardin Feb 03, 2010 5:13 PM
# re: Can You Really Measure Customer Satisfaction?
Hmm... I think there's a hidden trap there in the word "quantitatively."

I started thinking - maybe a focus group? But then thought you could end up optimizing something like how people felt about their relationship with the framework's brand, or some other tangent.

Occured to me then even with the best of intentions any measure of a nebulous concept like performance or satisfaction runs the huge risk of becoming an end of its own rather than the means to measure something else.

Like when outsourcing a call center. By the numbers - minutes per call, percent of issues closed on first call, etc. You could be great on paper. But when you seek to optimize those numbers you lose site of the qualitative values. Like the actual service you get is awful.

Requesting Gravatar... Louis DeJardin Feb 03, 2010 5:26 PM
# re: Can You Really Measure Customer Satisfaction?
For example, you wouldn't want to be in a position where an increase in the number of bugs get from customer feedback year over year equals better service on paper. ;)
Requesting Gravatar... Romage Feb 03, 2010 6:58 PM
# re: Can You Really Measure Customer Satisfaction?
Actions speak louder than words. Check on the number of downloads of MVC. I think a separate quantitative measure could be by creating some "how to" vids, and monitor the downloads. You can also see how many twitter / blog followers you have etc.
Requesting Gravatar... Filip Ekberg Feb 03, 2010 8:49 PM
# re: Can You Really Measure Customer Satisfaction?
You can't really measure success by counting the downloads, I bet there are a lot of downloads that are never really installed. Even those who are satisfied and re-download them.

So without using any of those "Poll"-sites out there, maybe there should be a public ( even more public one ) bug-tracker where the community actually can help each other out.

And re-use the "Did this help you"-button from MSDN on all your videos and put a little "1 - 5"-star-rater next to it.

That would probably cover a little bit more than just doing an online survey..
Requesting Gravatar... Kyle Hodgson Feb 03, 2010 8:54 PM
# re: Can You Really Measure Customer Satisfaction?
Wonder if MVC would be visible in say a Netcraft survey. Does IIS indicate MVC somehow in HTTP headers?
Requesting Gravatar... Craig Stuntz Feb 03, 2010 10:16 PM
# re: Can You Really Measure Customer Satisfaction?
Phil,

I ran a query against the Stack Overflow public data dump which shows a steady and continuous increase in MVC questions per month:

http://stackql.net/default.aspx?qid=42654

SO just released another dump, but it doesn't look like it's on stackql.net, yet, so this only goes through mid-September 2009.

Still, I hope it's helpful.
Requesting Gravatar... Mat Roberts Feb 03, 2010 10:23 PM
# re: Can You Really Measure Customer Satisfaction?
I haven't read this book, but it sounds like the type of thing your trying to do:

www.amazon.co.uk/.../ref=sr_1_1

Requesting Gravatar... Javier Lozano Feb 03, 2010 11:15 PM
# re: Can You Really Measure Customer Satisfaction?
Dude, give them this link: http://bit.ly/a8bz3u Thats should do the trick. :)
Requesting Gravatar... Eric Feb 03, 2010 11:15 PM
# re: Can You Really Measure Customer Satisfaction?
An attempt to do a Twitter search for tweets like this and this.
Requesting Gravatar... Andrei Rinea Feb 03, 2010 11:24 PM
# re: Can You Really Measure Customer Satisfaction?
@Kyle Hodgson : There is (at least in MVC 1.0) an HTTP header called "X-AspNetMvc-Version".
Requesting Gravatar... Emmanuel Feb 03, 2010 11:52 PM
# re: Can You Really Measure Customer Satisfaction?
I say check the message boards, discussion groups, stackoverflow, etc. Not how many questions are being asked, but how many users participating, answering each other questions, recommending different solutions. How vibrant is the community, how many other user are helping other users. If you're satisfied with a product, you'll recommend it to your friend, you'll help others implement it.
Requesting Gravatar... Travis Illig Feb 04, 2010 12:06 AM
# re: Can You Really Measure Customer Satisfaction?
I'd avoid using polls since the participants aren't a statistically random set of people. You'll probably end up with either an improbably large number of negative results (since people love registering their discontent) or an improbably large number of positive results (very vocal supporters... which would help you from a corporate perspective but wouldn't accurately represent satisfaction). Online polls where you voluntarily participate aren't really objective.

getsatisfaction.com is probably the best suggestion I've seen. More accessible than your typical issue tracker and allows people to leave both positive and negative feedback.
Requesting Gravatar... Portman Feb 04, 2010 12:12 AM
# re: Can You Really Measure Customer Satisfaction?
Yes, you can measure customer satisfaction. Airlines, Amazon.com, and the American Medical Association do it all the time. And that's just the "A"s.

But the prerequisite to measuring is collecting data. And that's your problem with Microsoft developer tools like MVC.

how can I measurably demonstrate that [...] my team is responding to customer feedback?


Where is the customer feedback coming from? Current answer: all the frickin internet. It's on blog comments, Twitter, MSDN, Connect, StackOverflow, private emails, etc. There's no one official place for MVC developers to provide feedback. Therefore you can't collect any data, and you can't measure.

You need to pick one place where your customers are supposed to go to complain. Then you can measure how responsive you're being to feedback and how satisfied your customers are. Talk to some PMs with Amazon's customer service team - they are very good at this.
Requesting Gravatar... ciro Feb 04, 2010 12:36 AM
# re: Can You Really Measure Customer Satisfaction?
I'm a big fan of immediate feedback and I like the idea to give to the users some intruments to express their opinions and make them feel that they are listened. I like put an informal rudimental rating system directly inside the 'About' dialog and possibly show in real time the number of downloads to build up a sense of community. I envision a (possible not) introusive way where customers can rate single features directly inside the software, or at least prepare a special 'beta' version with the feedback process directly engraved in my projects
Requesting Gravatar... Andrew Hay Feb 04, 2010 12:47 AM
# re: Can You Really Measure Customer Satisfaction?
There are services that measure the tone and mood of the conversation on the Internet. The goal of the service is to discover conversations about your area of interest and then measure the mood, influence and trend. Corporations and government organizations really get into this stuff when they stand to gain/lose a lot based on public perception.

It's not too far of a stretch to turn something like this, http://waggeneredstrom.com/nn/demo/index.html, into a measurement device for conversations around ASP.Net, et al. There are other providers that offer this service too.

The license might be out of your budget range, but it would be a sweet graph to see.
Requesting Gravatar... John Feb 04, 2010 3:17 AM
# re: Can You Really Measure Customer Satisfaction?
Good luck.

I'm not a cynical person, but my first take on this is that this will just be another number for bean counters to pass to their bosses who pass to their bosses, etc.

The reason that I call them numbers for bean counters is that they're arbitrarily created and evaluated, but don't do actually do anything. Suppose you score an 8 out of 10, how much would getting a 9 affect MVC use? A 10? How much money would Microsoft potentially lose if it drops to a 6? A 5? How does the number affected investments?

The real challenge is to find metrics that can be related higher level metrics, which comes from a methodology known as The Balanced Scorecard. You should be able to relate the success of your efforts to the success of your customer metrics. And you should then be able to relate your customer metrics to profitability.

Find a customer measure that can be continuously measured. How, in turn, does satisfaction affect adoption and, presumably, profitability? Maybe you want to look at adoption rate among alpha developers. Maybe you want to do analysis of people actually using MVC -- are they using anti-patterns or bad practices, because they don't know about better ones?

Oops, sorry, I forgot that Steve Ballmer is in charge there. (I still remember Ballmer declaring Vista a tremendous success and the iPhone a flop, because it didn't have a keyboard). In that case, send out a survey to all MVPs.
Requesting Gravatar... Roberto Sebestyen Feb 04, 2010 4:02 AM
# re: Can You Really Measure Customer Satisfaction?
Take a look at ITIL and Service Desk and Incident Management. Although that may be overkill for a small project, it is indispensable to large corporations. Check out http://www.thinkitsm.com/
Requesting Gravatar... rickj Feb 04, 2010 4:32 AM
# re: Can You Really Measure Customer Satisfaction?
Check out how many people are using the forum the more people using it the more problems there are that need to be solved I think you guys are doing a great job and I allways get the support
and access to training to get me through put in one positive vote for me
Thanks
Requesting Gravatar... Dave Feb 04, 2010 4:32 AM
# re: Can You Really Measure Customer Satisfaction?
I would just like to add that we only build sites
in mvc now but to really know the world wide
opinion your best bet is prob a simple poll
cant beat em for instant results and voting on
features. Keep up the good work
Requesting Gravatar... Craig McGuff Feb 04, 2010 4:46 AM
# re: Can You Really Measure Customer Satisfaction?
Instead of trying to think outside the square, try thinking inside it - Microsoft is a big company, with lots of other software projects that probably have the same or similar measurement/reporting requirements. Without knowing the internal politics, I would try and speak to someone else in a similar role that has already done this (in a way that satisfies the powers that be, and is productive). Reuse is not just about software :)
Requesting Gravatar... Pablo Cifuentes Feb 04, 2010 5:05 AM
# re: Can You Really Measure Customer Satisfaction?
Exposed additional training,video and blogs, then gage the number of hits and comments. I believe the number of potencial learners and audience can provide you with an important hint to the number of people joining the MVC club. Always include two side to every asp net forum question, adding more information of solution on both platforms.

Continue with the great work and people with eventually value the product as yet again an important tool to our daily work.
Requesting Gravatar... Klaus Graefensteiner Feb 04, 2010 5:59 AM
# re: Can You Really Measure Customer Satisfaction?
You only hear from a satisfied customer when she buys your product or service again.
Requesting Gravatar... Asad Ali Butt Feb 04, 2010 6:04 AM
# re: Can You Really Measure Customer Satisfaction?
using downloads as metrics could be just a part of the measurement stretegy.

There are customers who will download, install, test and use.
Then there are clients which download , install and test.
Then there are clients which download and test only.
Then those who download and install only, but never have time for it and then those who only download and then forget untill after there PM announces that some project is going to have a touch with ASP.NET MVC.

I feel all downloads account for only 25 / 30 % of real users

your real figures should consist of real customer. A client is one who is your client for life time of the product, not who just downloads.
Requesting Gravatar... NC Feb 04, 2010 9:10 AM
# re: Can You Really Measure Customer Satisfaction?
IM NOT SATISFIED!!!!

Your making something extra for me to learn. Im not happy.

/end
Requesting Gravatar... Anthony Feb 04, 2010 10:54 AM
# re: Can You Really Measure Customer Satisfaction?
Maybe also by how many emails you and your team get over asp.net mvc. If you are getting lots of emails and helping people out then thats a good thing. It takes time for people to write emails asking questions, so I would think that this is something worth looking at.

Also maybe the number of questions and answer on stackoverflow... it shows how many questions are asked for particular topics, have a look and see what people are taking about.

Just a few thoughts.
Requesting Gravatar... Andri Haraldsson Feb 04, 2010 11:03 AM
# re: Can You Really Measure Customer Satisfaction?
Wow. I'm actually working on a product that in part addresses exactly this problem, using MVC as the front-end.

The comments so far have pointed out a lot of the short-comings. The one that talked about balanced scorecard even went so far as to try to talk about meaning of the counted things, as opposed to just how to count things.

Here's what I'm working on. Ask people what is important to them. Get real priorities (that's the magic sauce we got). Let them tell you which is more important, features, or stability, or standard's compliance, or extendibility, or whatever. Because if you don't know what's most important to people, you cannot really measure their satisfaction. I mean you can get a number, but not much insight. If everyone tells you they are really happy with the decor in your take-out pizza joint, that's probably not going to contribute as much to actual satisfaction as the quality of your toppings. And if you just asked about decor and toppings -- and then averaged the two to come up with one number. That would be almost as bad. And it's what most people do.

So, what our product does (beta coming out in March -- you can try it for free) is that it creates a simple matrix. You ask customers to prioritize the issues that are most important to them, and then tell you how well you are performing on those issues. Combined with demographic info on respondents, these answers can give you very deep insight into what's going on. We can even track info over time to see if there is a noticeable response/chance after a new release.

If you're interested, I'd be happy to share more. Just email me at my firstname@mylastname.com

Requesting Gravatar... Tarik Guney Feb 04, 2010 1:01 PM
# re: Can You Really Measure Customer Satisfaction?
Frankly, me and thousands of people like me don't like to fill out surveys as they are boring. The best way would be a integrated environment into products which collects data about how customers are using the specific products and on which parts they are mostly spending their times. I don't know it would be violating the privacy policies though. When people are working, if they allow, system can ask short questions about the part they are using and send them to the creator team along with the data collected during system use.

For example, lets say, a customer is using Visual Studio. The customer is writing codes within it, he/she probably uses the Intellisense. What if he doesn't like it ? He will probably disable this feature and here is the data you should collect. While he is disabling, system can ask a quick question about the reason and provides a little text box to allow him to write some suggestions. It would be easy and mostly unboring way instead of handing over 100 question surveys.
Requesting Gravatar... Dhananjay Goyani Feb 04, 2010 2:37 PM
# re: Can You Really Measure Customer Satisfaction?
I am not saying anything fundamentally different than what is been talked already in the comments. Apart from no. of downloads, blogs, connect, etc here are my two cents:

1) Google search results for '.NET MVC' or something. More results means more voice meaning more coverage. You can also present .NET MVC book stat, may be. I believe that people positive-negative expression ration is 80-20. On the net people want to share their experiences and specifically what went well (positive part - 80%) rather than what went poor (neg part - 20%). So, more google results (and increasing...) obviously means good thing. Say, how many people actively consuming Card Space and in turn how the product is performing. Compare google data over a period. (Obviously, you cant do that right away, its a continuous process...)

2) Stackoverflow. Yes, it can turn as powerful tool on your side. I know you (and your team) actively follow the threads and try to answer queries in as satisfying manner as you can. You can present your team scores (and also include non MS people but active in MVC community) as to show how much effort you put answering community - and its outcome.

Thanks!
Dhananjay
Requesting Gravatar... Mahyar Pasarzangene Feb 04, 2010 2:58 PM
# re: Can You Really Measure Customer Satisfaction?
Hi !
I think you can not even accomplish this task , i was someday in Razi university of Kermanshah Iran , and our computer science professor Dr.Montazeri always have problem with unmeasurable world of computer and he was saying because of this fact we should never call the "computer science" , the "computer engineering" !
Requesting Gravatar... Abdu Feb 05, 2010 8:08 AM
# re: Can You Really Measure Customer Satisfaction?
Measuring the number of downloads doesn't mean much. Most it can say is a lot of people are downloading it. A number has to be a percentage of another number or a rank to be meaningful. How many people who downloaded it are actually using it?

The amount of "flood" of incoming bugs is an indication. Feeling the general sentiment in the forums on the web is another. Are people in general complaining or happy and satisfied?

Are the common pain points people talk about attack narrow or wide aspects of your product?
Requesting Gravatar... Adam Ramshaw Feb 06, 2010 8:29 AM
# re: Can You Really Measure Customer Satisfaction?
Before you ask yourself how to measure customer satisfaction (and there are many ways to do it) you need to ask why you want to measure it. If you're thinking that customer satisfaction is a driver of product take up you're probably going to be disappointed. There is quite a lot of research to show that customer sat is not greatly aligned with product take-up or revenue growth (or anything really).

If you want to measure a variable that is more often aligned to growth then I suggest that you look towards Net Promoter Score. This has some good support.

How to measure it? I'd suggest sending an email to downloaders a few days after they had downloaded and ask them the NPS question: How likely would you be to recommend... to a friend or colleague" along with "Why do you say that". Just two questions. You could use Survey Monkey to do it if you like. Then do the NPS maths and look at the comments.

This is a pretty robust way to start down the path towards really understanding what is driving user loyalty. We use it a lot for our clients.
Requesting Gravatar... Visual C# Kicks Feb 08, 2010 8:38 AM
# re: Can You Really Measure Customer Satisfaction?
I like polls but they are a hassle since you need a very large number of people to get a realistic sample.
Requesting Gravatar... cttoy Feb 08, 2010 2:22 PM
# re: Can You Really Measure Customer Satisfaction?
The ASP.NET MVC is good, but the windows & database are expensive, I am satisfied with ASP.NET MVC, but it is not so easy to make a start business website.
So, when to make a linux/unix version of ASP.NET MVC? (mono project is slower than MS)
No limits, Just Use.
Requesting Gravatar... Glenn Feb 10, 2010 7:53 AM
# re: Can You Really Measure Customer Satisfaction?
Just like Adam, I too believe that NPS is the best way to gauge how customers view your product and willingness to act on feedback.
Requesting Gravatar... Nour Feb 13, 2010 8:42 PM
# re: Can You Really Measure Customer Satisfaction?
You may think to create a special custom feedback system for getting developers feedback. You may need to use third-party feedback systems like: Zodiac.NET to create some special kind of surveys.
Requesting Gravatar... SEOINKÖLN Mar 08, 2010 6:55 PM
# re: Can You Really Measure Customer Satisfaction?
intressanter beitrag
Requesting Gravatar... Jeremy Mar 17, 2010 5:53 AM
# re: Can You Really Measure Customer Satisfaction?
I would start by reading this book: www.amazon.com/.../188516730X/

This will show you what the right thing to measure is, and if I remember correctly give a couple of suggestions how to measure it.
Requesting Gravatar... Arvid de Sep 19, 2010 5:22 AM
# re: Can You Really Measure Customer Satisfaction?
@ Joshua, thank for your tip. Getsatisfaction.com is a great site, I think this is a good answer for this post.
Requesting Gravatar... Sean Carty Dec 29, 2011 3:00 AM
# re: Can You Really Measure Customer Satisfaction?
Like others have said, I think there are a bevy of ways to get some measurable customer satisfaction data, Including GetSatisfaction.com, and since this question has been asked a year ago a number of other great resources have blossomed. My own site, StepByStepMarketing.com, has its own section dedicated just to this topic at www.stepbystepmarketing.com/... We have highlighted a good number of high quality ideas and external services you can use. I know I am a little late for the original poster, but hopefully this will help others with similar questions that end up here.

What do you have to say?

(will show your gravatar)
Please add 5 and 2 and type the answer here: