Tuesday, February 16, 2010

You are not your user. No matter how good you think you are.

Listen up, people. This is why -

- quantity is not quality
- you are not your user.

The lesson for today on participant sampling is Google Buzz. Google has been working on Buzz for some time. And it's a cool idea. Integrating the sharing of photos, status updates, conversations, and email is a thing a lot of us have been looking for. Buzz makes lots of automatic connections. That's what integrating applications means.

BUT. One of the features of Buzz was that it would automatically connect you to people whom you have emailed in Gmail. On the surface, a great idea. A slick idea, which worked really well with 20,000 Google employees.

Large samples do not always generate quality data
Twenty thousand. Feedback from 20,000 people is a lot of data. How many of us would kill to have access to 20,000 people? So. How can such a large sample be bad? Large samples can definitely generate excellent data on which to make superfine design decisions. Amazon and Netflix use very large samples for very specialized tests. There's discussion everywhere, including at the recent Interaction10 conference in Savannah, about cheap methods for doing remote, unmoderated usability testing with thousands of people. More data seems like a good idea.

If you have access to 20,000 people and you can handle the amount of data that could come out of well designed research from that sample, go for it. But it has to be the right sample.

Look outside yourself (and your company)
Google employees are special. They're very carefully selected by the company. They have skills, abilities, and lives that are very different from most people outside Google. So, there's the bias of being selected to be a Googler. And then there's indoctrination as you assimilate into the corporate culture. It's a rarified environment.

But Google isn't special in this way. Every organization selects its employees carefully. Every organization has a culture that new people undergo indoctrination and assimilation for, or they leave. In aggregate, the people in an organization begin to behave similarly and think similarly. They aspire to the same things, like wanting products to work.

But what about 37 Signals and/or Apple? They don't do testing at all. (We don't actually know this for sure. They may not call it testing.) They design for themselves and their products are very successful in the marketplace. I think that those companies do know a lot about their customers. They've observed. They've studied. And, over time, they do adjust their designs (look at the difference in interaction design in the iPod from first release in 2001 to now). Apple has also had its failures (Newton, anyone?).

The control thing
By not using an outside sample, Google ran into a major interaction design problem. About as big as it gets. This is a control issue, not a privacy issue, though the complaints were about over sharing. One of the cardinal rules of interaction design is to always let the user feel she's in control. By taking control of users' data, Buzz invaded users' privacy. That's the unfortunate outcome in this case, and now, users will trust Google less. It's difficult to regain trust. But I digress.

The moral of today's lesson: Real users always surprise us
Google miscalculated when it assumed that everyone you email is someone you want to share things with, and that you might want those people connected to one another. In a work setting, this might be true. In a closed community like a corporation, this might be true. But the outside world is much messier.

For example, I have an ex. He emails me. Sometimes, I even email him back. But I don't want to share things with him anymore. We're not really friends. I don't want to connect him to my new family.

Even testing with friends and family might have exposed the problem. Google has a Trusted Tester program. Though there are probably some biases in that sample because of the association with Google employees, they are not Google employees. This makes friends and family who use Gmail one step closer to typical users. But Google didn't use Trusted Testers for Buzz.

You get to choose your friends in real life. Google could have seen this usage pattern pretty quickly just by testing with a small sample who live beyond the Google garden walls.


  1. Thanks for such a great blog post. I think this is the wake up call that many of us need to take notice of. So much detail gets polished over so often, and QA and Testing are the last positions filled in companies, and the first elements squidged when budgets get squeezed. I see it so often :(

  2. So true (as usual). I printed your post to take with me to Starbucks to read over my morning coffee (just on the basis of the heading), and on my way heard a discussion on WBUR of Googles misstep which left me wondering "How could they not have tested it? How could you make such an obvious goof like that, which probably squandered hundreds of millions of dollars in future advertising value?" (Something they only had one shot at getting right....) And then I read you piece, which nailed it. Thanks!

  3. Thanks, guys.

    Why does this happen - not just to Google, but to organizations of all types, sizes, and maturity levels?

    I have a few thoughts about that. There is, of course the budget thing. 'We don't have a lot of money, so we're going to just grab people inside the company who aren't involved in the project.'

    There's the pulling rank thing. 'Yeah, I don't care what the data says, I've got a hunch about this.' Which brings us to the lack of trust (respect?) of the data or the person who gathered it.

    And then there's the possibility that the test methods just weren't that good and there's this market window looming, and it's just too much to resist.

    Most teams I talk to who ask me about using internal people are in the first situation. There just isn't a lot of money, and the theory is that some testing is better than no testing. I believe that's true. I really believe that's true. And I think that testing with internal users is a fine place to start: to practice a test protocol before using it with external users. I think that testing with internal users is fine if you're using a few, and you're mixing internal with external over time.

    Usability testing is about getting data to inform design decisions. That means getting out of the bubble you're in to get good data to design from.

  4. I think that you are right in general that regardless of how well you think you know the users you need outside judgement. However, in the case of Google Buzz, I believe that this was not an accidental mistake, in fact fairly sure that it was done this way on purpose, to expose the largest number of people to Google Buzz automatically.

  5. I am absolutely in agreement with you and Google of all places should have known better. In other corporate cultures, particularly where user experience is new and "different" (perceived as scary, strange, etc), there are often legal restrictions on speaking to real or potential customers. Large corporations fear lawsuits and shy away from any potential risk associated with observing customers test new products.

    I'll be passing this post around at work to inspire a fearless environment where users are a real part of the process.

  6. Thanks for the wonderful case study, Dana.

    What I find interesting about the Google Buzz case is the particulars of the problem they missed. Testing 20,000 Google employees probably did a lot to identify usability breakdowns, interaction wonkiness, and missing features, but missed the undesirability of a social feature in an app that is social at its core.

    Depending on the context, the feature could be exciting or frustrating. For my work email account I can see myself wanting to automatically exchange status updates with my contacts; for my personal email account, not so much.

    One of the most important aspects of any usability test is defining what you want to learn from the test. When you've done that you can decide what kind of test and population are right for the project.

    I still think that there are cases when internal usability testing can be helpful, but what we see in this case is that it's not sufficient.

  7. Great post, Dana! Love the breakdown of the problem that Buzz encountered, it's comforting (and scary) that the biggest players in the business can still get it so wrong.

    I feel that "You are not your user" is one of the most thrown around but least understood principles of UX. I expanded on the dangers of not getting the reasoning behind it on my blog, let me know what you think:

  8. Mattias, thanks for your comment and for your blog piece. You add some really useful stuff to the conversation.