Monday, December 19, 2011

What's the best way to find people for user research and usability testing?

There are lots of great sources of participants for usability studies and other user research. The key: know what behavior you want to learn about. For example:
  • Playing online games
  • Voting
  • Planning for retirement
  • Shopping for a new car
  • Treating a chronic illness

Note that there’s nothing about demographics here.

After you identify the behaviors you want to learn about -- preferably by observing people using a design rather than just asking them about it -- brainstorming ideas for where to find them can be fun. There are loads of options.

1. Let the snowball work its magic for you

Friends and family. You can invite friends and family to take part in your study, themselves, if they do the behavior you want to learn about. This is a great way to protect secrets because you can swear them to secrecy in a way that you can’t with outsiders, just by keeping them close. They’ll keep their commitment to you.

These folks can also be an excellent source of participants who you don’t know. That is, you can put the word out to your personal network, describe what you’re looking for, and let them do the work. If you can include some statement about the value of the user research you’re doing, people may be motivated to take part without other compensation.

User groups, community organizations, social clubs. Calling the heads of clubs and groups from churches or unions to chess clubs, charitable organizations, affinity groups, professional associations -- basically any group you can think of -- will often net you one super interested person who will pass the word. These networks work really well if your study has special relevance to the group, but many people may respond just because they’re curious.

Online social networks. Yep, you can put it out to Twitter, Facebook, Google+, or whatever you’re into at the moment to find people, too. The people who follow you in these spaces may not be the people you want as participants, but they know people who are appropriate.

We call this “snowball recruiting” because it puts the word out and that takes on a momentum of its own. Now, it doesn’t happen automatically. You will have to nudge people and put the call for participants out several times. For the clubs and groups, phoning (I know, that is terribly old fashioned) is going to get you way better response and results than sending an email to the group’s email address.

2. Intercept people

This works in real life and online. In real life, it helps to approach people where they’re doing or would normally do the behavior, but you don’t have to do it that way. For example, if you want to learn about how people make decisions about what car to buy, you might want to hang out at a dealership (with permission, of course). I recently conducted a study on election ballot designs where the research teams approached people in libraries or at farmers markets. People were not voting in those places, but they were receptive to a smiling, charming person approaching them with a clipboard and a “Would you give us 15 minutes to learn about ballot design?” In about 2 hours, each team had observed 6 or 7 people using two different ballots.

If your design is online, an excellent tool to ask people nicely if they’d let you watch them while they use your website is Ethnio. It’s made by the brilliant people at Bolt | Peters, who use it constantly, themselves.

3. Craigslist

Craigslist gets its own category because its charms are different from the other options. On the plus side, the reach is great, the response will be huge in most cities, and the cost is little or nothing. On the minus side, there are some, shall we say, interesting people who use Craigslist. So, you’ll have to do some diligent filtering. Fortunately, the response is usually so large that you can be very discriminating. For example, at my company, we automatically delete any email that comes back without answering all of the questions we asked people to answer in our ad. Don’t put your phone number in the ad.

Getting people to find you 

Two or three weeks before you want to do your sessions, put the word out or post your ad to Craigslist. Be sure to include:

Who you’re looking for. Make it something like, “We’re looking for people who play first-person shooter games,” or “We’re looking for people who are thinking about buying a new car in 2012,” or “Know someone who has had a triple bypass? We’d like to interview them.”

When you want to do the sessions. “We’re conducting 1-hour interviews in person January 20-23.”

What your prerequisite information is. Here, you might want occupation and location information, but you definitely want contact information.

How to contact you. By phone, email, Twitter, whatever.

Create a version you can fit into 140 characters, first. Add detail for the other channels you’re going to spread the word through. Don’t be afraid to change it up a bit as you learn what is attracting people and what questions they have.

What not to do

I have two other recommendations:

Don’t use a screener. List the behaviors and motivations you want to make sure that people have for your study for yourself, but don’t create multiple choice questions. Doing a screener -- which most people are very, very bad at -- is just an invitation for people to game their way into your study. Screeners eliminate edge people who might be really interesting for you to interview.

Instead, narrow down the candidates, and call them up. Spend time talking with them on the phone, asking open-ended questions. This accomplishes several awesome things. First, you get bonus user research. You may learn things about people that you hadn’t thought about before that will enrich your research and your designs. Second, it’s very unlikely you’ll end up with someone in the study who is not appropriate. Third, the candidate gets very engaged and is much more likely to show up for the sessions. If you hire an agency, they’re going to recruit 12 participants so you can end up with 8 good sessions. When we recruit this way we almost never have to replace anyone. Our “show rate” over 7 years is about 94%.

Don’t use an agency. Agencies have panels of people who have signed up to be in studies. These are the same people who answer surveys to get prizes. Do you want those people? Probably not. Agencies will have to use a screener. They will ask only the questions you include in your screener. They don’t know anything about your domain, or about how user research is done. They don’t care at all about what you want to end up knowing. All they care about is putting a body in a seat. Now, having said that, if you have an agency you know and like and have a relationship with, you can get excellent results. Most aren’t set up to help you in this way.

A word about biased samples

Most of us don’t have time or budget to do a truly random, representative sample. So, we use samples that are people who present themselves to us; user research is all about convenience samples. There is nothing wrong with this. You just have to be aware of the sources and what their biases might be. Using a combination of sources will even out the biases, and from there you should end up with reasonably reliable data.

This post originated as an answer to a question on


  1. Interesting perspective. While the hands-on, personal attention for each participant is appealing, sometimes it's just not feasible for high volume work, and would be difficult to standardize.

    I hear your objections to using agencies and screeners, but we've found that crafting careful screeners and working with trusted agencies as partners also yields great results (our show rate is around 95% as well).

    It *is* possible to find smart, articulate, motivated people with a screener, and the process is more efficient and repeatable. That said, it's always great to have a variety of methods at your disposal.

  2. I agree with Kathi: interesting perspective.

    I have a lot of trouble with the "let the snowball work its magic for you" approach though, perhaps because of my training as a statistician! Seems pretty certain to me that any snowball-related sample is going to be biased *because* of the association with you, the researcher. Friends, family, colleagues, social networks...all are going to attract people who are more like you than not, even a degree or two removed.

    I also think demographics can serve a useful purpose for ensuring bias is minimised. For example, if your target market is "general public", then getting a cross-section of income, education, employment status etc, that matches the population, is going to help ensure representativeness. That's where agencies — with high quality procedures and methods for weeding out expert participants — can really add value.

    Thanks for the opportunity to comment.

  3. Jessica,

    Thanks for your comments. You seem to assume that most user research and usability testing needs representative samples. But as I point out, most of this type of research *can't* be. How would you do a test big enough to have a representative sample for Gmail? Not possible. Not really desirable, even.

    All that said, I'll take a sample from a snowball recruit -- remember, they're 2 or 3 steps removed from me, I'm not going to have someone in a study who I know -- over a group of undergrads any day. :)

    Keep up the good work!