Thursday, January 5, 2012
Four secrets of getting great participants who show up
What if you had a near-perfect participant show rate for all your studies? The first time it happens, it’s surprising. The next few times, it’s refreshing -- a relief. Teams that do great user research start with the recruiting process, and they come to expect near perfect attendance.
Secret 1: Participants are people, not data points
The people who opt in to a study have rich, complex lives that offer rich, complex experiences that a design may or may not fit into. People don’t always fit nicely into the boxes that screening questionnaires create.
Screeners can be constraining not in a good way. An agency that isn’t familiar with your design or your audience or both -- and may not be experienced with user research -- may eliminate people who could be great in user research or usability testing. Teams we work with find that participants who are selected through open-ended interviews conducted voice-to-voice become engaged and invested in the study. The conversation helps the participant know they’re interesting to you, and that makes them feel wanted. The team learns about variations in the user profile that they might want to design for.
Secret 2: Participants are in the network
Let’s say the source is panel or a database (versus a customer list). People who sign up to be in panels or recruiting databases tend to be people who take part in studies to make easy money. Many are the kind of people who fill out surveys to win prizes. These people might be good participants, or they might not.
Teams that find study participants through personal, professional, and community networks find that when the network snowball of connections works, people respond because they’re interested and have something to offer (or a problem you might solve for them with your design).
They also come partially pre-screened. Generally, your friends of friends of friends don’t want to embarrass the people who referred them. If the call for participants is clear and compelling, the community coordinator at the church, school, club, union, or team will remember to mention the study as soon as they encounter someone they know who might fit. Don’t worry: the connections soon get far enough away from you and your direct network that your data will be just as objective and clean as can be.
Secret 3: Participants want to help you
They want to be picked for your team. They want to share their experiences and demonstrate their expertise. When teams are open to the wide range of participants’ experiences, they learn from participants during screening. Those selected become engaged in the research. These are participants who call when they’re going to be late, or apologize for having to switch times. They want to work with you. One team we worked with had a participant call from a car accident before calling the police. (They rescheduled!)
Secret 4: Participants need attention
You know all the details that go into a study. Participants need confirmation and reminding. Teams that send detailed email confirmations get respectable show rates. Teams that send email confirmations, and then email reminders just before the sessions get good show rates. Teams that send email confirmations, email reminders, and then call the participants to remind them in a friendly, inviting tone get stellar show rates.
Some teams use the call before the session to start the “official” research. Rather than the recruiter doing the final call, the researcher phones to explain the study and the roles, and ask some of the warm up questions you might normally start a regular session with. These researchers establish a relationship with the participant. They also get a head start, leaving more time when they’re face-to-face with a participant to observe behavior rather than interview.
Perfect attendance is worth the effort
When all the scheduled participants show up, the gold stars come not only for efficient use of the time in the lab and keeping clients and team members eyes and ears with users. It’s likely the team ends up with better, more appropriate, more informative participants, overall. That means better, more reliable data to inform design decisions.
Tuesday, January 3, 2012
The true costs of no-shows
One of the first things people say when they call up looking for help with recruiting is that they want to recruit “12 for 8” or “20 for 15”. They know what they want to end up with. They’ve got to get data. Managers are showing up to observe. They’ve gone through a lot to get a study to happen at all. They don’t want to risk putting a study together only to get less data than they need. So, compensating for a show rate of between 60% and 80% means over-recruiting.
Even though a recruiting agency probably won’t charge for no-shows, those no-shows can be costly in lots of ways.
Stress. First, there’s the anguish of the moments before you know whether the participant will show. It’s not only the researcher who’s nervous. Nobody knows what to do. Time is ticking away. The session is running late late. Observers get antsy. The careful planning and scheduling of putting on a study feels like it is falling apart. That can be agonizing. When the participant arrives, it’s difficult to center yourself and get on with the session.
Embarrassment. There is nothing like the humiliation of having participants not show up after nudging, cajoling, and begging a team to show up to observe their users. The campaign of marketing user research disintegrates as the promise of enlightenment fades. It’s like promising fireworks but the fuses won’t light. The negative energy can be palpable and extremely difficult to recover from. Claiming that this is out of you control will not help the credibility of the study.
Loss of productivity. When you find out there is a longer-than-planned gap between sessions, conducting shadow puppet shows or showing cat videos to fill the time will not win the day. The researcher has got to get the team engaged in something productive that will help everyone feel like they’re still getting value out of being there. Without a participant to observe, it’s tough to keep the team from thinking you’re not wasting their time. That’s difficult to pull off for even the most experienced of researchers.
Loss of trust. Let’s not forget the pain of losing momentum as observations are interrupted. Say you got a few sessions done. Everyone is charged up with the revelations they’ve witnessed. Epiphany is everywhere. And then, a no-show. It can suck the air out of the room. Clients and teams can become disenchanted and impatient. They may lose confidence in the process and the results.
By comparison, the amount of money going for lab time when you’re not running sessions -- or paying for even more lab time and recruiting to make up sessions -- seems small. The damage of no-shows is long lasting. Yet, most of us treat no-shows as a cost of doing studies. They just happen. It’s the recruiting agency’s fault. We can’t control the participants.
What if you had a near-perfect participant show rate? The first time it happens, it’s surprising. The next few times, it’s refreshing -- a relief. Teams that do great user research start with the recruiting process, and they come to expect near perfect attendance. In the next post, I’ll share their secrets for getting great participants who show up.
Even though a recruiting agency probably won’t charge for no-shows, those no-shows can be costly in lots of ways.
Stress. First, there’s the anguish of the moments before you know whether the participant will show. It’s not only the researcher who’s nervous. Nobody knows what to do. Time is ticking away. The session is running late late. Observers get antsy. The careful planning and scheduling of putting on a study feels like it is falling apart. That can be agonizing. When the participant arrives, it’s difficult to center yourself and get on with the session.
Embarrassment. There is nothing like the humiliation of having participants not show up after nudging, cajoling, and begging a team to show up to observe their users. The campaign of marketing user research disintegrates as the promise of enlightenment fades. It’s like promising fireworks but the fuses won’t light. The negative energy can be palpable and extremely difficult to recover from. Claiming that this is out of you control will not help the credibility of the study.
Loss of productivity. When you find out there is a longer-than-planned gap between sessions, conducting shadow puppet shows or showing cat videos to fill the time will not win the day. The researcher has got to get the team engaged in something productive that will help everyone feel like they’re still getting value out of being there. Without a participant to observe, it’s tough to keep the team from thinking you’re not wasting their time. That’s difficult to pull off for even the most experienced of researchers.
Loss of trust. Let’s not forget the pain of losing momentum as observations are interrupted. Say you got a few sessions done. Everyone is charged up with the revelations they’ve witnessed. Epiphany is everywhere. And then, a no-show. It can suck the air out of the room. Clients and teams can become disenchanted and impatient. They may lose confidence in the process and the results.
By comparison, the amount of money going for lab time when you’re not running sessions -- or paying for even more lab time and recruiting to make up sessions -- seems small. The damage of no-shows is long lasting. Yet, most of us treat no-shows as a cost of doing studies. They just happen. It’s the recruiting agency’s fault. We can’t control the participants.
What if you had a near-perfect participant show rate? The first time it happens, it’s surprising. The next few times, it’s refreshing -- a relief. Teams that do great user research start with the recruiting process, and they come to expect near perfect attendance. In the next post, I’ll share their secrets for getting great participants who show up.
Monday, December 19, 2011
What's the best way to find people for user research and usability testing?
There are lots of great sources of participants for usability studies and other user research. The key: know what behavior you want to learn about. For example:
Note that there’s nothing about demographics here.
After you identify the behaviors you want to learn about -- preferably by observing people using a design rather than just asking them about it -- brainstorming ideas for where to find them can be fun. There are loads of options.
1. Let the snowball work its magic for you
Friends and family. You can invite friends and family to take part in your study, themselves, if they do the behavior you want to learn about. This is a great way to protect secrets because you can swear them to secrecy in a way that you can’t with outsiders, just by keeping them close. They’ll keep their commitment to you.
These folks can also be an excellent source of participants who you don’t know. That is, you can put the word out to your personal network, describe what you’re looking for, and let them do the work. If you can include some statement about the value of the user research you’re doing, people may be motivated to take part without other compensation.
User groups, community organizations, social clubs. Calling the heads of clubs and groups from churches or unions to chess clubs, charitable organizations, affinity groups, professional associations -- basically any group you can think of -- will often net you one super interested person who will pass the word. These networks work really well if your study has special relevance to the group, but many people may respond just because they’re curious.
Online social networks. Yep, you can put it out to Twitter, Facebook, Google+, or whatever you’re into at the moment to find people, too. The people who follow you in these spaces may not be the people you want as participants, but they know people who are appropriate.
We call this “snowball recruiting” because it puts the word out and that takes on a momentum of its own. Now, it doesn’t happen automatically. You will have to nudge people and put the call for participants out several times. For the clubs and groups, phoning (I know, that is terribly old fashioned) is going to get you way better response and results than sending an email to the group’s info@whatever.com email address.
2. Intercept people
This works in real life and online. In real life, it helps to approach people where they’re doing or would normally do the behavior, but you don’t have to do it that way. For example, if you want to learn about how people make decisions about what car to buy, you might want to hang out at a dealership (with permission, of course). I recently conducted a study on election ballot designs where the research teams approached people in libraries or at farmers markets. People were not voting in those places, but they were receptive to a smiling, charming person approaching them with a clipboard and a “Would you give us 15 minutes to learn about ballot design?” In about 2 hours, each team had observed 6 or 7 people using two different ballots.
If your design is online, an excellent tool to ask people nicely if they’d let you watch them while they use your website is Ethnio. It’s made by the brilliant people at Bolt | Peters, who use it constantly, themselves.
3. Craigslist
Craigslist gets its own category because its charms are different from the other options. On the plus side, the reach is great, the response will be huge in most cities, and the cost is little or nothing. On the minus side, there are some, shall we say, interesting people who use Craigslist. So, you’ll have to do some diligent filtering. Fortunately, the response is usually so large that you can be very discriminating. For example, at my company, we automatically delete any email that comes back without answering all of the questions we asked people to answer in our ad. Don’t put your phone number in the ad.
Getting people to find you
Two or three weeks before you want to do your sessions, put the word out or post your ad to Craigslist. Be sure to include:
Who you’re looking for. Make it something like, “We’re looking for people who play first-person shooter games,” or “We’re looking for people who are thinking about buying a new car in 2012,” or “Know someone who has had a triple bypass? We’d like to interview them.”
When you want to do the sessions. “We’re conducting 1-hour interviews in person January 20-23.”
What your prerequisite information is. Here, you might want occupation and location information, but you definitely want contact information.
How to contact you. By phone, email, Twitter, whatever.
Create a version you can fit into 140 characters, first. Add detail for the other channels you’re going to spread the word through. Don’t be afraid to change it up a bit as you learn what is attracting people and what questions they have.
What not to do
I have two other recommendations:
Don’t use a screener. List the behaviors and motivations you want to make sure that people have for your study for yourself, but don’t create multiple choice questions. Doing a screener -- which most people are very, very bad at -- is just an invitation for people to game their way into your study. Screeners eliminate edge people who might be really interesting for you to interview.
Instead, narrow down the candidates, and call them up. Spend time talking with them on the phone, asking open-ended questions. This accomplishes several awesome things. First, you get bonus user research. You may learn things about people that you hadn’t thought about before that will enrich your research and your designs. Second, it’s very unlikely you’ll end up with someone in the study who is not appropriate. Third, the candidate gets very engaged and is much more likely to show up for the sessions. If you hire an agency, they’re going to recruit 12 participants so you can end up with 8 good sessions. When we recruit this way we almost never have to replace anyone. Our “show rate” over 7 years is about 94%.
Don’t use an agency. Agencies have panels of people who have signed up to be in studies. These are the same people who answer surveys to get prizes. Do you want those people? Probably not. Agencies will have to use a screener. They will ask only the questions you include in your screener. They don’t know anything about your domain, or about how user research is done. They don’t care at all about what you want to end up knowing. All they care about is putting a body in a seat. Now, having said that, if you have an agency you know and like and have a relationship with, you can get excellent results. Most aren’t set up to help you in this way.
A word about biased samples
Most of us don’t have time or budget to do a truly random, representative sample. So, we use samples that are people who present themselves to us; user research is all about convenience samples. There is nothing wrong with this. You just have to be aware of the sources and what their biases might be. Using a combination of sources will even out the biases, and from there you should end up with reasonably reliable data.
This post originated as an answer to a question on Quora.com.
- Playing online games
- Voting
- Planning for retirement
- Shopping for a new car
- Treating a chronic illness
Note that there’s nothing about demographics here.
After you identify the behaviors you want to learn about -- preferably by observing people using a design rather than just asking them about it -- brainstorming ideas for where to find them can be fun. There are loads of options.
1. Let the snowball work its magic for you
Friends and family. You can invite friends and family to take part in your study, themselves, if they do the behavior you want to learn about. This is a great way to protect secrets because you can swear them to secrecy in a way that you can’t with outsiders, just by keeping them close. They’ll keep their commitment to you.
These folks can also be an excellent source of participants who you don’t know. That is, you can put the word out to your personal network, describe what you’re looking for, and let them do the work. If you can include some statement about the value of the user research you’re doing, people may be motivated to take part without other compensation.
User groups, community organizations, social clubs. Calling the heads of clubs and groups from churches or unions to chess clubs, charitable organizations, affinity groups, professional associations -- basically any group you can think of -- will often net you one super interested person who will pass the word. These networks work really well if your study has special relevance to the group, but many people may respond just because they’re curious.
Online social networks. Yep, you can put it out to Twitter, Facebook, Google+, or whatever you’re into at the moment to find people, too. The people who follow you in these spaces may not be the people you want as participants, but they know people who are appropriate.
We call this “snowball recruiting” because it puts the word out and that takes on a momentum of its own. Now, it doesn’t happen automatically. You will have to nudge people and put the call for participants out several times. For the clubs and groups, phoning (I know, that is terribly old fashioned) is going to get you way better response and results than sending an email to the group’s info@whatever.com email address.
2. Intercept people
This works in real life and online. In real life, it helps to approach people where they’re doing or would normally do the behavior, but you don’t have to do it that way. For example, if you want to learn about how people make decisions about what car to buy, you might want to hang out at a dealership (with permission, of course). I recently conducted a study on election ballot designs where the research teams approached people in libraries or at farmers markets. People were not voting in those places, but they were receptive to a smiling, charming person approaching them with a clipboard and a “Would you give us 15 minutes to learn about ballot design?” In about 2 hours, each team had observed 6 or 7 people using two different ballots.
If your design is online, an excellent tool to ask people nicely if they’d let you watch them while they use your website is Ethnio. It’s made by the brilliant people at Bolt | Peters, who use it constantly, themselves.
3. Craigslist
Craigslist gets its own category because its charms are different from the other options. On the plus side, the reach is great, the response will be huge in most cities, and the cost is little or nothing. On the minus side, there are some, shall we say, interesting people who use Craigslist. So, you’ll have to do some diligent filtering. Fortunately, the response is usually so large that you can be very discriminating. For example, at my company, we automatically delete any email that comes back without answering all of the questions we asked people to answer in our ad. Don’t put your phone number in the ad.
Getting people to find you
Two or three weeks before you want to do your sessions, put the word out or post your ad to Craigslist. Be sure to include:
Who you’re looking for. Make it something like, “We’re looking for people who play first-person shooter games,” or “We’re looking for people who are thinking about buying a new car in 2012,” or “Know someone who has had a triple bypass? We’d like to interview them.”
When you want to do the sessions. “We’re conducting 1-hour interviews in person January 20-23.”
What your prerequisite information is. Here, you might want occupation and location information, but you definitely want contact information.
How to contact you. By phone, email, Twitter, whatever.
Create a version you can fit into 140 characters, first. Add detail for the other channels you’re going to spread the word through. Don’t be afraid to change it up a bit as you learn what is attracting people and what questions they have.
What not to do
I have two other recommendations:
Don’t use a screener. List the behaviors and motivations you want to make sure that people have for your study for yourself, but don’t create multiple choice questions. Doing a screener -- which most people are very, very bad at -- is just an invitation for people to game their way into your study. Screeners eliminate edge people who might be really interesting for you to interview.
Instead, narrow down the candidates, and call them up. Spend time talking with them on the phone, asking open-ended questions. This accomplishes several awesome things. First, you get bonus user research. You may learn things about people that you hadn’t thought about before that will enrich your research and your designs. Second, it’s very unlikely you’ll end up with someone in the study who is not appropriate. Third, the candidate gets very engaged and is much more likely to show up for the sessions. If you hire an agency, they’re going to recruit 12 participants so you can end up with 8 good sessions. When we recruit this way we almost never have to replace anyone. Our “show rate” over 7 years is about 94%.
Don’t use an agency. Agencies have panels of people who have signed up to be in studies. These are the same people who answer surveys to get prizes. Do you want those people? Probably not. Agencies will have to use a screener. They will ask only the questions you include in your screener. They don’t know anything about your domain, or about how user research is done. They don’t care at all about what you want to end up knowing. All they care about is putting a body in a seat. Now, having said that, if you have an agency you know and like and have a relationship with, you can get excellent results. Most aren’t set up to help you in this way.
A word about biased samples
Most of us don’t have time or budget to do a truly random, representative sample. So, we use samples that are people who present themselves to us; user research is all about convenience samples. There is nothing wrong with this. You just have to be aware of the sources and what their biases might be. Using a combination of sources will even out the biases, and from there you should end up with reasonably reliable data.
This post originated as an answer to a question on Quora.com.
Tuesday, June 28, 2011
Usability testing is HOT
For many of us, usability testing is a necessary evil. For others, it’s too much work, or it’s too disruptive to the development process. As you might expect, I have issues with all that. It’s unfortunate that some teams don’t see the value in observing people use their designs. Done well, it can be an amazing event in the life of a design. Even done very informally, it can still show up useful insights that can help a team make informed design decisions. But I probably don’t have to tell you that.
Usability testing can be enormously elevating for teams at all stages of UX maturity. In fact, there probably isn’t nearly enough of it being done. Even on enlightened teams that know about and do usability tests, they’re probably not doing it often enough. There seems to be a correlation between successful user experiences and how often and how much the designers and developers spend time observing users. (hat tip Jared Spool)
Observing people using early designs can be energizing as designers and developers get a chance to see reactions to ideas. I’ve seen teams walk away with insights from observing people use their designs that they couldn’t have got any other way – and then make better designs than they’ve ever made. Close to launch, it is exciting – yes, exciting – to see a design perform as useful, usable, and desirable.
I’ve been negative on usability testing and our failure of imagination regarding bringing the method up to date, lately. But there’s a lot of good to any basic usability test. In fact, I went looking for the worth, the value, the alluring in usability testing a few weeks ago when I asked on Quora, “What’s the sexiest thing about usability testing?”
Some of the answers surprised me. Some of the answers were more about what people love about usability testing than what makes it seductive. But let’s go with seductive. People who find usability testing hot say it’s about data that can end the opinion wars, revelations and surprises, and getting perspective about real use, motivations, and context of use. Okay. We’re nerds.
The kiss of data
We always learn from users. Of course, we could just ask. But observing is so much more interesting. People do unpredictable things; they create workarounds, hacks, and alternative paths to make tools fit for their use.
This is the best case I can think of for watching rather than asking. From this observing, we get data. Juicy, luscious data like verbal protocols, task success rates, and physical behavior. This package makes it much easier to make good design decisions because we know have evidence on which to create theories about what should work better. There’s nothing like having hard evidence for going with a design direction – or changing direction.
Voyeuristic revelations
When designers, developers, and stakeholders of all persuasions get to observe people using a design – especially the first time – there’s often an “ah ha!” moment. (That’s the clean version.) Observers exclaim, “Wow, that was amazing!” when they see something surprising, both the good and the bad. The reaction that follows a completed usability study often is, “Damn. I wish we’d done this years ago. It would have saved us a ton of rework!” After watching one over-qualified participant struggle with a design recently, I heard a client say, “If that guy can’t do it, we’re in serious trouble.” That’s powerful.
When participants are surprised, that’s when the real fun begins. Not everyone likes surprises in their user interfaces, especially if they’re not the delightful Easter egg kind. While a team hopes not to hear, “I feel lost and abandoned,” you’ve got to wonder how bad it’s been when a participant squeals, “Oh, my gosh! This is so much better! When can I have it?!” Those eureka moments can reveal what to do to improve a design or an experience.
Relationship dynamics
One of the magical things about observing users working with a design is that suddenly, disputes within the team melt away. Chances are, the disputing parties were both wrong because neither (unless they have a ton of experience already observing these kinds of users in this domain doing this task) could accurately predict how the user would behave and perform.
Now, even with observations from watching just one user, there’s data on which to base design decisions. Data trumps gut. Data outweighs feelings. Data can put to rest those endless, circular discussions where inevitably, the person with the biggest paycheck or the most important title wins. The opinion wars come to an end.
When the whole team is involved in deciding what to test and observing sessions, everyone can share in making and carrying out agreed design decisions. Whenever a question comes up where no one knows but everyone has an opinion, the answer in a team doing usability testing is, “Let’s do some user research on that,” or “Let’s find out what users do.”
For the love of users
It’s so easy to get caught up in the business goals and issues with the underlying technology of a design. It’s so easy to stay in the safe bubble of the office, cranking out code, designs, plans, and reports. It’s easy to lose touch with users.
Teams that spend a couple of hours observing their users every few weeks keep that connection. They fall in love with their users. They relish the chance to see for themselves why people do the things they do with designs.
Getting out of their own heads, a successful team uses usability testing to get perspective, learn about users’ contexts, and remember the people and their stories. For these teams, usability testing is inspiring. And that’s hot.
What’s sexy about usability testing?
Observing people use a design can be revelatory. It turns up the volume on design by helping teams make informed design decisions. What’s sexy about usability testing? Data for evidence-based design. Ending opinion wars. Knowing users from observations and surprises. Getting perspective and knowledge of context of use.
The UX equivalent of a romantic dinner or a walk on the beach? Perhaps not, even for a geek girl like me. But it can be exciting, fun, funny, encouraging, and empowering. Just what you want from a relationship. That’s pretty seductive, if you ask me.
Monday, March 21, 2011
Oh, Etsy. How could you?
During the last holiday season, I called Land's End. I hardly ever call; I'm a huge fan of their online experience. I wanted to send a special order to my mother, putting two matching things in the same gift box. Landsend.com isn't really set up to do that, but the site instructed me that I could do it, so I called.
The sales rep was friendly and efficient, and very helpful. She pulled up my order-in-progress and put everything in the box that I wanted to be in my mother's gift. When she asked me if there was anything else she could help me with, I blithely said, "You could check my mother's account and tell me what she's sending me for Christmas." The sales rep giggled, teased me a little bit by telling me that she could see my mother's account, but told me I would have to wait until UPS delivered it to find out. She protected the relationship between the seller and the buyer. She also protected the relationship between two buyers – me and my mother -- at least for that episode. If I wanted to find out what I was getting for Christmas, I'd either have to wheedle it out of my mother or wait.
Respect and research. That's all I ask.
Facebook had a go at Beacon, a service that broadcasted out to all your friends the purchases you've made outside of Facebook, without permission. The Federal Trade Commission sued, and Facebook eventually settled and took down the "service" in 2009.
Facebook has a history of screwing with the privacy of its users. Beacon was a prime example. The main problem here is the lack of permission. And that's the case for Etsy's new People Search, too.
The designers of Etsy decided it was a good idea to make everyone on Etsy searchable by name, including buyers. So, if you have ever bought anything on Etsy, you can now be found there by anyone else either by your real name or your username. Your whole profile is viewable, including your purchase history. Not only that, it'll all show up in Google search results.
The idea is that buyers would form social "circles" on the site to share information about their purchases.
Uninformed by research, guided by gut
These are the kinds of things that happen when an organization puts business goals before customer goals. It's also the kind of thing that can happen when an executive wakes up one day and says, We want to be one of the cool kids. And right now, to be one of the cool kids, you have to have social media. How do we do that?
What made Etsy think it needed a social layer on its beautiful, engaging site? It's the kind of thing that happens when teams decide to strap social on rather than looking at the conversation they're already having with customers and that customers are already having with one another.
I'm sure a lot of thought went into this decision of Etsy's. I fear this is a vacuum-sealed decision. Here's my imagined scenario, a scenario I've seen played out in other, similar decisions at other, similar organizations: Management, who forgets that their site is not the center of the universe for anyone outside that room, went to the product manager and asked for some of the social awesomesauce that is out there to turn up the buzz a notch. The product manager brainstormed with the team. The best idea they could come up with is to get customers to talk about the fun, beautiful, interesting stuff they'd bought on Etsy online with one another. (Never mind that we already have Twitter for this.)
Where is making use of the conversation that Etsy is already having with its customers or that buyers and sellers are already having together? They probably can't make use of these conversations because they haven't observed them. Where's the research to support this design decision?
Why this is on my usability testing blog
It's hard for me to believe that if Etsy had conducted user research and even informal but realistic usability testing on the idea that they would not have quickly seen the privacy violation. They could have avoided the damage control they now have to deal with because of the breach of trust they've had with buyers who already love the experience of shopping there.
How Etsy could have avoided the problem and discovered a possibly great idea for engaging buyers even more
1. Analyze the risks of a social media strategy to users' privacy, security, and trust. Where was the business plan for allowing search of users? How does having social "circles" support the business model, exactly? How would the social media strategy be supported on the back end? More than all that, let's look at others who have gone before us: Beaon on Facebook and Boden USA come to mind. What happened there? What could the Etsy team learn from those mistakes? Oh, and, why duplicate Facebook in any way?
2. Proof the concept with real people who shop on Etsy. This is pure conjecture based on my experiences with other organizations: Etsy may have thought that to up their game and get people more engaged in the site, they needed to get buyers talking with one another and not just to sellers. Charming idea. But how do you find out if people find that useful?
Focus groups? If there were focus groups, I'm just going to guess here that participants liked the idea, but there was no exploration of the implications of this profile information being public rather than private. Not ideal.
What else could they have done? Invited friends and family. This approach still perhaps is not optimum, because friendly participants might not have exposed the privacy problems. They are, after all, friends and family, so there's automatic trust and wanted connections there already. How about rolling it out to a very small number of key buyers -- 3 or 5 -- and watch what happens for a week or a month as they connect to their people, or until something bad and unintended happens?
3. Conduct usability testing with real people in real contexts to learn the ripples to real relationships. Let's say they did usability testing. Did they bring in real buyers to use a working prototype with their own data? Did it occur to anyone that now my ex can Google me (like he does) and find out that I bought my sister a Star Wars crochet pattern, or my current paramour a hand made can coozie? Or what about the fact that my clients could see all the personal things on my Etsy wish list?
A usability test with a limited "circle" on a closed sandbox (like a walled-off development or testing server) for a couple of weeks might have given them some clues about what might work and what might not.
Etsy, I love you, but I have to go now
Not only will Etsy have to clean up its own site by making the social opt-in, but they'll also have to figure out a way to recover buyers' privacy. How does a web organization reclaim data that is now not in its control? If they could invent a big Web eraser to drag behind them as they invite buyers back to the site, they might have a chance.
Sources and resources:
http://arstechnica.com/web/news/2011/03/etsy-users-irked-after-buyers-purchases-exposed-to-the-world.ars
http://forums.penny-arcade.com/showthread.php?t=138661
Monday, February 28, 2011
Involving older adults in design of the user experience: Inclusive design
Despite the reality of differences due to aging, research has also shown that in many cases, we do not need a separate design for people who are age 50+. We need better design for everyone.
Everyone performs better on web sites where the interaction matches users' goals; where navigation and information are grouped well; where navigation elements are consistent and follow conventions; where writing is clear, straightforward, in the active voice, and so on. And, much of what makes up good design for younger people helps older adults as well.
For example, we know that most users, regardless of age, are more successful finding information in broad, shallow information architectures than they are with deep, narrow hierarchies. When web sites make their sites easier to use for older adults, all of their users perform better in usability studies. The key is involving older adults in user research and usability testing throughout design and development.
There are some important considerations in working with older adults in studies. Remembering the points below will ensure that you and your participants have a good experience and you get the data you need to inform design decisions.
Finding participants: Understanding older adults before you recruit
In many places in the world, older adults outnumber people in other age groups. The question is, how do you find the right people to take part in studies? They can be difficult to get to.
We found that these approaches did not work well:
These ideas did work well:
Recruiting older adults
Recruiting participants in their 70s and 80s is more difficult than recruiting participants in their 50s. The oldest candidates are less receptive to strangers phoning them, and they don't check email as frequently.
Recruiting by phone. Phoning is important. Plan to phone potential participants at least once (or have your recruiters do so). You need to quickly establish credibility and trustworthiness, to assure potential participants that you are not selling anything, and to establish a connection by letting them know where you got their names. When you can do that, potential participants are often glad to hear from a real person. It is also easier for them to determine legitimacy and to ask questions about the study on the phone. They'll use your answers to help them decide whether they want to take part.
Another reason for phoning potential participants is so you can judge their English language skills and whether they are hearing impaired. (You may well want to include limited English speakers and hearing impaired users in your study; if you do, you want to be aware of these specifics about the participants before they come.)
Recruiting by email. Email can be very efficient for younger participants; it's less so with adults in their later 60s and 70s and older. Give yourself more time for these older participants; they generally don't check their email more than a couple of times per week. This happens for a variety of reasons: They don't feel the need to check mail frequently. They use a computer at a senior centre or a library. They have limited time available through their Internet service provider.
As suspicious as older adults are of telemarketers, they are also vigilant about spam. If your email address is unknown to them, without an appropriately descriptive subject line, they may delete it. Always put on a very clear subject line.
Screening older adults
Screening older adults demands specificity. Many older users when asked the question "what do you do online?" answer "email." They often don't think about practical activities such as banking or bill paying online as "using the web."
Many older users are also not as familiar with the language of the web as younger users are. They don't distinguish between the Internet and the web. They don't always know the difference between the web browser and the web page.
We found that self-reported data about frequency of use and numbers of hours spent online were not good indicators of proficiency, either. For example, we had one participant who spent 60 hours per week online. We didn't find out until the session started that her sole use of the web was playing games on four web sites that her friend had set up as separate shortcuts on her desktop.
So, asking a variety of specific questions to gauge potential participants' familiarity with the web can help the person recruiting participants for a study make judgments about how suited the person might be for the study. Even if you're looking for a mix of proficiency levels, you still have to be able to determine where in the range a potential participant fits.
Tech savviness matrices
An assessment we’ve found to work well asks about frequency of use over a broad range of types of interactions older people can take part in on the web. Here are some example assessment grids we have used.
Scheduling sessions with older adults
Scheduling sessions with older participants can present some logistical challenges that you might not think about in studies involving younger participants.
They arrive early. Because many older people are retired (or at least have ample free time), they almost always arrive for their sessions early – up to an hour early. Be sure to have someone to greet them and set up a comfortable place for them to wait.
They bring their spouses. Older participants often bring their spouses or a friend with them. They may have travelled some distance to get to the session; they may have planned activities for after their session; or they simply may not like driving alone. Have magazines, a phone, and a comfortable chair available for the spouse or friend.
They do best in the morning. Even though people in their later 60s, 70s, and 80s are vital and energetic, they usually have more—and better—attention to give earlier in the day. Try to schedule people who are in their late 60s, 70s and 80s in the morning and save any afternoon sessions for participants in their 50s or early 60s. We don't recommend running evening sessions.
They don't like driving in rush hour. If you are holding sessions in a central place (rather than meeting participants in their homes or workplaces), schedule the sessions outside of peak traffic times, if possible.
Reminding older adults of important points before they come
Reminders about one-to-one sessions. Participants can become nervous and uncomfortable if they realize after arriving that they will be the only participant in the session. Usability studies are still fairly new to the general population. Recruiting firms often recruit for focus groups, and participants who come through these firms often assume that they are coming to a focus group.
Reminders about videotaping and observations. Although a good practice is to ask for permission to record and to have people observing the sessions when you recruit, people tend to forget that. Make sure that the person who calls the participant to confirm the session also tells the participant that:
Special reminders for older adults.
Working with older adults during sessions
Many older participants won't know what to expect coming into a usability study session. Be clear in setting their expectations and be firm but polite about keeping the session focused on what you're trying to find out.
Make participants comfortable. Be respectful without being patronizing. You can be a neutral moderator but still be polite. "Please" and "thank you" are important. Many older adults expect more statements of politeness like these than younger participants do.
Older participants also deserve extra consideration, politeness, and detailed information about the session. They will feel more comfortable if they know what to expect up front:
Keep them on track tactfully. Most of the participants we've had in sessions are interesting, charming, and very talkative. Many older participants have a lot of stories to tell. Their stories say a lot about who they are, and where they have been— and often provide a context for interpreting data.
But it may be easy for participants to get off track during the session, and while it may feel awkward or mean sometimes, it is the moderator's job to keep the participant focused on the task, talking about it, and getting data for the study. This is the main reason for warning participants in the introduction to the session that you may interrupt them and that you may stop tasks before they've completed them.
Listen for their beliefs about computers and the web. Many people who are in their late 60s and older never used computers at work. This means they have no previous experience from which to make inferences about how a computer or an application might work. Many learn how to use computers and the Internet through friends, family, and neighbours. They inherit the superstitions and myths that those people have developed to help themselves work around problems. Then the older adults bring these myths into sessions with them, and you'll hear about them as task-solving strategies and workarounds. It's important to capture these; they are part of the users' reality and we have to deal with these beliefs when we design web sites.
Be careful of the words you use; avoid computer jargon. Older computer users rarely know much about computer-related terminology, so you should avoid using these terms during your sessions. Older participants often don't know the names of widgets such as drop-down boxes or cascading menus. Most of our participants also had little knowledge of web-related terminology. For example, they weren't sure about terms such as "link," "URL," and "login." Many were unclear about the meanings of "online community" and "message boards." "Browsing" wasn't always meaningful in the context of a feature called "browse by topic." The word "emoticon" and the concept behind it were completely foreign to most of our participants. This means that you must pay close attention to what participants do and point at on the screen or device.
Give them time. Older participants almost always take longer to do tasks than younger participants. And, although they seem to struggle, the oldest participants also expect using the computer to be difficult. Plan for tasks to take much longer for older participants than they would for younger participants—up to 25 percent longer in our experience.
Help participants understand the time constraints of the session by explaining the session format in your introduction. Also, wait longer to prompt than you normally might. You might also consider giving participants permission to ask for hints when you introduce the session.
If necessary, hold the think-aloud and ask participants to reflect later. A classic technique in usability testing is to ask participants to think aloud while they work through tasks toward a goal. When tasks become complex or difficult, participants may stop talking. Use your best judgment about nudging them to tell you what they're thinking. For some participants with short-term memory loss or other cognitive impairment (such as that caused by pain medication), your asking for their thoughts may interrupt their task enough that it causes them to make errors. In those cases, you may get more usable data without the think aloud protocol by asking participants to reflect later.
Don't lead even when you want to. If, as a session moderator, you have a soft spot in your heart at all for participants, working with older participants will exercise that spot a lot. You may be tempted to give hints; worse, you may lead them in ways you don't intend. Be patient and firm but polite while keeping to your agenda.
When appropriate, teach something at the end. If the session has been difficult for the participant, or, if there is some small thing that would make using the computer or the web easier, take a little time at the end of the session to teach the participant something. For example, show participants how to change the text size in their browsers and shortcuts for copying and pasting and printing.
Including older adults in user research and usability studies: Older, wiser, wired
Older adults don’t behave differently from younger people online. Just thinking about old age as a collection of disabilities is old business. The new world of designing for older adults is about creating web sites and other technology that is useful and desirable as well as accessible to the broadest range of users. Older adults as a cohort are living longer than their parents because they’re healthier, and many will be affluent because they’ve been saving up for a lifetime – this means they have time, money, and motivation to be online.
Neither a monolithic view of older adults nor an entirely separate design for older adults is necessary. Younger designers developing web sites for older adults need to learn more about older adults’ life experiences. For example, many older adults don’t perceive themselves as old. And so, all technology design – not just designs for older adults – should involve users.
Everyone performs better on web sites where the interaction matches users' goals; where navigation and information are grouped well; where navigation elements are consistent and follow conventions; where writing is clear, straightforward, in the active voice, and so on. And, much of what makes up good design for younger people helps older adults as well.
For example, we know that most users, regardless of age, are more successful finding information in broad, shallow information architectures than they are with deep, narrow hierarchies. When web sites make their sites easier to use for older adults, all of their users perform better in usability studies. The key is involving older adults in user research and usability testing throughout design and development.
There are some important considerations in working with older adults in studies. Remembering the points below will ensure that you and your participants have a good experience and you get the data you need to inform design decisions.
Finding participants: Understanding older adults before you recruit
In many places in the world, older adults outnumber people in other age groups. The question is, how do you find the right people to take part in studies? They can be difficult to get to.
We found that these approaches did not work well:
- Community web sites, message boards, or chat sessions. The oldest old tend not to take part in these groups, so posting ads in those places is not a fruitful way to find participants.
- Senior centers and community colleges. These are places that offer classes in using computers. If you want computer and web novices for a study, they might be good places to find appropriate participants. They are not good sources if you want to observe people with enough web experience to see them working at a web site without teaching them.
- Flyers at a senior center, when they did not make clear that we were recruiting for a study. Many older people are much more cautious and skeptical than younger people. They are often fearful of being cheated or "taken." For example, we had put up flyers at a senior center from which we got no response; later we learned that people thought we might be trying to sell them something.
- Cold calling from a database. This is probably again because older people are afraid that they may be scammed into buying something.
These ideas did work well:
- Calling with a personal connection. If we could say that a mutual acquaintance had suggested the contact, potential participants were much more receptive to hearing about the study and considering taking part. It is important to establish credibility and trust with the potential participant.
- Being careful in the initial call to say where we had gotten the contact information and that we weren't selling anything.
Recruiting older adults
Recruiting participants in their 70s and 80s is more difficult than recruiting participants in their 50s. The oldest candidates are less receptive to strangers phoning them, and they don't check email as frequently.
Recruiting by phone. Phoning is important. Plan to phone potential participants at least once (or have your recruiters do so). You need to quickly establish credibility and trustworthiness, to assure potential participants that you are not selling anything, and to establish a connection by letting them know where you got their names. When you can do that, potential participants are often glad to hear from a real person. It is also easier for them to determine legitimacy and to ask questions about the study on the phone. They'll use your answers to help them decide whether they want to take part.
Another reason for phoning potential participants is so you can judge their English language skills and whether they are hearing impaired. (You may well want to include limited English speakers and hearing impaired users in your study; if you do, you want to be aware of these specifics about the participants before they come.)
Recruiting by email. Email can be very efficient for younger participants; it's less so with adults in their later 60s and 70s and older. Give yourself more time for these older participants; they generally don't check their email more than a couple of times per week. This happens for a variety of reasons: They don't feel the need to check mail frequently. They use a computer at a senior centre or a library. They have limited time available through their Internet service provider.
As suspicious as older adults are of telemarketers, they are also vigilant about spam. If your email address is unknown to them, without an appropriately descriptive subject line, they may delete it. Always put on a very clear subject line.
Screening older adults
Screening older adults demands specificity. Many older users when asked the question "what do you do online?" answer "email." They often don't think about practical activities such as banking or bill paying online as "using the web."
Many older users are also not as familiar with the language of the web as younger users are. They don't distinguish between the Internet and the web. They don't always know the difference between the web browser and the web page.
We found that self-reported data about frequency of use and numbers of hours spent online were not good indicators of proficiency, either. For example, we had one participant who spent 60 hours per week online. We didn't find out until the session started that her sole use of the web was playing games on four web sites that her friend had set up as separate shortcuts on her desktop.
So, asking a variety of specific questions to gauge potential participants' familiarity with the web can help the person recruiting participants for a study make judgments about how suited the person might be for the study. Even if you're looking for a mix of proficiency levels, you still have to be able to determine where in the range a potential participant fits.
Tech savviness matrices
An assessment we’ve found to work well asks about frequency of use over a broad range of types of interactions older people can take part in on the web. Here are some example assessment grids we have used.
Scheduling sessions with older adults
Scheduling sessions with older participants can present some logistical challenges that you might not think about in studies involving younger participants.
They arrive early. Because many older people are retired (or at least have ample free time), they almost always arrive for their sessions early – up to an hour early. Be sure to have someone to greet them and set up a comfortable place for them to wait.
They bring their spouses. Older participants often bring their spouses or a friend with them. They may have travelled some distance to get to the session; they may have planned activities for after their session; or they simply may not like driving alone. Have magazines, a phone, and a comfortable chair available for the spouse or friend.
They do best in the morning. Even though people in their later 60s, 70s, and 80s are vital and energetic, they usually have more—and better—attention to give earlier in the day. Try to schedule people who are in their late 60s, 70s and 80s in the morning and save any afternoon sessions for participants in their 50s or early 60s. We don't recommend running evening sessions.
They don't like driving in rush hour. If you are holding sessions in a central place (rather than meeting participants in their homes or workplaces), schedule the sessions outside of peak traffic times, if possible.
Reminding older adults of important points before they come
Reminders about one-to-one sessions. Participants can become nervous and uncomfortable if they realize after arriving that they will be the only participant in the session. Usability studies are still fairly new to the general population. Recruiting firms often recruit for focus groups, and participants who come through these firms often assume that they are coming to a focus group.
Reminders about videotaping and observations. Although a good practice is to ask for permission to record and to have people observing the sessions when you recruit, people tend to forget that. Make sure that the person who calls the participant to confirm the session also tells the participant that:
- "You will be videotaped and observed by people you won't be able to see during the session."
- "This is a one-on-one session. You will be the only participant in the study room with a moderator."
Special reminders for older adults.
- Computer glasses. Many participants will have special glasses for using the computer. So another important reminder is, "Don't forget your computer glasses!"
- Eat first. Also, for long sessions— anything longer than 45 minutes— ask participants to make sure they eat before they arrive. Because many participants expect to take part in focus groups rather than individual sessions, they also expect to be fed. If you have snacks available, try to have fruit and nuts or other relatively healthy food. Many older participants are diabetic.
Working with older adults during sessions
Many older participants won't know what to expect coming into a usability study session. Be clear in setting their expectations and be firm but polite about keeping the session focused on what you're trying to find out.
Make participants comfortable. Be respectful without being patronizing. You can be a neutral moderator but still be polite. "Please" and "thank you" are important. Many older adults expect more statements of politeness like these than younger participants do.
Older participants also deserve extra consideration, politeness, and detailed information about the session. They will feel more comfortable if they know what to expect up front:
- Clearly explain the session plan, timing, and what they can expect.
- Warn participants that you'll interrupt them and that you may stop them before they have completed tasks.
- Schedule breaks for long sessions (and tell them they can take breaks whenever they need to).
- Have them practice thinking aloud.
- Consider including a practice task to help participants understand how the session will work.
- Take account of beliefs that participants may have learned or created about how to work with computers.
- Remember that older participants often are not versed in computer and web terminology, so avoid using this jargon when working with them.
- Be extra patient with older participants; wait longer than you normally might to prompt; consider giving participants permission to ask for hints.
- If participants stop talking, consider letting them continue that way; try reflecting on the task later.
- Teach participants something at the end of the session.
Keep them on track tactfully. Most of the participants we've had in sessions are interesting, charming, and very talkative. Many older participants have a lot of stories to tell. Their stories say a lot about who they are, and where they have been— and often provide a context for interpreting data.
But it may be easy for participants to get off track during the session, and while it may feel awkward or mean sometimes, it is the moderator's job to keep the participant focused on the task, talking about it, and getting data for the study. This is the main reason for warning participants in the introduction to the session that you may interrupt them and that you may stop tasks before they've completed them.
Listen for their beliefs about computers and the web. Many people who are in their late 60s and older never used computers at work. This means they have no previous experience from which to make inferences about how a computer or an application might work. Many learn how to use computers and the Internet through friends, family, and neighbours. They inherit the superstitions and myths that those people have developed to help themselves work around problems. Then the older adults bring these myths into sessions with them, and you'll hear about them as task-solving strategies and workarounds. It's important to capture these; they are part of the users' reality and we have to deal with these beliefs when we design web sites.
Be careful of the words you use; avoid computer jargon. Older computer users rarely know much about computer-related terminology, so you should avoid using these terms during your sessions. Older participants often don't know the names of widgets such as drop-down boxes or cascading menus. Most of our participants also had little knowledge of web-related terminology. For example, they weren't sure about terms such as "link," "URL," and "login." Many were unclear about the meanings of "online community" and "message boards." "Browsing" wasn't always meaningful in the context of a feature called "browse by topic." The word "emoticon" and the concept behind it were completely foreign to most of our participants. This means that you must pay close attention to what participants do and point at on the screen or device.
Give them time. Older participants almost always take longer to do tasks than younger participants. And, although they seem to struggle, the oldest participants also expect using the computer to be difficult. Plan for tasks to take much longer for older participants than they would for younger participants—up to 25 percent longer in our experience.
Help participants understand the time constraints of the session by explaining the session format in your introduction. Also, wait longer to prompt than you normally might. You might also consider giving participants permission to ask for hints when you introduce the session.
If necessary, hold the think-aloud and ask participants to reflect later. A classic technique in usability testing is to ask participants to think aloud while they work through tasks toward a goal. When tasks become complex or difficult, participants may stop talking. Use your best judgment about nudging them to tell you what they're thinking. For some participants with short-term memory loss or other cognitive impairment (such as that caused by pain medication), your asking for their thoughts may interrupt their task enough that it causes them to make errors. In those cases, you may get more usable data without the think aloud protocol by asking participants to reflect later.
Don't lead even when you want to. If, as a session moderator, you have a soft spot in your heart at all for participants, working with older participants will exercise that spot a lot. You may be tempted to give hints; worse, you may lead them in ways you don't intend. Be patient and firm but polite while keeping to your agenda.
When appropriate, teach something at the end. If the session has been difficult for the participant, or, if there is some small thing that would make using the computer or the web easier, take a little time at the end of the session to teach the participant something. For example, show participants how to change the text size in their browsers and shortcuts for copying and pasting and printing.
Including older adults in user research and usability studies: Older, wiser, wired
Older adults don’t behave differently from younger people online. Just thinking about old age as a collection of disabilities is old business. The new world of designing for older adults is about creating web sites and other technology that is useful and desirable as well as accessible to the broadest range of users. Older adults as a cohort are living longer than their parents because they’re healthier, and many will be affluent because they’ve been saving up for a lifetime – this means they have time, money, and motivation to be online.
Neither a monolithic view of older adults nor an entirely separate design for older adults is necessary. Younger designers developing web sites for older adults need to learn more about older adults’ life experiences. For example, many older adults don’t perceive themselves as old. And so, all technology design – not just designs for older adults – should involve users.
Monday, February 7, 2011
Bonus research: Do the recruiting yourself
There are some brilliant questions on Quora. This morning, I was prompted to answer one about recruiting.
The question asker asked, How do I recruit prospective customers to shadow as a part of a user-centered design approach? The asker expanded, thusly:
The question asker asked, How do I recruit prospective customers to shadow as a part of a user-centered design approach? The asker expanded, thusly:
I'm interested in shadowing prospective customers in order to better understand how my tool can fit into their life and complement, supplement, or replace the existing tools that they use. How do I find prospective customers? How do I convince them to let me shadow them?Seemed like a very thoughtful question. I have some experience with recruiting for field studies and other user research, so I thought I might share my lessons learned. Here's my answer. Would love to hear yours.
I've learned many recruiting lessons the hard way. Recruiting can be challenging, but it can also be really interesting. I always find that I learn a lot in the recruiting process. Here are my pointers:
Do the recruiting yourself. This gives you bonus research data about your people, and you may learn things you hadn't anticipated that will influence how you conduct the study. It also starts a relationship with people that gets them invested in taking part. It's the start of a conversation with you and your organization. They're more likely to trust your motives and give you a deeper, richer view of their lives.
Focus on behavior, not demographics. If you want people to use your tool to do a particular thing, look for people who already do that somehow. For example, if you want people to use your design to store their photos and music and other content, find people who have a lot of that type of content and who are concerned about losing it. If you want people to use your design to generate invoices, find people who are doing that now and observe what they do to generate invoice when they're in the mode of doing it. If you want people to use your design to remember to take their medicines at the right time in the right dosage, find people who have persistent conditions that need medication and who have been diagnosed with the type of condition you want to help them deal with.
You'll see that I never once mentioned age, sex, income, location, education level, marital status - or any of those things that marketers go on. Because it doesn't usually matter. What matters for UX design is behavior. Do people do the thing you want to make a design for?
Be as generous as you can with incentive/honorarium/compensation. People love gifts. They also love cash. The point is to show your appreciation for their help. They're doing you a huge favor by spending time with you and letting you see their lives. Offer something your kind of people want.
Be flexible about days, times, number of hours. If you want to shadow people, you have to do it when they're in the mode of doing the things you want to observe. My experience is that most interesting behavior (unless it is truly work related) doesn't happen between 8am and 5pm Monday through Friday.
Get participants to opt in to the study. Sourcing for a field study is one of the most challenging steps to recruiting. A carefully worded ad on Craigslist can attract the right kind of people. Going through community organizations or professional associations can work well.
We practice something we call "snowball recruiting." That means putting the word out among friends, family, and colleagues about the type of people you're looking for and what you want to do in the session and asking your close ties to help you find people. Advantage: Participants come lightly pre-screened, so you know they're not nuts. and they're likely to be reliable because someone you know sent them. Snowball recruiting will also help establish you as legit with potential participants. The reputation management goes both ways. Disadvantage: This approach can take a little longer to generate leads. For a shadowing or field study, it can take 3-4 weeks to get people to come to you, screen them, and then schedule them to do the session you ultimately want to do.
You can recruit from Craigslist in a much shorter period, because the response is usually great, depending on the geographic area you're recruiting in. However, there's more filtering and more screening to be done.
Remember that every sample has biases. UX research is based on convenience samples rather than random or representative samples. That's fine because you're focusing on behavior and performance rather than generalized opinions and preferences. But the sources do influence the expertise and world view that people have. For example, if you're observing people's behavior around online security and you recruit just from Craigslist, you may find that most of the people who respond are not very concerned with protecting their online accounts. Or you might find the opposite.
Be genuinely, authentically interested in the people you want to observe without being creepy. These people are letting you into their lives. So, explain what you do and what you're hoping to learn from the study. Treat them as your partner in answering your research questions. After all, you can't find out what you want to know without them.
Follow up. Send a hand-written thank-you note. Yes, this is in addition to the incentive.
I've written a lot about recruiting for user research and usability tests. You can see all those articles here: http://usabilitytestinghowto.blogspot.com/search/label/recruiting
Do the recruiting yourself. This gives you bonus research data about your people, and you may learn things you hadn't anticipated that will influence how you conduct the study. It also starts a relationship with people that gets them invested in taking part. It's the start of a conversation with you and your organization. They're more likely to trust your motives and give you a deeper, richer view of their lives.
Focus on behavior, not demographics. If you want people to use your tool to do a particular thing, look for people who already do that somehow. For example, if you want people to use your design to store their photos and music and other content, find people who have a lot of that type of content and who are concerned about losing it. If you want people to use your design to generate invoices, find people who are doing that now and observe what they do to generate invoice when they're in the mode of doing it. If you want people to use your design to remember to take their medicines at the right time in the right dosage, find people who have persistent conditions that need medication and who have been diagnosed with the type of condition you want to help them deal with.
You'll see that I never once mentioned age, sex, income, location, education level, marital status - or any of those things that marketers go on. Because it doesn't usually matter. What matters for UX design is behavior. Do people do the thing you want to make a design for?
Be as generous as you can with incentive/honorarium/compensation. People love gifts. They also love cash. The point is to show your appreciation for their help. They're doing you a huge favor by spending time with you and letting you see their lives. Offer something your kind of people want.
Be flexible about days, times, number of hours. If you want to shadow people, you have to do it when they're in the mode of doing the things you want to observe. My experience is that most interesting behavior (unless it is truly work related) doesn't happen between 8am and 5pm Monday through Friday.
Get participants to opt in to the study. Sourcing for a field study is one of the most challenging steps to recruiting. A carefully worded ad on Craigslist can attract the right kind of people. Going through community organizations or professional associations can work well.
We practice something we call "snowball recruiting." That means putting the word out among friends, family, and colleagues about the type of people you're looking for and what you want to do in the session and asking your close ties to help you find people. Advantage: Participants come lightly pre-screened, so you know they're not nuts. and they're likely to be reliable because someone you know sent them. Snowball recruiting will also help establish you as legit with potential participants. The reputation management goes both ways. Disadvantage: This approach can take a little longer to generate leads. For a shadowing or field study, it can take 3-4 weeks to get people to come to you, screen them, and then schedule them to do the session you ultimately want to do.
You can recruit from Craigslist in a much shorter period, because the response is usually great, depending on the geographic area you're recruiting in. However, there's more filtering and more screening to be done.
Remember that every sample has biases. UX research is based on convenience samples rather than random or representative samples. That's fine because you're focusing on behavior and performance rather than generalized opinions and preferences. But the sources do influence the expertise and world view that people have. For example, if you're observing people's behavior around online security and you recruit just from Craigslist, you may find that most of the people who respond are not very concerned with protecting their online accounts. Or you might find the opposite.
Be genuinely, authentically interested in the people you want to observe without being creepy. These people are letting you into their lives. So, explain what you do and what you're hoping to learn from the study. Treat them as your partner in answering your research questions. After all, you can't find out what you want to know without them.
Follow up. Send a hand-written thank-you note. Yes, this is in addition to the incentive.
I've written a lot about recruiting for user research and usability tests. You can see all those articles here: http://usabilitytestinghowto.blogspot.com/search/label/recruiting
Subscribe to:
Posts (Atom)