Involving older adults in design of the user experience: Inclusive design

Despite the reality of differences due to aging, research has also shown that in many cases, we do not need a separate design for people who are age 50+. We need better design for everyone.

Everyone performs better on web sites where the interaction matches users' goals; where navigation and information are grouped well; where navigation elements are consistent and follow conventions; where writing is clear, straightforward, in the active voice, and so on. And, much of what makes up good design for younger people helps older adults as well.

For example, we know that most users, regardless of age, are more successful finding information in broad, shallow information architectures than they are with deep, narrow hierarchies. When web sites make their sites easier to use for older adults, all of their users perform better in usability studies. The key is involving older adults in user research and usability testing throughout design and development.

There are some important considerations in working with older adults in studies. Remembering the points below will ensure that you and your participants have a good experience and you get the data you need to inform design decisions.


Finding participants: Understanding older adults before you recruit

In many places in the world, older adults outnumber people in other age groups. The question is, how do you find the right people to take part in studies? They can be difficult to get to.

We found that these approaches did not work well:

• Community web sites, message boards, or chat sessions. The oldest old tend not to take part in these groups, so posting ads in those places is not a fruitful way to find participants.
  
• Senior centers and community colleges. These are places that offer classes in using computers. If you want computer and web novices for a study, they might be good places to find appropriate participants. They are not good sources if you want to observe people with enough web experience to see them working at a web site without teaching them.
  
• Flyers at a senior center, when they did not make clear that we were recruiting for a study. Many older people are much more cautious and skeptical than younger people. They are often fearful of being cheated or "taken." For example, we had put up flyers at a senior center from which we got no response; later we learned that people thought we might be trying to sell them something.
  
• Cold calling from a database. This is probably again because older people are afraid that they may be scammed into buying something.

These ideas did work well:

• Calling with a personal connection. If we could say that a mutual acquaintance had suggested the contact, potential participants were much more receptive to hearing about the study and considering taking part. It is important to establish credibility and trust with the potential participant.
  
• Being careful in the initial call to say where we had gotten the contact information and that we weren't selling anything.

Recruiting older adults

Recruiting participants in their 70s and 80s is more difficult than recruiting participants in their 50s. The oldest candidates are less receptive to strangers phoning them, and they don't check email as frequently.

Recruiting by phone. Phoning is important. Plan to phone potential participants at least once (or have your recruiters do so). You need to quickly establish credibility and trustworthiness, to assure potential participants that you are not selling anything, and to establish a connection by letting them know where you got their names. When you can do that, potential participants are often glad to hear from a real person. It is also easier for them to determine legitimacy and to ask questions about the study on the phone. They'll use your answers to help them decide whether they want to take part.

Another reason for phoning potential participants is so you can judge their English language skills and whether they are hearing impaired. (You may well want to include limited English speakers and hearing impaired users in your study; if you do, you want to be aware of these specifics about the participants before they come.)

Recruiting by email. Email can be very efficient for younger participants; it's less so with adults in their later 60s and 70s and older. Give yourself more time for these older participants; they generally don't check their email more than a couple of times per week. This happens for a variety of reasons: They don't feel the need to check mail frequently. They use a computer at a senior centre or a library. They have limited time available through their Internet service provider.

As suspicious as older adults are of telemarketers, they are also vigilant about spam. If your email address is unknown to them, without an appropriately descriptive subject line, they may delete it. Always put on a very clear subject line.



Screening older adults

Screening older adults demands specificity. Many older users when asked the question "what do you do online?" answer "email." They often don't think about practical activities such as banking or bill paying online as "using the web."

Many older users are also not as familiar with the language of the web as younger users are. They don't distinguish between the Internet and the web. They don't always know the difference between the web browser and the web page.

We found that self-reported data about frequency of use and numbers of hours spent online were not good indicators of proficiency, either. For example, we had one participant who spent 60 hours per week online. We didn't find out until the session started that her sole use of the web was playing games on four web sites that her friend had set up as separate shortcuts on her desktop.

So, asking a variety of specific questions to gauge potential participants' familiarity with the web can help the person recruiting participants for a study make judgments about how suited the person might be for the study. Even if you're looking for a mix of proficiency levels, you still have to be able to determine where in the range a potential participant fits.



Tech savviness matrices
An assessment we’ve found to work well asks about frequency of use over a broad range of types of interactions older people can take part in on the web. Here are some example assessment grids we have used.

Scheduling sessions with older adults

Scheduling sessions with older participants can present some logistical challenges that you might not think about in studies involving younger participants.

They arrive early. Because many older people are retired (or at least have ample free time), they almost always arrive for their sessions early – up to an hour early. Be sure to have someone to greet them and set up a comfortable place for them to wait.

They bring their spouses. Older participants often bring their spouses or a friend with them. They may have travelled some distance to get to the session; they may have planned activities for after their session; or they simply may not like driving alone. Have magazines, a phone, and a comfortable chair available for the spouse or friend.

They do best in the morning. Even though people in their later 60s, 70s, and 80s are vital and energetic, they usually have more—and better—attention to give earlier in the day. Try to schedule people who are in their late 60s, 70s and 80s in the morning and save any afternoon sessions for participants in their 50s or early 60s. We don't recommend running evening sessions.

They don't like driving in rush hour. If you are holding sessions in a central place (rather than meeting participants in their homes or workplaces), schedule the sessions outside of peak traffic times, if possible.


Reminding older adults of important points before they come

Reminders about one-to-one sessions. Participants can become nervous and uncomfortable if they realize after arriving that they will be the only participant in the session. Usability studies are still fairly new to the general population. Recruiting firms often recruit for focus groups, and participants who come through these firms often assume that they are coming to a focus group.

Reminders about videotaping and observations. Although a good practice is to ask for permission to record and to have people observing the sessions when you recruit, people tend to forget that. Make sure that the person who calls the participant to confirm the session also tells the participant that:

• "You will be videotaped and observed by people you won't be able to see during the session."
  
• "This is a one-on-one session. You will be the only participant in the study room with a moderator."

Special reminders for older adults.
 

•  Computer glasses. Many participants will have special glasses for using the computer. So another important reminder is, "Don't forget your computer glasses!"
  
• Eat first. Also, for long sessions— anything longer than 45 minutes— ask participants to make sure they eat before they arrive. Because many participants expect to take part in focus groups rather than individual sessions, they also expect to be fed. If you have snacks available, try to have fruit and nuts or other relatively healthy food. Many older participants are diabetic.

Working with older adults during sessions

Many older participants won't know what to expect coming into a usability study session. Be clear in setting their expectations and be firm but polite about keeping the session focused on what you're trying to find out.

Make participants comfortable. Be respectful without being patronizing. You can be a neutral moderator but still be polite. "Please" and "thank you" are important. Many older adults expect more statements of politeness like these than younger participants do.

Older participants also deserve extra consideration, politeness, and detailed information about the session. They will feel more comfortable if they know what to expect up front:

• Clearly explain the session plan, timing, and what they can expect.
• Warn participants that you'll interrupt them and that you may stop them before they have completed tasks.
• Schedule breaks for long sessions (and tell them they can take breaks whenever they need to).
• Have them practice thinking aloud.
• Consider including a practice task to help participants understand how the session will work.
• Take account of beliefs that participants may have learned or created about how to work with computers.
• Remember that older participants often are not versed in computer and web terminology, so avoid using this jargon when working with them.
• Be extra patient with older participants; wait longer than you normally might to prompt; consider giving participants permission to ask for hints.
•  If participants stop talking, consider letting them continue that way; try reflecting on the task later.
• Teach participants something at the end of the session.

Keep them on track tactfully. Most of the participants we've had in sessions are interesting, charming, and very talkative. Many older participants have a lot of stories to tell. Their stories say a lot about who they are, and where they have been— and often provide a context for interpreting data.

But it may be easy for participants to get off track during the session, and while it may feel awkward or mean sometimes, it is the moderator's job to keep the participant focused on the task, talking about it, and getting data for the study. This is the main reason for warning participants in the introduction to the session that you may interrupt them and that you may stop tasks before they've completed them.

Listen for their beliefs about computers and the web. Many people who are in their late 60s and older never used computers at work. This means they have no previous experience from which to make inferences about how a computer or an application might work. Many learn how to use computers and the Internet through friends, family, and neighbours. They inherit the superstitions and myths that those people have developed to help themselves work around problems. Then the older adults bring these myths into sessions with them, and you'll hear about them as task-solving strategies and workarounds. It's important to capture these; they are part of the users' reality and we have to deal with these beliefs when we design web sites.

Be careful of the words you use; avoid computer jargon. Older computer users rarely know much about computer-related terminology, so you should avoid using these terms during your sessions. Older participants often don't know the names of widgets such as drop-down boxes or cascading menus. Most of our participants also had little knowledge of web-related terminology. For example, they weren't sure about terms such as "link," "URL," and "login." Many were unclear about the meanings of "online community" and "message boards." "Browsing" wasn't always meaningful in the context of a feature called "browse by topic." The word "emoticon" and the concept behind it were completely foreign to most of our participants. This means that you must pay close attention to what participants do and point at on the screen or device.

Give them time. Older participants almost always take longer to do tasks than younger participants. And, although they seem to struggle, the oldest participants also expect using the computer to be difficult. Plan for tasks to take much longer for older participants than they would for younger participants—up to 25 percent longer in our experience.

Help participants understand the time constraints of the session by explaining the session format in your introduction. Also, wait longer to prompt than you normally might. You might also consider giving participants permission to ask for hints when you introduce the session.

If necessary, hold the think-aloud and ask participants to reflect later. A classic technique in usability testing is to ask participants to think aloud while they work through tasks toward a goal. When tasks become complex or difficult, participants may stop talking. Use your best judgment about nudging them to tell you what they're thinking. For some participants with short-term memory loss or other cognitive impairment (such as that caused by pain medication), your asking for their thoughts may interrupt their task enough that it causes them to make errors. In those cases, you may get more usable data without the think aloud protocol by asking participants to reflect later.

Don't lead even when you want to. If, as a session moderator, you have a soft spot in your heart at all for participants, working with older participants will exercise that spot a lot. You may be tempted to give hints; worse, you may lead them in ways you don't intend. Be patient and firm but polite while keeping to your agenda.

When appropriate, teach something at the end. If the session has been difficult for the participant, or, if there is some small thing that would make using the computer or the web easier, take a little time at the end of the session to teach the participant something. For example, show participants how to change the text size in their browsers and shortcuts for copying and pasting and printing.



Including older adults in user research and usability studies: Older, wiser, wired

Older adults don’t behave differently from younger people online. Just thinking about old age as a collection of disabilities is old business. The new world of designing for older adults is about creating web sites and other technology that is useful and desirable as well as accessible to the broadest range of users. Older adults as a cohort are living longer than their parents because they’re healthier, and many will be affluent because they’ve been saving up for a lifetime – this means they have time, money, and motivation to be online.

Neither a monolithic view of older adults nor an entirely separate design for older adults is necessary. Younger designers developing web sites for older adults need to learn more about older adults’ life experiences. For example, many older adults don’t perceive themselves as old. And so, all technology design – not just designs for older adults – should involve users.

Researcher as director: scripts and stage direction

For most teams, the moderator of user research sessions is the main researcher. Depending on the comfort level of the team, the moderator might be a different person from session to session in the same study. (I often will moderate the first few sessions of a study and then hand the moderating over to the first person on the design team who feels ready to take over.)

To make that work, it's a good practice to create some kind of checklist for the sessions, just to make sure that the team's priorities are addressed. For a field study or a formative usability test, a checklist might be all a team needs. But if the team is working on sussing out nuanced behaviors or solving subtle problems, we might want a bit more structure.

A couple of the teams I work with ensure that everything is lined up and that *anyone* on the team could conduct the sessions by creating detailed scripts that include stage direction. Here are a couple of samples:

Whether the team is switching up moderators or it's the same person conducting all the sessions, creating a script for the session that includes logistics is a good idea:

• think through all the logistics, ideally, together with the team
• make sure the sessions are conducted consistently, from one to the next
• back up the main researcher in case something drastic happens -- someone else could easily fill in

Logistics rehearsal

When you walk through, step by step, what's supposed to happen during a session, it helps everyone visualize the steps, pacing, and who should be doing what. My client teams use the stage direction in the script as a check to make sure everything is being covered to reach the objectives of the sessions. It's also a good way to review what tools, data, and props you might need.

Estimate timing

Teams often ask me about timing. When they get through a draft of a script that includes stage directions, they get a pretty solid feeling pretty quickly for what is going to take how long. From this they can assign timing estimates and make decisions about whether they want participants to keep going on a task after the estimated time is reached or redirect to the next task.

Mapping out location flow

It's easy to overlook the physical or geographic flow - what a director would call blocking - of a session. Where does the participant start the session? In a waiting room, at her desk, or somewhere else? Will you change locations within a room or building during the session? How do you get from one place to the next?

Consistency and rigor

Including stage directions in a script for a user research session can help reviewer-stakeholders understand what to expect. More importantly, the stage directions act as reminders to the moderator so she's doing the same things with and saying the same things to every participant in the study. This means nothing gets left out deliberately and nothing gets added that wasn't agreed on ahead of time. (For example, the team could identify some area to observe for and put a prompt in the script for the moderator to ask follow-up questions that are not specifically scripted, depending on what the participant does.)

Insurance

Any really good project manager is going to have a Plan B. With a script that includes detailed stage directions, anyone who has been involved in the planning of a study should be able to pick up the script and moderate a session. The people I worked with at Tec-Ed called this "the bus test" (as in, If you get hit by a bus we still have to do this work).

Some teams I work with want to spread out and run simultaneous sessions. The stage directions can help ensure consistency across moderators. (Rehearse and refine if you're really going to do this.)

Finally, when it comes time to write the report about the insights the team gained, the script -- with its stage directions -- can help answer the questions that often come asking why things were done the way they were done or why the data says what it says.

Stage it

Each person in a session is an actor, whether participant or observer. The moderator is the director. If the script for a study includes instructions for all the actors in the session as well as the director in addition to documenting what words to say, everyone involved will give a great performance.

Overcoming fear of moderating UX research sessions

It always happens: Someone asks me about screwing up as an amateur facilitator/moderator for user research and usability testing sessions. This time, I had just given a pep talk to a bunch of user experience professionals about sharing responsibility with the whole team for doing research. "But what if the (amateur) designer does a bad job of moderating the session?"

What not to do

There are numerous ways in which a moderator can foul things up. Here are just a few possibilities that might render the data gathered useless:

• Leading the participant
• Interrupting or intervening at the wrong time
• Teaching or training rather than observing and listening
• Not following a script or checklist
• Arguing with the participant

Rolf Molich and Chauncey Wilson put together an extensive list of the many wrong things moderators could do. There are dozens of behaviors on the list. I have committed many of these sins myself at some point. It's embarrassing, but it is not the end of the world. So, here, let's talk about what to do to be the best possible moderator in your first session.

Your role as a moderator 

To be the best  moderator you can be, remember that there are three basic roles of the moderator in user research and usability testing. When Carolyn Snyder worked for User Interface Engineering, she codified these:

Flight attendant. Though you might think that your priority is collecting data, your number one job during the session is to see to the comfort and safety of the participant. Make sure this person is comfortable, is appreciated, and knows she can stop at any time. Set up a relaxed situation that is still focused on the goal of learning from the person.

Sportscaster. The line of sight and the acoustics of the session situation aren't always ideal for the observers. Because the observers from your team will be helping you take notes and analyze the data, you can help them by talking just enough so they can keep their places in the session. For example, if the participant is vague about a UI element in pointing out goods and bads, simply echo the last couple of words the participant said to get them to clarify or expand.

Scientist. The moderator is usually the person who designed the study and will be responsible for analyzing the data that comes out of it. This means managing any recordings to ensure the privacy of the participants, tracking notes and data gathering from observers, and pulling observations and data together so the team can come to a design direction based on the evidence gathered.

(Hat tip to Carolyn Snyder and Jared Spool for the moderator roles.)

Who should moderate UX sessions?

Who makes a good moderator? Anyone who is a quick learner, is a good listener, can build rapport with a participant, and has a good memory. Typically, there isn't a lot of time to know all the nuances of a UI before going into a usability test. Likewise, if you're in the field doing basic ethnographic research, you may learn characteristics of the participants or the environment that inform the rest of the interview direction. Handling those on-the-fly perceptions will help everyone get value out of the session.

The listening is important for asking insightful follow-up questions as participants think aloud. Getting clarification on comments made, drilling in a bit to get to specifics, and always keeping in mind "why is this behavior happening" will come to you from listening and (gently) questioning.

Rapport with the participant is key to creating trust. The participant is always trying to get a reading from you about whether what he's doing is correct and whether what he's giving you is what you want. Even a newbie to moderating can be friendly, objective, and neutral at the same time. (It may take some practice.)

Remembering what happened early in the session will help you ask useful follow up questions later in the session. Remembering the main, interesting behaviors will help you work with the observers after the session is over and you're all telling the story of what happened, especially if you have assigned someone else to take notes while you concentrate on running the session. 

If there's someone on the design team besides you who has these characteristics, that's who you want to moderate sessions, no matter what their regular job is.

How to be a great UX moderator

Keeping those roles and attributes in mind, this is what I tell clients and workshop attendees about how to be a good moderator. You can pass the list below to your team's candidate.

Be willing to let go of ownership of the design. When you're doing field research, you may enter the session with design ideas in mind. Try not to. Instead, let the heft of the data over sessions help build the ideas. If you're testing a design, as soon as you put a design in front of another person, you no longer own it, the person you're showing it to or who is using it owns it. In that act, you have specifically asked for reactions and interactions. Open yourself up to the possibilities.

Shut up. After you've explained the purpose of the session, and explained your role, and the roles of the other people in the room, stop talking. Even when there are silences, don't be too quick to fill them. Wait. Count to 20 slowly and silently before you say anything. Chances are, something interesting has happened by then and you won't have to open your mouth.

Listen. This is not the same as shutting up. Listening is about being present. Be fully attentive so you can not only hear the words, but process their meaning. Be empathetic to the participant and what she's trying to do. 

Suspend judgment. This is one of the hardest things to do, but it is also the most important. You have invited the participant to help you learn about what you're designing. If you have shut up, and listened well, and the participant is appropriate for the study, then let go of assessing what is happening in that moment. Give yourself time to process later. This will also prevent you from asking inappropriate questions during the session that may betray your feelings about a participant or what she has to say about the design.

Plan ahead. Script, create checklists, and read Beth Loring and Joe Dumas's book, Moderating Usability Tests. You may feel silly using a script, but really, there's absolutely nothing wrong with doing that. Having a script to follow means you say the same things the same way to every participant. You can also ensure that you've hit everything on the list that the team wanted to learn about. Finally, scripting out what to say and thinking through the checklist of focus questions will give everyone a better feeling for how much can reasonably be covered during a given session.

Rehearse. If you don't set up a pilot session, then your first "real" session will end up being the rehearsal. Practice in a dry run by yourself, out loud. Record it. Change the script if you need to, making sure that the words you say make sense and feel authentic. Then find someone down the hall or in the next cube to play your participant and try the script out again. Make changes if you need to.

Do enough sessions. As you moderate each session, you will get better at it. Remember, it's not about you. Though you may feel awkward doing this in front of your team, and reading from a script, they're not paying attention to you, they're paying attention to the participant. If you feel like you have made a mistake – you said the wrong thing, or asked a question the wrong way, or you led the participant somehow, keep going. And then, go do another session. You don't have to throw everything out from the session you made a mistake in. Salvage what you can and move on.

It's a chance for team members to get closer to the participants

The whole object of doing user research is for the team to learn about current experiences. More data, even data gathered sloppily, is better than a tiny bit of data gathered expertly. And the new moderator will get better at it. No one is born to the role; moderating well is a set of learned skills. And I think that anyone can learn them with time, practice, and coaching. Find your next moderator on your design team. And keep up the good work.  

Other resources you might find useful:

Common Problems in Usability Test Facilitation by Rolf Molich and Chauncey Wilson

Moderating Usability Tests: Principles and Practices for Interacting (Interactive Technologies), by Joseph S. Dumas and Beth A. Loring

Remote Research: Real Users, Real Time, Real Research (Rosenfeld Media), by Nate Bolt and Tony Tulathimutte

Seven Common Usability Testing Mistakes, by Jared Spool

Moderating with Multiple Personalities: 3 Roles for Facilitating Usability Tests, by Jared Spool

Usability Tools Podcast: Moderating Usability Tests, Part 1

Jared Spool and Brian Christiansen

Effectively Moderating Usability Tests, with Beth Loring

Recorded virtual seminar from UIE ($149)

Beware the Hawthorne Effect

In a clear and thoughtful article in the May 3, 2007 Journal of Usability Studies (JUS) put out by the Usability Professionals’ Association, Rich Macefield blasts the popular myths around the legendary Hawthorne effect. He goes on to explain very specifically how no interpretation of the Hawthorne effect applies to usability testing.

Popular myth – and Mayo’s (1933) original conclusion – says that human subjects in any kind of research will perform better just because they’re aware they’re being studied.

Several researchers have reviewed the original study that generated the finding, and they say that’s not what really happened. Parsons (1974) was the first to say that the improvement in performance of subjects in the original study was more likely due to feedback they got from the researchers about their performance and what they learned from getting that feedback.

Why it doesn’t apply to usability tests

Macefield convincingly demonstrates why the Hawthorne effect just doesn’t figure in to well designed and professionally executed usability tests:

• The Hawthorne studies were longitudinal, most usability tests are not.
  
  
• The subjects were experts, most participants are novices at something in a usability test because what they are using is new.
  
  
• The metrics used in the Hawthorne studies were different from most usability tests.
  
  
• The subjects in the Hawthorne studies had horrible, boring jobs, so they may have been motivated to perform better because of attention they got from researchers; it’s possible in usability tests that participants are experiencing unwanted interruptions by being included or that they’re just doing the test to get paid.
  
  
• The Hawthorne subjects may have thought that taking part in the study would improve their chances for raises or promotions; the days of usability test participants thinking that their participating in studies might help them get jobs are probably over.

What about feedback and learning effects?

We want feedback to be part of a good user interface, don’t we? Yes. And we want people to learn from using an interface, don’t we? Again, yes. But, as Macefield says, let’s make sure that all the feedback and learning from a usability test comes from the UI and not the researcher/moderator. Instead, get to the cause of problems from qualitative data such as the verbal protocol from participants’ thinking aloud to see how they’re thinking about the problem.

Look at effects across tasks or functions

Macefield suggests that if you’re getting grief, add a control group to compare against and then look at performance across tasks. For example, you might expect that the test group (using an “improved” UI) would be more efficient or effective in all elements of a test than a control group. But it’s possible that the test group did better on one task but both groups had a similar level of problems on a different task. If this happens, it is unlikely that the moderator has given feedback or prompted learning to create the effect of improved performance because the effect should be global across tasks across groups.

Macefield closes the article with a couple of pages that could be a lesson out of Defense Against the Dark Arts, setting out very specific ways to argue against any assertion that your findings might be “contaminated.” But don’t just zoom to the end of the piece. The value of the article is in knowing the whole story.

Moderating tips and techniques

Getting the right information from the participant can be a difficult. As the moderator, you must attend to many things besides what the participant doing and saying. Focusing on a few specific behaviors of your own will help you have a better test.

Focus your attention on what’s happening now

• Quickly build rapport with the participant
• Listen attentively
• Be open to what might happen in a session – be ready to learn from the participant

Tips for being a better moderator

Be the neutral observer – avoid priming or teaching. If you’re too close to the product or the domain, you may train participants without realizing it by using keywords in your task scenarios or materials.

Observe at the expense of collecting data, if you must. It is difficult to take notes and to watch the participant at the same time. If things are happening quickly or you find yourself missing things the participant is saying or doing, just stop taking notes. Instead, listen and spend time between sessions making notes about what happened. Go through your recordings later if you need to, or ask observers to share their notes.

Play dumb – don’t answer questions. If participants perceive that you are an expert on the product, they may ask you questions about it or look for your approval on actions. Instead, let her know that you are learning too, and that you’ll note her questions but won’t always be able to answer them.

Flex the script and test plan. Even after you pilot test your test, you may have to adjust on-the-fly when participants do unpredictable things. That’s okay. You’re learning important things that fit into your aggregate patterns of use.

Practice and get feedback. Ask co-workers and observers to give you feedback about how you conduct sessions and how you ask questions.

Your own self-awareness is your best tool for moderating test sessions successfully. Following these guidelines should help you get valid, reliable data from your participants, even if your attention is slightly divided.

When to ask participants to think out loud

I was taught that one of the most important aspects of moderating usability study sessions was to encourage participants to think out loud as they worked on tasks. While the technique is good and useful in many usability test situations, it isn’t always the best approach.

Get data about how and why people do things
This “verbal protocol,” as it is known, can be an extremely useful thing to have. If the participant is good at thinking aloud, you all hear about how how she is forming the task and how she is thinking about reaching her goal. You will hear about why she is doing the things she is doing, and the words she uses to describe it all. You also will get verbal feedback about how the participant feels about what is happening because she may say that she’s frustrated, or annoyed or even happy.

What the data means
Hearing how a participant forms a task tells you whether the designers and the user are thinking of (modeling) the task in the same way.

Hearing why a participant is taking a particular step tells you where your design does and does not support users’ goals.

Hearing the words gives you labels for navigation, links, and buttons. Your information architecture should match the participant’s vocabulary.

Hearing the emotion tells you how severe a problem may be.

These are all good things to know.


How to get a good think-aloud
Some people think aloud naturally, or at least will verbalize their questions and frustrations just because there’s someone else in the room (that would be you, the moderator). But most people need to be primed to do it, and some even need practice doing it.

In your introduction to the session, ask participants to tell you what’s going through their minds as they do tasks.

Consider incorporating a practice task that lasts for half a minute, just to get participants to try it out. Encourage them and quickly move on.

When you describe the task scenario to participants, remind them to think aloud.

During the task, when something seems to be frustrating, annoying or hindering -- and the participant isn’t talking -- ask her to tell you what she’s thinking.



Know that there’s more going on than you can hear

People filter automatically
Participants can’t tell you everything they’re thinking. And really, you don’t want that. Humans can process on a number of cognitive tracks at the same time. Most study participants will automatically be able to distinguish between what is related to the situation and what isn’t.

This is a test
They also may filter what they tell you beyond this basic distinction. For example, they want to do well. Although you tell participants you are not testing them, a participant might feel some level of test, even if she’s just in competition with The Machine.

Participants may fear failure or embarrassment. In usability studies, people often persist at times when they would normally ask for help.

People tend to give positive feedback
Participants want to give you a good session. People are conditioned to say and do things for the approval of others. They want the moderator to approve of their performance.


Participants take responsibility for bad design
People who are novices at a task or are working with something outside their experience may excuse the design by taking responsibility for a design problem. For example, they may say they could do it now that they (have failed and) have done it once. Or they just need more time to learn the site. This is especially common among older adults who are unsure of their computer or other appropriate skills.


When you might not want to use think-aloud
There are times when using think-aloud can conflate or dilute your data. There are other situations in which using think-aloud is just difficult, or won’t work for the type of participants you have in your study.

Time on task
If you want to measure how much time it takes people to complete a task because you are particularly concerned with efficiency, introducing think-aloud is probably a bad idea. Talking about what you’re thinking slows you down while you choose words to convey your ideas about what’s happening and why.

Audio feedback in a user interface
Some interfaces incorporate audio feedback to indicate statuses or modes. These auditory cues may be overlapped by the participant talking so the participant may miss something important happening – or you might. Also, many blind people and people with severe vision impairments use screen readers to use software and web sites. If you’re tuned in, you can learn things by listening to the screen reader as it works. And, although most of the people with visual disabilities who use screen readers who I have observed can listen and talk at the same time (like sighted people can see or read and talk at the same time), as a sighted moderator, my auditory channel is challenged by listening to both the screen reader and the participant at the same time.

You’re interrupting a taxed thought process
People who have short-term memory loss, are medicated, or have other cognitive limitations tend to stop talking when they encounter obstacles to reaching their goals. You might be tempted to prompt these people to “tell me what you’re thinking,” but try not to. They’re concentrating on working around the obstacle. If you watch closely, you can see their solution unfold. After it does, then ask them about how they got to it.


An alternative to think-aloud: Retrospective review
“Retrospective review” is just a fancy name for asking people to tell you what happened after the fact. Go back to a particular point in the task, set the context, and ask the participant to tell you what was happening. For example, say something like this: “When you got to this point on the registration form [pointing to a field], you stopped talking. Tell me about what you were trying to do and what was happening.” The participant may revise what happened, but you will have good notes and the memory of someone who was observing closely, not trying to perform, so you can pinpoint issues that you thought were happening. Invite the participant to correct your perceptions.

If you have the tools and time available, you can go to the video recording so the participant can see what he did and respond to that by giving you a play-by-play commentary.



It’s a great tool, used at the right time with the right participants
Think-aloud or verbal protocol can give you rich data about vocabulary and effectiveness of design. From it, you can also get some impression of the severity of problems for a particular participant or the level of satisfaction for someone who had a positive experience. Use think-aloud in exploratory or formative studies to help you understand how users are modeling their tasks. Consider carefully whether to use it in other situations, though, to ensure that you’re not adding to the cognitive load that the participant is already experiencing.