Usability Testing

Four secrets of getting great participants who show up

2012-01-05T10:14:00.000-08:00

What if you had a near-perfect participant show rate for all your studies? The first time it happens, it’s surprising. The next few times, it’s refreshing -- a relief. Teams that do great user research start with the recruiting process, and they come to expect near perfect attendance.

Secret 1: Participants are people, not data points
The people who opt in to a study have rich, complex lives that offer rich, complex experiences that a design may or may not fit into. People don’t always fit nicely into the boxes that screening questionnaires create.

Screeners can be constraining not in a good way. An agency that isn’t familiar with your design or your audience or both -- and may not be experienced with user research -- may eliminate people who could be great in user research or usability testing. Teams we work with find that participants who are selected through open-ended interviews conducted voice-to-voice become engaged and invested in the study. The conversation helps the participant know they’re interesting to you, and that makes them feel wanted. The team learns about variations in the user profile that they might want to design for.

Secret 2: Participants are in the network
Let’s say the source is panel or a database (versus a customer list). People who sign up to be in panels or recruiting databases tend to be people who take part in studies to make easy money. Many are the kind of people who fill out surveys to win prizes. These people might be good participants, or they might not.

Teams that find study participants through personal, professional, and community networks find that when the network snowball of connections works, people respond because they’re interested and have something to offer (or a problem you might solve for them with your design).

They also come partially pre-screened. Generally, your friends of friends of friends don’t want to embarrass the people who referred them. If the call for participants is clear and compelling, the community coordinator at the church, school, club, union, or team will remember to mention the study as soon as they encounter someone they know who might fit. Don’t worry: the connections soon get far enough away from you and your direct network that your data will be just as objective and clean as can be.

Secret 3: Participants want to help you
They want to be picked for your team. They want to share their experiences and demonstrate their expertise. When teams are open to the wide range of participants’ experiences, they learn from participants during screening. Those selected become engaged in the research. These are participants who call when they’re going to be late, or apologize for having to switch times. They want to work with you. One team we worked with had a participant call from a car accident before calling the police. (They rescheduled!)

Secret 4: Participants need attention
You know all the details that go into a study. Participants need confirmation and reminding. Teams that send detailed email confirmations get respectable show rates. Teams that send email confirmations, and then email reminders just before the sessions get good show rates. Teams that send email confirmations, email reminders, and then call the participants to remind them in a friendly, inviting tone get stellar show rates.

Some teams use the call before the session to start the “official” research. Rather than the recruiter doing the final call, the researcher phones to explain the study and the roles, and ask some of the warm up questions you might normally start a regular session with. These researchers establish a relationship with the participant. They also get a head start, leaving more time when they’re face-to-face with a participant to observe behavior rather than interview.

Perfect attendance is worth the effort
When all the scheduled participants show up, the gold stars come not only for efficient use of the time in the lab and keeping clients and team members eyes and ears with users. It’s likely the team ends up with better, more appropriate, more informative participants, overall. That means better, more reliable data to inform design decisions.

The true costs of no-shows

2012-01-03T12:35:00.000-08:00

One of the first things people say when they call up looking for help with recruiting is that they want to recruit “12 for 8” or “20 for 15”. They know what they want to end up with. They’ve got to get data. Managers are showing up to observe. They’ve gone through a lot to get a study to happen at all. They don’t want to risk putting a study together only to get less data than they need. So, compensating for a show rate of between 60% and 80% means over-recruiting.

Even though a recruiting agency probably won’t charge for no-shows, those no-shows can be costly in lots of ways.

 Stress. First, there’s the anguish of the moments before you know whether the participant will show. It’s not only the researcher who’s nervous. Nobody knows what to do. Time is ticking away. The session is running late late. Observers get antsy. The careful planning and scheduling of putting on a study feels like it is falling apart. That can be agonizing. When the participant arrives, it’s difficult to center yourself and get on with the session.

Embarrassment. There is nothing like the humiliation of having participants not show up after nudging, cajoling, and begging a team to show up to observe their users. The campaign of marketing user research disintegrates as the promise of enlightenment fades. It’s like promising fireworks but the fuses won’t light. The negative energy can be palpable and extremely difficult to recover from. Claiming that this is out of you control will not help the credibility of the study.

Loss of productivity. When you find out there is a longer-than-planned gap between sessions, conducting shadow puppet shows or showing cat videos to fill the time will not win the day. The researcher has got to get the team engaged in something productive that will help everyone feel like they’re still getting value out of being there. Without a participant to observe, it’s tough to keep the team from thinking you’re not wasting their time. That’s difficult to pull off for even the most experienced of researchers.

Loss of trust. Let’s not forget the pain of losing momentum as observations are interrupted. Say you got a few sessions done. Everyone is charged up with the revelations they’ve witnessed. Epiphany is everywhere. And then, a no-show. It can suck the air out of the room. Clients and teams can become disenchanted and impatient. They may lose confidence in the process and the results.

By comparison, the amount of money going for lab time when you’re not running sessions -- or paying for even more lab time and recruiting to make up sessions -- seems small. The damage of no-shows is long lasting. Yet, most of us treat no-shows as a cost of doing studies. They just happen. It’s the recruiting agency’s fault. We can’t control the participants.

What if you had a near-perfect participant show rate? The first time it happens, it’s surprising. The next few times, it’s refreshing -- a relief. Teams that do great user research start with the recruiting process, and they come to expect near perfect attendance. In the next post, I’ll share their secrets for getting great participants who show up.

What's the best way to find people for user research and usability testing?

2011-12-19T08:42:00.000-08:00

There are lots of great sources of participants for usability studies and other user research. The key: know what behavior you want to learn about. For example:

Playing online games
Voting
Planning for retirement
Shopping for a new car
Treating a chronic illness

Note that there’s nothing about demographics here.

After you identify the behaviors you want to learn about -- preferably by observing people using a design rather than just asking them about it -- brainstorming ideas for where to find them can be fun. There are loads of options.

1. Let the snowball work its magic for you

Friends and family. You can invite friends and family to take part in your study, themselves, if they do the behavior you want to learn about. This is a great way to protect secrets because you can swear them to secrecy in a way that you can’t with outsiders, just by keeping them close. They’ll keep their commitment to you.

These folks can also be an excellent source of participants who you don’t know. That is, you can put the word out to your personal network, describe what you’re looking for, and let them do the work. If you can include some statement about the value of the user research you’re doing, people may be motivated to take part without other compensation.

User groups, community organizations, social clubs. Calling the heads of clubs and groups from churches or unions to chess clubs, charitable organizations, affinity groups, professional associations -- basically any group you can think of -- will often net you one super interested person who will pass the word. These networks work really well if your study has special relevance to the group, but many people may respond just because they’re curious.

Online social networks. Yep, you can put it out to Twitter, Facebook, Google+, or whatever you’re into at the moment to find people, too. The people who follow you in these spaces may not be the people you want as participants, but they know people who are appropriate.

We call this “snowball recruiting” because it puts the word out and that takes on a momentum of its own. Now, it doesn’t happen automatically. You will have to nudge people and put the call for participants out several times. For the clubs and groups, phoning (I know, that is terribly old fashioned) is going to get you way better response and results than sending an email to the group’s info@whatever.com email address.

2. Intercept people

This works in real life and online. In real life, it helps to approach people where they’re doing or would normally do the behavior, but you don’t have to do it that way. For example, if you want to learn about how people make decisions about what car to buy, you might want to hang out at a dealership (with permission, of course). I recently conducted a study on election ballot designs where the research teams approached people in libraries or at farmers markets. People were not voting in those places, but they were receptive to a smiling, charming person approaching them with a clipboard and a “Would you give us 15 minutes to learn about ballot design?” In about 2 hours, each team had observed 6 or 7 people using two different ballots.

If your design is online, an excellent tool to ask people nicely if they’d let you watch them while they use your website is Ethnio. It’s made by the brilliant people at Bolt | Peters, who use it constantly, themselves.

3. Craigslist

Craigslist gets its own category because its charms are different from the other options. On the plus side, the reach is great, the response will be huge in most cities, and the cost is little or nothing. On the minus side, there are some, shall we say, interesting people who use Craigslist. So, you’ll have to do some diligent filtering. Fortunately, the response is usually so large that you can be very discriminating. For example, at my company, we automatically delete any email that comes back without answering all of the questions we asked people to answer in our ad. Don’t put your phone number in the ad.

Getting people to find you

Two or three weeks before you want to do your sessions, put the word out or post your ad to Craigslist. Be sure to include:

Who you’re looking for. Make it something like, “We’re looking for people who play first-person shooter games,” or “We’re looking for people who are thinking about buying a new car in 2012,” or “Know someone who has had a triple bypass? We’d like to interview them.”

When you want to do the sessions. “We’re conducting 1-hour interviews in person January 20-23.”

What your prerequisite information is. Here, you might want occupation and location information, but you definitely want contact information.

How to contact you. By phone, email, Twitter, whatever.

Create a version you can fit into 140 characters, first. Add detail for the other channels you’re going to spread the word through. Don’t be afraid to change it up a bit as you learn what is attracting people and what questions they have.

What not to do

I have two other recommendations:

Don’t use a screener. List the behaviors and motivations you want to make sure that people have for your study for yourself, but don’t create multiple choice questions. Doing a screener -- which most people are very, very bad at -- is just an invitation for people to game their way into your study. Screeners eliminate edge people who might be really interesting for you to interview.

Instead, narrow down the candidates, and call them up. Spend time talking with them on the phone, asking open-ended questions. This accomplishes several awesome things. First, you get bonus user research. You may learn things about people that you hadn’t thought about before that will enrich your research and your designs. Second, it’s very unlikely you’ll end up with someone in the study who is not appropriate. Third, the candidate gets very engaged and is much more likely to show up for the sessions. If you hire an agency, they’re going to recruit 12 participants so you can end up with 8 good sessions. When we recruit this way we almost never have to replace anyone. Our “show rate” over 7 years is about 94%.

Don’t use an agency. Agencies have panels of people who have signed up to be in studies. These are the same people who answer surveys to get prizes. Do you want those people? Probably not. Agencies will have to use a screener. They will ask only the questions you include in your screener. They don’t know anything about your domain, or about how user research is done. They don’t care at all about what you want to end up knowing. All they care about is putting a body in a seat. Now, having said that, if you have an agency you know and like and have a relationship with, you can get excellent results. Most aren’t set up to help you in this way.

A word about biased samples

Most of us don’t have time or budget to do a truly random, representative sample. So, we use samples that are people who present themselves to us; user research is all about convenience samples. There is nothing wrong with this. You just have to be aware of the sources and what their biases might be. Using a combination of sources will even out the biases, and from there you should end up with reasonably reliable data.

This post originated as an answer to a question on Quora.com.

Usability testing is HOT

2011-06-28T09:00:00.000-07:00

For many of us, usability testing is a necessary evil. For others, it’s too much work, or it’s too disruptive to the development process. As you might expect, I have issues with all that. It’s unfortunate that some teams don’t see the value in observing people use their designs. Done well, it can be an amazing event in the life of a design. Even done very informally, it can still show up useful insights that can help a team make informed design decisions. But I probably don’t have to tell you that.

Usability testing can be enormously elevating for teams at all stages of UX maturity. In fact, there probably isn’t nearly enough of it being done. Even on enlightened teams that know about and do usability tests, they’re probably not doing it often enough. There seems to be a correlation between successful user experiences and how often and how much the designers and developers spend time observing users. (hat tip Jared Spool)

Observing people using early designs can be energizing as designers and developers get a chance to see reactions to ideas. I’ve seen teams walk away with insights from observing people use their designs that they couldn’t have got any other way – and then make better designs than they’ve ever made. Close to launch, it is exciting – yes, exciting – to see a design perform as useful, usable, and desirable.

I’ve been negative on usability testing and our failure of imagination regarding bringing the method up to date, lately. But there’s a lot of good to any basic usability test. In fact, I went looking for the worth, the value, the alluring in usability testing a few weeks ago when I asked on Quora, “What’s the sexiest thing about usability testing?”

Some of the answers surprised me. Some of the answers were more about what people love about usability testing than what makes it seductive. But let’s go with seductive. People who find usability testing hot say it’s about data that can end the opinion wars, revelations and surprises, and getting perspective about real use, motivations, and context of use. Okay. We’re nerds.

The kiss of data

We always learn from users. Of course, we could just ask. But observing is so much more interesting. People do unpredictable things; they create workarounds, hacks, and alternative paths to make tools fit for their use.

This is the best case I can think of for watching rather than asking. From this observing, we get data. Juicy, luscious data like verbal protocols, task success rates, and physical behavior. This package makes it much easier to make good design decisions because we know have evidence on which to create theories about what should work better. There’s nothing like having hard evidence for going with a design direction – or changing direction.

Voyeuristic revelations

When designers, developers, and stakeholders of all persuasions get to observe people using a design – especially the first time – there’s often an “ah ha!” moment. (That’s the clean version.) Observers exclaim, “Wow, that was amazing!” when they see something surprising, both the good and the bad. The reaction that follows a completed usability study often is, “Damn. I wish we’d done this years ago. It would have saved us a ton of rework!” After watching one over-qualified participant struggle with a design recently, I heard a client say, “If that guy can’t do it, we’re in serious trouble.” That’s powerful.

When participants are surprised, that’s when the real fun begins. Not everyone likes surprises in their user interfaces, especially if they’re not the delightful Easter egg kind. While a team hopes not to hear, “I feel lost and abandoned,” you’ve got to wonder how bad it’s been when a participant squeals, “Oh, my gosh! This is so much better! When can I have it?!” Those eureka moments can reveal what to do to improve a design or an experience.

Relationship dynamics

One of the magical things about observing users working with a design is that suddenly, disputes within the team melt away. Chances are, the disputing parties were both wrong because neither (unless they have a ton of experience already observing these kinds of users in this domain doing this task) could accurately predict how the user would behave and perform.

Now, even with observations from watching just one user, there’s data on which to base design decisions. Data trumps gut. Data outweighs feelings. Data can put to rest those endless, circular discussions where inevitably, the person with the biggest paycheck or the most important title wins. The opinion wars come to an end.

When the whole team is involved in deciding what to test and observing sessions, everyone can share in making and carrying out agreed design decisions. Whenever a question comes up where no one knows but everyone has an opinion, the answer in a team doing usability testing is, “Let’s do some user research on that,” or “Let’s find out what users do.”

For the love of users

It’s so easy to get caught up in the business goals and issues with the underlying technology of a design. It’s so easy to stay in the safe bubble of the office, cranking out code, designs, plans, and reports. It’s easy to lose touch with users.

Teams that spend a couple of hours observing their users every few weeks keep that connection. They fall in love with their users. They relish the chance to see for themselves why people do the things they do with designs.

Getting out of their own heads, a successful team uses usability testing to get perspective, learn about users’ contexts, and remember the people and their stories. For these teams, usability testing is inspiring. And that’s hot.

What’s sexy about usability testing?

Observing people use a design can be revelatory. It turns up the volume on design by helping teams make informed design decisions. What’s sexy about usability testing? Data for evidence-based design. Ending opinion wars. Knowing users from observations and surprises. Getting perspective and knowledge of context of use.

The UX equivalent of a romantic dinner or a walk on the beach? Perhaps not, even for a geek girl like me. But it can be exciting, fun, funny, encouraging, and empowering. Just what you want from a relationship. That’s pretty seductive, if you ask me.

Oh, Etsy. How could you?

2011-03-21T14:17:00.000-07:00

During the last holiday season, I called Land's End. I hardly ever call; I'm a huge fan of their online experience. I wanted to send a special order to my mother, putting two matching things in the same gift box. Landsend.com isn't really set up to do that, but the site instructed me that I could do it, so I called.

The sales rep was friendly and efficient, and very helpful. She pulled up my order-in-progress and put everything in the box that I wanted to be in my mother's gift. When she asked me if there was anything else she could help me with, I blithely said, "You could check my mother's account and tell me what she's sending me for Christmas." The sales rep giggled, teased me a little bit by telling me that she could see my mother's account, but told me I would have to wait until UPS delivered it to find out. She protected the relationship between the seller and the buyer. She also protected the relationship between two buyers – me and my mother -- at least for that episode. If I wanted to find out what I was getting for Christmas, I'd either have to wheedle it out of my mother or wait.

Respect and research. That's all I ask.

Facebook had a go at Beacon, a service that broadcasted out to all your friends the purchases you've made outside of Facebook, without permission. The Federal Trade Commission sued, and Facebook eventually settled and took down the "service" in 2009.

Facebook has a history of screwing with the privacy of its users. Beacon was a prime example. The main problem here is the lack of permission. And that's the case for Etsy's new People Search, too.

The designers of Etsy decided it was a good idea to make everyone on Etsy searchable by name, including buyers. So, if you have ever bought anything on Etsy, you can now be found there by anyone else either by your real name or your username. Your whole profile is viewable, including your purchase history. Not only that, it'll all show up in Google search results.

The idea is that buyers would form social "circles" on the site to share information about their purchases.

Uninformed by research, guided by gut

These are the kinds of things that happen when an organization puts business goals before customer goals. It's also the kind of thing that can happen when an executive wakes up one day and says, We want to be one of the cool kids. And right now, to be one of the cool kids, you have to have social media. How do we do that?

What made Etsy think it needed a social layer on its beautiful, engaging site? It's the kind of thing that happens when teams decide to strap social on rather than looking at the conversation they're already having with customers and that customers are already having with one another.

I'm sure a lot of thought went into this decision of Etsy's. I fear this is a vacuum-sealed decision. Here's my imagined scenario, a scenario I've seen played out in other, similar decisions at other, similar organizations: Management, who forgets that their site is not the center of the universe for anyone outside that room, went to the product manager and asked for some of the social awesomesauce that is out there to turn up the buzz a notch. The product manager brainstormed with the team. The best idea they could come up with is to get customers to talk about the fun, beautiful, interesting stuff they'd bought on Etsy online with one another. (Never mind that we already have Twitter for this.)

Where is making use of the conversation that Etsy is already having with its customers or that buyers and sellers are already having together? They probably can't make use of these conversations because they haven't observed them. Where's the research to support this design decision?

Why this is on my usability testing blog

It's hard for me to believe that if Etsy had conducted user research and even informal but realistic usability testing on the idea that they would not have quickly seen the privacy violation. They could have avoided the damage control they now have to deal with because of the breach of trust they've had with buyers who already love the experience of shopping there.

How Etsy could have avoided the problem and discovered a possibly great idea for engaging buyers even more

1. Analyze the risks of a social media strategy to users' privacy, security, and trust. Where was the business plan for allowing search of users? How does having social "circles" support the business model, exactly? How would the social media strategy be supported on the back end? More than all that, let's look at others who have gone before us: Beaon on Facebook and Boden USA come to mind. What happened there? What could the Etsy team learn from those mistakes? Oh, and, why duplicate Facebook in any way?

2. Proof the concept with real people who shop on Etsy. This is pure conjecture based on my experiences with other organizations: Etsy may have thought that to up their game and get people more engaged in the site, they needed to get buyers talking with one another and not just to sellers. Charming idea. But how do you find out if people find that useful?

Focus groups? If there were focus groups, I'm just going to guess here that participants liked the idea, but there was no exploration of the implications of this profile information being public rather than private. Not ideal.

What else could they have done? Invited friends and family. This approach still perhaps is not optimum, because friendly participants might not have exposed the privacy problems. They are, after all, friends and family, so there's automatic trust and wanted connections there already. How about rolling it out to a very small number of key buyers -- 3 or 5 -- and watch what happens for a week or a month as they connect to their people, or until something bad and unintended happens?

3. Conduct usability testing with real people in real contexts to learn the ripples to real relationships. Let's say they did usability testing. Did they bring in real buyers to use a working prototype with their own data? Did it occur to anyone that now my ex can Google me (like he does) and find out that I bought my sister a Star Wars crochet pattern, or my current paramour a hand made can coozie? Or what about the fact that my clients could see all the personal things on my Etsy wish list?

A usability test with a limited "circle" on a closed sandbox (like a walled-off development or testing server) for a couple of weeks might have given them some clues about what might work and what might not.

Etsy, I love you, but I have to go now

Not only will Etsy have to clean up its own site by making the social opt-in, but they'll also have to figure out a way to recover buyers' privacy. How does a web organization reclaim data that is now not in its control? If they could invent a big Web eraser to drag behind them as they invite buyers back to the site, they might have a chance.

Sources and resources:

http://arstechnica.com/web/news/2011/03/etsy-users-irked-after-buyers-purchases-exposed-to-the-world.ars

http://forums.penny-arcade.com/showthread.php?t=138661

Involving older adults in design of the user experience: Inclusive design

2011-02-28T11:06:00.000-08:00

Despite the reality of differences due to aging, research has also shown that in many cases, we do not need a separate design for people who are age 50+. We need better design for everyone.

Everyone performs better on web sites where the interaction matches users' goals; where navigation and information are grouped well; where navigation elements are consistent and follow conventions; where writing is clear, straightforward, in the active voice, and so on. And, much of what makes up good design for younger people helps older adults as well.

For example, we know that most users, regardless of age, are more successful finding information in broad, shallow information architectures than they are with deep, narrow hierarchies. When web sites make their sites easier to use for older adults, all of their users perform better in usability studies. The key is involving older adults in user research and usability testing throughout design and development.

There are some important considerations in working with older adults in studies. Remembering the points below will ensure that you and your participants have a good experience and you get the data you need to inform design decisions.

Finding participants: Understanding older adults before you recruit

In many places in the world, older adults outnumber people in other age groups. The question is, how do you find the right people to take part in studies? They can be difficult to get to.

We found that these approaches did not work well:

Community web sites, message boards, or chat sessions. The oldest old tend not to take part in these groups, so posting ads in those places is not a fruitful way to find participants.
Senior centers and community colleges. These are places that offer classes in using computers. If you want computer and web novices for a study, they might be good places to find appropriate participants. They are not good sources if you want to observe people with enough web experience to see them working at a web site without teaching them.
Flyers at a senior center, when they did not make clear that we were recruiting for a study. Many older people are much more cautious and skeptical than younger people. They are often fearful of being cheated or "taken." For example, we had put up flyers at a senior center from which we got no response; later we learned that people thought we might be trying to sell them something.
Cold calling from a database. This is probably again because older people are afraid that they may be scammed into buying something.

These ideas did work well:

Calling with a personal connection. If we could say that a mutual acquaintance had suggested the contact, potential participants were much more receptive to hearing about the study and considering taking part. It is important to establish credibility and trust with the potential participant.
Being careful in the initial call to say where we had gotten the contact information and that we weren't selling anything.

Recruiting older adults

Recruiting participants in their 70s and 80s is more difficult than recruiting participants in their 50s. The oldest candidates are less receptive to strangers phoning them, and they don't check email as frequently.

Recruiting by phone. Phoning is important. Plan to phone potential participants at least once (or have your recruiters do so). You need to quickly establish credibility and trustworthiness, to assure potential participants that you are not selling anything, and to establish a connection by letting them know where you got their names. When you can do that, potential participants are often glad to hear from a real person. It is also easier for them to determine legitimacy and to ask questions about the study on the phone. They'll use your answers to help them decide whether they want to take part.

Another reason for phoning potential participants is so you can judge their English language skills and whether they are hearing impaired. (You may well want to include limited English speakers and hearing impaired users in your study; if you do, you want to be aware of these specifics about the participants before they come.)

Recruiting by email. Email can be very efficient for younger participants; it's less so with adults in their later 60s and 70s and older. Give yourself more time for these older participants; they generally don't check their email more than a couple of times per week. This happens for a variety of reasons: They don't feel the need to check mail frequently. They use a computer at a senior centre or a library. They have limited time available through their Internet service provider.

As suspicious as older adults are of telemarketers, they are also vigilant about spam. If your email address is unknown to them, without an appropriately descriptive subject line, they may delete it. Always put on a very clear subject line.

Screening older adults

Screening older adults demands specificity. Many older users when asked the question "what do you do online?" answer "email." They often don't think about practical activities such as banking or bill paying online as "using the web."

Many older users are also not as familiar with the language of the web as younger users are. They don't distinguish between the Internet and the web. They don't always know the difference between the web browser and the web page.

We found that self-reported data about frequency of use and numbers of hours spent online were not good indicators of proficiency, either. For example, we had one participant who spent 60 hours per week online. We didn't find out until the session started that her sole use of the web was playing games on four web sites that her friend had set up as separate shortcuts on her desktop.

So, asking a variety of specific questions to gauge potential participants' familiarity with the web can help the person recruiting participants for a study make judgments about how suited the person might be for the study. Even if you're looking for a mix of proficiency levels, you still have to be able to determine where in the range a potential participant fits.

Tech savviness matrices
An assessment we’ve found to work well asks about frequency of use over a broad range of types of interactions older people can take part in on the web. Here are some example assessment grids we have used.

Scheduling sessions with older adults

Scheduling sessions with older participants can present some logistical challenges that you might not think about in studies involving younger participants.

They arrive early. Because many older people are retired (or at least have ample free time), they almost always arrive for their sessions early – up to an hour early. Be sure to have someone to greet them and set up a comfortable place for them to wait.

They bring their spouses. Older participants often bring their spouses or a friend with them. They may have travelled some distance to get to the session; they may have planned activities for after their session; or they simply may not like driving alone. Have magazines, a phone, and a comfortable chair available for the spouse or friend.

They do best in the morning. Even though people in their later 60s, 70s, and 80s are vital and energetic, they usually have more—and better—attention to give earlier in the day. Try to schedule people who are in their late 60s, 70s and 80s in the morning and save any afternoon sessions for participants in their 50s or early 60s. We don't recommend running evening sessions.

They don't like driving in rush hour. If you are holding sessions in a central place (rather than meeting participants in their homes or workplaces), schedule the sessions outside of peak traffic times, if possible.

Reminding older adults of important points before they come

Reminders about one-to-one sessions. Participants can become nervous and uncomfortable if they realize after arriving that they will be the only participant in the session. Usability studies are still fairly new to the general population. Recruiting firms often recruit for focus groups, and participants who come through these firms often assume that they are coming to a focus group.

Reminders about videotaping and observations. Although a good practice is to ask for permission to record and to have people observing the sessions when you recruit, people tend to forget that. Make sure that the person who calls the participant to confirm the session also tells the participant that:

"You will be videotaped and observed by people you won't be able to see during the session."
"This is a one-on-one session. You will be the only participant in the study room with a moderator."

Special reminders for older adults.

Computer glasses. Many participants will have special glasses for using the computer. So another important reminder is, "Don't forget your computer glasses!"
Eat first. Also, for long sessions— anything longer than 45 minutes— ask participants to make sure they eat before they arrive. Because many participants expect to take part in focus groups rather than individual sessions, they also expect to be fed. If you have snacks available, try to have fruit and nuts or other relatively healthy food. Many older participants are diabetic.

Working with older adults during sessions

Many older participants won't know what to expect coming into a usability study session. Be clear in setting their expectations and be firm but polite about keeping the session focused on what you're trying to find out.

Make participants comfortable. Be respectful without being patronizing. You can be a neutral moderator but still be polite. "Please" and "thank you" are important. Many older adults expect more statements of politeness like these than younger participants do.

Older participants also deserve extra consideration, politeness, and detailed information about the session. They will feel more comfortable if they know what to expect up front:

Clearly explain the session plan, timing, and what they can expect.
Warn participants that you'll interrupt them and that you may stop them before they have completed tasks.
Schedule breaks for long sessions (and tell them they can take breaks whenever they need to).
Have them practice thinking aloud.
Consider including a practice task to help participants understand how the session will work.
Take account of beliefs that participants may have learned or created about how to work with computers.
Remember that older participants often are not versed in computer and web terminology, so avoid using this jargon when working with them.
Be extra patient with older participants; wait longer than you normally might to prompt; consider giving participants permission to ask for hints.
If participants stop talking, consider letting them continue that way; try reflecting on the task later.
Teach participants something at the end of the session.

Keep them on track tactfully. Most of the participants we've had in sessions are interesting, charming, and very talkative. Many older participants have a lot of stories to tell. Their stories say a lot about who they are, and where they have been— and often provide a context for interpreting data.

But it may be easy for participants to get off track during the session, and while it may feel awkward or mean sometimes, it is the moderator's job to keep the participant focused on the task, talking about it, and getting data for the study. This is the main reason for warning participants in the introduction to the session that you may interrupt them and that you may stop tasks before they've completed them.

Listen for their beliefs about computers and the web. Many people who are in their late 60s and older never used computers at work. This means they have no previous experience from which to make inferences about how a computer or an application might work. Many learn how to use computers and the Internet through friends, family, and neighbours. They inherit the superstitions and myths that those people have developed to help themselves work around problems. Then the older adults bring these myths into sessions with them, and you'll hear about them as task-solving strategies and workarounds. It's important to capture these; they are part of the users' reality and we have to deal with these beliefs when we design web sites.

Be careful of the words you use; avoid computer jargon. Older computer users rarely know much about computer-related terminology, so you should avoid using these terms during your sessions. Older participants often don't know the names of widgets such as drop-down boxes or cascading menus. Most of our participants also had little knowledge of web-related terminology. For example, they weren't sure about terms such as "link," "URL," and "login." Many were unclear about the meanings of "online community" and "message boards." "Browsing" wasn't always meaningful in the context of a feature called "browse by topic." The word "emoticon" and the concept behind it were completely foreign to most of our participants. This means that you must pay close attention to what participants do and point at on the screen or device.

Give them time. Older participants almost always take longer to do tasks than younger participants. And, although they seem to struggle, the oldest participants also expect using the computer to be difficult. Plan for tasks to take much longer for older participants than they would for younger participants—up to 25 percent longer in our experience.

Help participants understand the time constraints of the session by explaining the session format in your introduction. Also, wait longer to prompt than you normally might. You might also consider giving participants permission to ask for hints when you introduce the session.

If necessary, hold the think-aloud and ask participants to reflect later. A classic technique in usability testing is to ask participants to think aloud while they work through tasks toward a goal. When tasks become complex or difficult, participants may stop talking. Use your best judgment about nudging them to tell you what they're thinking. For some participants with short-term memory loss or other cognitive impairment (such as that caused by pain medication), your asking for their thoughts may interrupt their task enough that it causes them to make errors. In those cases, you may get more usable data without the think aloud protocol by asking participants to reflect later.

Don't lead even when you want to. If, as a session moderator, you have a soft spot in your heart at all for participants, working with older participants will exercise that spot a lot. You may be tempted to give hints; worse, you may lead them in ways you don't intend. Be patient and firm but polite while keeping to your agenda.

When appropriate, teach something at the end. If the session has been difficult for the participant, or, if there is some small thing that would make using the computer or the web easier, take a little time at the end of the session to teach the participant something. For example, show participants how to change the text size in their browsers and shortcuts for copying and pasting and printing.

Including older adults in user research and usability studies: Older, wiser, wired

Older adults don’t behave differently from younger people online. Just thinking about old age as a collection of disabilities is old business. The new world of designing for older adults is about creating web sites and other technology that is useful and desirable as well as accessible to the broadest range of users. Older adults as a cohort are living longer than their parents because they’re healthier, and many will be affluent because they’ve been saving up for a lifetime – this means they have time, money, and motivation to be online.

Neither a monolithic view of older adults nor an entirely separate design for older adults is necessary. Younger designers developing web sites for older adults need to learn more about older adults’ life experiences. For example, many older adults don’t perceive themselves as old. And so, all technology design – not just designs for older adults – should involve users.

Bonus research: Do the recruiting yourself

2011-02-07T07:27:00.000-08:00

There are some brilliant questions on Quora. This morning, I was prompted to answer one about recruiting.

The question asker asked, How do I recruit prospective customers to shadow as a part of a user-centered design approach? The asker expanded, thusly:

I'm interested in shadowing prospective customers in order to better understand how my tool can fit into their life and complement, supplement, or replace the existing tools that they use. How do I find prospective customers? How do I convince them to let me shadow them?

Seemed like a very thoughtful question. I have some experience with recruiting for field studies and other user research, so I thought I might share my lessons learned. Here's my answer. Would love to hear yours.

I've learned many recruiting lessons the hard way. Recruiting can be challenging, but it can also be really interesting. I always find that I learn a lot in the recruiting process. Here are my pointers:

Do the recruiting yourself. This gives you bonus research data about your people, and you may learn things you hadn't anticipated that will influence how you conduct the study. It also starts a relationship with people that gets them invested in taking part. It's the start of a conversation with you and your organization. They're more likely to trust your motives and give you a deeper, richer view of their lives.

Focus on behavior, not demographics. If you want people to use your tool to do a particular thing, look for people who already do that somehow. For example, if you want people to use your design to store their photos and music and other content, find people who have a lot of that type of content and who are concerned about losing it. If you want people to use your design to generate invoices, find people who are doing that now and observe what they do to generate invoice when they're in the mode of doing it. If you want people to use your design to remember to take their medicines at the right time in the right dosage, find people who have persistent conditions that need medication and who have been diagnosed with the type of condition you want to help them deal with.

You'll see that I never once mentioned age, sex, income, location, education level, marital status - or any of those things that marketers go on. Because it doesn't usually matter. What matters for UX design is behavior. Do people do the thing you want to make a design for?

Be as generous as you can with incentive/honorarium/compensation. People love gifts. They also love cash. The point is to show your appreciation for their help. They're doing you a huge favor by spending time with you and letting you see their lives. Offer something your kind of people want.

Be flexible about days, times, number of hours. If you want to shadow people, you have to do it when they're in the mode of doing the things you want to observe. My experience is that most interesting behavior (unless it is truly work related) doesn't happen between 8am and 5pm Monday through Friday.

Get participants to opt in to the study. Sourcing for a field study is one of the most challenging steps to recruiting. A carefully worded ad on Craigslist can attract the right kind of people. Going through community organizations or professional associations can work well.

We practice something we call "snowball recruiting." That means putting the word out among friends, family, and colleagues about the type of people you're looking for and what you want to do in the session and asking your close ties to help you find people. Advantage: Participants come lightly pre-screened, so you know they're not nuts. and they're likely to be reliable because someone you know sent them. Snowball recruiting will also help establish you as legit with potential participants. The reputation management goes both ways. Disadvantage: This approach can take a little longer to generate leads. For a shadowing or field study, it can take 3-4 weeks to get people to come to you, screen them, and then schedule them to do the session you ultimately want to do.

You can recruit from Craigslist in a much shorter period, because the response is usually great, depending on the geographic area you're recruiting in. However, there's more filtering and more screening to be done.

Remember that every sample has biases. UX research is based on convenience samples rather than random or representative samples. That's fine because you're focusing on behavior and performance rather than generalized opinions and preferences. But the sources do influence the expertise and world view that people have. For example, if you're observing people's behavior around online security and you recruit just from Craigslist, you may find that most of the people who respond are not very concerned with protecting their online accounts. Or you might find the opposite.

Be genuinely, authentically interested in the people you want to observe without being creepy. These people are letting you into their lives. So, explain what you do and what you're hoping to learn from the study. Treat them as your partner in answering your research questions. After all, you can't find out what you want to know without them.

Follow up. Send a hand-written thank-you note. Yes, this is in addition to the incentive.

I've written a lot about recruiting for user research and usability tests. You can see all those articles here: http://usabilitytestinghowto.blogspot.com/search/label/recruiting

Usability testing is broken: Rethinking user research for social interaction design

2010-11-28T16:34:00.000-08:00

How many of you have run usability tests that look like this: Individual, one-hour sessions, in which the participant is performing one or more tasks from a scenario that you and your team have come up with, on a prototype, using bogus or imaginary data. It’s a hypothetical situation for the user, sometimes, they’re even role-playing.

Anyone? That’s what I thought. Me too. I just did it a couple of weeks ago.

But that model of usability testing is broken. Why? Because one of the first things we found out is that the task we were asking people to do - doing some basic financial estimates based on goals for retirement - involved more than the person in the room with me.

For the husbands, the task involved their wives because the guys didn’t actually know what the numbers were for the household expenses. For the women, it was their children, because they wanted to talk to them about medical expenses and plans for assisted living. For younger people it was their parents or grandparents, because they wanted to learn from them how they’d managed to save enough to help them through school and retire, too.

There’s a conversation there. There’s a support network there. And that’s what’s broken about usability testing. It always has been.

I first started thinking about this when Google launched Buzz. Buzz used Gmail users’ contacts to automatically generate online social networks to connect users’ most frequent contacts. Google employees - 20,000 of them - had been using Buzz inside the garden walls for a year. A nice, big sample. The problem became evident, however, when Buzz was let into the wild -- almost immediately. One example: A blogger who calls herself Harriet is one of the most famous cases. She wrote about how one of her most frequent correspondents in Gmail was her boyfriend. Another was her abusive ex-husband. Now they were publicly connected, and this made her very, very unhappy. In fact, the post was titled, Fuck You, Google.

There might have been no harm done in the retirement planning study. But there might. Would the 31-year-old who broke down crying in the session because her mother was in late-stage ALS have had a better experience if we’d tested in her context, where she could work with her closest advisor - her dad? Might it have been a calming process, where she felt in control and became engaged in envisioning her independent future because someone she trusted could give her perspective that I could not? Maybe.

As for Buzz, Harriet certainly wasn’t pleased, and she was left with a mess to clean up. How to unconnect two people who were now connected?

When companies started doing usability testing with regularity in the 1980s, it was about finding design problems in what now look like fairly simple UIs that frustrated or hindered users. It was one person, one machine, as the human did a usually work-based task. That’s why it was called computer-human interaction.

But today, technology large and small is fully integrated into peoples’ lives in a much more ephemeral, less compartmentalized way. It is rare to sit next to a corded phone holding the handset only talking and listening.

When you look at what the social web is, there are some characteristics that I think we’re not taking into account very well in doing usability tests:

- It’s about relationships among people
- In context
- Conducted fluidly, across time across time and space, continuously

I also think that people who are new to usability testing and user research are not going to do a good job of testing social interaction design, because constructing a study cannot be very formal or controlled. Measuring what’s happening is much more complex. And scope and scale make a difference. Testing Buzz with 20,000 Googlers for a year wasn’t enough; it took letting it out to a million people who hadn’t drunk the Koolaid to find the *real* problems, the real frustrations, the real hindrances that truly affect uptake and adoption.

The nature of online is social
Let’s back up and talk about a key definition. What I mean by “social” is anything that someone does that changes the behavior of someone else.

This is how I can say that being online *is* social. Email is social. Publishing a flat HTML document is social. Putting something on a calendar is social. Everything is social. Social isn’t the special secret sauce that you pour on top of an experience. Social is already there. Choosing a bank is social. Planning a vacation is social. Buying an appliance is social. I SMSd a series of photos to my boyfriend the other day, of me in different eyeglass frames because I couldn’t decide by myself. This was *not* a computer-centered, or an app-centered interaction. This was a decision being made by two people, a conversation, mediated by fully integrated technology in fluid activities in different contexts for two people. It was social.

Social isn’t sauce. It’s sustenance. It’s already there, and we’re not seeing it. So we’re not researching it, and we’re definitely not testing for it.

We have to stop thinking about human-computer interaction. That model by default is too limiting. Look around. It’s really about human relationships and interactions mediated by technology. Technology is supporting the communication, not driving it. Ask any parent who has used Facetime or Skype to have a video chat with their baby for the first time.

Scale is the game changer
Discount usability testing is great for some things, but what we’re really studying when doing user research and usability testing for the social web is social behavior. And that takes scale. That takes connections. That takes observing people’s real networks and understanding what makes those work, what makes those friends, family, colleagues, neighbors, acquaintances, associates, clients, vendors, pen pals, drinking buddies, partners for life, or friends with benefits.

Those are rich, life-framing relationships that affect how someone interacts with a user interface that most of us are not even scratching the surface of when when we “micro-test” a commenting feature on an online invitation web site.

“Task” doesn’t mean what you think it means
For the retirement planning tool, I did a little interview to start the session that I hoped would set some context for the participant to do the behavior that I wanted to observe. But it was woefully inadequate. Don’t get me wrong, the client wasn’t unhappy; they thought it was a cool technique. But as soon as I learned who the participant went to for financial advice, where was I? Putting the participant in a situation where they had to pretend. They did, and they did a fair job of it. But it was lacking.

But tasks are the wrong unit. What we’re asking people to do in usability tests is like attending a cocktail party while grocery shopping. Even with an interview, even with careful recruiting, it’s incongrous. There are very few discrete tasks in life. Instead there are activities that lead people to goals. Multiple activities and goals might be intermixed on the way to achieving any one of them. In the meantime, the technology is completely integrated, ambient, almost incidental. LIke asking your bf which eyeglass frames look nerdy-sexy, versus just nerdy.

The activity of interest isn’t computer based. Look at retirement planning. It’s *retirement planning*! That’s not the task. The activity is planning for the future, a future in which you have no real idea of what is going to happen, but you have hopes, aspirations.

Using Buzz is not a task. It’s not an activity. People who use Buzz don’t have the goal of connecting to other people, not in that deliberate way. They’re saying, hey, I’ve read something interesting you might be interested in, too. The task isn’t “sharing.” It’s putting themselves out in the world hoping that people they care about will notice. How do you make that scenario in a usability test?

Satisfaction may now equal user control
The ISO measures of usability are efficiency, effectiveness, and satisfaction. What is effective about having Tweetdeck open all day long while you’re also writing a report, drafting emails, taking part in conference calls, attending virtual seminars, going back to writing a report, calling your mother?

When the efficiency measure went into the ISO definition, most people were measuring time on task. But if you don’t have a discrete task, how do you measure time?

Satisfaction may be the most important thing in the end for the social web, and that may be the degree to which the user feels she has control of the activities she’s doing while she’s using your tool. How much is the UI forcing her to learn the tool, versus quickly integrating it into her life?

Measuring success in the social web often defies what we’ve been taught to count for data. How do you measure engagement in the social web? Is it about time on the site? I could lurk on Twitter.com or Facebook all day. Am I engaged? Is it about minutes spent pursuing and perusing content? Is it about how likely someone is to recommend something to someone else? I wrote my first product review, EVER last week, for a pair of jeans on the Lands End web site. Am I engaged with the site? I would say no.

We have to look hard at the goodness of conventional metrics. They’re not translating to anything meaningful, I don’t think, because we’ve been thinking about all this all wrong - or not enough. What is goodness to a user? Control of her life. Control of her identify. Control of her information.

Users are continuously designing your UI
What does Task mean, what does Success mean, how do you measure new features that users create with your UI on the fly? Twitter has hashtags and direct messages. Users created those. Facebook is continuously being hacked for fresh activities. Look at commenting systems on blog posts or articles. Spammers, yes, but people are also talking to one another, arguing, flirting, solving problems, telling their own stories. No matter what you build, and what your intentions were in designing it, users are going to hijack it to make it useful to them. How do you test for that?

Ideas from smart people
I had all these questions and more when I met with a bunch of smart people who have been working in researching the social web. Out of that discussion came some great stories about what people had tried and worked, and what had not worked so well.

For creating task scenarios for usability tests, getting participants to tell stories of specific interactions helped. Doing long interviews helped learn context, scope, priorities, connections. Getting people to talk about their online profiles and explain relationships helped set the scene for activities. Getting them to use their own log-ins with their real relationships helped everyone know whether the outcomes were useful, usable, and desirable. Whether the outcomes were satisfying and even enriching.

Some of the people in this informal workshop also offered these ideas:

Screen sharing with someone outside the test lab or test situation
Making video diaries and then reviewing them retrospectively
Developing and testing at the same time, with users
Including friends or other connections in the same test session, setting up multi-user sessions
Sampling the experience in the same way that Flow was discovered: prompting people by SMS at select or random moments to ask people to report their behavior

There’s also bodystorming, critical incident analysis, co-designing paper or other prototypes. A few things seemed clear through that discussion. To make user research and usability testing useful to designers, we have to rethink how we’re doing it. It’s got to reflect reality a bit better, which means it takes more from social science and behavioral science than psychology. It takes more time. It takes more people. It takes a wider view of task, success, and engagement. And we’re just beginning to figure all that out.

Rethink research and testing

Everything is social. Scale is the game changer. Tasks aren’t what you think they are. User satisfaction may be about control. Users are continuously designing your UI. I invite you to work with me on rethinking how we’re doing user research and usability testing for what’s really happening in the world: fluid, context-dependent, relationships mediated by technology.

I want to thank Brynn Evans, Chris Messina, Nate Bolt, Ben Gross, Erin Malone, and Jared Spool for spending the better part of a day talking with me about their experiences in researching social. These musings come from that cooperative, ahem, social effort.

IDEA 2010 Discount Code

2010-09-07T16:24:00.000-07:00

You could go to the IDEA 2010 conference web site and key in the Konami Code (up up down down left right left right b a) to find out the phrase to enter when you register to get a 10% discount. Or, you could just type in FuzzyBunny in the promo code field.

(By the way, keying in the Konami Code on the web site seems to work in Safari but not in Firefox.)

See you in Philly! There I'll be doing a break out session with Yoni Knoll about one of the oldest design problems around, voting:

Design Studio for World Peace: Designers affecting change in elections

Did you know our 43rd president was chosen because of a decision made about font size? Yep, in 2000, thousands of voters did not vote as they intended because of the way their ballot was designed.

If you're interested in the voting experience, service design, and world peace, come to our breakout. There, you'll get a chance to redesign a ballot used in a real election, discuss the design constraints, and learn how you can get involved in election design where you live.

Researcher as director: scripts and stage direction

2010-08-19T15:24:00.000-07:00

For most teams, the moderator of user research sessions is the main researcher. Depending on the comfort level of the team, the moderator might be a different person from session to session in the same study. (I often will moderate the first few sessions of a study and then hand the moderating over to the first person on the design team who feels ready to take over.)

To make that work, it's a good practice to create some kind of checklist for the sessions, just to make sure that the team's priorities are addressed. For a field study or a formative usability test, a checklist might be all a team needs. But if the team is working on sussing out nuanced behaviors or solving subtle problems, we might want a bit more structure.

A couple of the teams I work with ensure that everything is lined up and that *anyone* on the team could conduct the sessions by creating detailed scripts that include stage direction. Here are a couple of samples:

Whether the team is switching up moderators or it's the same person conducting all the sessions, creating a script for the session that includes logistics is a good idea:

think through all the logistics, ideally, together with the team
make sure the sessions are conducted consistently, from one to the next
back up the main researcher in case something drastic happens -- someone else could easily fill in

Logistics rehearsal

When you walk through, step by step, what's supposed to happen during a session, it helps everyone visualize the steps, pacing, and who should be doing what. My client teams use the stage direction in the script as a check to make sure everything is being covered to reach the objectives of the sessions. It's also a good way to review what tools, data, and props you might need.

Estimate timing

Teams often ask me about timing. When they get through a draft of a script that includes stage directions, they get a pretty solid feeling pretty quickly for what is going to take how long. From this they can assign timing estimates and make decisions about whether they want participants to keep going on a task after the estimated time is reached or redirect to the next task.

Mapping out location flow

It's easy to overlook the physical or geographic flow - what a director would call blocking - of a session. Where does the participant start the session? In a waiting room, at her desk, or somewhere else? Will you change locations within a room or building during the session? How do you get from one place to the next?

Consistency and rigor

Including stage directions in a script for a user research session can help reviewer-stakeholders understand what to expect. More importantly, the stage directions act as reminders to the moderator so she's doing the same things with and saying the same things to every participant in the study. This means nothing gets left out deliberately and nothing gets added that wasn't agreed on ahead of time. (For example, the team could identify some area to observe for and put a prompt in the script for the moderator to ask follow-up questions that are not specifically scripted, depending on what the participant does.)

Insurance

Any really good project manager is going to have a Plan B. With a script that includes detailed stage directions, anyone who has been involved in the planning of a study should be able to pick up the script and moderate a session. The people I worked with at Tec-Ed called this "the bus test" (as in, If you get hit by a bus we still have to do this work).

Some teams I work with want to spread out and run simultaneous sessions. The stage directions can help ensure consistency across moderators. (Rehearse and refine if you're really going to do this.)

Finally, when it comes time to write the report about the insights the team gained, the script -- with its stage directions -- can help answer the questions that often come asking why things were done the way they were done or why the data says what it says.

Stage it

Each person in a session is an actor, whether participant or observer. The moderator is the director. If the script for a study includes instructions for all the actors in the session as well as the director in addition to documenting what words to say, everyone involved will give a great performance.

Voting: The 233-year-old design problem

2010-08-11T11:30:00.000-07:00

Whenever I talk about my involvement in voting and elections, whoever I'm chatting with invariably asks:

Why is it so hard to design a good looking ballot that prevents voters from making mistakes?
Why doesn't the whole country use the same voting system?
Don't they have UX people working on elections?

Design in elections is a wicked problem. And I think designers will be fascinated with learning what the constraints really are. So, I decided to take it to big audience - South by Southwest Interactive 2011. I need your help to get on the program. Please go to the 'panel picker' and vote for my session.

Just to give you a head start, here's the description:

Do you think the "butterfly ballot" was an isolated problem? How did the hanging chad become a world-class design problem? Did you know our 43rd president was chosen because of a decision made about font size? By someone who was not a trained designer? Did you know that the presidential election in 2000 was not the first - or last - time that design problems affected the outcome of an election? If you're trained in design, interested in fair elections, or looking for a way to affect world peace, come to this panel. This is probably the most important panel in the Free World. And we're not kidding.

Please vote Yes, This rocks! before August 27, 2010.

[Yes, this is cross-posted to my blog about ballot design and accessibility. Sorry if that was confusing.]

Is your team stuck in a bubble?

2010-08-02T08:17:00.000-07:00

This happens. The team is heads down, just trying to do work, to make things work, and then you realize it. Perspective is gone.

Recently I gave a couple of talks about usability testing and collaboratively analyzing data. There was a guy in the first row who was super attentive as I showed screen shots of web sites and walked the attendees through tasks that regular people might try to do on the sites.

Sweat beaded on his brow. His hands came up to his forehead in the way that someone who has had a sudden realization reacts. He put his hand over his mouth. I assumed he was simply passionate about web design and was feeling distressed about the crimes this web site committed against its users.

Turns out, he was the web site's owner.

This I found out at a break. When people started filing in from lunch to start the next session, this fellow appeared in my second session. I had time to talk with attendees, so I decided to approach him. "Hi. I noticed you were in my first session. Glad you're back. I hope the first was useful." He said yes, he had found it useful. But he frowned. "You look puzzled. Do you have a question I didn't answer?"

The bubble is insidious

"No," he said. "But it's clear that I have been -- along with a whole lot of other people -- out of touch."

"Oh? You got some insights today, already?"

"Some especially applicable insights, actually. The site you used this morning as your example is the site I work on every day." He gave a sad grin.

I knew this day would come. I would get caught out critiquing or running a demonstration on a site for which the owner was present. That day had arrived.

"I should have talked with you beforehand," I said. "The site has some classic problems. That's why I chose it as an example. It is one of dozens of sites in this domain that have similar issues. If I did or said anything that embarrassed you or your team, I apologize."

He sighed. "Not at all. You can't be embarrassed by something you weren't aware of." He went on, "We hadn't looked at the site at all from the point of view of users outside the organization. We've been in a bubble."

He actually seemed grateful. "Ah. That explains it," I said.

We chatted some more about the political pressures and the technology constraints that his team -- most teams -- faced in creating a great web site and maintaining it. There had been some usability testing on intranets and even on extranets. But it was a few years ago. And the audience for the public-facing web site was different from the internal-facing web apps.

Perspective comes from observing real users doing real stuff

The best tool for resolving disputes within a design team, for making design decisions based on data rather than opinion, is sitting next to someone who is a real person who wants to accomplish something as they use your design to do it.

Some people call this usability testing. Call it whatever you want (except "user testing"). You can make it simple or complex, but when boiled down to its essence there are three ingredients:

- Someone to try out your design.
- Somewhere to test.
- Something to study.

That's it. You can do it by the book, or you can do it very simply and ad hoc. The insights come from observing, first hand. I've seen just an hour of observation get many teams out of their own, customized bubbles.

Supporting great design: features of bubble prevention

Fortunately, my new friend stayed for the second session, in which I gave my recipe for supporting great experiences:

- Each phase includes input from users.

- The team is made up of people each with multiple skills from various disciplines.

- Management of the team is supportive an enlightened about the importance of the user experience.

- Everyone is willing to learn as they go along.

- The team has defined their usability goals and knows how they will measure their success.

Note that of the five attributes, two are directly about perspective (input from users; learning). Another two are about creating an infrastructure for getting and using that perspective (multidisciplinary team; setting usability goals). The remaining one (enlightened management) means there's support for getting and keeping perspective.

The importance of perspective cannot be overstated. Teams that meet with users regularly – every week or every month – turn out great experiences. Observing users regularly, at every phase of a design, gives a team evidence on which to make design decisions. More importantly, that act of being present with users, can bring the team together, enlighten management further, and give a needed break from the rarefied space most of us work in every day.

Get out of your head and into your users' .

Usability isn't just about eliminating frustration anymore

2010-06-18T12:24:00.000-07:00

[This is an excerpt of an article published in UX Magazine on June 16, 2010.]

I'm a devotee of TED talks. I was once assigned to watch several TED talks to deconstruct what made each a good or a bad presentation. TED topics are wide-ranging, though they generally relate to the categories that make up the "TED" acronym: Technology, Entertainment, and Design. I tend to stick to the design topics, but during my research I came across a video of Martin Seligman talking about positive psychology.

Happiness is a topic I've been interested in for a while. According to Darrin McMahon, author of Happiness: A History, happiness is a relatively new construct in the history of humanness. It's only been in the last 250 years or so in the West that we've been safe and healthy enough to think about how we feel emotionally.

After watching Seligman and skimming through McMahon, it occurred to me that the experience design field is undergoing a similar evolution. When I started out at the same time the field was starting out in the early 1980s, teams did usability testing on products to learn where the users would encounter problems, and where they would get lost or frustrated. It was a kind of human factors quality check, and it was often done two seconds before launch. No changes could be made, but these teams could at least incorporate what they found out into training, which almost everyone needed to use technology back then.

Fast forward to somewhere between 1997 and 2005, and the usability testing dynamic shifted. Teams started performing usability testing earlier and earlier in the product development process to inform their designs. Usability professionals were still concerned with identifying issues post facta to eliminate existing problems and frustrations, but they also learned about user behaviors and habits and used this to influence design decisions earlier in the product development cycle. This was a very important step in the evolution of user research. It happened because teams started testing early designs and prototypes and using what they learned to refine designs well before they launched.

Only recently has technology improved, proliferated, and become cheap enough that design is less and less about solving problems, testing less and less about eliminating frustration. It's all becoming more and more about making a good experience for users. Sure, we're still eliminating frustration, but it's happening earlier in the design process. Finally, design is at the table with engineering and business. Now we talk about experience design rather than just usability. Now it's not good enough to just be usable. The design has to fit into peoples' lives. It actually has to make people happy, and anticipate their needs.

It seems we're all aspiring to an ideal of designing for delight, but most of us are landing somewhere short of delight, but at least better than frustrated. Users can use our designs, but they're not excited about it. (Okay, you can't be excited about everything.) We want users to trust our designs, but there's still too much overhead for users to easily reach their goals.

... Read the entire article at UX Magazine.

Overcoming fear of moderating UX research sessions

2010-06-08T10:22:00.000-07:00

It always happens: Someone asks me about screwing up as an amateur facilitator/moderator for user research and usability testing sessions. This time, I had just given a pep talk to a bunch of user experience professionals about sharing responsibility with the whole team for doing research. "But what if the (amateur) designer does a bad job of moderating the session?"

What not to do

There are numerous ways in which a moderator can foul things up. Here are just a few possibilities that might render the data gathered useless:

Leading the participant
Interrupting or intervening at the wrong time
Teaching or training rather than observing and listening
Not following a script or checklist
Arguing with the participant

Rolf Molich and Chauncey Wilson put together an extensive list of the many wrong things moderators could do. There are dozens of behaviors on the list. I have committed many of these sins myself at some point. It's embarrassing, but it is not the end of the world. So, here, let's talk about what to do to be the best possible moderator in your first session.

Your role as a moderator

To be the best moderator you can be, remember that there are three basic roles of the moderator in user research and usability testing. When Carolyn Snyder worked for User Interface Engineering, she codified these:

Flight attendant. Though you might think that your priority is collecting data, your number one job during the session is to see to the comfort and safety of the participant. Make sure this person is comfortable, is appreciated, and knows she can stop at any time. Set up a relaxed situation that is still focused on the goal of learning from the person.

Sportscaster. The line of sight and the acoustics of the session situation aren't always ideal for the observers. Because the observers from your team will be helping you take notes and analyze the data, you can help them by talking just enough so they can keep their places in the session. For example, if the participant is vague about a UI element in pointing out goods and bads, simply echo the last couple of words the participant said to get them to clarify or expand.

Scientist. The moderator is usually the person who designed the study and will be responsible for analyzing the data that comes out of it. This means managing any recordings to ensure the privacy of the participants, tracking notes and data gathering from observers, and pulling observations and data together so the team can come to a design direction based on the evidence gathered.

(Hat tip to Carolyn Snyder and Jared Spool for the moderator roles.)

Who should moderate UX sessions?

Who makes a good moderator? Anyone who is a quick learner, is a good listener, can build rapport with a participant, and has a good memory. Typically, there isn't a lot of time to know all the nuances of a UI before going into a usability test. Likewise, if you're in the field doing basic ethnographic research, you may learn characteristics of the participants or the environment that inform the rest of the interview direction. Handling those on-the-fly perceptions will help everyone get value out of the session.

The listening is important for asking insightful follow-up questions as participants think aloud. Getting clarification on comments made, drilling in a bit to get to specifics, and always keeping in mind "why is this behavior happening" will come to you from listening and (gently) questioning.

Rapport with the participant is key to creating trust. The participant is always trying to get a reading from you about whether what he's doing is correct and whether what he's giving you is what you want. Even a newbie to moderating can be friendly, objective, and neutral at the same time. (It may take some practice.)

Remembering what happened early in the session will help you ask useful follow up questions later in the session. Remembering the main, interesting behaviors will help you work with the observers after the session is over and you're all telling the story of what happened, especially if you have assigned someone else to take notes while you concentrate on running the session.

If there's someone on the design team besides you who has these characteristics, that's who you want to moderate sessions, no matter what their regular job is.

How to be a great UX moderator

Keeping those roles and attributes in mind, this is what I tell clients and workshop attendees about how to be a good moderator. You can pass the list below to your team's candidate.

Be willing to let go of ownership of the design. When you're doing field research, you may enter the session with design ideas in mind. Try not to. Instead, let the heft of the data over sessions help build the ideas. If you're testing a design, as soon as you put a design in front of another person, you no longer own it, the person you're showing it to or who is using it owns it. In that act, you have specifically asked for reactions and interactions. Open yourself up to the possibilities.

Shut up. After you've explained the purpose of the session, and explained your role, and the roles of the other people in the room, stop talking. Even when there are silences, don't be too quick to fill them. Wait. Count to 20 slowly and silently before you say anything. Chances are, something interesting has happened by then and you won't have to open your mouth.

Listen. This is not the same as shutting up. Listening is about being present. Be fully attentive so you can not only hear the words, but process their meaning. Be empathetic to the participant and what she's trying to do.

Suspend judgment. This is one of the hardest things to do, but it is also the most important. You have invited the participant to help you learn about what you're designing. If you have shut up, and listened well, and the participant is appropriate for the study, then let go of assessing what is happening in that moment. Give yourself time to process later. This will also prevent you from asking inappropriate questions during the session that may betray your feelings about a participant or what she has to say about the design.

Plan ahead. Script, create checklists, and read Beth Loring and Joe Dumas's book, Moderating Usability Tests. You may feel silly using a script, but really, there's absolutely nothing wrong with doing that. Having a script to follow means you say the same things the same way to every participant. You can also ensure that you've hit everything on the list that the team wanted to learn about. Finally, scripting out what to say and thinking through the checklist of focus questions will give everyone a better feeling for how much can reasonably be covered during a given session.

Rehearse. If you don't set up a pilot session, then your first "real" session will end up being the rehearsal. Practice in a dry run by yourself, out loud. Record it. Change the script if you need to, making sure that the words you say make sense and feel authentic. Then find someone down the hall or in the next cube to play your participant and try the script out again. Make changes if you need to.

Do enough sessions. As you moderate each session, you will get better at it. Remember, it's not about you. Though you may feel awkward doing this in front of your team, and reading from a script, they're not paying attention to you, they're paying attention to the participant. If you feel like you have made a mistake – you said the wrong thing, or asked a question the wrong way, or you led the participant somehow, keep going. And then, go do another session. You don't have to throw everything out from the session you made a mistake in. Salvage what you can and move on.

It's a chance for team members to get closer to the participants

The whole object of doing user research is for the team to learn about current experiences. More data, even data gathered sloppily, is better than a tiny bit of data gathered expertly. And the new moderator will get better at it. No one is born to the role; moderating well is a set of learned skills. And I think that anyone can learn them with time, practice, and coaching. Find your next moderator on your design team. And keep up the good work.

Other resources you might find useful:

Common Problems in Usability Test Facilitation by Rolf Molich and Chauncey Wilson

Moderating Usability Tests: Principles and Practices for Interacting (Interactive Technologies), by Joseph S. Dumas and Beth A. Loring

Remote Research: Real Users, Real Time, Real Research (Rosenfeld Media), by Nate Bolt and Tony Tulathimutte

Seven Common Usability Testing Mistakes, by Jared Spool

Moderating with Multiple Personalities: 3 Roles for Facilitating Usability Tests, by Jared Spool

Usability Tools Podcast: Moderating Usability Tests, Part 1

Jared Spool and Brian Christiansen

Effectively Moderating Usability Tests, with Beth Loring

Recorded virtual seminar from UIE ($149)

Making sense of the data: Collaborative data analysis

2010-04-26T16:29:00.000-07:00

I've often said that most of the value in doing user research is in spending time with users -- observing them, listening to them. This act, especially if done by everyone on the design team, can be unexpectedly enlightening. Insights are abundant.

But it's data, right? Now that the team has done this observing, what do you know? What are you going to do with what you know? How do you figure that out?

The old way: Tally the data, write a report, make recommendations

This is the *usual* sequence of things after the sessions with users are done: finish the sessions; count incidents; tally data; summarize data; if there's enough data, do some statistical analysis; write a report listing all the issues; maybe apply severity ratings; present the report to the team; make recommendations for changes to the user interface; wait to see what happens.

There are a couple of problems with this process, though. UXers feel pressure to analyze the data really quickly. They complain that no one reads the report. And if you're an outside consultant, there's often no good way of knowing whether your recommendations will be implemented.

And, the researcher owns the data. I say this like it's a problem because it is. Although the team may have observed sessions and now have some image of the users in their heads, the researcher is responsible for representing what happened by reporting on the data and drawing conclusions. The users are re-objectified by this distance between the sessions and the design direction. And, the UXer is now in the position of *suggesting* to designers what to do. How well is that working for you? Teams I work with find it, well, difficult.

The better way: Tell stories, come to consensus on priority, discuss theories

Teams that consistently turn out great experiences do one cool thing with data from user research and usability testing. They Talk To One Another. A lot. That's the process.

Okay, there's a slightly more systematic way to approach collaborative analysis. I've happened on a combination of techniques that work really well, with a major hat tip to User Interface Engineering, which originated most of the techniques in this process. As I've tried these techniques with teams, I've monkeyed with them a bit, iterating improvements. So here's my take:

- Tell stories

- Do a KJ analysis on the observations from the sessions

- Explore the priority observations

- Brainstorm inferences

- Examine the weight of evidence to form opinions

- Develop theories about the design

Tell stories to share experiences with users

This is the simplest thing, ever. These teams can't wait to tell their teammates what happened in sessions. Some teams set up debrief scrums. Some teams send around emails. Some teams use wikis or blogs. The content? A 300-word description of the person in the session, what she did, what was surprising, and anything else interesting about what the observers heard and saw. (Ideally, the structure for the story comes from the focus questions or answers the research questions that the study was designed for.)

Do a KJ analysis to come to consensus on priority issues

KJs, as they're affectionately known by devotees, are powerful, short sessions in which teams democratically prioritize observations. There are two keys to this technique. First, there's no point in being there unless you've observed sessions with users. Second, because there's no discussion at all until the last step, every CxO in the company can be there and have no more influence on the design than anyone else in the room. This lack of discussion also means that the analysis happens super fast. (I've done this with as many as 45 people, who generated about a thousand observations, and we were done in 45 minutes.)

Explore the priority observations

What did the team see? What did the team hear? After the KJ bubbles the priorities up, either with the whole group of observers or with a subset, pull the key observations from the issues and drill in. Again, this is only what people heard and what they saw (no interpreting). The team usually will have different views on the same issue. That's good. Getting the different perspectives of business people, technologists, and designers on what they observed will get things ready for the next step.

Brainstorm inferences

These are judgments and guesses about why the things the team observed happening. The question the team should be asking themselves about each observation is What's the gap between the user's behavior and what the user interface design supports?

So, say you saw users click choices in a list of items that they wanted to compare. But they never found the button to activate the comparison. When the team looks at the gap between what users did or what they were looking for and how the UI is designed, what can you infer from that? Don't hold back. Put all the ideas out there. But remember, we're not going to solutions, yet. We're still guessing about *What happened*.

Examine the weight of evidence to form opinions

By now the team has pored through what they heard and what they saw. They've drawn inferences about what might be happening in the gap between behavior and UI. *Why* are these things happening?

A look at the data will tell you. Note that by now you're analyzing a relatively small subset of the data from the study because through this process you've eliminated a lot of the noise. The team should now be asking, How many incidents were there of the issue? Which participants had the issue? Are there any patterns or trends that give more weight to some of the inferences the team came up with than other inferences?

By collaborating on this examination of the weight of evidence, the team shares points of view, generates feasible solutions, and forms a group opinion on the diagnosis.

Develop theories about the design

By now the team should have inferences with heft. That is, the winning inferences have ample data to support them. Having examined that evidence, the team can easily form a consensus opinion about why the issue is an issue. It's time to determine a design direction. What might solve the design problem?

In essence, this decision is a theory. The team has a new hypothesis -- based on evidence from the study they've just done -- about the remedies to issues. And it's time to implement those theories and, guess what, test them with users.

Did you see the report?

Look, Ma, no report! The team has been involved in all of the data analysis. They've bought in, signed up, signed off. And, they're moving on. All without a written report. All without recommendations from you.

What's valuable is having the users in the designers' heads

Getting the team to spend time with users is a first step. Observing users, listening to users will be enlightening. Keeping the users in the heads of designers through the process of design is more difficult. How do you do that? Collaborate on analyzing observations; explore inferences together; weigh the evidence as a group. From this, consensus on design direction reveals itself.

+ + + + + + + + + + + + + + + +

I stole all these ideas. Yep. User Interface Engineering gets all the credit for these awesome techniques. I just repackaged them. To see the original work, check out these links:

Group Activities to Demonstrate Usability and Design
The KJ-Technique: A Group Process for Establishing Priorities
The Road to Recommendation

But this isn't going to stop me from talking about these techniques. In fact, I'm going to talk about them a lot. So come learn more at these events:

UX Lx, Lisbon, Portugal, May 12-14. "Making Smart Design Decisions" on Wednesday, May 12 at 3:30pm. http://www.ux-lx.com/index.html
Wharton UIConf 2010, July 21-22. "Making Sense of Usability Results" and "Usability Testing in the WILD" on July 22 at 10:45am and 1:30pm, respectively. http://web.wharton.upenn.edu/uiconf2010/

You are not your user. No matter how good you think you are.

2010-02-16T09:35:00.000-08:00

Listen up, people. This is why -

- quantity is not quality
- you are not your user.

The lesson for today on participant sampling is Google Buzz. Google has been working on Buzz for some time. And it's a cool idea. Integrating the sharing of photos, status updates, conversations, and email is a thing a lot of us have been looking for. Buzz makes lots of automatic connections. That's what integrating applications means.

BUT. One of the features of Buzz was that it would automatically connect you to people whom you have emailed in Gmail. On the surface, a great idea. A slick idea, which worked really well with 20,000 Google employees.

Large samples do not always generate quality data
Twenty thousand. Feedback from 20,000 people is a lot of data. How many of us would kill to have access to 20,000 people? So. How can such a large sample be bad? Large samples can definitely generate excellent data on which to make superfine design decisions. Amazon and Netflix use very large samples for very specialized tests. There's discussion everywhere, including at the recent Interaction10 conference in Savannah, about cheap methods for doing remote, unmoderated usability testing with thousands of people. More data seems like a good idea.

If you have access to 20,000 people and you can handle the amount of data that could come out of well designed research from that sample, go for it. But it has to be the right sample.

Look outside yourself (and your company)
Google employees are special. They're very carefully selected by the company. They have skills, abilities, and lives that are very different from most people outside Google. So, there's the bias of being selected to be a Googler. And then there's indoctrination as you assimilate into the corporate culture. It's a rarified environment.

But Google isn't special in this way. Every organization selects its employees carefully. Every organization has a culture that new people undergo indoctrination and assimilation for, or they leave. In aggregate, the people in an organization begin to behave similarly and think similarly. They aspire to the same things, like wanting products to work.

But what about 37 Signals and/or Apple? They don't do testing at all. (We don't actually know this for sure. They may not call it testing.) They design for themselves and their products are very successful in the marketplace. I think that those companies do know a lot about their customers. They've observed. They've studied. And, over time, they do adjust their designs (look at the difference in interaction design in the iPod from first release in 2001 to now). Apple has also had its failures (Newton, anyone?).

The control thing
By not using an outside sample, Google ran into a major interaction design problem. About as big as it gets. This is a control issue, not a privacy issue, though the complaints were about over sharing. One of the cardinal rules of interaction design is to always let the user feel she's in control. By taking control of users' data, Buzz invaded users' privacy. That's the unfortunate outcome in this case, and now, users will trust Google less. It's difficult to regain trust. But I digress.

The moral of today's lesson: Real users always surprise us
Google miscalculated when it assumed that everyone you email is someone you want to share things with, and that you might want those people connected to one another. In a work setting, this might be true. In a closed community like a corporation, this might be true. But the outside world is much messier.

For example, I have an ex. He emails me. Sometimes, I even email him back. But I don't want to share things with him anymore. We're not really friends. I don't want to connect him to my new family.

Even testing with friends and family might have exposed the problem. Google has a Trusted Tester program. Though there are probably some biases in that sample because of the association with Google employees, they are not Google employees. This makes friends and family who use Gmail one step closer to typical users. But Google didn't use Trusted Testers for Buzz.

You get to choose your friends in real life. Google could have seen this usage pattern pretty quickly just by testing with a small sample who live beyond the Google garden walls.

User interfaces make Fast Company's biggest design moments of the last decade

2010-01-15T10:21:00.000-08:00

Of the 14 items that Fast Company chose to include in its selection of the biggest design moments of the years 2000-2009, there were five that were notable for their importance as breakthroughs in user interface and user experience design.

Three were specifically related to technology. I'm guessing that most of us will not think of the others as either technology or as part of a user experience design.

iPod

There was, of course Apple's iPod, introduced in 2001. The iPod isn't a great piece of audio technology, and never has been. But it is a cool, fun, fashion statement that integrates nearly seamlessly with Apple's other products, especially iTunes. So there's the experience. But the iPod also created a new look for selecting songs on a small device, which interestingly, has evolved over the decade from a multi-button (circle within a circle) user interface physical click to a slick, integrated multi-purpose magic circle that's closer to gestural.

2001
2009

Apple.com

Wii

In 2006 the Wii rocked video gaming by making games interactive in a very different way – no more joy stick, no more keyboard – and creating games that inspired people to be more physically active.

Nintendo.com

Multi-touch screens and gestural interfaces, large and small

Can you say, "I love my iPhone"? I do. Giving the feeling of interacting more directly with the phone, music, games, photos, and other applications, touch screens are finding their way into nearly every UI with a screen. Always good? Maybe not. But it's a good experiment, probably on the way to something better. I just wish someone could figure out a way for me to use my iPhone when it's 30 degrees out and I have gloves on.

multi-touch-screen.com

Blackberry.com

The above technologies all fit neatly into many lives these days and affect a much larger user experience. These elegant UIs are on things that most people would consider non-essential. Design made big inroads in healthcare, democracy, and sustainability, too. But there were two more important user interface design accomplishments featured by Fast Company that made huge differences in how many people take medicine and how many more will vote in elections in the future.

Clear Rx from Target
Deborah Adler created a beautifully designed package for Target's Design For All campaign in 2005. But the package is just a vehicle for clearer, more readable labels with larger type that make taking medicine safer. In addition, Adler created other effective and beautiful, brilliant ways of identifying which drug is in the bottle and who should be taking it, with cleverly labeled the tops for quick recognition, and color coding by family member.

Target.com

Design for Democracy

Perhaps the most important design moment of the last decade was the US presidential election in 2000 because it inspired the AIGA to institutionalize a project that had started in 1998 called Design for Democracy. The project was commissioned by the US Elections Assistance Commission to create design guidelines, templates, and specifications for optical scan ballots, election signage, and other election-related materials. I am very proud of having been involved as a consulting expert on the research led by Mary Quandt for the Effective Designs for the Administration of Federal Elections to develop the guidelines, which were published in 2007.

Call your county elections office and ask them to adopt the guidelines now. Offer to help them do it. Counties that have already adopted these best practice design guidelines for ballots have encountered happier, more confident voters, and lower costs for litigation and recounts – improving the experience of elections all the way around.

aiga.org/design-for-democracy

May Design have fewer serious problems to solve and more fun to create in the new decade.

EDITED 1/21/2010 to add:
Scot Marvin wrote to tell me that his brilliant wife, Tara Starr, had cleverly doctored a pair of regular, knitted gloves to use with her iPhone, by sewing lovely flowers on the fingertips using conductive thread. The thread produces a working connection between the finger and the iPhone through the glove. Love it.

Plain language is important. Reward the best of it.

2010-01-13T11:49:00.000-08:00

The Center for Plain Language just announced its first awards program.

Nominations are due Feb. 15, and you can find the nomination forms here: www.centerforplainlanguage.org/awards

It's a simple form to fill out, and this year there is no entry fee.

The Center is calling its award for good plain language and design the ClearMark Award.
There will also be an award for confusing language, called the WonderMark Award. (Hmmm. wonder what that could mean?)

The center is looking for submissions from anyone who has worked on, read, or knows of outstandingly good or unclear language use in government (any level), nonprofits, and corporate America.

For the WonderMarkAward, we are especially interested in humorous examples or examples that you know caused someone harm because of a lack of clarity.

The Center will present the awards at the first ClearMark Awards ceremony at the National Press Club, Thursday evening, April 29. We expect to feature a comedic entertainer, so please join us for an evening of satiric fun. A link to purchase tickets will go up on the website soon.

If you have any questions, please contact Annetta Cheek at acheek@patriot.net.

What to do with the data: Moving from observations to design direction

2009-12-07T11:07:00.000-08:00

What is data but observation? Observations are what was seen and what was heard. As teams work on early designs, the data is often about obvious design flaws and higher order behaviors, and not necessarily tallying details. In this article, let's talk about tools for working with observations made in exploratory or formative user research.

Many teams have a sort of intuitive approach to analyzing observations that relies on anecdote and aggression. Whoever is the loudest gets their version accepted by the group. Over the years, I've learned a few techniques for getting past that dynamic and on to informed inferences that lead to smart design direction and creating solution theories that can then be tested.

Collaborative techniques give better designs

The idea is to collaborate. Let's start with the assumption that the whole design team is involved in the planning and doing of whatever the user research project is.

Now, let's talk about some ways to expedite analysis and consensus. Doing this has the side benefit of minimizing reporting – if everyone involved in the design direction decisions has been involved all along, what do you need reporting for? (See more about this in the last section of this article.)

Some collaborative analysis techniques I've seen work really well with teams are:

- Between-session debriefs

- Rolling issues lists

- K-J analysis

- Cross-matching rolling issues lists with K-Js

Between-session debriefs

Do you just grind through sessions until you're through them all, only to end up having an excruciatingly long meeting with the team where you're having to re-play every session because no one was there but you?

Schedule extra time between sessions

Try this: Schedule more time than usual between sessions. If you usually schedule 15 minutes between usability test sessions, for example, then next time, schedule 30 minutes. Use the additional time to debrief with observers.

If the team sees that there will be discussion in between the sessions that will help move the design forward, they're more engaged. If team members are already observing sessions, then this gives you a chance to manage the conversations that they're already having.

Knowing that you're going to want to debrief between sessions, the team is more likely to come to more sessions and to pay full attention. They'll learn that if they're at the sessions, they get more say in the design outcome, and the design outcomes will make more sense. If they don't attend, they don't get as much say, simply because they've observed less and have less evidence for their inferences.

All you have to do is get the team to talk about what they saw and what they heard, and what was most surprising about that. Save the design solutions for later, unless you're doing rapid iterative testing.

Play 'guess the reason'

To get teams in the practice of sticking to discussing observations rather than jumping to design conclusions, I've tried playing a game called "Guess the Reason" with them. It's easy. Show a user interface – just one page or screen or panel – and describe the behavior observed. Then ask the team to guess why that happened. It's a brainstorming activity. The first person to go to a design solution has to put money in the team's drink fund. You can use the same system during your own debriefs, which can make it fun (and profitable).

Rolling issues lists

I've written about these before. Simply put, this technique gets the team further engaged in the collecting of observations and takes the burden off the moderator/researcher.

Gather whiteboard and markers

The idea is that those observations that come out in the debrief get written down on a white board that all the observers can see. Each observation gets tracked to the participants who had the issues. As the moderator, you start the list, but as the sessions go on, you encourage the team to add their own.

Natural consensus through debrief disucssion

As team members add, and the team talks about the observations that go onto the list, there's a natural consensus building that goes on. Does everyone agree that this is something we want to track? Does everyone agree that this way of talking about it makes sense to everyone?

Draws out what is important to the team

When I moderate user research sessions, doing this often means that I don't have to take notes at all because the team is recording what is important to them. As they're doing that, I also get to see what is important to the different roles on the team.

K-J analysis

I admit that I stole this idea from User Interface Engineering (UIE). But it's one of the most powerful tools in the collaboration toolbox. Jared Spool has an excellent article (that doubles as a script) about this technique.

When I do K-Js in workshops, everyone gets really excited. It's an amazing tool that objectively, democratically identifies what the high priority items are from subjective data.

The technique was invented by Jiro Kawakita to help his co-workers quickly come to consensus on priorities by getting them to discuss only what was really important to the whole team. There are 8 steps:

1. Create a focus question. For a usability test, it might be, "What are the most important changes to make to improve the user experience for this design?" In workshops, I often choose a more philosophical question, like, "What obstacles to teams face in implementing user experience design practices in their organizations?"

2. Get the group together. When I use this technique with teams at the end of a user research project, I invite only people who observed at least one session.

3. Put data or opinions on sticky notes. For the user research focus question, I ask for specific, one-sentence observations that are clear enough for other people to understand. (Team members often bring their computers with them to go through the notes they took during sessions.)

4. Put the sticky notes on the wall. Everyone puts their sticky notes up, in random order on one wall, while reviewing what other people are also putting on the wall. Allow no discussion.

5. Group similar items. This step is like affinity diagramming. Pick up a sticky; find something that seems related to it. Move to another wall, and put those two stickies on the wall, one above the other to form a column. All team members do this step together. Keep going until all the stickies have moved from one wall to the other and all the stickies are in a column. No discussion.

6. Name each column. Using a different color of stickies now, everyone in the room writes down a name for each group and puts their name on the wall above the appropriate column. Everyone must name every column, except if someone else has already stuck up a name that was exactly what you had written down. No discussion.

7. Vote for the most important columns. Everyone writes down what they think are the 3 most important columns. Next, they vote by marking 3 Xs for their most important group, 2 Xs for the second most important, and 1 X for their third most important group. Again, no discussion.

8. Tally the votes, which ranks the columns. On a flip chart or a white board, number a list from 20 to 1. Pull all the column name stickies that have votes and stick them next to the number of votes that are on the sticky. Now the facilitator can read off to the team which groups had the highest votes and thus are the highest priority. Now is the opportunity for discussion as the team determines which stickies can be combined. The decision to combine stickies – and thus, what the most important topics are – must be unanimous.

You're done. Very cool. Now the team knows exactly what to focus on to discuss, resolve, and remedy. And, if you're doing a report, you now know what to bother to report on. (See what I meant about reporting?)

Cross-match the rolling issues with the K-J

If your team or your management is into validation, you can now go back to your desk and compare what came out of the rolling issues with what the K-J generated. My experience so far has been that they match up. And it isn't because everyone at the K-J was primed by being at all the debriefs between sessions. People who observed remotely often contribute to the K-Js live, so you'd think that their data might change the K-J results. Your mileage may vary, but so far, mine matches up.

Directed collaboration is fun and generates better design solutions

When you help the team review together what they saw and heard during user research sessions, there is more likely to be consensus, buy-in, and a shared vision of the design direction. In testing early designs especially, consensus, buy-in, and shared vision are crucial to ending up with great user experiences. Collaborative techniques for analyzing observations turn work into fun, and take the pressure off the researcher to generate results. Because everyone on the team was involved in generating observations and setting priorities, everyone can move on quickly to making informed decisions that lead to coordinated, smart designs.

P.S. To anyone who gets my email newsletter whose email address was in the CC field rather than the BCC field: I apologize. The production manager (me) must still have been asleep, and the QA manager (me) didn't catch that. We'll try to get it right next time.

Easier data gathering: Techniques of the pros

2009-10-21T09:51:00.000-07:00

In an ideal world, we'd have one person moderating a user research session and at least one other person taking notes or logging data. In practice it often just doesn't work out that way. The more people I talk to who are doing user research, the more often I hear from experienced people that they're doing it all: designing the study, recruiting participants, running sessions, taking notes, analyzing the data, and reporting.

I've learned a lot from the people I've worked with on studies. Two of these lessons are key:

- Doing note taking well is really hard.

- There are ways to make it easier, more efficient, and less stressful.

Today, I'm going to talk about a couple of the techniques I've learned over the years (yes, I'll give credit to those I, um, borrowed from so you can go to the sources) for dealing with stuck participants, sticking to the data you want to report on, and making it easy to see patterns.

Graduated prompting

Say you're testing a process that has several steps and you want to see the whole thing, end-to-end. This is not realistic. In real life, if someone gets stuck in a process, they're going to quit and go elsewhere. But you have a test to do. So you have to give hints. Why not turn that into usable data? Track not only where in the user interface people get stuck, but also how much help they need to get unstuck.

This is also an excellent technique for scavenger hunt tasks – you can learn a lot about where the trigger words are not working or where there are too many distractions from the happy path or people are simply going to need more help from the UI.

Here's what I learned from Tec-Ed about what to do when a participant is stuck but you need them to finish:

- First, ask participants to simply try again.

- If participants are unable to move forward, give a hint about where to look: "I noticed that you seem to be focused mostly in this area (pointing). What if you look elsewhere?"

- If participants are still stuck and want to give up or say they would call someone, let them call a "help desk" or, depending on the study, give a stronger hint without being specific.

- Finally, you may have to get specific.

The idea is to note where in the UI you're giving the hints and how many for any particular hindrance. This gives you weighted evidence for any given participant and then some great input to design decisions as you look at the data across participants.

Pick lists

You may say this is cheating. But don't you feel like you have a pretty good idea of what's going to happen when a participant uses a design? This technique is about anticipating what's going to happen without projecting to participants what the possibilities are. Make a list of all the possible wrong turns you can imagine. Or at least the ones you care about fixing.

Being able to do this comes from awareness and the researcher's experience with lots of user interfaces. This is not easy to do if you've only done one or two studies. But as you get more observations under your belt, looking ahead gets easier. That is, most of us are paying attention to the happy path as the optimum success in a task, but then have to take lots of notes about any deviation from that path. If you look at what the success and error conditions are as you design a study, you can create list to check off to make data gathering quicker and less taxing as you're doing both that and moderating.

Here's an example from a study I did with Ginny Redish researching the language of instructions on ballots. This is on a touch screen, so "touch" is the interaction with the screen, not with an actual candidate:

There are a lot of things wrong with this particular example: having the same word at the beginning of many of the possible selections does not make the list easy to skim and scan; there are too many items in the list (we ended up not using all of those error conditions as data points). As we designed the test, we were interested in what voters did. But as we analyzed the data and reported, we realized that what didn't matter for this test as much as whether they got it right the first time or made mistakes and recovered or made mistakes and never recovered. That would have been a different pick list.

But all was not lost. We still got what we wanted out of the pick lists. This is what they ended up looking like as note-taking devices:

Judged ratings

Usually in a formative or exploratory study, you can get your participants to tell you what you need to know. But sometimes you have to decide what happened from other evidence: how the participant behaved, what they did to solve a problem or move forward, where they pointed to.

As a researcher, as a moderator, you're making decisions all the time. Is this data? Is it not? Am I going to want to remember this later, or is it not needed?

After we realized that we were just going to make a judgment anyway, Christine Perfetti and I came up with a shortcut for making those kinds of decisions. Really, what we're doing is assisting judgments that experienced researchers have probably automatized. That is, after dozens or hundreds of observations, you've stored away a large library of memories of participant behaviors that act as evidence of particular types of problems.

To make these on-the-fly judgments, Christine and I borrowed a bunch of techniques from Jared Spool at UIE and used variations of them for a study we worked on together. As the moderator of the day asked, "What do you expect to happen when you click on that?" and followed up with, "How close is this to what you expected?" the note taker for the day recorded something like this:

Another way to use this trick is to ask, "The task is [X]. How close do you feel you are now to getting to where you want to be? Warmer? Cooler?"

I also think that most of us collect too much data. Well, okay, I often do. Then I wonder what to do with it. I've found that when I really focus on the research questions, I can boil the data collecting down significantly. So here's a minimalist note-taking device: I created a one-sheet data collector that covered three tasks and helped me document a pass/fail decision for voting system documentation. You can quibble about some of the labeling in the example below, but I was happy to have one piece of paper that collected what happened along with how I know that, and what that combination of things happening means.

It attempts to encapsulate the observation-inference-theory process all in one place.

Again, if you haven't done much user research or usability testing, you may not be happy with this approach. And, let's not forget how valuable qualitative data like quotes is. But you more experienced people out there may find that codifying the judgments you're making this way makes it much quicker to note what happened and what it might mean, expediting analysis.

Shortcuts are not for sissies

Most user research is not run in an ideal world. Note taking in user research is one of the most difficult skills to learn. Luckily, I have had great people to work with who shared their secrets for making that part of the research less stressful and more efficient.

Graduated prompting is a way to quantify the hints you give participants when you need them to complete tasks in a process or continue on a path in a scavenger hunt.

Judged ratings are based on observations and evidence that fall into defined success and error criteria.

Got several dozen hours under your research belt? Focus the data collecting. Try these techniques for dealing with stuck participants, sticking to the data you want to report on, and making it easy to see patterns.

:: :: :: :: :: :: :: :: NEWS FLASH :: :: :: :: :: :: :: ::

Come to UI 14 in Boston November 1-3 and get a discount on me. Type in promotional code CHISNELL when you register and you'll get $50 off each day. If you sign up for all three days, you'll also get a set of Bose headphones. Sweeeet.

Do it! You know you want to. Mastering the Art of User Research.

Getting the right people in the room: Recruiting

2009-09-18T08:41:00.000-07:00

Nothing is more important to a study than the appropriateness of the participants. But many of us actively resist being involved in recruiting. Is it fear? Is it loathing? Is it the time it takes?

On September 30th, I'm giving a UIE virtual seminar about getting the right participants in the room. I'll talk about recruiting for user research and usability tests:

how to get participants to help you find other participants
how to source creatively
how to eliminate no-shows
why you (as the researcher) should really consider doing the recruiting yourself.

Watch a preview! Then sign up. (Check out the bundle UIE has put together: My virtual seminar, along with Beth Loring's upcoming virtual seminar about moderating usability tests, along with a bonus report from UIE, Recruiting without fear -- for an excellent price.)

September 30, 1:30pm Eastern Time. See you then!

Mastering the art of user resarch - a new seminar for UI 14

2009-09-17T14:46:00.000-07:00

This month, we're back to school. No, I'm not doing a graduate degree. But I am doing a lot of teaching. Today it's UI 14 that I want to tell you about: a special seminar I've developed, and a discount on registration.

Mastering the Art of User Research:
Stress-free, Advanced Techniques for Creating Great Designs

For folks who feel solid on basic methodology for usability testing and user research, I've developed "Mastering the Art of User Research" just in time for User Interface 14 (or UI 14 if you're on friendly terms with UIE, producers of the conference), being held in Boston, Massachusetts on November 1-3.

I developed this seminar because I was thinking of you.

You know there's an art to making research valuable to the team and integral to design. You know how to design and conduct a good study, but you're thinking there just have to be shortcuts to make the process smarter. I bet you're the kind of person who is always teasing out ways to get the most out of every opportunity you have to be with users -- without trading off rigor.

In this seminar, I'll step through advanced techniques that will help you master your craft. You'll learn ways seasoned pros speed up planning, data collecting, and reporting. These technques will lighten your load, net you more and better data than you're getting with your current practices, and -- the best part -- will get the team excited about the design direction the data suggests.

Because you're actually asking more of your participants and your team, you'll have better, richer relationships with them. Participants will help you learn more than ever. The team will help you know what they care about and what to do about it.

You'll learn how to:

- Get participants to recruit and document data for you.
- Get a head start on data analysis during sessions.
- Get consensus on observations on the fly.
- Get the design team to prioritize issues in record time.
- Generate short, focused communications that will help the team internalize results.

Come having practiced the classic processes for planning, designing, carrying out, and analyzing data for usability tests and the more common user research methods and techniques. This is not learning to cook cheese soup in Home Ec. This is advanced knife skills and saucing at the Cordon Bleu. This is riffing on a classic recipe to create something that is easier, quicker, tastier. I'm thinking 30-minute cassoulet.

Registration discount: Use my name and get $50 off of each day of registration

When you go to www.uie.com/events/uiconf/2009/register to sign up for UI 14, in the Promotion code box, just type in CHISNELL. If you register for all three days of the conference, you get $150 off the full registration AND a set of Bose headphones. Very cool.

UI 14 has an awesome line-up of speakers. Check out the program and register with the CHISNELL promotion code here: http://www.uie.com/events/uiconf/2009/.

See you in Boston in November.

Thanks for letting me interrupt you briefly. Now, back to our regularly scheduled program.

Beyond frustration: 3 levels of happy design

2009-08-20T08:11:00.000-07:00

Most of us in the UX disciplines are here: Users can use the design, but they're not excited about it. You already know about eliminating frustration. The team is constantly working to remove obstacles and hindrances that prevent users from reaching their goals. Ideally, through remedying those problems, you'll gain users' trust. But it's more than that. The sites we'll look at here know that trust is tied to how well the design team anticipates users' needs.

Isn't it your secret wish that you'd like users to be in love with your design? You'd like them to enjoy using it. Look forward to using it. Maybe even miss it when they're not able to get to it.

Creating remarkable design
The thing is, creating a lovable, remarkable experience versus a just satisfactory one can seem unattainable. As Jared Spool has said, "Too often, we choose a design because it's doable, not because it's the best we could do." We want to make the ship date. Or there are internal or external pressures or constraints on the team and the design.

But there are companies making their customers happy – so happy that the users engage willingly, sometimes even with longing. It comes down to three levels of happiness, according to psychological research done by Martin Seligman and others. The levels of happiness are progressive and commingled, but they come down to these:

Mindfulness
Flow
Meaning

Let's talk about how this manifests in user experience design.

Three levels of happiness in user experience design

Mindfulness in design is about a pleasing awareness. In relationships, it can mean infatuation. It's knowing that this is good, it makes me happy. It's satisfying.

Flow, from work done by Mihaly Csikszentmihalyi, comes from immersion, absorption, the beta waves rocking the brain so you forget about the task you're doing to become one with what is happening. There's no time between tool time and goal time.

Meaning comes from a feeling of fellowship, contributing, and making the world a better place. As I say, I didn't make this up. Most of these ideas come from Martin Seligman, with help from Don Norman, and Stuart Brown. (Read to the end of the newsletter for links to sources from them that inspired me). It occurred to me after studying them and watching users perform with designs, that there was a connection there.

A counter example
First, let's talk about what not to do.

Here, the traveler is checking in for a flight online. The page she's just come from has asked all the necessary questions about carrying hazardous materials, checking bags, and so on. Wanting to move on to getting the best possible seat, there's this:

Note the defaults. There are interesting interactions here that go well beyond frustrating to downright annoying. Yeah, let's buy a couple of thousand "award" miles right now, which is an "option." What happens when our traveler clicks the Add option button without thinking? She's sent to a shopping cart. Further and further away from the goal of checking in to a flight. Not the optimum moment for selling something.

What this feels like is an act of desperation. We can't make you like us, but we can make you spend money, even if we have to trick you into doing it. That is not a happy experience.

Today, I'm going to focus on three favorites: TripIt, Netflix, and Zipcar.

Mindfulness: TripIt
This first level of design builds in as much positive emotion as possible, helping people have the skills to engage in that positive experience. That is, having positive experiences and being aware enough that you're savoring them. A beautiful afternoon. A perfect summer peach. Looking into the eyes of someone you love.

TripIt.com shows consideration and thoughtfulness – there's a pleasing awareness of the user throughout the user interface, not only of who the user is, but that the person has relationships other than the one with TripIt. It tries to anticipate the user's needs.

Here, the page shows some standard stuff. When the user logs in, there's a picture, and a welcome, but there are other nice features, like showing upcoming and past trips on a map, listing the trips of people she's shared itineraries with, and offering an option to share all trips in particular relationships.

Now drilling into an itinerary, TripIt anticipates needs like the best travel agent would, with a link to rent a car – which is a very subtle cross sell, at just the moment when the user might be thinking about that – along with other useful trip information so the user can find her way around on arriving.

It's really easy to get used to this kind of attention. Users begin to expect this level of happiness pretty quickly. Soon, the relationship is taken for granted by the user.

Flow: Netflix
How many times have you been to your Netflix queue only to look up a half hour later realizing that you've just spent 30 minutes rating movies? Here we have contentment, understanding, immersion, and trust. Time stands still in a good way.

Using Netflix, I've seen users become completely absorbed. It starts by clicking a star in an email. It's a simple call to action.

The star ratings take you to the Netflix web site, where it now offers you other movies it thinks you will like, asking you to rate the ones you've already seen.

It's a natural extension of the same task. Even people who say that they don't rate things on other sites have said in my research that they just get completely sucked in. They rate a few movies, and look up to find that 30 or 40 minutes have flown by. But it's not like spending time on other sites (like Facebook). This is useful. The more you rate, the better the suggestions are that the site makes.

Netflix even talks about love – they're not called suggestions, they're called "Movies You'll Love." It does seem to adore users, and users adore it as it continues to offer them things that they are likely to like. As users add movies to queues, it offers yet more. Every page of movies is like a group of gifts. What's not to like about that? Give me more and keep it coming.

Meaning: Zipcar
Yes, people get meaning from the web, and not just from social media. Meaning comes from belonging, reflectiveness, contributing, making a difference, commitment – being involved in something bigger than you.

Zipsters – that's what people who sign up for Zipcar are called – who I've talked to about their experiences with the service are radiant when they describe their interactions with the company, glowing with purpose and pleasure at the same time.

Zipcar encompasses mindfulness and flow as well as meaning – the nirvana of experience design.

Reserving a car is pleasant, even cute.

The flow of Zipcar comes from how quickly and easily it is incorporated into members' lives. The people in charge of the experience seem to have taken into account everything important, from how easy it is to reserve to how easy it is to extend a reservation, all the while keeping a fun, whimsical tone.

But it goes well beyond just ease of use. The idea of Zipcar is simple: rent a car by the hour. This works great for people who live in expensive, dense, urban areas where owning a car isn't necessary and sometimes is downright inconvenient. Zipcar's commercial premise is inexpensive convenience. But everything about using the service says "sharing." The rules are simple, and all about being considerate of the other people who use "your" car.

Zipsters at a cocktail mixer sponsored by Zipcar (to which no one drove, by the way), many of whom said they joined just for the convenience of the service, said that they loved the environmental, sustainable aspects of being a member. This goes beyond just social networking online. People who had never known one another before met face to face because they were excited about their experiences with the service so far, and found themselves effortlessly becoming involved with the company and one another, signing up for community based activities sponsored by Zipcar. This was the germ of a real community forming within real neighborhoods, mediated through a service that is self-served on the Web. Customers see themselves as members, but forget about themselves instead thinking about making a difference in the world. Magic.

Design happiness in to the experience
Want your users to fall in love with your designs? Fall in love with your users. The companies I've showcased here are just a few that have done that. And it shows. That's how users become excited about designs – being desired is very seductive.

:: :: :: :: :: ::> Related links
TED Talks: The psychology of happiness - Martin Seligman
http://www.ted.com/talks/lang/eng/martin_seligman_on_the_state_of_psychology.html

TED Talks: Play is more than fun - Stuart Brown
http://www.ted.com/talks/lang/eng/stuart_brown_says_play_is_more_than_fun_it_s_vital.html

TED Talks : Happy design - Stefan Sagmeister
http://www.ted.com/talks/lang/eng/stefan_sagmeister_shares_happy_design.html

Emotional Design: Why We Love (or Hate) Everyday Things, by Donald Norman

Happiness: A History, by Darrin M. McMahon

Flow: The Psychology of Optimal Experience by Mihaly Csikszentmihalyi

:: :: :: :: ::

:::> Come see me!
Hear me speak Sept. 21 and 22 at the EdUI conference in at the University of Virginia, where I'll be talking about "Usability Testing Without The Scary."

I'll be capping off the summer by going back to college! Well, actually I'll be speaking at one. The University of Virginia in Charlottesville, Virginia is hosting a two-day conference for web professionals in higher ed. The speaking line-up looks pretty exciting, featuring Jared Spool, Molly Holzschlag, Derek Featherstone, Dan Rubin, and of course yours truly, among others. There'll be a broad coverage of a lot of universally useful information about usability, accessibility, design, coding, and social media, with a little higher-ed-specific content thrown in as well. View the full program at http://eduiconf.org.

:::> I have moved!
After 20 years in the San Francisco Bay Area, I've bugged out. As of September 1, I will be operating out of my new office and home in Andover, Massachusetts. I'm excited about this move. It's big! You can still find me at www.usabilityworks.net, email me at dana@usabilityworks.net, on Twitter as danachis, and on the phone at 415.519.1148.

Testing in the wild, seizing opportunity

2009-06-30T08:17:00.000-07:00

When I say “usability test,” you might think of something that looks like a psych experiment, without the electrodes (although I’m sure those are coming as teams think that measuring biometrics will help them understand users’ experiences). Anyway, you probably visualize a lab of some kind, with a user in one room and a researcher in another, watching either through a glass or a monitor.

It can be like that, but it doesn’t have to. In fact, I’d argue that for early designs it shouldn’t be like that at all. Instead, usability testing should be done wherever and whenever users normally do the tasks they’re trying to do with a design.

Usability testing: A great tool
It’s only one technique in the toolbox, but in doing usability testing, teams get crisp, detailed snapshots about user behavior and performance. As a bonus, gathering data from users through observing them do tasks can resolve conflict within a design team or assist in decision-making. The whole point is to inform the design decisions that teams are making already.

Lighten up the usability testing methodology
Most teams I know start out thinking that they’re going to have a hard time fitting usability testing into their development process. All they want is to try out early ideas, concepts and designs or prototypes with users. But reduced to its essence, usability testing is simple:

Develop a test plan and design
Find participants
Gather the data by conducting sessions
Debrief with the team

That test plan/design? It can be a series of lists or a table. It doesn’t have to be a long exposition. As long as the result is something that everyone on the team understands and can agree to, you have written enough. After that, improvising is encouraged.

The individual sessions should be short and focused on only one or two narrow issues to explore.

But why bother to do such a quick, informal test?
First, doing any sort of usability test is good for getting input from users. The act of doing it gets the team one step closer to supporting usable design. Next, usability testing can be a great vehicle for getting the whole team excited about gathering user data. There is nothing like seeing a user use your design without intervention.

Most of the value in doing testing – let’s say about 70% – comes from just watching someone use a design. Another valuable aspect is the team working together to prepare for a usability test. That is, thinking about what Big Question they want answered and how to answer it. When those two acts align, having the team discuss together what happened in the sessions just comes naturally.

When not to do testing in the wild: Hard problems or validation
This technique is great for proving concepts or exploring issues in formative designs. It is not the right tool if the team is facing subtle, nuanced, or difficult questions to answer. In those cases, it’s best to go with more rigor and a test design that puts controls on the many possible variables.

Why? Well, in a quick, ad hoc test in the wild, the sample of participants may be too small. If you have seized a particular opportunity (say, with a seatmate on an airplane or a bus, as I have been known to do – yeah, you really don’t want me to sit next to you on a cross-country flight), a sample of one may not be enough to instill confidence with the rest of the team.

It might also happen, because the team is still forming ideas, that the approach in conducting sessions is not consistent from session to session. When that goes on, it isn’t bad necessarily. It can just mean that it’s difficult to draw meaningful inferences about what the usability problems are and how to remedy them.

If the team is okay with all that and ready to say, “let’s just do it!” to usability testing in the wild, then you can just do more sessions.

So, there are tradeoffs
What might a team have to consider in doing quick, ad hoc tests in the wild rather than a larger, more formal usability test? If you’re in the right spot in a design, for me doing usability testing in the wild is a total win:

You have some data, rather than no data (because running a larger, formal test is daunting or anti-Agile).
The team gets a lot of energy out of seeing people use the design, rather than arguing among themselves in the bubble of the conference room.
Quick, ad hoc testing in the wild snugs nicely into nearly any development schedule; a team doesn’t have to carve out a lot of time and stop work to go do testing.
It can be very inexpensive (or even free) to go to where users are to do a few sessions, quickly.

Usability testing at its essence: something, someone, and somewhere
Just a design, a person who is like the user, and an appropriate place – these are all a team needs to gather data to inform their early designs. I’ve seen teams whip together a test plan and design in an hour and then send a couple of team members to go round up participants in a public place (cafes, trade shows, sporting events, lobbies, food courts). Two other team members conduct 15- to 20-minute sessions. After a few short sessions, the team debriefs about what they saw and heard, which makes it simple to agree on a design direction.

It’s about seizing opportunity
There’s huge value in observing users use a design that is early in its formation. Because it’s so cheap, and so quick, there’s little risk of making a mistake in making inferences from the observations because a team can compensate for any shortcomings of the informality of the format by doing more testing – either more sessions, or another round of testing as follow-up. See a space or time and use it. It only takes four simple steps.

Tools for plotting a future course of design, checking progress

2009-06-08T08:58:00.000-07:00

“Let’s check this against the Nielsen guidelines for intranets,” she said. We were three quarters of the way through completing wireframes for a redesign. We had spent 4 months doing user research, card sorting, prototyping, iterating, and testing (a lot). At the time, going back to the Nielsen Norman Group guidelines seemed like a really good idea. “Okay,” I said. “I’m all for reviewing designs from different angles.”

There are 614 guidelines.

This was not a way to check designs to see if the team had gone in the right design direction.

Are you designing or inspecting?
They are not interchangeable, guidelines and heuristics, but many UXers treat them that way. It’s common to hear someone saying that they’re doing a heuristic evaluation against X guidelines. But it doesn’t quite work like that.

Designing is an act of creation, whether you’re doing research, drawing on graph paper, or coding CSS. Inspecting is an act of checking, of examining, often with some measure in mind.

Guidelines are statements of direction. They’re about looking to the future and what you want to incorporate in the design. Guidelines are aspirational, like these:

Add, update, and remove content frequently.
Provide persistent navigation controls.
Index all intranet pages.
Provide org charts that can be viewed onscreen as well as printed.*

Heuristics challenge a design with questions. The purpose of heuristics is to provide a way to “test” a design in the absence of data by making an inspection. Heuristics are about enforcement, like these:

Visibility of system status: The system should always keep users informed about what is going on…
Match between system and the real world: The system should speak the users' language….
User control and freedom: The system should provide a clearly marked "emergency exit" to leave the unwanted state … **

Creating or diagnosing?
Heuristics are often cast as pass/fail tests. Does the UI comply or not? While you could use the usability.gov guidelines to evaluate web site designs, they were developed as tools for designing. They present things to think about as teams make decisions.

Both guidelines and heuristics are typically broad and interpretable. They’re built to apply to nearly interface. But they come into play at different points in a design project. Guidelines are things to think about in reaching a design; they are considerations and can interact with one another in interesting ways. Heuristics are usually diagnostic and generally don’t interact.

Don’t design by guidelines alone
For example, on the intranet project, we looked at guidelines about the home page. One directive says to put the most important new information on the home page, and the next one says to include key features and company news on the home page. A third says to include tools with information that changes every day. But earlier in the list of guidelines, we see a directive to be “judicious about having a designated ‘quick links’ area.” Guidelines may feel complementary to one another or some may seem to cancel others out. Taken together, there’s a set of complex decisions to make just about the home page.

And it was too late on our intranet to pay attention to every guideline. The decisions had been made, based on stakeholder input, business requirements, and technology constraints, as well as user requirements. Though we were thoughtful and thorough in designing, anyone scoring our site against the guidelines might not give us good marks.

Don’t evaluate by heuristics alone
Likewise, when looking at heuristics such as “be consistent,” there’s a case for conducting usability tests with real users. For example, on the intranet I was working on, one group in the client company was adamant about having a limited set of page templates, with different sections of the site meeting strict requirements for color, look, and feel. But in usability testing, participants couldn’t tell where they were in the site when they moved from section to section.

Guidance versus enforcement
What are you looking for at this point in your design project? In the intranet project, we were much closer to an evaluative mode than a creation mode (though we did continue to iterate). We needed something to help us measure how far we had come. Going back to the guidelines was not the checkpoint we were looking for.

We sallied forth. The client design team decided instead to create “heuristics” from items from the user and business requirements lists generated at the beginning of the project, making a great circle and a thoughtful cycle of research, design, and evaluation.

I don’t know whether the intranet we designed meets all of the guidelines. But users tell us and show us every day that it is easier, faster, and better than the old intranet. For now, that’s enough of a heuristic.

* From "Intranet Usability: Design Guidelines from Studies with Intranet Users" by Kara Pernice Coyne, Amy Schade, and Jakob Nielsen

** From Jakob Nielsen's 10 heuristics, see http://www.useit.com/papers/heuristic/heuristic_list.html

Related:

Where do heuristics come from?

What are you asking for when you ask for heuristic evaluation?

:: :: :: :: :: :: :: :: :: ::

Note: I'm moving!
After 20 years in the San Francisco Bay Area, I'm bugging out. As of September 1, I will be operating out of my new office and home in Andover, Massachusetts. I'm excited about this move. It's big!

You can still find me at www.usabilityworks.net, email me at dana@usabilityworks.net, on Twitter as danachis, and on the phone at 415.519.1148.

Webinar June 3: Quick, easy, and insightful - usability testing in the wild

2009-05-18T15:05:00.000-07:00

Ever want to do a usability test but felt you couldn't because you didn't have a lab? Or couldn't record sessions? Or have something special to test?

Fear no more. You can learn how to do informal, ad hoc usability tests in the formative stages of design that will help you gain knowledge about your users and hone your design direction.

This 2-hour webinar is being offered by the US General Services Administration. C'mon along. The price couldn't be better at $75 for non-government attendees, $50 for government employees.

See the course details and register!

Where do heuristics come from?

2009-05-05T19:41:00.000-07:00

Recently I had the honor and pleasure of working on a project for the National Institute of Standards and Technology (NIST) to develop style guidelines for voting system documentation. Yawner, right? Not at all, it turns out. It made me think about where guidelines and heuristics come from for all kinds of design. Yes, if you live in the United States, you paid for me to find this out. Thank you.

What I learned in the process of developing style guidelines for voting system documentation (which, astonishingly took about a year) is that most heuristics -- accepted principles -- used in evaluating user interfaces come from three sources: Lore or folk wisdom, specialist experience, and research.

Though style guidelines for content are important, I’m going to talk about each of these sources of heuristics with various design examples. I’m sure you’ll see something that you’ve encountered before.

Lore or folk wisdom
First come guidance from “They,” as in “They say…,” for which no one knows the true source. For example: “Feed a cold, starve a fever.” “Never end a sentence with a preposition.” “Limit the number of items in the main navigation to seven plus or minus two.”

Where do these come from? Someone’s belief that this is a good practice. They may have heard something or done something that they think supports the practice, but there’s really not basis in fact for any of these.

A New York Times article published on February 13, 2007 by Anahad O’Connor says that recent research about whether to eat a lot when you have a cold and fast when you have a fever is inconclusive. No one seems to know how this one started. It may just feel like there’s some inherent logic to it.

Not ending sentences with prepositions was encoded by a British guy named Henry Fowler in 1926. He was a crotchety, proscriptionist pedant but his book was a best seller. People wanted guidance about how to speak and write “properly,” especially in class-conscious England. So, a rule to not use words like “to,” “in,” “for,” “with,” or “on” as the last word in a sentence became wildly popular as a marker of a well bred, well educated person. But it was really just Fowler’s personal preference, and today the usage seems affected.

My favorite example in the web design world is a guideline about limiting the number of items in a navigation menu or list to five to seven items. Most people don’t know where this came from – if they did, they’d know that this isn’t the best use of that “rule” and imposing it actually won’t make the design usable. This one does originate in research, specifically, an article published in 1956 by George Miller in the Psychological Review called “The magical number seven, plus or minus two: Some limits on our capacity for processing information.” (You can see a reproduction of the article here: http://www.musanim.com/miller1956/) The findings from the research Miller describes are about working memory. The lore passed down from that article is that humans can only hold about seven things in their short-term memory at a time. BUT, Miller heavily, heavily qualifies this as “suggestive” and an “estimate.” More importantly, what in the world does this have to do with designing web site navigation? Nothing. Navigation is persistent. We’re not asking people to remember from section to section where they can go. It’s right there for them to see and use. The number of items in navigation should be determined by the data from research about the users and their task goals.

If you catch yourself saying, “they say,” or “I’ve heard” when making an observation about a design issue when you’re inspection mode, you may be caught without a lot of support for your point. Basing an inspection on your own experiences observing users can hold more authority – but not as much as research-based guidelines or heuristics.

Specialist experience
Older adults who use the Web need high contrast and large targets. If they are not expert Web users, they can be easily distracted, so to ensure that they’re successful, we should design in smooth task paths and clear labeling that doesn’t use jargon.

One of my special interests since about 2003 has been Web design for older adults. I’ve internalized the design principles above (as well as many others) after watching dozens of people who are age 55 or older use a variety of web sites. I am confident that implementing a design that takes these design principles into account will make the design easier for older people to use than designs that use subtle colors layered on one another, small buttons and links, and cluttery page layout with trendoid headings and labels. Though I have observed many types of people using lots of different kinds of web sites, I have specialist experience from watching one audience try to do typical tasks on a variety of web sites.

Specialist experience means expertise in a particular domain or product. You get it only after hours and hours and hours of seeing the same kinds of things happen to the same types of users. Basing an inspection on specialist experience is definitely a step up from working from lore, but if you haven’t distilled what you have found in the many hours of observing a type of users using a site or type of site, then you may be working from hunches and opinion that could make it difficult for you to justify the evaluation recommendations.

Evidence from research
Some things that experienced designers have internalized do have data to support them: Eliminate horizontal scrolling. Design for working memory limitations. Facilitate scanning.

These all rate a 5 for strength of evidence in the guidelines on usability.gov. (I’ll get to the rating thing in a minute.) Usability.gov started as a project at the National Cancer Institute (NCI), which is part of the US Department of Health and Human Services. NCI needed help designing usable cancer information web sites. A simple goal.

On the way, the NCI team realized that not all guidelines were equal. Some guidelines were supported by a lot of data from multiple studies (like the high scoring heuristics above). Some guidelines might come from only one study. Still, evidence was evidence and NCI wanted to use “quantified, peer-reviewed web site design guidelines,” which they found simply didn’t exist. And as far as I know, there’s nothing like the resource NCI created at usability.gov.

To reach their goal, NCI put together panels of experts to review research. The panelists then who rated each guideline for strength of evidence (which among other considerations was “cumulative and compelling” for a 5 out of 5).

The idea was that teams could use the ratings to help them make design decisions. But the guidelines were not meant to be a substitute for usability testing. Why not? The main reason was that the guidelines at usability.gov were developed for information-rich web sites (versus e-commerce or transaction-based sites) with content about major illnesses. That’s fairly specialized. But when you read through the 500 guidelines that NCI identified, it is obvious that almost all could apply to many types of web sites or many types of pages within sites. Your mileage may vary.

The basis of the heuristic matters
As the folks at NCI learned in developing usability.gov and I learned in the work for NIST*, provenance is important. This is true of all implicit and explicit heuristics applied in design decisions.

Learning about where heuristics come from – lore or folk wisdom; specialist experience; or research – helped me understand better where some of the teams I’ve worked with were coming from as they developed design principles. Sometimes they based the principles on lore, sometimes on expertise. Rarely did they go to the research.

Expertise is good, but research is better. Research-based heuristics simply have more heft: credibility, specificity, and applicability.

Still, there’s no substitute for primary research. Firsthand observation of your users in their context reveals subtleties of behavior that even research-based heuristics can’t match. And if your research of your users in their context contradicts the known research, what do you do? (You don’t get two guesses to answer this question.) If you go with what your users do, then even the most deeply researched heuristics are at best a poor substitute for doing the right thing.

* I couldn't have made the discoveries I did on that project without Susan Becker, my project partner, who did most of the heavy lifting.

:: :: :: :: ::

Related links

You can pore through the evidence-based guidelines for usability developed by the National Cancer Institute (NCI) at www.usability.gov.

You might also want to check out other sets of research-based heuristics and guidelines. I've worked on a couple of them:

www.aarp.org/olderwiserwired holds a set of heuristics for designing web sites for older adults. The National Institute on Aging (NIA) has just published an update of its guidelines for designing senior-friendly websites at http://www.nia.nih.gov/HealthInformation/Publications/website.htm.

vote.nist.gov now links to style guidelines for voting system documentation, which are based on research in technical communication, information design, usability, and instructional design. Click on the Publications link to download the PDF of the guidelines and to view other design research related to voting systems.

Oh, and you might want to read Miller's article to judge for yourself: http://www.musanim.com/miller1956/

What are you asking for when you ask for a heuristic evaluation?

2009-04-16T09:39:00.000-07:00

Every usability professional I know gets requests to do heuristic evaluations. But it isn’t always clear that the requester actually knows what is involved in doing a heuristic evaluation. Some clients who have asked me to do them have picked up the term “heuristic evaluation” somewhere but often are not clear on the details. Typically, they have mapped “heuristic evaluation” to “usability audit,” or something like that. It’s close enough to start a conversation.

Unfortunately, the request usually suggests that a heuristic evaluation can substitute for usability tests. I chat with the person, starting by talking about what a heuristic evaluation is, what you get out of it, and how it compares to what you find out in a usability test.

How do you do a heuristic evaluation?

Let’s talk about what a “classic” heuristic evaluation is. When Jakob Nielsen and Rolf Molich published the method in 1990, these two really smart guys were trying to distill some of the basic principles that make a user interface usable to its audience. They came up with 10 “accepted usability principles” (heuristics) that, when multiple evaluators applied them to any user interface, should reveal gaps in the design of the UI that could cause problems for users.

Armed with the Nielsen checklist of accepted usability principles – heuristics – someone who had never seen the UI before and who was not necessarily knowledgeable about the domain should be able to determine whether any UI complied with these 10 commandments of usable user interface design. If three or four or five people sat down for an hour or two and inspected an interface separately, they could come up with piles of problems. Then they could compare their lists, normalize the issues, and then hand a list off to the engineers to go fix.

What do you get out of a heuristic evaluation?

Let’s say that the person who called me the other day was asking for a review in the form of a heuristic evaluation to resolve a conflict on the team. The conflict on this team was about the page flow: What should the order of steps in the process be: the same as site X or the same as site Y? Should the up-sell be at the beginning or the end of the purchase process? ‘Could you please review the UI and just tell us what to do because we don’t have time and money to do a usability test.’

Several of the Nielsen heuristics might apply. Some probably don’t. For example, did the success of the page flow require users to remember things from step to step (recognition rather than recall)? Were there any shortcuts for return customers (flexibility and efficiency of use)? Where might users get bogged down, distracted, or lost (aesthetic and minimalist design)? By applying these heuristics, what have we found out?

The flow might require people to remember something from one step to another. The way the heuristic is written, requiring this of users is always bad. But it might not be.

The flow might not have shortcuts for expert users. The way the heuristic is written, not having short cuts is bad. But it might not be.

There may be places in the flow that slow people down. The way the heuristic is written, you always want users to be able to do tasks quickly. But you might not.

And I don’t think we have resolved the conflict on the team.

When applying what I call “checklist usability” in a heuristic evaluation to learn what the flaws and frustrations of a design might be, the outcome is a determination of whether the user interface complies to the heuristics. It is an inspection, not an evaluation. It is not about the user experience. It’s not even about task performance, which is what the underlying question was in the team’s conflict: Will users do better with this flow versus that flow? If we interrupt them, will they still complete a purchase? Any inspection method that claims to answer those kinds of questions is just guessing.

A team may learn about some design flaws, but the frustrations could remain stubbornly hidden -– unless the reviewer has already observed many users trying to reach goals using this site or process, or something very like it in the same domain. Even then, there’s a huge risk that a single inspector or even a small group of inspectors -- who are applying very general guidelines, are not like the users, and are not actually using the design as part of the inspection – will miss flaws that will be task-stoppers. Worse, they may identify things that don’t comply with the heuristics that should not be changed.

How does heuristic evaluation compare to usability testing?

Heuristic evaluation was codified about 1990, at a time when it was expensive to get access to users. It wasn’t uncommon for people to have to be trained to use the technology being evaluated before they could sit down in a usability lab to perform some tasks. The whole concept of there being a user interface was pretty new. Conventions were just settling into place.

Usability testing has been around since at least the 1980s, but began to be widely practiced about the same time Nielsen and Molich published their heuristic evaluation method. While usability testing probably needs some updating as a method, the basic process still works well. It is pretty inexpensive to get access to users. User interfaces to technology are everywhere. For many of the applications of technology that I test, users don’t need special training.

Heuristic evaluation may help a team know whether their UI complies to someone else’s guidelines. But observing people using a design in a usability test gives a team primary data for making design decisions for their users using their design – especially in a world evolved far beyond command line entry and simple GUIs to options like touch screens, web 2.0, and ubiquitous connectivity. Separately and in combination these and other design decisions present subtle, complex problems of usability. For me, observing people using a design will always trump an inspection or audit for getting solid evidence to determine a design direction. There is nothing like that “ah ha!” moment when a user does something unexpected to shed light on how well a design works.

What counts: Measuring the effectiveness of your design

2009-03-07T09:00:00.000-08:00

Let’s say you’re looking at these behaviors in your usability test:

Where do participants start the task?
How easily do participants find the right form? How many wrong turns do they take on the way? Where in the navigation do they make wrong turns?
How easily and successfully do they recognize the form they need on the gallery page?
How well do participants understand where they are in the site?

How does that turn into data from which to make design decisions?

What counts?
It’s all about what counts. What did the team observe that shows that these things happened or did not happen?

Say the team does 10 individual usability test sessions. There were 5 major “scavenger hunt” tasks. Everyone has their own stack of yellow stickies that they’ve written down observations on. (Observations of behavior, only – there should be no interpreting, projecting, guessing, or inferring yet.) Or, say the team has kept a rolling issues list. All indications are that the team is in consensus about what happened.

Example 1: Entry points
Here’s an example. For the first task, Find an account open form, the first thing the team wanted to observe for was whether participants started out where we thought they should (Forms), and if not, where participants did start.

The data looked like this:

Seven of the 10 started out at Forms – great. That’s what the team expected based on the outcomes of card sorts. But 3 participants didn’t. But those 3 all started out at the same place. (First inference: Now the team knows there is strong scent in one link and some scent in another link.)

Example 2: Tracking navigation paths – defining “wrong turn”
Now, what about the wrong turns? In part, this depends on how the team defines “wrong turn.”

What you’re finding out in exploratory tests with early designs is where users go. Is that wrong? Not necessarily. Think of it in the same way that some landscapers and urban planners do about where to put walkways in a park. Until you can see where the traffic patterns are, there’s not a lot of point in paving. The data will tell you where to put the paths outside where the team projects the path should be.

As each session goes on, the team tracks where participants went. The table below actually tracks the data for multiple issues to explore:

How many wrong turns do they take on the way?
Where in the navigation do they make wrong turns?
How easily and successfully do they recognize the form they need on the gallery page?

The data looked like this:

Everyone ended up at the right place. Some participants even took the path that the team expected everyone to take: Forms / Account Open / Form #10.

But the participants who started out at Products had to go back to the main navigation to get to the right place. There’s a decision to make. The team could count those as “wrong turns” or they could look at them as a design opportunity. That is, the team could put a link to Forms on the Product page – from the point of view of the user, they’re still on the “right” path and the design has prevented the user from making a mistake.

Account Open is a gallery page. Kits is the beginning of a wizard. Either way, the right form is available in the next step and all the participants chose the right one.

Measures: Everything counts
So, how do you count what counts? The team counted errors (“wrong turns”) and task successes. How important are the counts? The team could have gone with their impressions and what they remembered. There’s probably little enough data to be able to do that. In smaller tests, your team might be comfortable with that. But in larger tests – anything over a few participants – observers typically remember the most recent sessions the best. Earlier sessions either fade in memory or the details become fuzzy. So tracking data for every session can keep the whole team honest. When there are numbers, the team can decide together what to do with them.

What we saw
This team learned that we got the high-level information architecture pretty close to right – most participants recognized where to enter the site to find the forms. We also learned that gallery pages were pretty successful; most participants picked the right thing the first or second time. It was easy to see all of this in tracking and counting what participants did.

Consensus on observations in real time: Keeping a rolling list of issues

2009-02-27T08:33:00.000-08:00

Design teams often need results from usability studies yesterday. Teams I work with always want to start working on observations right away. How to support them while giving good data and ensuring that the final findings are valid?

Teams that are fully engaged in getting feedback from users – teams that share a vision of the experience they want their users to have – have often helped me gather data and evaluate in the course of the test. In chatting with Livia Labate, I learned that the amazing team at Comcast Interactive Media (CIM) came to the same technique on their own. Crystal Kubitsky of CIM was good enough to share photos of CIM's progress through one study. Here’s how it works:

1. Start noting observations right away
After two or three participants have tried the design, we take a longer break to debrief about what we have observed so far. In that debrief, each team member talks about what he or she has observed. We write it down on a white board and note which participants had the issue. The team works together to articulate what the observation or issue was.

2. Add observations to the list between sessions
After each succeeding session, as the team generates observations, I add observations to that list, including the numbers for each participant who had the issue. We note any variations on each observation – they may end up all in one, or they may branch off, depending on what else we see.

Here’s an example observation from one study. We noted on the first day of testing that

Participants talked about location, but scrolled past the map without interacting with it to get to the search results (the map may not look clickable)

The team and I later added the participant numbers for those who we observed doing this:

Participants talked about location, but scrolled past the map without interacting with it to get to the search results (the map may not look clickable) PP, P1, P3

Each day of testing, the team and I add more observations and more participant numbers. The CIM team debriefs to review top observations, highlighting what they learned and color-coding participants or segments as they capture rolling observations:

3. Invite team member observers to add observations to the list themselves
As the team gets better at articulating the issues they have seen in a test session, it is my experience that they start adding to the list on their own. Often one of the observers voluntarily takes over adding to the list. This helps generate even more buy-in from the team and means that I can concentrate on focusing the discussion on the issues we agreed to explore when we planned and designed the test.

At the end of the last session, it’s easy then to move to a design direction meeting because the team has observed sessions, articulated issues, and already analyzed that data together.

The CIM team documents which participants had which behaviors in the table at the top right of the photo above.

What makes it a “rolling” list of observations and issues
There are three things that are “rolling” about the list. First, the team adds issues to the list as they see new things come up (or that you didn’t notice before, or seemed like a one-off problem). Second, the team adds participant numbers for each of the issues as the test goes along. Third, the team refines the descriptions of the issues as they learn more from each new participant.

Double-check the data, if there’s time
Unless the team is in a very rapid, iterative, experimental mode, I still go back and tally all of the official data that I collected during each session. I want to be sure that there are no major differences between the rolling issues list and the final report. Usually, the rolling issues list and the final data match pretty closely because the team took care in defining the research questions and issues to explore together when we planned and designed the test.

Doing the rolling list keeps teams engaged, informed, and invested. It helps the researcher cross-check data later, and it gives designers and developers something to work from right away.

London calling: Web Geek Gathering March 8, the Pitcher & Piano Trafalgar

2009-02-26T08:05:00.000-08:00

Proving that the world really can be a small place, an amazing convergence of geekiness is scheduled for Sunday afternoon March 8 in London at the Pitcher & Piano pub in Trafalgar Square when I have the honor of basking in the glow of two Stars of Geekdom, Jared Spool and Eric Meyer, for a Web Geek Gathering.

C’mon along. Carsonified is sponsoring, so the pints will be flowing. Please do RSVP. It is a pub (though a swanky looking one), after all, so space is limited. I’ll have a couple of copies if Handbook of Usability Testing Second Edition to give away. Would love to see you.

Looking for love: Deciding what to observe for

2009-02-17T07:30:00.000-08:00

Last winter I worked with a team that wanted to find out whether a prototype they had designed for a new intranet worked for users. Their new design was a radical change from the site that had been in place for five years and in use by 8,000 users. Going to this new design was a big risk. What if users didn’t like it? Worse, what if they couldn’t use it?

We went on tour. Not to show the prototype, but to test it. Leading up to this moment we had done heaps of user research: stakeholder interviews, field observations (ethnography, contextual inquiry – pick your favorite name), card sorting, taxonomy testing. We learned amazing things, and as our talented interaction designer started translating all that into wireframes, we got pressure to show them. We knew what we were doing. But we wanted to be sure. So we made the wireframes clickable and strung them together to make them feel like they were doing something. And then we asked (among other things):

How well does the design support the tasks of each user group?
How easily do users move through the site for typical tasks?
Where do they take wrong turns? What trigger words are missing? What trigger words are wrong?

Validating the research
In some ways, you could look at this as a validation test – not validating the design necessarily, but instead validating the user research we had done. Did we interpret our observations correctly by making the right inferences, in turn getting us to the design we got to?

What was possible: where the design might break
To find out, we had to answer those Big Questions. What were the issues within them that we wanted to investigate? Let’s take an example: How easily do users move through the site for typical tasks? We wanted to know whether users took the same path we wanted them to take, and if they didn’t, why not. On a task to find forms to open a brokerage account, we listed the possible issues. Users might

start at the wrong place in the site
get lost
pick the wrong form
not recognize they’ve reached the right place

From that discussion of the disasters that we could imagine came a list of behaviors to observe for, or as my friends at Tec-Ed say, issues to explore:

Where do participants start the task?
How easily do participants find the right form? How many wrong turns do they take on the way? Where in the navigation do they make wrong turns?
How easily and successfully do they recognize the form they need on the gallery page?
How well do participants understand where they are in the site?

What we saw
From these questions, we learned that we got the high-level information architecture right – most participants recognized where to enter the site to find the forms. We also learned that there were a couple of spots in the task path that had a combination of weak trigger words and other distractions that drew attention away from the things that would have gotten participants to the goal more quickly. But the groupings on the gallery page were pretty successful; most participants picked the right thing the first or second time. It was easy to see all of this in the way participants performed, but we also heard clues from them about what they were looking for and why.

And, by the way, the participants loved it. We knew because they said so.

Popping the big question(s): How well? How easily? How valuable?

2009-02-10T09:57:00.000-08:00

When teams decide to do usability testing on a design, it is often because there’s some design challenge to overcome. Something isn’t working. Or, there’s disagreement among team members about how to implement a feature or a function. Or, the team is trying something risky. Going to the users is a good answer. Otherwise, even great teams can get bogged down. But how do you talk about what you want to find out? Testing with users is not binary – you probably are not going to get an up or down, yes or no answer. It’s a question of degree. Things will happen that were not expected. The team should be prepared to learn and adjust. That is what iterating is for (in spite of how Agile talks about iterations).

Ask: How well
Want to find out whether something fits into the user’s mental model? Think about questions like these:

How well does the interaction/information information architecture support users’ tasks?
How well do headings, links, and labels help users find what they’re looking for?
How well does the design support the brand in users’ minds?

Ask: How easily
Want to learn whether users can quickly and easily use what you have designed? Here are some questions to consider:

How easily and successfully do users reach their task goals?
How easily do users recognize this design as belonging to this company?
How easily and successfully do they find the information they’re looking for?
How easily do users understand the content?
How easy is it for users to understand that they have found what they were looking for?
How easy or difficult is it for them to understand the content?

Ask: How valuable

What do users find useful about the design?
What about the design do they value and why?
What comments do participants have about the usefulness of the feature?

Ask: What else?

What questions do your users have that the content is not answering?
What needs do they have that the design is not addressing?
Where do users start the task?

Teams that think of their design issues this way find that their users show them what to do in the way they perform with a design. Rarely is the result of usability testing an absolute win or lose for a design. Instead, you get clues about what’s working – and what’s not – and why. From that, you can make a great design.

Thinking inside the right box: Developing tasks for usability test participants

2009-01-19T11:54:00.000-08:00

One question I get in workshops on usability testing is How do I get participants to do the tasks I want them to do?

On further discussion, we find (the attendee and I) that this question is really asking two things:

How do I use usability testing to exercise the design?
How do I motivate users to try things I want them to try?

Thinking outside-in versus thinking inside-out

Teams get to a point where they have to make decisions or settle disagreements about which direction to go with a design. The natural – and good – thing is to go to the users and collect a little data. You have this thing you want to test. This is thinking from the inside out, from the point of view of the system or design you’re working on.

Users have goals they want to reach. So, you have to think from their point of view – that is, from the outside, looking in.

It’s easy to get caught up in asking test participants to try particular design features without fitting that trying-out into a realistic situation for the participant. Teams do it all the time. Here’s example of inside-out thinking in setting up a task:

Task for the participant from the designer’s point of view: We’ve added a map to our search so you can see where our product outlets are. Here. Try it out and tell us what you think.

You watch and listen, but what happens? The data is about a reaction to something that is out of context of use. Here’s a response from a test I did in a similar situation:

Participant response:

That’s cool. I like the idea of having a map. This one looks good. But I don’t know that city, so I don’t know what these locations are in relation to. Hmm. And look, the little numbers in the bubbles show me… something. What are they numbered from? What makes one number 1 and another one number 10? When I hover over those bubbles, it shows me more information, but I can’t see the other locations on the map now.

What do you do now?

Compare that situation to this.

Task scenario for the participant from a user’s point of view: Ever had problems with your cell phone? Okay, imagine that you’re in a city other than the one you live in. You’re there visiting your family (insert appropriate occasion here), so you don’t know where the stores are where you could take your phone to be fixed or exchanged. But you do have access to the Web. What do you do now?

Participant response:

Man, I’ve had that happen. First, I went to the site for the company I get service through. This isn’t my computer so I don’t have bookmarks set up to go to my account. Okay, so I type in the main site address and then I look for some way to find retail locations. I’m on the site now, but I’m not seeing what I’m looking for. Do I have to log in first? No, that would be stupid. They wouldn’t make me do that, would they? Where’s the link for stores? I swear I’ve used it before. Huh. Oh, here it is at the top. It’s a tiny link. Click. I’m there. Great. I’m going to enter my mother’s zip code to see where the stores are near her. Woo! I got a map. Cool. I can instantly see where there are locations within range. I don’t know the neighboring towns very well, though, so I’ll have to zoom in at some point. Hmmm. What’s the address of the nearest store? I need directions now…

See how much richer that data is? Let us deconstruct what’s going on here.

The task scenario sets up a situation that the participant can relate to and you hope he’s motivated to do, and leaves it open-ended. (You can adjust the scenario to fit the participants’ experiences and motivations and still get consistent data across participants.) This way, you can see much more natural behavior, thought processes, and performance. And you get some seriously cool stuff along the way.

First, you learn that the participant goes to the site enough to bookmark it in the browser. You also got help with your information architecture, in trigger words for labels: “retail locations” and “stores.” The participant is telling you the vocabulary he uses to articulate the task goal. You want to use those words in your interface and search terms.

Next, you observe that the way to get to store locations on the site isn’t immediately obvious because the participant doesn’t see it right away and wonders if he has to log into an account. This may be an artifact of the task, or it may be a design issue. Over a few sessions, you should be able to tell whether the position, size, design, or proximity of the widget should be changed.

Then, he enters a zip code to get to a map. That works. The participant’s interaction with the map tells you that he grocked it right away. At a glance, he got what he needed and it made sense. Yay for you!

Now he’s using the map in his context to reach his goal. You can use what happens next to further refine your design.

It’s okay to start out thinking about tasks by localizing test scenarios to certain areas of the design as long as you turn them around to look at the localized design problems from the larger point of view of the user – from the outside, in.

Yes or No: Make your recruiter smarter

2009-01-16T10:05:00.000-08:00

In response to my last post about writing effective screeners, c_perfetti asks:

I agree open-ended questions in a screener are best.

But one reason some usability professionals use 'yes/no' questions is because they don't have confidence that the external recruiters can effectively assess what an acceptable open ended answer would be.

In some cases, they may find that asking a 'yes/no' question is the safer approach.

How would you handle this concern?

You asked a great open-ended question! What you need is a smarter recruiter.

There are two things you can do to make your recruiter smarter: brief her on the study, and give her the answers.

Brief your recruiter

Basically what we’re talking about is giving your recruiter enough literacy in the domain you’re in to be intelligent when screening rather than a human SurveyMonkey. You can make them work smarter for you by doing two things:

Spend 15 minutes before the recruit starts explaining to the recruiting agency the purpose and goals of the study, the format of the sessions, what you’re hoping to find out, and who the participant is. For this last, you should be able to give the agency a one- or two-word envisionment of the participant: “The participant has recently been diagnosed with high cholesterol or diabetes or both and has to make some decisions about what to do going forward. She hasn’t done much research yet, but maybe a little.”
Insist that the agency work with you. Tell them to call you after the first two interviews they do and walk through how it went. Questions will come up. Encourage them to call you and ask questions rather than guessing or interpreting for themselves.

With this training done, you can trust your recruiting agency a bit more. If you continue to work with the agency, over time they’ll learn more about what you want, but you’ll also have a relationship that is more collaborative.

Tell the recruiter what the answers might be

Now, to your question about Yes/No.

Using Yes/No leads to one of two things: inviting the respondent to cheat by just saying “yes!” or scaring the respondent into giving the “wrong” answer because it might be bad or embarrassing to give the “right” answer. In the screening interview, this can be scary or accusatory to the respondent: “Do you have high cholesterol?” (And saying “no” would disqualify him from the study.) Or just super easy to say “yes” because the question is too broad or ambiguous. “Do you download movies from the Web?” could be stretched to mean ‘watch videos on YouTube,’ or bit torrenting adult entertainment, but what it means is ‘Do you use a service from which you get on-demand or instant access to commercial, Hollywood movies and then watch them?’

If it’s the main qualifier for the study – Do you do X? – that can be avoided by putting out the call for participants the right way. Check the headlines on craigslist.org (usually in Jobs/ETC or in Volunteers), for example. There you’ll see pre-qualifying titles on the postings, and that’s the place to put the question, “Do you have high cholesterol?” or “Do you use a headphone with your mobile phone?” You still have to verify by asking open-ended questions.

If you find yourself wanting to ask a Yes/No question:

Craft an open-ended question and provide what several possible right answers might be for the recruiters to use as reference (but not something they should read to respondents). Possible alternative script for the recruiter:

“Tell me about the last cholesterol test you had. What did the doctor say?”
[Recruiter: Listen for answers like this
___ He said that I’m okay but I should probably watch what I eat and get more exercise. My total cholesterol was <-200>.
___ He said that if I didn’t make a change I’d have to start taking meds/a prescription/away my cheese. My total cholesterol was <200-239>.
___ He said that I am a high risk for heart disease. I could have a heart attack. My total cholesterol was <240+>]

Think of one key question that would call the respondent out on fibbing to get into the study. For a gaming company, we wanted people who had experience with a particular game. Anyone can look up the description of a game online and come up with plausible answers. We added in a question asking what the respondent’s favorite character was and why. Our client provided a list of possible answers: names and powers. The responses were fascinating and indicated deeper knowledge of the game than a cheater could get from the cover art or the YouTube trailer.

The short answer: You should still avoid Yes/No questions in screeners. First, think about what you’re really asking and what you want to find out by asking it. Is it really a yes/no question? Then train your recruiter a little bit beforehand, and anticipate what the answers to the open-ended questions might be.

Why your screener isn't working

2009-01-15T12:12:00.000-08:00

I get that not every researcher wants to or has time to do her own recruiting of participants. Recruiting always seems like an ideal thing to outsource to someone else. As the researcher, you want to spend your time designing, doing, and analyzing research.

So, you find an agency to do the recruiting. Some are very appealing: They're cheap, they're quick, and they have big databases of people. You send requirements, they send a list of people they've scheduled.

How do you get the most out of an agency doing the recruiting? Write a great screener -- and test it. How do you get a great screener? Here are a few tips.

Seven screener best practices

Focus questions on the behavior you want to see in the test. For example, for a hotel reservations website, you might want to know Does the person book his own travel online? For a website for a hospital network, the behavior might be Does the person have a condition we treat? Is the person looking for treatment?
Limit the number of questions. If the question does not qualify or disqualify a respondent for the study, take the question out. If you want to collect information besides the selection criteria, develop a background questionnaire for the people selected for the study.
Think about how you're going to use the data collected from the screener. Are you going to compare user groups based on the answers to screener questions? For example, if you're asking in your screener for people who are novices, intermediates, and experts with your product, are you actually going to have a large enough sample of participants to compare the data you collect in the usability test? If not, don't put requirements in your screener for specific numbers of participants with those qualities. Instead, ask for a mix.
Avoid Yes/No responses. This is difficult to do, but worthwhile. Yes/No questions are very easy for respondents to guess what the "right" answer is to get into the study. In combination, a series of gamed Yes/No responses can make a respondent look like he fits your profile when he really doesn't.
Ask open-ended questions if at all possible. This gets respondents to volunteer information in answer to a real question rather than picking the "right" choice from a list of options that the recruiter reads to them. You can give the recruiter the choices you think people will come up with and a pick list for the recruiter to use to note the data. But the recruiter should not read the list to the respondent. For example, on the hospital website, you might ask, "Tell me about your health right now. What were the last three things you visited a doctor for?"
Avoid using number of hours or frequency as a measure of or a proxy for expertise. I was looking for tech savvy people for one study. One respondent told us she spent 60 hours a week on the Web. When she got into the lab, it was clear she didn't know how to use a browser. When I asked her what she does on the Web, she said this computer didn't look like hers at all. That she starts in a place where she clicks on a picture and it brings up her favorite game. Turns out, her son-in-law had set up a series of shortcuts on her desktop. She knew the games were on the Web, but that was all she knew about the Web.
Watch the insider jargon. If you're using industry or company terms for products or services that you want to test, you may prime respondents for what you're looking for and lead them to giving the right answer. Again, open-ended questions can help here. This is where you start looking at your product from the user's point of view.

Need help developing a screener? Need help with doing recruiting? Contact me about recruiting services my company offers. We've got a great process and a 90% show rate.

Testing in the wild defined

2009-01-08T08:26:00.000-08:00

Lately I’ve been talking a lot about “usability testing in the wild.” There are a lot of people out there who make their livings as usability practitioners. Those people know that the conventional way to do usability testing is in a laboratory setting. If you have come to this blog from outside the world of user experience research, that may never have occurred to you.

Some of the groups I’ve been working with recently do all their testing in the wild. That is, they never set foot in a lab, but instead conduct evaluations wherever their users normally do the tasks the groups are interested in observing. That setting could be a grocery store, City Hall, on the bus, or at a home or workplace – or any number of other places.

A “wild” usability test sometimes has another feature: it is lightly planned or even ad hoc. Just last night I was on a flight from Boston to San Francisco. I’ve been working with a team to develop a web site that lists course offerings and a way to sign up to take the courses. As I was working through the navigation and checking wireframes, the guy in the seat next to me couldn’t help looking over at my screen. He asked me about the site and the offerings, explaining that they looked like interesting topics. I didn’t have a prototype, but I did have the wireframes. So, after we talked for a moment about what he did for a living and what seemed interesting about the topics listed, I showed him the wireframe for the first page of the site and said, “Okay, from the list of courses here, is there something you would want to take?” He said yes, so I said, “What do want to do next, then?” He told me and I showed him the next appropriate wireframe. And we were off.

I learned heaps for the team about whether this user found the design useful and what he valued about it. It also gave me some great input for a more formal usability test later. Testing in the wild is great for early testing of concepts and ideas you have about a design. It’s one quick, cheap way to gain insights about designs so teams can make better design decisions.

Another awesome event you should know about: Web App Summit

2009-01-06T07:39:00.000-08:00

Made a resolution to improve your work skills? One really great event to do that at is UIE's Web App Summit.

The Summit is April 19-22, 2009, in Newport Beach, CA. There's a ton of great information at http://webappsummit.com. But let me give you a little preview. Though there are killer shorter sessions, the highlight of the Summit is two days of intensive workshops with world class, rockin' speakers:

Web 2.0 Strategy and Design—Riccardo La Rosa and Steve Mulder
Communicating Design: Essential Deliverables for Highly Effective Design Teams—Dan Brown
Web App Anatomy: Effective Interaction Design with Frameworks—Robert Hoekman
Web Application Form Design—Luke Wroblewski

Designing Great Interactive Experiences for Everyone: Implementing Ajax and Accessibility—Derek Featherstone
Achieving Reuse with Patterns and Components—Nathan Curtis
Web Standards for Web Applications—Molly Holzschlag
Wireframing and Prototyping for Highly Interactive Web Apps—James Box and Richard Rutter

If you decide to register, let them know you heard about it from me! Sign up right away to get a discount and an iPod nano.

Disclosure: You'll notice I'm not speaking at this. I'd go even if I didn't have personal connections to UIE.

Events: Speaking about usability testing in 2009

2009-01-01T17:56:00.000-08:00

I hate staying home. Here’s a list of the events I’m signed up to speak at in 2009. So far.

Feb 6	Washington, DC	NASED	Usability Testing Ballots
April 1	Seattle, WA	WritersUA	Conducting Usability Tests “in the Wild”
May 3-6	Atlanta, GA	STC	Rewriting the Voting Experience On Election Day
June 3	Online Webinar	Web Manager University	Quick, Easy & Insightful: Conducting Usability Testing in the Wild
June 12	Portland, OR	UPA	Improving the User Experience of Voting
July 7-11	Spokane, WA	IACREOT	Improving the User Experience of Voting
July 19-24	San Diego, CA	HCI International	User Experience in Elections: Poll Workers

Insights quickly and cheaply

2008-11-26T12:22:00.000-08:00

After I gave a day-long seminar and a short talk at UI 13, I sat down with Tim Keirnan of Design Critique to talk about doing usability testing in the wild for quick, cheap insights from users. Download that podcast.

Recruiting 101: Treat your test participants like humans

2008-11-26T12:11:00.000-08:00

One of the most often asked questions I get at talks and workshops is What about recruiting – how do I do a better job of that part of a usability test? One way is to ensure that you’re remembering that the people you recruit are humans. I wrote about this topic for Boxes & Arrows.

Just vote.

2008-11-04T06:45:00.000-08:00

Though many people who are eligible to vote were hindered (but not prevented from) registering; though there are obstacles to getting to precincts like having to work or not having transportation; though we have all read and heard the many stories about problems with voting machines -- a vote has rarely counted for so much in the history of America.

Please vote today.

If you will vote on paper: fill in the bubble completely, or join the arrow.

If you will vote on an electronic machine: check the review screen and the paper record if there is one.

And get your "I voted!" sticker.

Ditch the book - Come to a virtual seminar on "usability testing in the wild"

2008-10-20T09:22:00.000-07:00

I'm excited about getting to do a virtual seminar with the folks at User Interface Engineering (www.uie.com) on Wednesday, October 22 at 1 pm Eastern Time. I'll be talking about doing "minimalist" usability tests -- boiling usability testing down to its essence and doing just what is necessary to gather data to inform design decisions.

If you use my promo code when you sign up for the session -- DCWILD -- you can get in for the low, low price of $99 ($30 off the regular price of $129). Listen and watch in a conference room with all your team mates and get the best deal ever.

For more about the virtual seminar, see the full description.

Usability testing in the wild – ballots

2008-09-14T09:18:00.000-07:00

I’ve been busy the last few weeks doing some of the most challenging usability testing I’ve ever done. There were three locations where I did day-long test sessions. But that wasn’t the challenging part. The adventure came in testing ballots for the November election.

What was wild about it?
This series of tests came together through a project with the Brennan Center for Justice and the Usability Professionals’ Association. The Brennan Center released a report in July called Better Ballots, which reviewed ballot designs and instructions, finding that

hundreds of thousands of voters have been disenfranchised by ballot design problems
there has been little or no federal or state guidance on ballot design that might have been helpful to elections officials who define and design ballots at the local level
usability testing is the best way to ensure that voters can use ballots to vote as they intend

Also in the report, the Brennan Center strongly urged election officials to conduct usability tests on ballots. The recommendation to include usability testing in the ballot design process is a major revelation in the election world. The UPA Voting and Usability Project has developed the LEO Usability Test Kit to help local elections officials to do their own simple, quick usability tests of ballot designs.

But not all local elections officials were ready to do their own usability tests, and some wanted objective outsiders to help evaluate ballots for this particular, important upcoming election.

I did tests in three locations -- Marin County, California, Los Angeles County, California, and the home of Las Vegas in Clark County, Nevada -- with about 40 participants across the three locations. Several other UPA volunteers conducted tests and reviews in Florida, New Hampshire, and Ohio. In addition, UPAers trained local elections officials on usability testing and the LEO Test Kit in Ohio, Iowa, and a couple of other spots I can’t think of right now.

Pulling together a test in just a few days, including recruiting and scheduling participants
The Brennan Center report was released toward the end of July. Most ballots must be ready to print or roll out right now, the middle of September. The Brennan Center sent the report to every election department in the US and the response was great. Most requests came in in August, so among the five or six UPA Usability and Voting Project members available, we scrambled to cover the requests for tests.

We had the assistance of one of the Brennan Center staff to help coordinate recruiting, although it took some pretty serious networking to get people in to sessions on short notice, often within a few days.

The Brennan Center covered the expenses, but the time and effort spent by the people who worked with local elections officials and conducted the sessions was purely pro bono.

Not knowing what I would be testing until I walked onto the site
For two out of the three tests, I hadn’t seen exactly what I was going to be testing until I walked in the door of the election department. (I got the other ballot two days before the test.) This happened for a couple of reasons. Sometimes the local election official didn’t have a lot of information about what could be evaluated and how that might happen. Sometimes the ballot wasn’t ready until the last minute because of final filing deadlines or other constraints. Sometimes it was all of the above.

Fortunately, the main task is pretty straightforward: Vote! Use the ballot as you normally would. But there are neat variations. Are there write-ins possible? On an electronic voting machine, how do you change a vote? What if you’re mailing in a ballot – what’s different about that and how do design and instructions have to compensate for not having poll workers available to ask questions of?

Giving immediate results and feedback
So, we got copies of ballots or something close to final on an electronic voting machine. We’ve met briefly with the local elections officials (and often with their advisory committees). We’ve recruited participants (sometimes off the street). We’ve conducted 8 or 10 or 15 20-minute sessions in one day. Now it’s time to roll up what we saw in the sessions and to talk with the person who owns the ballot about how the evaluations went.

Handling enthusiastic observers and activists
A lot of people are concerned with the usability, accessibility, and security of ballots and voting systems. You probably are. Some are more concerned about it than others. Those are the people who show up to observe sessions. They’re well informed, they’re enthusiastic, and they’re skeptical. The observers and activists (many signed up to be test participants) were also keenly interested in understanding this activity. How was this different from focus groups or reviews by experts? How do we know that the problems we’ve witnessed are generalizable to other voters in the jurisdiction?

The good news: Mostly, the ballots worked pretty well. The local elections officials usually have the ability to make small changes at this stage and they were willing, especially to improve instructions to voters. By doing this testing, we were able to effect change and to make voting easier for many, many voters. (LA County alone has more than 3 million registered voters.)

Links:
Brennan Center for Justice report Better Ballots
http://www.brennancenter.org/content/resource/better_ballots/

UPA’s Voting and Usability Project
http://www.usabilityprofessionals.org/civiclife/voting/
voting@usabilityprofessionals.org

LEO Usability Testing Kit
http://www.usabilityprofessionals.org/civiclife/voting/leo_testing.html

Ethics guidelines for usability and design professionals working in elections
http://www.usabilityprofessionals.org/civiclife/voting/ethics.html

Information about being a poll worker
http://www.eac.gov/voter/poll%20workers

EAC Effective Polling Place Designs
http://www.eac.gov/election/effective-polling-place-designs

EAC Election Management Guidelines
http://www.eac.gov/election/quick-start-management-guides

Retrospective review and memory

2008-08-19T08:20:00.000-07:00

One of my favorite radio programs (though I listen to it as a podcast) is Radiolab, “ a show about science,” which is a production of WNYC hosted by Robert Krulwich and Jad Abmurad and distributed by NPR. This show contemplates lots of interesting things from reason versus logic in decision making to laughter to lies and deception.

The show I listened to last night was about how memories are formed. Over time, several analogies have developed for human memory that seem to be related to the technology available at that time. Robert said he thinks of his memory as a filing cabinet. But Jad, who is somewhat younger than Robert, described his mind as a computer hard disk. Neurologists and cognitive scientists they talked to, though, said No, memory isn’t like that at all. In fact, we don’t store memories. We recreate them every time we think of them.

Huh, I thought. Knowing this has implications for user research. For example, there are several points at which usability testing relies on memory: the memory of the participant if we’re asking questions about the past behavior; the memory of the facilitator for taking notes, analyzing data, and drawing inferences; the memories of observers in discussions about what happened in sessions and what it means.

Using a think-aloud technique – getting participants to say what they’re thinking while working through a task – avoids some of this. You have a verbal protocol as “evidence.” If there’s disagreement about what happened among the team members, you can go back to the recording to review what the participant said as well as what they did.

But there are times when think-aloud is not the right technique, either because the participant cannot manage the divided attention of doing a task and talking about it at the same time, or because of other circumstances. In those situations, you might think about doing retrospective review, instead.

“Retrospective review” is just a fancy name for asking people to tell you what happened. If you have the tools and time available, you can go to a recording after a session, so the participant can see what she did and respond to that by giving you a play-by-play commentary.

As soon as participants start viewing or listening to the beginning of an episode – up to 48 hours after doing the task – they’ll remember having done it. They probably won’t be able to tell you how it ended. But they will be able to tell you what’s going to happen next.

And that’s the really useful thing about doing retrospective review. As the participant recreates the memory of the task, you can ask, “What happens next? What will you do next and why?” Pause. Listen. Take notes. And then start playing back the recording again. Sure enough, it’ll be like the participant said. Only now you know why.

Asking participants what happens next in their own stories also avoids most revisionist history. That is, if you ask participants to explain had what happened after they view it, they may rationalize what they did. This isn’t the same as remembering it.

Get a discount on UI 13 by using my name

2008-08-12T18:25:00.000-07:00

I'm excited about teaching a full-day workshop on "Usability Testing Guerilla Techniques: Collecting User Data on A Shoestring" at UI 13, which is happening in Cambridge, MA on October 13-16. For the full program, see http://www.uie.com/events/uiconf/2008/.

Much of the material for the session will be based on my new book (with Jeff Rubin), Handbook of Usability Testing Second Edition. BUT this time, we're focusing on the budgets of usability testing. If you have $50 what should you do with it? If you have $50,000 what's the best way to spend it in the context of usability testing?

I will share a minimal budget and schedule for a usability study, walking through the costs and schedule for planning, recruiting participants, designing tasks, moderating the study, and analyzing the results. I'll also show you how to produce effective results with even less money and time. We'll talk about what parts of the process you can live without, and which essentials you’ll need to keep.

When you register for the conference, use my last name -- chisnell -- in the promo code and get a discount! If you sign up using "chisnell" as the promotion code, you will receive $30 off each single day registration. If you sign up for all 4 days, you get $120 off *and* a free limited-edition UI13 Flip Ultra video camcorder.

Here's a secret: You don't have to sign up for my workshop to get the discount; you can sign up for any of the great offerings. But you can still use my name.

Pass it on...

Thanks. Looking forward to seeing you in Cambridge in October.

Call to action: Volunteer to test ballots or be a poll worker

2008-08-11T11:05:00.000-07:00

I am a member of the Usability Professionals' Association (UPA) Usability and Voting Project. We're gearing up now to help local elections officials test their ballots and other election materials for usability for the November election.

Ballots are being defined and designed now. Most have to be completed within a few weeks.

Want to help test ballots for usability? Step up! We'd love to have you. Questions? Write to us at voting@usabilityprofessionals.org.

In the meantime, a few pointers about working on elections.

1. Usability and Voting is non-partisan
First, and most importantly, this is a non-partisan effort. Whatever our personal political beliefs, they stay separate from this work. You can read our ethics guidelines for usability and design professionals on http://www.usabilityprofessionals.org/civiclife/voting/ethics.html

2. Sign up to be a poll worker
We know that it’s hard to find time to volunteer. But there is something that everyone in the US can do that takes just a day (or part of a day) of your time. Sign up to be a poll worker. With record turnout expected this November, new poll workers are needed more than ever to meet the target of 2 million poll workers set by the EAC. To find out how to be a poll worker, contact your local elections office or visit http://www.eac.gov/voter/poll%20workers

3. Volunteer to help your local election officials with usability testing
One way to help ensure that ballots get a good usability test is to volunteer to run the test yourself. If you do:

Please use the LEO Usability Testing Kit. Every election official has received a printed copy of the report, and it would be useful – especially in this first wave -- if we all used the materials they have already been introduced to, and spoke with one voice. Plus, your feedback will help improve the Kit. http://www.usabilityprofessionals.org/civiclife/voting/leo_testing.html
Remember that election officials are in the middle of their busiest, highest pressure work period – final preparations for a highly volatile presidential election. This is a time to offer to help, not offer blame.
Don't be discouraged if they say "not this time" – offer to contact them again in the spring for the off-season elections.
Before you make any recommendations, take a look at the EAC's Best Practices Guidelines for ballot designs as well as the recommendations in the Brennan Center report, Better Ballots. But, don't be surprised if you are told that it "can't be done". Not only do many voting systems have limitations that can get in the way, but elections are governed by state laws and regulations, as well as expectations from current practice. See http://www.eac.gov/election/effective-polling-place-designs and http://www.brennancenter.org/content/resource/better_ballots/

4. Keep in touch
Let us know if you have any contacts with elections folks. We're trying to keep track of any usability work, so we can get a full picture of the impact of our efforts.

We'll be happy to talk to anyone who is interested, and support them in any way we can. Our goal is to make better design and usability part of every aspect of elections, integrated into the normal processes.

Links:
LEO Usability Testing Kit
http://www.usabilityprofessionals.org/civiclife/voting/leo_testing.html

Ethics guidelines for usability and design professionals working in elections
http://www.usabilityprofessionals.org/civiclife/voting/ethics.html

Information about being a poll worker
http://www.eac.gov/voter/poll%20workers

EAC Effective Polling Place Designs
http://www.eac.gov/election/effective-polling-place-designs

EAC Election Management Guidelines
http://www.eac.gov/election/quick-start-management-guides

Brennan Center for Justice report Better Ballots
http://www.brennancenter.org/content/resource/better_ballots/

UPA’s Voting and Usability Project
http://www.usabilityprofessionals.org/civiclife/voting/
voting@usabilityprofessionals.org.

Getting ready for sessions: Don’t forget…

2008-08-09T16:47:00.000-07:00

There are a bunch of things to do to get ready for any test besides designing the test and recruiting participants.

make sure you know the design well enough to know what should happen as the participant uses it
copy any materials you need for taking notes
copy of all the forms and questionnaires for participants, including honorarium receipts
organize the forms in some way that makes sense for you. (I like a stand-up accordion file folder, in which I sort a set of forms for each participant into each slot. I stand up the unused sets and then when they’ve been filled out, they go back in on their sides.)
check in with Accounting or whoever on money for honoraria or goodies for give-aways
get a status report from the recruiter
double-check the participant mix
make sure you have contact information for each participant
check that you have all the equipment, software, or whatever that you need for the participant to be able to do tasks
run through the test a couple of times yourself
double-check the equipment you’re going to use (I use a digital audio recorder, so I need memory sticks for that, along with rechargeable batteries)
charge all the batteries
double-check the location

Which gets us to where you’re going to do the sessions. But let’s talk about that later.

Where usability testing fits into your research strategy

2008-07-29T09:16:00.000-07:00

What, you don’t have a research strategy? Let’s think about the future here.

It’s not uncommon – and not bad – to be working in the present, reacting to the ever-growing demand for usability testing in your organization. “Ever-growing” is good. But when Jared Spool asked me to do a podcast with him recently to talk about what I think makes the difference between a good user experience team and a great user experience team, it got me thinking.

The recipe, based on my observations in dozens of corporations, comes down to these three main ingredients:

Vision
Strategy
Involvement

Vision is an overused word, but here I mean that you and your team have visualized the ideal customer experience -- no limits, no constraints. Imagine the best possible interactions a customer could have with your organization at every touch point. Write it down.

Strategy means that you have a plan for reaching the vision. Over the long term, you can learn about and take into account customers’ contexts and goals while matching those up to the goals and objectives of the business.

Involvement calls all interested people in the business together (and that really should be everyone from management to design to development to support and anyone else in the organization) to embrace the vision and carry out the strategy across disciplines.

But I haven’t said much about usability testing yet. Where does it fit in? Everywhere. Part of my strategy would be to teach as many people in the organization to do usability testing as possible. You probably can’t do all the testing that is wanted (let alone needed). If you teach others to do it and coach them along the way, the customer ultimately benefits as the organization gains a closer, smarter understanding of the customer experience and can make evidence-based decisions about how to get to the ideal experience it shares a vision of.

Usability testing and democracy: evaluating ballot designs makes the headlines

2008-07-21T17:33:00.000-07:00

Today the Brennan Center for Justice at the law school at NYU released a major report about the impact of poor ballot designs and unclear instructions on voters and the importance of usability testing.

Among the highlights is an overview of the Usability Professionals’ Association (UPA) usability testing kit for local election officials (the LEO Usability Testing Kit). Members of the UPA Usability in Civic Life Project are working with Brennan Center to provide direct training for election officials.

The report is titled Better Ballots, and can be found on the Brennan Center site:
http://www.brennancenter.org/

http://www.brennancenter.org/content/resource/better_ballots/

The report released today, and 3 articles in USA Today and the New York Times highlight it.

Study: Poor ballot designs still affect U.S. elections
http://www.usatoday.com/news/politics/election2008/2008-07-20-ballots_N.htm

Ballot designs are 'literacy test for voters'
http://www.usatoday.com/news/politics/election2008/2008-07-20-ballot-inside_N.htm

Influx of Voters Expected to Test New Technology
http://www.nytimes.com/2008/07/21/us/21voting.html

Stop writing reports

2008-07-20T12:20:00.000-07:00

I’ve had a few questions from readers lately about standardizing reports of usability test results. Why is there no report template in the Handbook? There’s no “template” for a final report because I think you probably shouldn’t be writing reports. Or at least written reports should be minimal. Mini.mal. Though the outline should basically be what’s in the Handbook, what you put in your report depends on

The test design and plan
What your team needs and can use

And let’s use “report” in the loosest possible way: delivering information to others. That’s it. Your report doesn’t have to be a long, prose-based, descriptive tome. (Not that there’s anything wrong with that.) And the delivery method doesn’t have to be paper.

That leaves a lot of options, from an email with a bulleted list of items, to a “top line” post on a blog or wiki that lightly covers the main trends and patterns. In the middle of the range might be a classic usability test report that describes results and findings in some detail. (I personally dislike slide decks as reports, but a lot of organizations do them.) These will all work for any type of test. For summative tests, you may want to go as far as the CIF, or Common Industry Format, established by the International Standards Organization. BUT if your team has observed the sessions and attended the debriefs, you probably don’t need much of a report. They won’t read it; everything has been discussed and decided already. Whatever you deliver is simply a record of that set of decisions and agreements.

Making it easy to collect the data you want to collect

2008-07-15T19:21:00.000-07:00

As I have said before, taking notes is rife with danger. It’s so tempting to just write down everything that happens. But you probably can’t deal with all that data. First, it’s just too much. Second, it’s not organized. Let’s go back to the example from the last post. The research question was Do people make more errors on one version of the system than the other? And we chose these measures to find out the answer:

Count of all incorrect selections (errors)

Count and location of incorrect menu choices
Count and location of incorrect buttons selected

Count of errors of omission
Count and location of visits to online help
Number and percentage of tasks completed incorrectly

Here’s what I did. First, I listed the page (sometimes I put in a screen shot), with the correct action and then the possible (predicable, known) errors, like this:

I hope you can see how to turn this into a pick list. You can just check these items off or number them if people do more than one of the errors. For my note taking forms, I added a choice for “Other” in case something happened that I hadn’t anticipated. This is from a different study, but you get the idea: This is what your notes end up looking like. Even if you’re not doing statistical analyses, it’s easy to hold up these pages together across participants to see how many had problems and how severe the problems were.

Translating research questions to data

2008-07-09T18:44:00.000-07:00

There’s an art to asking a question and then coming up with a way to answer it. I find myself asking, What do you want to find out? The next question is How do we know what the answer is?

Maybe the easiest thing is to take you through an example.

Forming the right question
On a study I’m working on now, we have about 10 research questions, but the heart of the research is about this one:

Do people make more errors on one version of the system than the other?

Note that this is not a hypothesis, which would be worded something more like, “We expect people to make more mistakes and to be more likely to not complete tasks on the B version of the system than on the A version of the system.” (Some would argue that there are multiple hypotheses embedded in that statement.)

But in our study, we’re not out to prove or disprove anything. Rather, we just want to compare two versions to see what works well about each one and what doesn’t.

Choosing data to answer the question
There are dozens of possible measures you can look at in a usability test. Here are just a few examples:

Number and percentage of tasks completed correctly with and without prompts or assistance
Number and type of prompts given
Number and percentage of tasks completed incorrectly
Count of all incorrect selections (errors)
Count of errors of omission
Count of incorrect menu choices
Count of incorrect icons selected
Count of calls to the help desk
Count of user manual accesses
Count of visits to the index
Count of visits to the table of contents
Count of “negative comments or mannerisms”
Time required to access information in the manual
Time required to access information in online help
Time needed to recover from error(s)
Time spent reading a specific section of a manual
Time spent talking to help desk
Time to complete each task

Which data will answer the question? We’re mainly concerned about numbers of errors people make, but at some point we might want to know the types of errors. So we can eliminate any of the time measures.

So, we decided to count incorrect selections. We’re also going to count it every time people leave something out that they should have done. This of course means that we have to know what they should have done, which isn’t always possible in a very formative test. Here, in a summative test, we do.

In our study, we’re not giving hints if people get stuck, so we’re not counting prompts or assistance.

We do want to know where in each system people have questions or problems, so in addition to tracking incorrect steps, we’re going to track where in the system people go to the online help and whether they complete the tasks correctly after going to help.

So here’s our list of data measures to answer the research question: Did people make more errors on one version of the system than the other?

Count of all incorrect selections (errors)
Count and location of incorrect menu choices
Count and location of incorrect buttons selected
Count of errors of omission
Count and location of visits to online help
Number and percentage of tasks completed incorrectly

How do you take notes on that? I’ll talk about that in the next post.

Data collecting: Tips and tricks for taking notes

2008-07-01T11:12:00.000-07:00

A common mistake newbies make when they’re conducting their first usability tests is taking verbatim notes.

Note taking for summative tests can be pretty straightforward. For those you should have benchmark data that you’re comparing against or at least clear success criteria. In that case, data collecting could (and probably should) be done mostly by the recording software (such as Morae). But for formative or exploratory tests, note taking can be more complex.

Why is it so tempting to write down everything?
Interesting things keep happening! Just last week I was the note taker for a summative test in which I noticed (after about 30 sessions), that women and men seemed to be holding the stylus for marking what we were testing differently and that it seemed that difference was causing a specific category of errors.

But the test wasn’t about using the hardware. This issue wasn’t something we had listed in our test plan as a measure. It was interesting, but not something we could investigate for this test. We will include it as an incidental observation in the report as something to research later.

Note-taking don'ts

Don’t take notes yourself if you are moderating the session if you can help it.
Don’t take verbatim notes. Ever. If you want that, record the sessions and get transcripts. (Or do what Steve Krug does, and listen to the recordings and re-dictate them into a speech recognition application.)
Don’t take notes on anything that doesn’t line up with your research questions.
Don’t take notes on anything that you aren’t going to report on (either because you don’t have time or it isn’t in the scope of the test).

Tips and tricks

DO get observers to take notes. This is, in part, what observers are for. Give them specific things to look for. Some usability specialists like to get observer notes on large sticky notes, which is handy for the debriefing sessions.
DO create pick lists, use screen shots, or draw trails. For example, for one study, I was trying to track a path through a web site to see if the IA worked. I printed out the first 3 levels of IA in nested lists in 2 columns so it fit on one page of a legal sized sheet of paper. Then I used colored highlighters to draw arrows from one topic label to the next as the participant moved through the site, numbering as I went. It was reasonably easy to transfer this data to Excel spreadsheets later to do further analysis.
DO get participants to take notes for you. If the session is very formative, get the participants to mark up wireframes, screen flows, or other paper widgets to show where they had issues. For example, you might want to find out if a flow of screens matches the process a user typically follows. Start the session asking the participant to draw a boxes-and-arrows diagram of their process. At the end of the session, ask the participant to revise the diagram to a) get any refinements they may have forgotten, b) see gaps between their process and how the application works, or c) some variation or combination of a and b.
DO think backward from the report. If you have written a test plan, you should be able to use that as a basis for the final report. What are you going to report on? (Hint: the answers to your research questions, using the measures you said you were going to collect.)

Observers are your friends

2008-05-30T11:03:00.000-07:00

Research that you do alone ends up in only your head. No matter how good the report, slide deck, or highlights video, not all the knowledge gets transferred to your teammates. This isn’t your fault. It just is.

So what to do? Enlist as many people on your team as possible to help you by observing your usability testing sessions. You can even give your observers jobs, such as time keeper if you’re measuring time on task. Or, if you are recording sessions, it could be an observer’s job to start and stop the recordings and to label and store them properly.

The key is to involve the other people on the team – even managers – so they can

Help you
Learn from participants
Share insights with you and other observers
Buy in
Reach consensus on what the issues are and how to solve them

Who should observe: Everyone
Ideally, everyone on the design and development team should observe sessions. Every designer, every programmer, every manager on the project should watch as real people use their designs.

Each of the observers should watch as many sessions as possible. (Okay, you could settle for two sessions.) Seeing only one session is just a snapshot; what happens in that session may not be similar to what happens in the other sessions. If the observer who saw only one participant is in an influential position in the organization, that one set of observations may outweigh others when it comes time to get some consensus from observers about what the issues are. (Hint: You don’t really want that.)

Training observers: Ground rules are essential
Observers can watch from a separate space – most labs are set up this way – or they can be present in the testing room with you and the participant during each session. Either way, you should brief them on

What to look for
How they might see it
How to behave with participants
What to do (and not do) with their observations

It probably isn’t enough just to hand out a list of rules if this is the first usability test these people will watch. Consider holding a short training session to explain how the usability test will be conducted and what they can expect in watching sessions. Be plain (but diplomatic, of course) about your expectations of your observers.

Here’s one of my favorite sets of guidelines for observers who will be in the testing room with you and the participant.

If observers will not be in the testing room with you and the participant but rather will be watching from a separate viewing room, ask observers to resist redesigning or finding solutions until the end of the study (or until the day’s debrief, whichever works best in your situation). This is another good reason to give each observer a little job – it keeps them occupied at different things that demand the attention that might otherwise be given to premature redesign.

Working with observers: Reaching consensus
No small part of working with observers is understanding what their separate stakes in the test are. When you know this, you can facilitate discussion among observers effectively. If you don’t know, you’re probably working harder than you need to to gain consensus.

Consensus is important. Reaching consensus among observers means that

There’s a shared vision of where the design should go. This is good for the team and the design.
Everyone uses the same descriptions for issues. Having a common language for talking about the usability issues among departments or groups will help the diverse group visualize the user experience together.
Nuances and exceptions have been discussed and agreed on. Anything that isn’t optimally feasible to fix can be negotiated among design and development team members in a daily debrief or final design direction meeting.
There should be no surprises when you publish the report. Because team members observed sessions and discussed the issues together, insights and outcomes in the report should reflect the agreed direction from those observations and group discussions.

Feeling lonely? Gather up some observers for your next usability test from your design and development teams. Make them feel welcome, give them duies, and work with them to understand the issues. They’ll thank you for it.

The importance of rehearsal

2008-05-20T07:33:00.000-07:00

You have designed a study. Everyone seems to be buying in. Scheduling participants is working out and the mix looks good. What’s left to be done except just doing the sessions? Three things:

Practice.
Practice.
Practice.

There are three rounds of practice that I do before I do a “real” session. Jeez, I can hear you say, why would I need to practice so much? Why would you, Dana, who have been doing usability testing for so many years, need to practice so much? I do it for a couple of reasons:

It gives me multiple opportunities to clarify the intent of the test, the tasks, and the data measures.
I can focus on observing the participant in each regular session because any kinks have been worked out.

Walk through the script and gather tools and materials
The first is to walk through my test plan and script. I read the script aloud even though I’m by myself. While I’m doing that, I do two things: adjust the wording to sound more natural, and gather tools and materials I’ll need to do the sessions.

Do a dress rehearsal or dry run
For the second round of practice, I do a dry run of the now refined script with someone I know filling the role of the participant. We do everything you would normally do in a session, from greeting and filling out forms, to doing tasks, to closing the session. I might occasionally stop the session to adjust the script or to make notes about what to do differently next time. I might even ask the participant (usually a friend, neighbor, or colleague) questions about whether the test is making sense. It’s a combination of dress rehearsal and “logic and accuracy” test to get the sequence down and to make sure you’ve got all the necessary pieces.

Pilot the protocol
Finally, there’s the pilot test session. In this pilot, I work with a “real” participant – someone who was screened and scheduled along with all of the other participants. I conduct the session in the same way I intend to conduct all of the following sessions. The twist this time is that observers from the design team should be present. At the end of the session, I debrief with them about the protocol.

Don’t waste good participant data
There have been times when I’ve been rushed by a client or was just too cavalier about going into a usability test and did not rehearse. I paid for it by having rough sessions that I couldn’t use all the data from. Every time it’s a reminder that preparation and practice are as important to getting good data as a good test design is.

Are you doing “user testing” or “usability testing”?

2008-05-13T08:51:00.000-07:00

Calling anything user testing just seems bad. Okay, contrary to the usual content on this blog – which I’ve tried to make about method and technique – this discussion is philosophical and political. If you feel it isn’t decent to talk about the politics of user research in public, then you should perhaps click away right now.

I know, talking about “users” opens up another whole discussion that we’re not going to have here, now. In this post, I want to focus on the difference between “usability testing” and “user testing” and why we should be specific.

When I say “usability test,” what I’m talking about is testing a design for how usable it is. Rather, how unusable it is, because that’s what we can measure: how hard is it to use; how many errors do people make; how frustrated do people feel when using it. Usability testing is about finding the issues that leave a design lacking. By observing usability test sessions, a team can learn about what the issues are and make inferences about why they are happening to then implement informed design solutions.

If someone says “user testing,” what does that mean? Let’s talk about the two words separately.

First, what’s a “user”? It is true that we ask people who use (or who might use) a design to take part in the study of how usable the design is, and some of us might refer to those people as “users” of the product.

Now, “testing” is about using some specified method for evaluating something. If you call it “user testing,” it sure sounds like you are evaluating users, even though what you probably mean to say is that you’re putting a design in front of users to see how they evaluate it. It’s shorthand, but I think it is the wrong shorthand.

If the point is to observe people interacting with a design to see where the flaws in the design are and why those elements aren’t successful, then you’re going beyond user testing. You’re at usability testing. That’s what I do as part of my user research practice. I try not to test the users in the process.