Usability testing is HOT

For many of us, usability testing is a necessary evil. For others, it’s too much work, or it’s too disruptive to the development process. As you might expect, I have issues with all that. It’s unfortunate that some teams don’t see the value in observing people use their designs. Done well, it can be an amazing event in the life of a design. Even done very informally, it can still show up useful insights that can help a team make informed design decisions. But I probably don’t have to tell you that.

Usability testing can be enormously elevating for teams at all stages of UX maturity. In fact, there probably isn’t nearly enough of it being done. Even on enlightened teams that know about and do usability tests, they’re probably not doing it often enough. There seems to be a correlation between successful user experiences and how often and how much the designers and developers spend time observing users. (hat tip Jared Spool)

Observing people using early designs can be energizing as designers and developers get a chance to see reactions to ideas. I’ve seen teams walk away with insights from observing people use their designs that they couldn’t have got any other way – and then make better designs than they’ve ever made. Close to launch, it is exciting – yes, exciting – to see a design perform as useful, usable, and desirable.

I’ve been negative on usability testing and our failure of imagination regarding bringing the method up to date, lately. But there’s a lot of good to any basic usability test. In fact, I went looking for the worth, the value, the alluring in usability testing a few weeks ago when I asked on Quora, “What’s the sexiest thing about usability testing?”

Some of the answers surprised me. Some of the answers were more about what people love about usability testing than what makes it seductive. But let’s go with seductive. People who find usability testing hot say it’s about data that can end the opinion wars, revelations and surprises, and getting perspective about real use, motivations, and context of use.  Okay. We’re nerds.

The kiss of data

We always learn from users. Of course, we could just ask. But observing is so much more interesting. People do unpredictable things; they create workarounds, hacks, and alternative paths to make tools fit for their use.

This is the best case I can think of for watching rather than asking. From this observing, we get data. Juicy, luscious data like verbal protocols, task success rates, and physical behavior. This package makes it much easier to make good design decisions because we know have evidence on which to create theories about what should work better. There’s nothing like having hard evidence for going with a design direction – or changing direction.

Voyeuristic revelations

When designers, developers, and stakeholders of all persuasions get to observe people using a design – especially the first time – there’s often an “ah ha!” moment. (That’s the clean version.) Observers exclaim, “Wow, that was amazing!” when they see something surprising, both the good and the bad. The reaction that follows a completed usability study often is, “Damn. I wish we’d done this years ago. It would have saved us a ton of rework!” After watching one over-qualified participant struggle with a design recently, I heard a client say, “If that guy can’t do it, we’re in serious trouble.” That’s powerful.

When participants are surprised, that’s when the real fun begins. Not everyone likes surprises in their user interfaces, especially if they’re not the delightful Easter egg kind. While a team hopes not to hear, “I feel lost and abandoned,” you’ve got to wonder how bad it’s been when a participant squeals, “Oh, my gosh! This is so much better! When can I have it?!” Those eureka moments can reveal what to do to improve a design or an experience.

Relationship dynamics

One of the magical things about observing users working with a design is that suddenly, disputes within the team melt away. Chances are, the disputing parties were both wrong because neither (unless they have a ton of experience already observing these kinds of users in this domain doing this task) could accurately predict how the user would behave and perform.

Now, even with observations from watching just one user, there’s data on which to base design decisions. Data trumps gut. Data outweighs feelings. Data can put to rest those endless, circular discussions where inevitably, the person with the biggest paycheck or the most important title wins. The opinion wars come to an end.

When the whole team is involved in deciding what to test and observing sessions, everyone can share in making and carrying out agreed design decisions. Whenever a question comes up where no one knows but everyone has an opinion, the answer in a team doing usability testing is, “Let’s do some user research on that,” or “Let’s find out what users do.”

For the love of users

It’s so easy to get caught up in the business goals and issues with the underlying technology of a design. It’s so easy to stay in the safe bubble of the office, cranking out code, designs, plans, and reports. It’s easy to lose touch with users.

Teams that spend a couple of hours observing their users every few weeks keep that connection. They fall in love with their users. They relish the chance to see for themselves why people do the things they do with designs.

Getting out of their own heads, a successful team uses usability testing to get perspective, learn about users’ contexts, and remember the people and their stories. For these teams, usability testing is inspiring. And that’s hot.

What’s sexy about usability testing?

Observing people use a design can be revelatory. It turns up the volume on design by helping teams make informed design decisions. What’s sexy about usability testing? Data for evidence-based design. Ending opinion wars. Knowing users from observations and surprises. Getting perspective and knowledge of context of use.

The UX equivalent of a romantic dinner or a walk on the beach? Perhaps not, even for a geek girl like me. But it can be exciting, fun, funny, encouraging, and empowering. Just what you want from a relationship. That’s pretty seductive, if you ask me.

Bonus research: Do the recruiting yourself

There are some brilliant questions on Quora. This morning, I was prompted to answer one about recruiting.

The question asker asked, How do I recruit prospective customers to shadow as a part of a user-centered design approach? The asker expanded, thusly:

I'm interested in shadowing prospective customers in order to better understand how my tool can fit into their life and complement, supplement, or replace the existing tools that they use. How do I find prospective customers? How do I convince them to let me shadow them?

Seemed like a very thoughtful question. I have some experience with recruiting for field studies and other user research, so I thought I might share my lessons learned. Here's my answer. Would love to hear yours.

I've learned many recruiting lessons the hard way. Recruiting can be challenging, but it can also be really interesting. I always find that I learn a lot in the recruiting process. Here are my pointers:

Do the recruiting yourself. This gives you bonus research data about your people, and you may learn things you hadn't anticipated that will influence how you conduct the study. It also starts a relationship with people that gets them invested in taking part. It's the start of a conversation with you and your organization. They're more likely to trust your motives and give you a deeper, richer view of their lives.

Focus on behavior, not demographics. If you want people to use your tool to do a particular thing, look for people who already do that somehow. For example, if you want people to use your design to store their photos and music and other content, find people who have a lot of that type of content and who are concerned about losing it. If you want people to use your design to generate invoices, find people who are doing that now and observe what they do to generate invoice when they're in the mode of doing it. If you want people to use your design to remember to take their medicines at the right time in the right dosage, find people who have persistent conditions that need medication and who have been diagnosed with the type of condition you want to help them deal with.

You'll see that I never once mentioned age, sex, income, location, education level, marital status - or any of those things that marketers go on. Because it doesn't usually matter. What matters for UX design is behavior. Do people do the thing you want to make a design for?

Be as generous as you can with incentive/honorarium/compensation. People love gifts. They also love cash. The point is to show your appreciation for their help. They're doing you a huge favor by spending time with you and letting you see their lives. Offer something your kind of people want.
 
Be flexible about days, times, number of hours. If you want to shadow people, you have to do it when they're in the mode of doing the things you want to observe. My experience is that most interesting behavior (unless it is truly work related) doesn't happen between 8am and 5pm Monday through Friday.

Get participants to opt in to the study. Sourcing for a field study is one of the most challenging steps to recruiting. A carefully worded ad on Craigslist can attract the right kind of people. Going through community organizations or professional associations can work well.

We practice something we call "snowball recruiting." That means putting the word out among friends, family, and colleagues about the type of people you're looking for and what you want to do in the session and asking your close ties to help you find people. Advantage: Participants come lightly pre-screened, so you know they're not nuts. and they're likely to be reliable because someone you know sent them. Snowball recruiting will also help establish you as legit with potential participants. The reputation management goes both ways. Disadvantage: This approach can take a little longer to generate leads. For a shadowing or field study, it can take 3-4 weeks to get people to come to you, screen them, and then schedule them to do the session you ultimately want to do.

You can recruit from Craigslist in a much shorter period, because the response is usually great, depending on the geographic area you're recruiting in. However, there's more filtering and more screening to be done.


Remember that every sample has biases. UX research is based on convenience samples rather than random or representative samples. That's fine because you're focusing on behavior and performance rather than generalized opinions and preferences. But the sources do influence the expertise and world view that people have. For example, if you're observing people's behavior around online security and you recruit just from Craigslist, you may find that most of the people who respond are not very concerned with protecting their online accounts. Or you might find the opposite.


Be genuinely, authentically interested in the people you want to observe without being creepy. These people are letting you into their lives. So, explain what you do and what you're hoping to learn from the study. Treat them as your partner in answering your research questions. After all, you can't find out what you want to know without them.

Follow up. Send a hand-written thank-you note. Yes, this is in addition to the incentive.



I've written a lot about recruiting for user research and usability tests. You can see all those articles here: http://usabilitytestinghowto.blogspot.com/search/label/recruiting

Is your team stuck in a bubble?

This happens. The team is heads down, just trying to do work, to make things work, and then you realize it. Perspective is gone.

Recently I gave a couple of talks about usability testing and collaboratively analyzing data. There was a guy in the first row who was super attentive as I showed screen shots of web sites and walked the attendees through tasks that regular people might try to do on the sites.

Sweat beaded on his brow. His hands came up to his forehead in the way that someone who has had a sudden realization reacts. He put his hand over his mouth. I assumed he was simply passionate about web design and was feeling distressed about the crimes this web site committed against its users.

Turns out, he was the web site's owner.

This I found out at a break. When people started filing in from lunch to start the next session, this fellow appeared in my second session. I had time to talk with attendees, so I decided to approach him. "Hi. I noticed you were in my first session. Glad you're back. I hope the first was useful." He said yes, he had found it useful. But he frowned. "You look puzzled. Do you have a question I didn't answer?"



The bubble is insidious

"No," he said. "But it's clear that I have been -- along with a whole lot of other people -- out of touch."

"Oh? You got some insights today, already?"

"Some especially applicable insights, actually. The site you used this morning as your example is the site I work on every day." He gave a sad grin.

I knew this day would come. I would get caught out critiquing or running a demonstration on a site for which the owner was present. That day had arrived.

"I should have talked with you beforehand," I said. "The site has some classic problems. That's why I chose it as an example. It is one of dozens of sites in this domain that have similar issues. If I did or said anything that embarrassed you or your team, I apologize."

He sighed. "Not at all. You can't be embarrassed by something you weren't aware of." He went on, "We hadn't looked at the site at all from the point of view of users outside the organization. We've been in a bubble."

He actually seemed grateful. "Ah. That explains it," I said.

We chatted some more about the political pressures and the technology constraints that his team -- most teams -- faced in creating a great web site and maintaining it.  There had been some usability testing on intranets and even on extranets. But it was a few years ago. And the audience for the public-facing web site was different from the internal-facing web apps.


Perspective comes from observing real users doing real stuff

The best tool for resolving disputes within a design team, for making design decisions based on data rather than opinion, is sitting next to someone who is a real person who wants to accomplish something as they use your design to do it.

Some people call this usability testing. Call it whatever you want (except "user testing"). You can make it simple or complex, but when boiled down to its essence there are three ingredients:

- Someone to try out your design.
-  Somewhere to test.
- Something to study.

That's it. You can do it by the book, or you can do it very simply and ad hoc. The insights come from observing, first hand. I've seen just an hour of observation get many teams out of their own, customized bubbles.


Supporting great design: features of bubble prevention

Fortunately, my new friend stayed for the second session, in which I gave my recipe for supporting great experiences:

- Each phase includes input from users.

- The team is made up of people each with multiple skills from various disciplines.

- Management of the team is supportive an enlightened about the importance of the user experience.

- Everyone is willing to learn as they go along.

- The team has defined their usability goals and knows how they will measure their success.

Note that of the five attributes, two are directly about perspective (input from users; learning). Another two are about creating an infrastructure for getting and using that perspective (multidisciplinary team; setting usability goals). The remaining one (enlightened management) means there's support for getting and keeping perspective.

The importance of perspective cannot be overstated. Teams that meet with users regularly – every week or every month – turn out great experiences. Observing users regularly, at every phase of a design, gives a team evidence on which to make design decisions. More importantly, that act of being present with users, can bring the team together, enlighten management further, and give a needed break from the rarefied space most of us work in every day.

Get out of your head and into your users' .