Wilder than testing in the wild: usability testing by flash mob

It was a spectacularly beautiful Saturday in San Francisco. Exactly the perfect day to do some field usability testing. But this was no ordinary field usability test. Sure, there’d been plenty of planning and organizing ahead of time. And there would be data analysis afterward. What made this test different from most usability tests?

• 16 people gathered to make 6 research teams 
• Most of the people on the teams had never met
• Some of the research teams had people who had never taken part in usability
  testing before 
• The teams were going to intercept people on the street, at libraries, in farmers’
• markets

Ever heard of Improv Everywhere? This was the UX equivalent. Researchers just appeared out of the crowd to ask people to try out a couple of designs and then talk about their experiences. Most of the interactions with participants were about 20 minutes long. That’s it. But by the time the sun was over the yardarm (time for cocktails, that is), we had data on two designs from 40 participants. The day was amazingly energizing.


How the day worked
The timeline for the day looked something like this:

8:00
Coordinator checks all the packets of materials and supplies

10:00
Coordinator meets up with all the researchers for a briefing

10:30
Teams head to their assigned locations, discuss who should lead, take notes, and intercept

11:00
Most teams reach their locations, check in with contacts (if there are contacts), set up

11:15-ish
Intercept the first participants and start gathering data

Break when needed

14:00
Finish up collecting data, head back to the meeting spot

14:30
Teams start arriving at the meeting spot with data organized in packets

15:00-17:00
Everybody debriefs about their experiences, observations

17:00
Researchers head home, energized about what they’ve learned

Later
Researchers upload audio and video recordings to an online storage space

On average, teams came back with data from 6 or 7 participants. Not bad for a 3-hour stretch of doing sessions.


The role of the coordinator
I was excited about the possibilities, about getting a chance to work with some old friends, and to expose a whole bunch of people to a set of design problems they had not been aware of before. If you have thought about getting everyone on your team to do usability testing and user research, but have been afraid of what might happen if you’re not with them, conducting a study by flash mob will certainly test your resolve. It will be a
lesson in letting go.

There was no way I could join a team for this study. I was too busy coordinating. And I wanted to be available in case there was some kind of emergency. (In fact, one team left the briefing without copies of the thing they were testing. So I jumped in a car to deliver to them.)

Though you might think that the 3-or-so hours of data collection might be dull and boring for the coordinator, there were all kinds of things for me to do: resolve issues with locations, answer questions about possible participants, reconfigure teams when people had to leave early. Cell phones were probably the most important tool of the day. 

I had to believe that the planning and organizing I had done up front would work for people who were not me. And I had to trust that all the wonderful people who showed up to be the flash mob were as keen on making this work as I was. (They were.)


Keys to making flash mob testing work
I am still astonished that a bunch of people would show up on a Saturday morning to conduct a usability study in the street without much preparation. If your team is half as excited about the designs you are working on as this team was, taking a field trip to do a flash mob usability test should be a great experience. That is the most important ingredient to making a flash mob test work: people to do research who are engaged with the project, and enthusiastic about getting feedback from users.

Contrary to what you might think, coordinating a “flash” test doesn’t happen out of thin air, or a bunch of friends declaring, “Let’s put on a show!” Here are 10 things that made the day work really well to give us quick and dirty data: 

1.    Organize up front
2.    Streamline data collection
3.    Test the data collection forms
4.    Minimize scripting
5.    Brief everyone on test goals, dos and don’ts
6.    Practice intercepting
7.    Do an inventory check before spreading out
8.    Be flexible
9.    Check in
10.    Reconvene the same day

Organize up front

Starting about 3 or 4 weeks ahead of time, pick the research questions, put together what needs to be tested, create the necessary materials, choose a date and locations, and recruit researchers.

Introduce all the researchers ahead of time, by email. Make the materials available to everyone to review or at least peek at as soon as possible. Nudge everyone to look at the stuff ahead of time, just to prepare.

Put together everything you could possibly need on The Day in a kit. I used a small roll-aboard suitcase to hold everything. Here’s my list:

• Pens (lots of them)
• Clipboards, one for each team
• Flip cameras (people took them but did most of the recording on their phones)
• Scripts (half a page)
• Data collecting forms (the other half of the page)
• Printouts of the designs, or device-accessible prototypes to test
• Lists of names and phone numbers for researchers and me
• Lists of locations, including addresses, contact names, parking locations, and public transit routes
• Signs to post at locations about the study
• Masking tape
• Badges for each team member – either company IDs, or nice printed pages with the first names and “Researcher” printed large
• A large, empty envelope

About 10 days ahead, I chose a lead for each of the teams (these were all people who I knew were experienced user researchers) and talked with them. I put all the stuff listed above in a large, durable envelope with the team lead’s name on it.

Streamline data collection

The sessions were going to be short, and the note-taking awkward because of doing this research in ad hoc places, so I wanted to make data collection as easy as possible. Working from a form I borrowed from Whitney Quesenbery, I made something that I hoped would be quick and easy to fill in and easy for me to understand what the data meant later.

Data collector for our flash mob usability test

The data collection form was the main thing I spent time on in the briefing before everyone went off to collect data. There are things I will emphasize more, next time, but overall, this worked pretty well. One note: It is quite difficult to collect qualitative data in the wild by writing things down. Better to audio record.

Test the data collection forms

While the form was reasonably successful, there were some parts of it that didn’t work that well. Though a version of the form had been used in other studies before, I didn’t ask enough questions about the success or failure of the open text (qualitative data) part of the form. I wanted that data desperately, but it came back pretty messy. Testing the data collection form with someone else would have told me what questions researchers would have about that (meta, no?), and I could have done something else. Next time.

Minimize scripting

Maximize participant time by dedicating as much time to the session as possible to their interacting with the design. That means that the moderator does nothing to introduce the session, instead relying on an informed consent form that one of the team members can administer to the next participant while the current one is finishing up.

The other tip here is to write out the exact wording for the session (with primary and follow up questions), and threaten the researchers with being flogged with a wet noodle if they don’t follow the script.

Brief everyone on test goals, dos and don’ts

All the researchers and I met up at 10am and had a stand-up meeting in which I thanked everyone profusely for joining me in the study. And then I talked about and took questions on:

• The main thing I wanted to get out of each session. (There was one key concept that we wanted to know whether people understood from the design.)
• How to use the data collection forms. (We walked through every field.)
• How to use the script. (“You must follow the script.”)
• How to intercept people, inviting them to participate. (More on this below.)
• Rules about recordings. (Only hands and voices, no faces.)
• When to check in with me. (When you arrive at your location; at the top of each hour, when you’re on the way back.)
• When and where to meet when they were done.

I also handed cash that the researchers could use for transit or parking or lunch, or just keep.

Practice intercepting people

Intercepting people to participate is the hardest part. You walk up to a stranger on the street asking them for a favor. This might not be bad in your town. But in San Francisco, there’s no shortage of competition. Homeless people, political parities registering voters, hucksters, buskers, and kids working for Greenpeace all wanting attention from passers-by. And there you are, trying to do a research study. So, how to get some attention without freaking people out? A few things that worked well:

• Put the youngest and/or best-looking person on the task.
• Smile and make eye contact.
• Using cute pets to attract people. Two researchers who own golden retrievers brought their lovely dogs with them, which was a nice icebreaker.
• Start off with what you’re not: “I’m not selling anything, and I don’t work for Greenpeace. I’m doing a research study.”
• Start by asking for what you want: “Would you have a few minutes to help us make ballots easier to use?”
• Take turns – it can be exhausting enduring rejection.

Do an inventory check before spreading out

Before the researchers went off to their assigned locations, I asked each team to check that they had everything they needed, which apparently was not thorough enough for one of my teams. Next time, I will ask each team to empty out the contents of the packet and check the contents. I’ll use the list of things I wanted to include in each team’s packet and my agenda items for the briefing to ask the teams to look for each item.

Be flexible

Even with lots of planning and organizing, things happen that you couldn’t have anticipated. Researchers don’t show up, or their schedules have shifted. Locations turn out to not be so perfect. Give teams permission to do whatever they think is the right thing to get the data – short of breaking the law.

Check in

Teams checked in when they found their location, between sessions, and when they were on their way back to the meeting spot. I wanted to know that they weren’t lost, that everything was okay, and that they were finding people to take part. Asking teams to check in also gave them permission to ask me questions or help them make decisions so they could get the best data, or tell me what they were doing that was different from the plan. Basically, it was one giant exercise in The Doctrine of No Surprise.

Reconvene the same day

I needed to get the data from the research teams at some point. Why not meet up again and share experiences? Turns out that the stories from each team were important to all the other teams, and extremely helpful to me. They talked about the participants they’d had and the issues participants ran into with the designs we were testing. They also talked about their experiences with testing this way, which they all seemed to love. Afterward, I got emails from at least half the group volunteering to do it again. They had all had an adventure, met a lot of new people, got some practice with skills, and helped the world be a become a better place through design.


Wilder than testing in the wild, but trust that it will work

On that Saturday in San Francisco the amazing happened: 16 people who were strangers to one another came together to learn from 40 users about how well a design worked for them. The researchers came out from behind their monitors and out of their labs to gather data in the wild. The planning and organizing that I did ahead of time let it feel like a flash mob event to the researchers, and it gave them room to improvise as long as they collected valid data. And it worked. (See the results.)


P.S. I did not originate this approach to usability testing. As far as I know, the first person to do it was Whitney Quesenbery in New York City in the autumn of 2010.

The true costs of no-shows

One of the first things people say when they call up looking for help with recruiting is that they want to recruit “12 for 8” or “20 for 15”. They know what they want to end up with. They’ve got to get data. Managers are showing up to observe. They’ve gone through a lot to get a study to happen at all. They don’t want to risk putting a study together only to get less data than they need. So, compensating for a show rate of between 60% and 80% means over-recruiting.

Even though a recruiting agency probably won’t charge for no-shows, those no-shows can be costly in lots of ways.   


Stress. First, there’s the anguish of the moments before you know whether the participant will show. It’s not only the researcher who’s nervous. Nobody knows what to do. Time is ticking away. The session is running late late. Observers get antsy. The careful planning and scheduling of putting on a study feels like it is falling apart. That can be agonizing. When the participant arrives, it’s difficult to center yourself and get on with the session. 

Embarrassment. There is nothing like the humiliation of having participants not show up after nudging, cajoling, and begging a team to show up to observe their users. The campaign of marketing user research disintegrates as the promise of enlightenment fades. It’s like promising fireworks but the fuses won’t light. The negative energy can be palpable and extremely difficult to recover from. Claiming that this is out of you control will not help the credibility of the study.

Loss of productivity. When you find out there is a longer-than-planned gap between sessions, conducting shadow puppet shows or showing cat videos to fill the time will not win the day. The researcher has got to get the team engaged in something productive that will help everyone feel like they’re still getting value out of being there. Without a participant to observe, it’s tough to keep the team from thinking you’re not wasting their time. That’s difficult to pull off for even the most experienced of researchers.

Loss of trust. Let’s not forget the pain of losing momentum as observations are interrupted. Say you got a few sessions done. Everyone is charged up with the revelations they’ve witnessed. Epiphany is everywhere. And then, a no-show. It can suck the air out of the room. Clients and teams can become disenchanted and impatient. They may lose confidence in the process and the results.

By comparison, the amount of money going for lab time when you’re not running sessions -- or paying for even more lab time and recruiting to make up sessions -- seems small. The damage of no-shows is long lasting. Yet, most of us treat no-shows as a cost of doing studies. They just happen. It’s the recruiting agency’s fault. We can’t control the participants.

What if you had a near-perfect participant show rate? The first time it happens, it’s surprising. The next few times, it’s refreshing -- a relief. Teams that do great user research start with the recruiting process, and they come to expect near perfect attendance. In the next post, I’ll share their secrets for getting great participants who show up.

Usability testing is HOT

For many of us, usability testing is a necessary evil. For others, it’s too much work, or it’s too disruptive to the development process. As you might expect, I have issues with all that. It’s unfortunate that some teams don’t see the value in observing people use their designs. Done well, it can be an amazing event in the life of a design. Even done very informally, it can still show up useful insights that can help a team make informed design decisions. But I probably don’t have to tell you that.

Usability testing can be enormously elevating for teams at all stages of UX maturity. In fact, there probably isn’t nearly enough of it being done. Even on enlightened teams that know about and do usability tests, they’re probably not doing it often enough. There seems to be a correlation between successful user experiences and how often and how much the designers and developers spend time observing users. (hat tip Jared Spool)

Observing people using early designs can be energizing as designers and developers get a chance to see reactions to ideas. I’ve seen teams walk away with insights from observing people use their designs that they couldn’t have got any other way – and then make better designs than they’ve ever made. Close to launch, it is exciting – yes, exciting – to see a design perform as useful, usable, and desirable.

I’ve been negative on usability testing and our failure of imagination regarding bringing the method up to date, lately. But there’s a lot of good to any basic usability test. In fact, I went looking for the worth, the value, the alluring in usability testing a few weeks ago when I asked on Quora, “What’s the sexiest thing about usability testing?”

Some of the answers surprised me. Some of the answers were more about what people love about usability testing than what makes it seductive. But let’s go with seductive. People who find usability testing hot say it’s about data that can end the opinion wars, revelations and surprises, and getting perspective about real use, motivations, and context of use.  Okay. We’re nerds.

The kiss of data

We always learn from users. Of course, we could just ask. But observing is so much more interesting. People do unpredictable things; they create workarounds, hacks, and alternative paths to make tools fit for their use.

This is the best case I can think of for watching rather than asking. From this observing, we get data. Juicy, luscious data like verbal protocols, task success rates, and physical behavior. This package makes it much easier to make good design decisions because we know have evidence on which to create theories about what should work better. There’s nothing like having hard evidence for going with a design direction – or changing direction.

Voyeuristic revelations

When designers, developers, and stakeholders of all persuasions get to observe people using a design – especially the first time – there’s often an “ah ha!” moment. (That’s the clean version.) Observers exclaim, “Wow, that was amazing!” when they see something surprising, both the good and the bad. The reaction that follows a completed usability study often is, “Damn. I wish we’d done this years ago. It would have saved us a ton of rework!” After watching one over-qualified participant struggle with a design recently, I heard a client say, “If that guy can’t do it, we’re in serious trouble.” That’s powerful.

When participants are surprised, that’s when the real fun begins. Not everyone likes surprises in their user interfaces, especially if they’re not the delightful Easter egg kind. While a team hopes not to hear, “I feel lost and abandoned,” you’ve got to wonder how bad it’s been when a participant squeals, “Oh, my gosh! This is so much better! When can I have it?!” Those eureka moments can reveal what to do to improve a design or an experience.

Relationship dynamics

One of the magical things about observing users working with a design is that suddenly, disputes within the team melt away. Chances are, the disputing parties were both wrong because neither (unless they have a ton of experience already observing these kinds of users in this domain doing this task) could accurately predict how the user would behave and perform.

Now, even with observations from watching just one user, there’s data on which to base design decisions. Data trumps gut. Data outweighs feelings. Data can put to rest those endless, circular discussions where inevitably, the person with the biggest paycheck or the most important title wins. The opinion wars come to an end.

When the whole team is involved in deciding what to test and observing sessions, everyone can share in making and carrying out agreed design decisions. Whenever a question comes up where no one knows but everyone has an opinion, the answer in a team doing usability testing is, “Let’s do some user research on that,” or “Let’s find out what users do.”

For the love of users

It’s so easy to get caught up in the business goals and issues with the underlying technology of a design. It’s so easy to stay in the safe bubble of the office, cranking out code, designs, plans, and reports. It’s easy to lose touch with users.

Teams that spend a couple of hours observing their users every few weeks keep that connection. They fall in love with their users. They relish the chance to see for themselves why people do the things they do with designs.

Getting out of their own heads, a successful team uses usability testing to get perspective, learn about users’ contexts, and remember the people and their stories. For these teams, usability testing is inspiring. And that’s hot.

What’s sexy about usability testing?

Observing people use a design can be revelatory. It turns up the volume on design by helping teams make informed design decisions. What’s sexy about usability testing? Data for evidence-based design. Ending opinion wars. Knowing users from observations and surprises. Getting perspective and knowledge of context of use.

The UX equivalent of a romantic dinner or a walk on the beach? Perhaps not, even for a geek girl like me. But it can be exciting, fun, funny, encouraging, and empowering. Just what you want from a relationship. That’s pretty seductive, if you ask me.

Oh, Etsy. How could you?

During the last holiday season, I called Land's End. I hardly ever call; I'm a huge fan of their online experience. I wanted to send a special order to my mother, putting two matching things in the same gift box. Landsend.com isn't really set up to do that, but the site instructed me that I could do it, so I called.

The sales rep was friendly and efficient, and very helpful. She pulled up my order-in-progress and put everything in the box that I wanted to be in my mother's gift. When she asked me if there was anything else she could help me with, I blithely said, "You could check my mother's account and tell me what she's sending me for Christmas." The sales rep giggled, teased me a little bit by telling me that she could see my mother's account, but told me I would have to wait until UPS delivered it to find out. She protected the relationship between the seller and the buyer. She also protected the relationship between two buyers – me and my mother -- at least for that episode. If I wanted to find out what I was getting for Christmas, I'd either have to wheedle it out of my mother or wait.

Respect and research. That's all I ask.

Facebook had a go at Beacon, a service that broadcasted out to all your friends the purchases you've made outside of Facebook, without permission. The Federal Trade Commission sued, and Facebook eventually settled and took down the "service" in 2009.

Facebook has a history of screwing with the privacy of its users. Beacon was a prime example. The main problem here is the lack of permission. And that's the case for Etsy's new People Search, too.

The designers of Etsy decided it was a good idea to make everyone on Etsy searchable by name, including buyers. So, if you have ever bought anything on Etsy, you can now be found there by anyone else either by your real name or your username. Your whole profile is viewable, including your purchase history. Not only that, it'll all show up in Google search results.

The idea is that buyers would form social "circles" on the site to share information about their purchases.

Uninformed by research, guided by gut

These are the kinds of things that happen when an organization puts business goals before customer goals. It's also the kind of thing that can happen when an executive wakes up one day and says, We want to be one of the cool kids. And right now, to be one of the cool kids, you have to have social media. How do we do that?

What made Etsy think it needed a social layer on its beautiful, engaging site? It's the kind of thing that happens when teams decide to strap social on rather than looking at the conversation they're already having with customers and that customers are already having with one another.

I'm sure a lot of thought went into this decision of Etsy's. I fear this is a vacuum-sealed decision. Here's my imagined scenario, a scenario I've seen played out in other, similar decisions at other, similar organizations: Management, who forgets that their site is not the center of the universe for anyone outside that room, went to the product manager and asked for some of the social awesomesauce that is out there to turn up the buzz a notch. The product manager brainstormed with the team. The best idea they could come up with is to get customers to talk about the fun, beautiful, interesting stuff they'd bought on Etsy online with one another. (Never mind that we already have Twitter for this.)

Where is making use of the conversation that Etsy is already having with its customers or that buyers and sellers are already having together? They probably can't make use of these conversations because they haven't observed them. Where's the research to support this design decision? 

Why this is on my usability testing blog

It's hard for me to believe that if Etsy had conducted user research and even informal but realistic usability testing on the idea that they would not have quickly seen the privacy violation. They could have avoided the damage control they now have to deal with because of the breach of trust they've had with buyers who already love the experience of shopping there.

How Etsy could have avoided the problem and discovered a possibly great idea for engaging buyers even more

1. Analyze the risks of a social media strategy to users' privacy, security, and trust. Where was the business plan for allowing search of users? How does having social "circles" support the business model, exactly? How would the social media strategy be supported on the back end? More than all that, let's look at others who have gone before us: Beaon on Facebook and Boden USA come to mind. What happened there? What could the Etsy team learn from those mistakes? Oh, and, why duplicate Facebook in any way?

2. Proof the concept with real people who shop on Etsy. This is pure conjecture based on my experiences with other organizations: Etsy may have thought that to up their game and get people more engaged in the site, they needed to get buyers talking with one another and not just to sellers. Charming idea. But how do you find out if people find that useful?

Focus groups? If there were focus groups, I'm just going to guess here that participants liked the idea, but there was no exploration of the implications of this profile information being public rather than private. Not ideal.

What else could they have done? Invited friends and family. This approach still perhaps is not optimum, because friendly participants might not have exposed the privacy problems. They are, after all, friends and family, so there's automatic trust and wanted connections there already. How about rolling it out to a very small number of key buyers -- 3 or 5 -- and watch what happens for a week or a month as they connect to their people, or until something bad and unintended happens?

3. Conduct usability testing with real people in real contexts to learn the ripples to real relationships. Let's say they did usability testing. Did they bring in real buyers to use a working prototype with their own data? Did it occur to anyone that now my ex can Google me (like he does) and find out that I bought my sister a Star Wars crochet pattern, or my current paramour a hand made can coozie? Or what about the fact that my clients could see all the personal things on my Etsy wish list?

A usability test with a limited "circle" on a closed sandbox (like a walled-off development or testing server) for a couple of weeks might have given them some clues about what might work and what might not.

Etsy, I love you, but I have to go now

Not only will Etsy have to clean up its own site by making the social opt-in, but they'll also have to figure out a way to recover buyers' privacy. How does a web organization reclaim data that is now not in its control? If they could invent a big Web eraser to drag behind them as they invite buyers back to the site, they might have a chance.

Sources and resources:


http://arstechnica.com/web/news/2011/03/etsy-users-irked-after-buyers-purchases-exposed-to-the-world.ars

http://forums.penny-arcade.com/showthread.php?t=138661

Involving older adults in design of the user experience: Inclusive design

Despite the reality of differences due to aging, research has also shown that in many cases, we do not need a separate design for people who are age 50+. We need better design for everyone.

Everyone performs better on web sites where the interaction matches users' goals; where navigation and information are grouped well; where navigation elements are consistent and follow conventions; where writing is clear, straightforward, in the active voice, and so on. And, much of what makes up good design for younger people helps older adults as well.

For example, we know that most users, regardless of age, are more successful finding information in broad, shallow information architectures than they are with deep, narrow hierarchies. When web sites make their sites easier to use for older adults, all of their users perform better in usability studies. The key is involving older adults in user research and usability testing throughout design and development.

There are some important considerations in working with older adults in studies. Remembering the points below will ensure that you and your participants have a good experience and you get the data you need to inform design decisions.


Finding participants: Understanding older adults before you recruit

In many places in the world, older adults outnumber people in other age groups. The question is, how do you find the right people to take part in studies? They can be difficult to get to.

We found that these approaches did not work well:

• Community web sites, message boards, or chat sessions. The oldest old tend not to take part in these groups, so posting ads in those places is not a fruitful way to find participants.
  
• Senior centers and community colleges. These are places that offer classes in using computers. If you want computer and web novices for a study, they might be good places to find appropriate participants. They are not good sources if you want to observe people with enough web experience to see them working at a web site without teaching them.
  
• Flyers at a senior center, when they did not make clear that we were recruiting for a study. Many older people are much more cautious and skeptical than younger people. They are often fearful of being cheated or "taken." For example, we had put up flyers at a senior center from which we got no response; later we learned that people thought we might be trying to sell them something.
  
• Cold calling from a database. This is probably again because older people are afraid that they may be scammed into buying something.

These ideas did work well:

• Calling with a personal connection. If we could say that a mutual acquaintance had suggested the contact, potential participants were much more receptive to hearing about the study and considering taking part. It is important to establish credibility and trust with the potential participant.
  
• Being careful in the initial call to say where we had gotten the contact information and that we weren't selling anything.

Recruiting older adults

Recruiting participants in their 70s and 80s is more difficult than recruiting participants in their 50s. The oldest candidates are less receptive to strangers phoning them, and they don't check email as frequently.

Recruiting by phone. Phoning is important. Plan to phone potential participants at least once (or have your recruiters do so). You need to quickly establish credibility and trustworthiness, to assure potential participants that you are not selling anything, and to establish a connection by letting them know where you got their names. When you can do that, potential participants are often glad to hear from a real person. It is also easier for them to determine legitimacy and to ask questions about the study on the phone. They'll use your answers to help them decide whether they want to take part.

Another reason for phoning potential participants is so you can judge their English language skills and whether they are hearing impaired. (You may well want to include limited English speakers and hearing impaired users in your study; if you do, you want to be aware of these specifics about the participants before they come.)

Recruiting by email. Email can be very efficient for younger participants; it's less so with adults in their later 60s and 70s and older. Give yourself more time for these older participants; they generally don't check their email more than a couple of times per week. This happens for a variety of reasons: They don't feel the need to check mail frequently. They use a computer at a senior centre or a library. They have limited time available through their Internet service provider.

As suspicious as older adults are of telemarketers, they are also vigilant about spam. If your email address is unknown to them, without an appropriately descriptive subject line, they may delete it. Always put on a very clear subject line.



Screening older adults

Screening older adults demands specificity. Many older users when asked the question "what do you do online?" answer "email." They often don't think about practical activities such as banking or bill paying online as "using the web."

Many older users are also not as familiar with the language of the web as younger users are. They don't distinguish between the Internet and the web. They don't always know the difference between the web browser and the web page.

We found that self-reported data about frequency of use and numbers of hours spent online were not good indicators of proficiency, either. For example, we had one participant who spent 60 hours per week online. We didn't find out until the session started that her sole use of the web was playing games on four web sites that her friend had set up as separate shortcuts on her desktop.

So, asking a variety of specific questions to gauge potential participants' familiarity with the web can help the person recruiting participants for a study make judgments about how suited the person might be for the study. Even if you're looking for a mix of proficiency levels, you still have to be able to determine where in the range a potential participant fits.



Tech savviness matrices
An assessment we’ve found to work well asks about frequency of use over a broad range of types of interactions older people can take part in on the web. Here are some example assessment grids we have used.

Scheduling sessions with older adults

Scheduling sessions with older participants can present some logistical challenges that you might not think about in studies involving younger participants.

They arrive early. Because many older people are retired (or at least have ample free time), they almost always arrive for their sessions early – up to an hour early. Be sure to have someone to greet them and set up a comfortable place for them to wait.

They bring their spouses. Older participants often bring their spouses or a friend with them. They may have travelled some distance to get to the session; they may have planned activities for after their session; or they simply may not like driving alone. Have magazines, a phone, and a comfortable chair available for the spouse or friend.

They do best in the morning. Even though people in their later 60s, 70s, and 80s are vital and energetic, they usually have more—and better—attention to give earlier in the day. Try to schedule people who are in their late 60s, 70s and 80s in the morning and save any afternoon sessions for participants in their 50s or early 60s. We don't recommend running evening sessions.

They don't like driving in rush hour. If you are holding sessions in a central place (rather than meeting participants in their homes or workplaces), schedule the sessions outside of peak traffic times, if possible.


Reminding older adults of important points before they come

Reminders about one-to-one sessions. Participants can become nervous and uncomfortable if they realize after arriving that they will be the only participant in the session. Usability studies are still fairly new to the general population. Recruiting firms often recruit for focus groups, and participants who come through these firms often assume that they are coming to a focus group.

Reminders about videotaping and observations. Although a good practice is to ask for permission to record and to have people observing the sessions when you recruit, people tend to forget that. Make sure that the person who calls the participant to confirm the session also tells the participant that:

• "You will be videotaped and observed by people you won't be able to see during the session."
  
• "This is a one-on-one session. You will be the only participant in the study room with a moderator."

Special reminders for older adults.
 

•  Computer glasses. Many participants will have special glasses for using the computer. So another important reminder is, "Don't forget your computer glasses!"
  
• Eat first. Also, for long sessions— anything longer than 45 minutes— ask participants to make sure they eat before they arrive. Because many participants expect to take part in focus groups rather than individual sessions, they also expect to be fed. If you have snacks available, try to have fruit and nuts or other relatively healthy food. Many older participants are diabetic.

Working with older adults during sessions

Many older participants won't know what to expect coming into a usability study session. Be clear in setting their expectations and be firm but polite about keeping the session focused on what you're trying to find out.

Make participants comfortable. Be respectful without being patronizing. You can be a neutral moderator but still be polite. "Please" and "thank you" are important. Many older adults expect more statements of politeness like these than younger participants do.

Older participants also deserve extra consideration, politeness, and detailed information about the session. They will feel more comfortable if they know what to expect up front:

• Clearly explain the session plan, timing, and what they can expect.
• Warn participants that you'll interrupt them and that you may stop them before they have completed tasks.
• Schedule breaks for long sessions (and tell them they can take breaks whenever they need to).
• Have them practice thinking aloud.
• Consider including a practice task to help participants understand how the session will work.
• Take account of beliefs that participants may have learned or created about how to work with computers.
• Remember that older participants often are not versed in computer and web terminology, so avoid using this jargon when working with them.
• Be extra patient with older participants; wait longer than you normally might to prompt; consider giving participants permission to ask for hints.
•  If participants stop talking, consider letting them continue that way; try reflecting on the task later.
• Teach participants something at the end of the session.

Keep them on track tactfully. Most of the participants we've had in sessions are interesting, charming, and very talkative. Many older participants have a lot of stories to tell. Their stories say a lot about who they are, and where they have been— and often provide a context for interpreting data.

But it may be easy for participants to get off track during the session, and while it may feel awkward or mean sometimes, it is the moderator's job to keep the participant focused on the task, talking about it, and getting data for the study. This is the main reason for warning participants in the introduction to the session that you may interrupt them and that you may stop tasks before they've completed them.

Listen for their beliefs about computers and the web. Many people who are in their late 60s and older never used computers at work. This means they have no previous experience from which to make inferences about how a computer or an application might work. Many learn how to use computers and the Internet through friends, family, and neighbours. They inherit the superstitions and myths that those people have developed to help themselves work around problems. Then the older adults bring these myths into sessions with them, and you'll hear about them as task-solving strategies and workarounds. It's important to capture these; they are part of the users' reality and we have to deal with these beliefs when we design web sites.

Be careful of the words you use; avoid computer jargon. Older computer users rarely know much about computer-related terminology, so you should avoid using these terms during your sessions. Older participants often don't know the names of widgets such as drop-down boxes or cascading menus. Most of our participants also had little knowledge of web-related terminology. For example, they weren't sure about terms such as "link," "URL," and "login." Many were unclear about the meanings of "online community" and "message boards." "Browsing" wasn't always meaningful in the context of a feature called "browse by topic." The word "emoticon" and the concept behind it were completely foreign to most of our participants. This means that you must pay close attention to what participants do and point at on the screen or device.

Give them time. Older participants almost always take longer to do tasks than younger participants. And, although they seem to struggle, the oldest participants also expect using the computer to be difficult. Plan for tasks to take much longer for older participants than they would for younger participants—up to 25 percent longer in our experience.

Help participants understand the time constraints of the session by explaining the session format in your introduction. Also, wait longer to prompt than you normally might. You might also consider giving participants permission to ask for hints when you introduce the session.

If necessary, hold the think-aloud and ask participants to reflect later. A classic technique in usability testing is to ask participants to think aloud while they work through tasks toward a goal. When tasks become complex or difficult, participants may stop talking. Use your best judgment about nudging them to tell you what they're thinking. For some participants with short-term memory loss or other cognitive impairment (such as that caused by pain medication), your asking for their thoughts may interrupt their task enough that it causes them to make errors. In those cases, you may get more usable data without the think aloud protocol by asking participants to reflect later.

Don't lead even when you want to. If, as a session moderator, you have a soft spot in your heart at all for participants, working with older participants will exercise that spot a lot. You may be tempted to give hints; worse, you may lead them in ways you don't intend. Be patient and firm but polite while keeping to your agenda.

When appropriate, teach something at the end. If the session has been difficult for the participant, or, if there is some small thing that would make using the computer or the web easier, take a little time at the end of the session to teach the participant something. For example, show participants how to change the text size in their browsers and shortcuts for copying and pasting and printing.



Including older adults in user research and usability studies: Older, wiser, wired

Older adults don’t behave differently from younger people online. Just thinking about old age as a collection of disabilities is old business. The new world of designing for older adults is about creating web sites and other technology that is useful and desirable as well as accessible to the broadest range of users. Older adults as a cohort are living longer than their parents because they’re healthier, and many will be affluent because they’ve been saving up for a lifetime – this means they have time, money, and motivation to be online.

Neither a monolithic view of older adults nor an entirely separate design for older adults is necessary. Younger designers developing web sites for older adults need to learn more about older adults’ life experiences. For example, many older adults don’t perceive themselves as old. And so, all technology design – not just designs for older adults – should involve users.

Usability testing is broken: Rethinking user research for social interaction design

How many of you have run usability tests that look like this: Individual, one-hour sessions, in which the participant is performing one or more tasks from a scenario that you and your team have come up with, on a prototype, using bogus or imaginary data. It’s a hypothetical situation for the user, sometimes, they’re even role-playing.

Anyone? That’s what I thought. Me too. I just did it a couple of weeks ago.

But that model of usability testing is broken. Why? Because one of the first things we found out is that the task we were asking people to do - doing some basic financial estimates based on goals for retirement - involved more than the person in the room with me.

For the husbands, the task involved their wives because the guys didn’t actually know what the numbers were for the household expenses. For the women, it was their children, because they wanted to talk to them about medical expenses and plans for assisted living. For younger people it was their parents or grandparents, because they wanted to learn from them how they’d managed to save enough to help them through school and retire, too.

There’s a conversation there. There’s a support network there. And that’s what’s broken about usability testing. It always has been.

I first started thinking about this when Google launched Buzz. Buzz used Gmail users’ contacts to automatically generate online social networks to connect users’ most frequent contacts. Google employees - 20,000 of them - had been using Buzz inside the garden walls for a year. A nice, big sample. The problem became evident, however, when Buzz was let into the wild -- almost immediately. One example: A blogger who calls herself Harriet is one of the most famous cases. She wrote about how one of her most frequent correspondents in Gmail was her boyfriend. Another was her abusive ex-husband. Now they were publicly connected, and this made her very, very unhappy. In fact, the post was titled, Fuck You, Google.

There might have been no harm done in the retirement planning study. But there might. Would the 31-year-old who broke down crying in the session because her mother was in late-stage ALS have had a better experience if we’d tested in her context, where she could work with her closest advisor - her dad? Might it have been a calming process, where she felt in control and became engaged in envisioning her independent future because someone she trusted could give her perspective that I could not? Maybe.

As for Buzz, Harriet certainly wasn’t pleased, and she was left with a mess to clean up. How to unconnect two people who were now connected?

When companies started doing usability testing with regularity in the 1980s, it was about  finding design problems in what now look like fairly simple UIs that frustrated or hindered users. It was one person, one machine, as the human did a usually work-based task. That’s why it was called computer-human interaction.

But today, technology large and small is fully integrated into peoples’ lives in a much more ephemeral, less compartmentalized way. It is rare to sit next to a corded phone holding the handset only talking and listening.

When you look at what the social web is, there are some characteristics that I think we’re not taking into account very well in doing usability tests:

- It’s about relationships among people
- In context
- Conducted fluidly, across time across time and space, continuously

I also think that people who are new to usability testing and user research are not going to do a good job of testing social interaction design, because constructing a study cannot be very formal or controlled. Measuring what’s happening is much more complex. And scope and scale make a difference. Testing Buzz with 20,000 Googlers for a year wasn’t enough; it took letting it out to a million people who hadn’t drunk the Koolaid to find the *real* problems, the real frustrations, the real hindrances that truly affect uptake and adoption.


The nature of online is social
Let’s back up and talk about a key definition. What I mean by “social” is anything that someone does that changes the behavior of someone else.

This is how I can say that being online *is* social. Email is social. Publishing a flat HTML document is social. Putting something on a calendar is social. Everything is social. Social isn’t the special secret sauce that you pour on top of an experience. Social is already there. Choosing a bank is social. Planning a vacation is social. Buying an appliance is social. I SMSd a series of photos to my boyfriend the other day, of me in different eyeglass frames because I couldn’t decide by myself. This was *not* a computer-centered, or an app-centered interaction. This was a decision being made by two people, a conversation, mediated by fully integrated technology in fluid activities in different contexts for two people. It was social.

Social isn’t sauce. It’s sustenance. It’s already there, and we’re not seeing it. So we’re not researching it, and we’re definitely not testing for it.

We have to stop thinking about human-computer interaction. That model by default is too limiting. Look around. It’s really about human relationships and interactions mediated by technology. Technology is supporting the communication, not driving it. Ask any parent who has used Facetime or Skype to have a video chat with their baby for the first time.

Scale is the game changer
Discount usability testing is great for some things, but what we’re really studying when doing user research and usability testing for the social web is social behavior. And that takes scale. That takes connections. That takes observing people’s real networks and understanding what makes those work, what makes those friends, family, colleagues, neighbors, acquaintances, associates, clients, vendors, pen pals, drinking buddies, partners for life, or friends with benefits.

Those are rich, life-framing relationships that affect how someone interacts with a user interface that most of us are not even scratching the surface of when when we “micro-test” a commenting feature on an online invitation web site.

“Task” doesn’t mean what you think it means
For the retirement planning tool, I did a little interview to start the session that I hoped would set some context for the participant to do the behavior that I wanted to observe. But it was woefully inadequate. Don’t get me wrong, the client wasn’t unhappy; they thought it was a cool technique. But as soon as I learned who the participant went to for financial advice, where was I? Putting the participant in a situation where they had to pretend. They did, and they did a fair job of it. But it was lacking.

But tasks are the wrong unit. What we’re asking people to do in usability tests is like attending a cocktail party while grocery shopping. Even with an interview, even with careful recruiting, it’s incongrous. There are very few discrete tasks in life. Instead there are activities that lead people to goals. Multiple activities and goals might be intermixed on the way to achieving any one of them. In the meantime, the technology is completely integrated, ambient, almost incidental. LIke asking your bf which eyeglass frames look nerdy-sexy, versus just nerdy.

The activity of interest isn’t computer based. Look at retirement planning. It’s *retirement planning*! That’s not the task. The activity is planning for the future, a future in which you have no real idea of what is going to happen, but you have hopes, aspirations.

Using Buzz is not a task. It’s not an activity. People who use Buzz don’t have the goal of connecting to other people, not in that deliberate way. They’re saying, hey, I’ve read something interesting you might be interested in, too. The task isn’t “sharing.” It’s putting  themselves out in the world hoping that people they care about will notice. How do you make that scenario in a usability test?

Satisfaction may now equal user control
The ISO measures of usability are efficiency, effectiveness, and satisfaction. What is effective about having Tweetdeck open all day long while you’re also writing a report, drafting emails, taking part in conference calls, attending virtual seminars, going back to writing a report, calling your mother?

When the efficiency measure went into the ISO definition, most people were measuring time on task. But if you don’t have a discrete task, how do you measure time?

Satisfaction may be the most important thing in the end for the social web, and that may be the degree to which the user feels she has control of the activities she’s doing while she’s using your tool. How much is the UI forcing her to learn the tool, versus quickly integrating it into her life?

Measuring success in the social web often defies what we’ve been taught to count for data. How do you measure engagement in the social web? Is it about time on the site? I could lurk on Twitter.com or Facebook all day. Am I engaged? Is it about minutes spent pursuing and perusing content? Is it about how likely someone is to recommend something to someone else? I wrote my first product review, EVER last week, for a pair of jeans on the Lands End web site. Am I engaged with the site? I would say no.

We have to look hard at the goodness of conventional metrics. They’re not translating to anything meaningful, I don’t think, because we’ve been thinking about all this all wrong - or not enough. What is goodness to a user? Control of her life. Control of her identify. Control of her information.

Users are continuously designing your UI
What does Task mean, what does Success mean, how do you measure new features that users create with your UI on the fly? Twitter has hashtags and direct messages. Users created those. Facebook is continuously being hacked for fresh activities. Look at commenting systems on blog posts or articles. Spammers, yes, but people are also talking to one another, arguing, flirting, solving problems, telling their own stories. No matter what you build, and what your intentions were in designing it, users are going to hijack it to make it useful to them. How do you test for that?



Ideas from smart people
I had all these questions and more when I met with a bunch of smart people who have been working in researching the social web. Out of that discussion came some great stories about what people had tried and worked, and what had not worked so well.

For creating task scenarios for usability tests, getting participants to tell stories of specific interactions helped. Doing long interviews helped learn context, scope, priorities, connections. Getting people to talk about their online profiles and explain relationships helped set the scene for activities. Getting them to use their own log-ins with their real relationships helped everyone know whether the outcomes were useful, usable, and desirable. Whether the outcomes were satisfying and even enriching.

Some of the people in this informal workshop also offered these ideas:

• Screen sharing with someone outside the test lab or test situation
  
• Making video diaries and then reviewing them retrospectively
  
• Developing and testing at the same time, with users
  
• Including friends or other connections in the same test session, setting up multi-user sessions
  
• Sampling the experience in the same way that Flow was discovered: prompting people by SMS at select or random moments to ask people to report their behavior
  

There’s also bodystorming, critical incident analysis, co-designing paper or other prototypes. A few things seemed clear through that discussion. To make user research and usability testing useful to designers, we have to rethink how we’re doing it. It’s got to reflect reality a bit better, which means it takes more from social science and behavioral science than psychology. It takes more time. It takes more people. It takes a wider view of task, success, and engagement. And we’re just beginning to figure all that out.

Rethink research and testing

Everything is social. Scale is the game changer. Tasks aren’t what you think they are. User satisfaction may be about control. Users are continuously designing your UI. I invite you to work with me on rethinking how we’re doing user research and usability testing for what’s really happening in the world: fluid, context-dependent, relationships mediated by technology.

I want to thank Brynn Evans, Chris Messina, Nate Bolt, Ben Gross, Erin Malone, and Jared Spool for spending the better part of a day talking with me about their experiences in researching social. These musings come from that cooperative, ahem, social effort. 

Is your team stuck in a bubble?

This happens. The team is heads down, just trying to do work, to make things work, and then you realize it. Perspective is gone.

Recently I gave a couple of talks about usability testing and collaboratively analyzing data. There was a guy in the first row who was super attentive as I showed screen shots of web sites and walked the attendees through tasks that regular people might try to do on the sites.

Sweat beaded on his brow. His hands came up to his forehead in the way that someone who has had a sudden realization reacts. He put his hand over his mouth. I assumed he was simply passionate about web design and was feeling distressed about the crimes this web site committed against its users.

Turns out, he was the web site's owner.

This I found out at a break. When people started filing in from lunch to start the next session, this fellow appeared in my second session. I had time to talk with attendees, so I decided to approach him. "Hi. I noticed you were in my first session. Glad you're back. I hope the first was useful." He said yes, he had found it useful. But he frowned. "You look puzzled. Do you have a question I didn't answer?"



The bubble is insidious

"No," he said. "But it's clear that I have been -- along with a whole lot of other people -- out of touch."

"Oh? You got some insights today, already?"

"Some especially applicable insights, actually. The site you used this morning as your example is the site I work on every day." He gave a sad grin.

I knew this day would come. I would get caught out critiquing or running a demonstration on a site for which the owner was present. That day had arrived.

"I should have talked with you beforehand," I said. "The site has some classic problems. That's why I chose it as an example. It is one of dozens of sites in this domain that have similar issues. If I did or said anything that embarrassed you or your team, I apologize."

He sighed. "Not at all. You can't be embarrassed by something you weren't aware of." He went on, "We hadn't looked at the site at all from the point of view of users outside the organization. We've been in a bubble."

He actually seemed grateful. "Ah. That explains it," I said.

We chatted some more about the political pressures and the technology constraints that his team -- most teams -- faced in creating a great web site and maintaining it.  There had been some usability testing on intranets and even on extranets. But it was a few years ago. And the audience for the public-facing web site was different from the internal-facing web apps.


Perspective comes from observing real users doing real stuff

The best tool for resolving disputes within a design team, for making design decisions based on data rather than opinion, is sitting next to someone who is a real person who wants to accomplish something as they use your design to do it.

Some people call this usability testing. Call it whatever you want (except "user testing"). You can make it simple or complex, but when boiled down to its essence there are three ingredients:

- Someone to try out your design.
-  Somewhere to test.
- Something to study.

That's it. You can do it by the book, or you can do it very simply and ad hoc. The insights come from observing, first hand. I've seen just an hour of observation get many teams out of their own, customized bubbles.


Supporting great design: features of bubble prevention

Fortunately, my new friend stayed for the second session, in which I gave my recipe for supporting great experiences:

- Each phase includes input from users.

- The team is made up of people each with multiple skills from various disciplines.

- Management of the team is supportive an enlightened about the importance of the user experience.

- Everyone is willing to learn as they go along.

- The team has defined their usability goals and knows how they will measure their success.

Note that of the five attributes, two are directly about perspective (input from users; learning). Another two are about creating an infrastructure for getting and using that perspective (multidisciplinary team; setting usability goals). The remaining one (enlightened management) means there's support for getting and keeping perspective.

The importance of perspective cannot be overstated. Teams that meet with users regularly – every week or every month – turn out great experiences. Observing users regularly, at every phase of a design, gives a team evidence on which to make design decisions. More importantly, that act of being present with users, can bring the team together, enlighten management further, and give a needed break from the rarefied space most of us work in every day.

Get out of your head and into your users' .

Testing in the wild, seizing opportunity

When I say “usability test,” you might think of something that looks like a psych experiment, without the electrodes (although I’m sure those are coming as teams think that measuring biometrics will help them understand users’ experiences). Anyway, you probably visualize a lab of some kind, with a user in one room and a researcher in another, watching either through a glass or a monitor.

It can be like that, but it doesn’t have to. In fact, I’d argue that for early designs it shouldn’t be like that at all. Instead, usability testing should be done wherever and whenever users normally do the tasks they’re trying to do with a design.


Usability testing: A great tool
It’s only one technique in the toolbox, but in doing usability testing, teams get crisp, detailed snapshots about user behavior and performance. As a bonus, gathering data from users through observing them do tasks can resolve conflict within a design team or assist in decision-making. The whole point is to inform the design decisions that teams are making already.


Lighten up the usability testing methodology
Most teams I know start out thinking that they’re going to have a hard time fitting usability testing into their development process. All they want is to try out early ideas, concepts and designs or prototypes with users. But reduced to its essence, usability testing is simple:

• Develop a test plan and design
• Find participants
• Gather the data by conducting sessions
• Debrief with the team

That test plan/design? It can be a series of lists or a table. It doesn’t have to be a long exposition. As long as the result is something that everyone on the team understands and can agree to, you have written enough. After that, improvising is encouraged.

The individual sessions should be short and focused on only one or two narrow issues to explore.


But why bother to do such a quick, informal test?
First, doing any sort of usability test is good for getting input from users. The act of doing it gets the team one step closer to supporting usable design. Next, usability testing can be a great vehicle for getting the whole team excited about gathering user data. There is nothing like seeing a user use your design without intervention.

Most of the value in doing testing – let’s say about 70% – comes from just watching someone use a design. Another valuable aspect is the team working together to prepare for a usability test. That is, thinking about what Big Question they want answered and how to answer it. When those two acts align, having the team discuss together what happened in the sessions just comes naturally.


When not to do testing in the wild: Hard problems or validation
This technique is great for proving concepts or exploring issues in formative designs. It is not the right tool if the team is facing subtle, nuanced, or difficult questions to answer. In those cases, it’s best to go with more rigor and a test design that puts controls on the many possible variables.

Why? Well, in a quick, ad hoc test in the wild, the sample of participants may be too small. If you have seized a particular opportunity (say, with a seatmate on an airplane or a bus, as I have been known to do – yeah, you really don’t want me to sit next to you on a cross-country flight), a sample of one may not be enough to instill confidence with the rest of the team.

It might also happen, because the team is still forming ideas, that the approach in conducting sessions is not consistent from session to session. When that goes on, it isn’t bad necessarily. It can just mean that it’s difficult to draw meaningful inferences about what the usability problems are and how to remedy them.

If the team is okay with all that and ready to say, “let’s just do it!” to usability testing in the wild, then you can just do more sessions.


So, there are tradeoffs
What might a team have to consider in doing quick, ad hoc tests in the wild rather than a larger, more formal usability test? If you’re in the right spot in a design, for me doing usability testing in the wild is a total win:

• You have some data, rather than no data (because running a larger, formal test is daunting or anti-Agile).
• The team gets a lot of energy out of seeing people use the design, rather than arguing among themselves in the bubble of the conference room.
• Quick, ad hoc testing in the wild snugs nicely into nearly any development schedule; a team doesn’t have to carve out a lot of time and stop work to go do testing.
• It can be very inexpensive (or even free) to go to where users are to do a few sessions, quickly.

Usability testing at its essence: something, someone, and somewhere
Just a design, a person who is like the user, and an appropriate place – these are all a team needs to gather data to inform their early designs. I’ve seen teams whip together a test plan and design in an hour and then send a couple of team members to go round up participants in a public place (cafes, trade shows, sporting events, lobbies, food courts). Two other team members conduct 15- to 20-minute sessions. After a few short sessions, the team debriefs about what they saw and heard, which makes it simple to agree on a design direction.


It’s about seizing opportunity
There’s huge value in observing users use a design that is early in its formation. Because it’s so cheap, and so quick, there’s little risk of making a mistake in making inferences from the observations because a team can compensate for any shortcomings of the informality of the format by doing more testing – either more sessions, or another round of testing as follow-up. See a space or time and use it. It only takes four simple steps.

Looking for love: Deciding what to observe for

Last winter I worked with a team that wanted to find out whether a prototype they had designed for a new intranet worked for users. Their new design was a radical change from the site that had been in place for five years and in use by 8,000 users. Going to this new design was a big risk. What if users didn’t like it? Worse, what if they couldn’t use it?

We went on tour. Not to show the prototype, but to test it. Leading up to this moment we had done heaps of user research: stakeholder interviews, field observations (ethnography, contextual inquiry – pick your favorite name), card sorting, taxonomy testing. We learned amazing things, and as our talented interaction designer started translating all that into wireframes, we got pressure to show them. We knew what we were doing. But we wanted to be sure. So we made the wireframes clickable and strung them together to make them feel like they were doing something. And then we asked (among other things):

• How well does the design support the tasks of each user group?
• How easily do users move through the site for typical tasks?
  
• Where do they take wrong turns? What trigger words are missing? What trigger words are wrong?

Validating the research
In some ways, you could look at this as a validation test – not validating the design necessarily, but instead validating the user research we had done. Did we interpret our observations correctly by making the right inferences, in turn getting us to the design we got to?

What was possible: where the design might break
To find out, we had to answer those Big Questions. What were the issues within them that we wanted to investigate? Let’s take an example: How easily do users move through the site for typical tasks? We wanted to know whether users took the same path we wanted them to take, and if they didn’t, why not. On a task to find forms to open a brokerage account, we listed the possible issues. Users might

• start at the wrong place in the site
• get lost
• pick the wrong form
• not recognize they’ve reached the right place

From that discussion of the disasters that we could imagine came a list of behaviors to observe for, or as my friends at Tec-Ed say, issues to explore:

• Where do participants start the task?
• How easily do participants find the right form? How many wrong turns do they take on the way? Where in the navigation do they make wrong turns?
• How easily and successfully do they recognize the form they need on the gallery page?
• How well do participants understand where they are in the site?

What we saw
From these questions, we learned that we got the high-level information architecture right – most participants recognized where to enter the site to find the forms. We also learned that there were a couple of spots in the task path that had a combination of weak trigger words and other distractions that drew attention away from the things that would have gotten participants to the goal more quickly. But the groupings on the gallery page were pretty successful; most participants picked the right thing the first or second time. It was easy to see all of this in the way participants performed, but we also heard clues from them about what they were looking for and why.

And, by the way, the participants loved it. We knew because they said so.

Testing in the wild defined

Lately I’ve been talking a lot about “usability testing in the wild.” There are a lot of people out there who make their livings as usability practitioners. Those people know that the conventional way to do usability testing is in a laboratory setting. If you have come to this blog from outside the world of user experience research, that may never have occurred to you.

Some of the groups I’ve been working with recently do all their testing in the wild. That is, they never set foot in a lab, but instead conduct evaluations wherever their users normally do the tasks the groups are interested in observing. That setting could be a grocery store, City Hall, on the bus, or at a home or workplace – or any number of other places.

A “wild” usability test sometimes has another feature: it is lightly planned or even ad hoc. Just last night I was on a flight from Boston to San Francisco. I’ve been working with a team to develop a web site that lists course offerings and a way to sign up to take the courses. As I was working through the navigation and checking wireframes, the guy in the seat next to me couldn’t help looking over at my screen. He asked me about the site and the offerings, explaining that they looked like interesting topics. I didn’t have a prototype, but I did have the wireframes. So, after we talked for a moment about what he did for a living and what seemed interesting about the topics listed, I showed him the wireframe for the first page of the site and said, “Okay, from the list of courses here, is there something you would want to take?” He said yes, so I said, “What do want to do next, then?” He told me and I showed him the next appropriate wireframe. And we were off.

I learned heaps for the team about whether this user found the design useful and what he valued about it. It also gave me some great input for a more formal usability test later. Testing in the wild is great for early testing of concepts and ideas you have about a design. It’s one quick, cheap way to gain insights about designs so teams can make better design decisions.

Insights quickly and cheaply

After I gave a day-long seminar and a short talk at UI 13, I sat down with Tim Keirnan of Design Critique to talk about doing usability testing in the wild for quick, cheap insights from users. Download that podcast.

Ditch the book - Come to a virtual seminar on "usability testing in the wild"

I'm excited about getting to do a virtual seminar with the folks at User Interface Engineering (www.uie.com) on Wednesday, October 22 at 1 pm Eastern Time. I'll be talking about doing "minimalist" usability tests -- boiling usability testing down to its essence and doing just what is necessary to gather data to inform design decisions.

If you use my promo code when you sign up for the session -- DCWILD -- you can get in for the low, low price of $99 ($30 off the regular price of $129). Listen and watch in a conference room with all your team mates and get the best deal ever.

For more about the virtual seminar, see the full description.

Usability testing in the wild – ballots

I’ve been busy the last few weeks doing some of the most challenging usability testing I’ve ever done. There were three locations where I did day-long test sessions. But that wasn’t the challenging part. The adventure came in testing ballots for the November election.

What was wild about it?
This series of tests came together through a project with the Brennan Center for Justice and the Usability Professionals’ Association. The Brennan Center released a report in July called Better Ballots, which reviewed ballot designs and instructions, finding that

• hundreds of thousands of voters have been disenfranchised by ballot design problems
• there has been little or no federal or state guidance on ballot design that might have been helpful to elections officials who define and design ballots at the local level
• usability testing is the best way to ensure that voters can use ballots to vote as they intend

Also in the report, the Brennan Center strongly urged election officials to conduct usability tests on ballots. The recommendation to include usability testing in the ballot design process is a major revelation in the election world. The UPA Voting and Usability Project has developed the LEO Usability Test Kit to help local elections officials to do their own simple, quick usability tests of ballot designs.

But not all local elections officials were ready to do their own usability tests, and some wanted objective outsiders to help evaluate ballots for this particular, important upcoming election.

I did tests in three locations -- Marin County, California, Los Angeles County, California, and the home of Las Vegas in Clark County, Nevada -- with about 40 participants across the three locations. Several other UPA volunteers conducted tests and reviews in Florida, New Hampshire, and Ohio. In addition, UPAers trained local elections officials on usability testing and the LEO Test Kit in Ohio, Iowa, and a couple of other spots I can’t think of right now.

Pulling together a test in just a few days, including recruiting and scheduling participants
The Brennan Center report was released toward the end of July. Most ballots must be ready to print or roll out right now, the middle of September. The Brennan Center sent the report to every election department in the US and the response was great. Most requests came in in August, so among the five or six UPA Usability and Voting Project members available, we scrambled to cover the requests for tests.

We had the assistance of one of the Brennan Center staff to help coordinate recruiting, although it took some pretty serious networking to get people in to sessions on short notice, often within a few days.

The Brennan Center covered the expenses, but the time and effort spent by the people who worked with local elections officials and conducted the sessions was purely pro bono.


Not knowing what I would be testing until I walked onto the site
For two out of the three tests, I hadn’t seen exactly what I was going to be testing until I walked in the door of the election department. (I got the other ballot two days before the test.) This happened for a couple of reasons. Sometimes the local election official didn’t have a lot of information about what could be evaluated and how that might happen. Sometimes the ballot wasn’t ready until the last minute because of final filing deadlines or other constraints. Sometimes it was all of the above.

Fortunately, the main task is pretty straightforward: Vote! Use the ballot as you normally would. But there are neat variations. Are there write-ins possible? On an electronic voting machine, how do you change a vote? What if you’re mailing in a ballot – what’s different about that and how do design and instructions have to compensate for not having poll workers available to ask questions of?

Giving immediate results and feedback
So, we got copies of ballots or something close to final on an electronic voting machine. We’ve met briefly with the local elections officials (and often with their advisory committees). We’ve recruited participants (sometimes off the street). We’ve conducted 8 or 10 or 15 20-minute sessions in one day. Now it’s time to roll up what we saw in the sessions and to talk with the person who owns the ballot about how the evaluations went.

Handling enthusiastic observers and activists
A lot of people are concerned with the usability, accessibility, and security of ballots and voting systems. You probably are. Some are more concerned about it than others. Those are the people who show up to observe sessions. They’re well informed, they’re enthusiastic, and they’re skeptical. The observers and activists (many signed up to be test participants) were also keenly interested in understanding this activity. How was this different from focus groups or reviews by experts? How do we know that the problems we’ve witnessed are generalizable to other voters in the jurisdiction?


The good news: Mostly, the ballots worked pretty well. The local elections officials usually have the ability to make small changes at this stage and they were willing, especially to improve instructions to voters. By doing this testing, we were able to effect change and to make voting easier for many, many voters. (LA County alone has more than 3 million registered voters.)

Links:
Brennan Center for Justice report Better Ballots
http://www.brennancenter.org/content/resource/better_ballots/

UPA’s Voting and Usability Project
http://www.usabilityprofessionals.org/civiclife/voting/
voting@usabilityprofessionals.org

LEO Usability Testing Kit
http://www.usabilityprofessionals.org/civiclife/voting/leo_testing.html

Ethics guidelines for usability and design professionals working in elections
http://www.usabilityprofessionals.org/civiclife/voting/ethics.html

Information about being a poll worker
http://www.eac.gov/voter/poll%20workers

EAC Effective Polling Place Designs
http://www.eac.gov/election/effective-polling-place-designs

EAC Election Management Guidelines
http://www.eac.gov/election/quick-start-management-guides