Wilder than testing in the wild: usability testing by flash mob

It was a spectacularly beautiful Saturday in San Francisco. Exactly the perfect day to do some field usability testing. But this was no ordinary field usability test. Sure, there’d been plenty of planning and organizing ahead of time. And there would be data analysis afterward. What made this test different from most usability tests?

• 16 people gathered to make 6 research teams 
• Most of the people on the teams had never met
• Some of the research teams had people who had never taken part in usability
  testing before 
• The teams were going to intercept people on the street, at libraries, in farmers’
• markets

Ever heard of Improv Everywhere? This was the UX equivalent. Researchers just appeared out of the crowd to ask people to try out a couple of designs and then talk about their experiences. Most of the interactions with participants were about 20 minutes long. That’s it. But by the time the sun was over the yardarm (time for cocktails, that is), we had data on two designs from 40 participants. The day was amazingly energizing.


How the day worked
The timeline for the day looked something like this:

8:00
Coordinator checks all the packets of materials and supplies

10:00
Coordinator meets up with all the researchers for a briefing

10:30
Teams head to their assigned locations, discuss who should lead, take notes, and intercept

11:00
Most teams reach their locations, check in with contacts (if there are contacts), set up

11:15-ish
Intercept the first participants and start gathering data

Break when needed

14:00
Finish up collecting data, head back to the meeting spot

14:30
Teams start arriving at the meeting spot with data organized in packets

15:00-17:00
Everybody debriefs about their experiences, observations

17:00
Researchers head home, energized about what they’ve learned

Later
Researchers upload audio and video recordings to an online storage space

On average, teams came back with data from 6 or 7 participants. Not bad for a 3-hour stretch of doing sessions.


The role of the coordinator
I was excited about the possibilities, about getting a chance to work with some old friends, and to expose a whole bunch of people to a set of design problems they had not been aware of before. If you have thought about getting everyone on your team to do usability testing and user research, but have been afraid of what might happen if you’re not with them, conducting a study by flash mob will certainly test your resolve. It will be a
lesson in letting go.

There was no way I could join a team for this study. I was too busy coordinating. And I wanted to be available in case there was some kind of emergency. (In fact, one team left the briefing without copies of the thing they were testing. So I jumped in a car to deliver to them.)

Though you might think that the 3-or-so hours of data collection might be dull and boring for the coordinator, there were all kinds of things for me to do: resolve issues with locations, answer questions about possible participants, reconfigure teams when people had to leave early. Cell phones were probably the most important tool of the day. 

I had to believe that the planning and organizing I had done up front would work for people who were not me. And I had to trust that all the wonderful people who showed up to be the flash mob were as keen on making this work as I was. (They were.)


Keys to making flash mob testing work
I am still astonished that a bunch of people would show up on a Saturday morning to conduct a usability study in the street without much preparation. If your team is half as excited about the designs you are working on as this team was, taking a field trip to do a flash mob usability test should be a great experience. That is the most important ingredient to making a flash mob test work: people to do research who are engaged with the project, and enthusiastic about getting feedback from users.

Contrary to what you might think, coordinating a “flash” test doesn’t happen out of thin air, or a bunch of friends declaring, “Let’s put on a show!” Here are 10 things that made the day work really well to give us quick and dirty data: 

1.    Organize up front
2.    Streamline data collection
3.    Test the data collection forms
4.    Minimize scripting
5.    Brief everyone on test goals, dos and don’ts
6.    Practice intercepting
7.    Do an inventory check before spreading out
8.    Be flexible
9.    Check in
10.    Reconvene the same day

Organize up front

Starting about 3 or 4 weeks ahead of time, pick the research questions, put together what needs to be tested, create the necessary materials, choose a date and locations, and recruit researchers.

Introduce all the researchers ahead of time, by email. Make the materials available to everyone to review or at least peek at as soon as possible. Nudge everyone to look at the stuff ahead of time, just to prepare.

Put together everything you could possibly need on The Day in a kit. I used a small roll-aboard suitcase to hold everything. Here’s my list:

• Pens (lots of them)
• Clipboards, one for each team
• Flip cameras (people took them but did most of the recording on their phones)
• Scripts (half a page)
• Data collecting forms (the other half of the page)
• Printouts of the designs, or device-accessible prototypes to test
• Lists of names and phone numbers for researchers and me
• Lists of locations, including addresses, contact names, parking locations, and public transit routes
• Signs to post at locations about the study
• Masking tape
• Badges for each team member – either company IDs, or nice printed pages with the first names and “Researcher” printed large
• A large, empty envelope

About 10 days ahead, I chose a lead for each of the teams (these were all people who I knew were experienced user researchers) and talked with them. I put all the stuff listed above in a large, durable envelope with the team lead’s name on it.

Streamline data collection

The sessions were going to be short, and the note-taking awkward because of doing this research in ad hoc places, so I wanted to make data collection as easy as possible. Working from a form I borrowed from Whitney Quesenbery, I made something that I hoped would be quick and easy to fill in and easy for me to understand what the data meant later.

Data collector for our flash mob usability test

The data collection form was the main thing I spent time on in the briefing before everyone went off to collect data. There are things I will emphasize more, next time, but overall, this worked pretty well. One note: It is quite difficult to collect qualitative data in the wild by writing things down. Better to audio record.

Test the data collection forms

While the form was reasonably successful, there were some parts of it that didn’t work that well. Though a version of the form had been used in other studies before, I didn’t ask enough questions about the success or failure of the open text (qualitative data) part of the form. I wanted that data desperately, but it came back pretty messy. Testing the data collection form with someone else would have told me what questions researchers would have about that (meta, no?), and I could have done something else. Next time.

Minimize scripting

Maximize participant time by dedicating as much time to the session as possible to their interacting with the design. That means that the moderator does nothing to introduce the session, instead relying on an informed consent form that one of the team members can administer to the next participant while the current one is finishing up.

The other tip here is to write out the exact wording for the session (with primary and follow up questions), and threaten the researchers with being flogged with a wet noodle if they don’t follow the script.

Brief everyone on test goals, dos and don’ts

All the researchers and I met up at 10am and had a stand-up meeting in which I thanked everyone profusely for joining me in the study. And then I talked about and took questions on:

• The main thing I wanted to get out of each session. (There was one key concept that we wanted to know whether people understood from the design.)
• How to use the data collection forms. (We walked through every field.)
• How to use the script. (“You must follow the script.”)
• How to intercept people, inviting them to participate. (More on this below.)
• Rules about recordings. (Only hands and voices, no faces.)
• When to check in with me. (When you arrive at your location; at the top of each hour, when you’re on the way back.)
• When and where to meet when they were done.

I also handed cash that the researchers could use for transit or parking or lunch, or just keep.

Practice intercepting people

Intercepting people to participate is the hardest part. You walk up to a stranger on the street asking them for a favor. This might not be bad in your town. But in San Francisco, there’s no shortage of competition. Homeless people, political parities registering voters, hucksters, buskers, and kids working for Greenpeace all wanting attention from passers-by. And there you are, trying to do a research study. So, how to get some attention without freaking people out? A few things that worked well:

• Put the youngest and/or best-looking person on the task.
• Smile and make eye contact.
• Using cute pets to attract people. Two researchers who own golden retrievers brought their lovely dogs with them, which was a nice icebreaker.
• Start off with what you’re not: “I’m not selling anything, and I don’t work for Greenpeace. I’m doing a research study.”
• Start by asking for what you want: “Would you have a few minutes to help us make ballots easier to use?”
• Take turns – it can be exhausting enduring rejection.

Do an inventory check before spreading out

Before the researchers went off to their assigned locations, I asked each team to check that they had everything they needed, which apparently was not thorough enough for one of my teams. Next time, I will ask each team to empty out the contents of the packet and check the contents. I’ll use the list of things I wanted to include in each team’s packet and my agenda items for the briefing to ask the teams to look for each item.

Be flexible

Even with lots of planning and organizing, things happen that you couldn’t have anticipated. Researchers don’t show up, or their schedules have shifted. Locations turn out to not be so perfect. Give teams permission to do whatever they think is the right thing to get the data – short of breaking the law.

Check in

Teams checked in when they found their location, between sessions, and when they were on their way back to the meeting spot. I wanted to know that they weren’t lost, that everything was okay, and that they were finding people to take part. Asking teams to check in also gave them permission to ask me questions or help them make decisions so they could get the best data, or tell me what they were doing that was different from the plan. Basically, it was one giant exercise in The Doctrine of No Surprise.

Reconvene the same day

I needed to get the data from the research teams at some point. Why not meet up again and share experiences? Turns out that the stories from each team were important to all the other teams, and extremely helpful to me. They talked about the participants they’d had and the issues participants ran into with the designs we were testing. They also talked about their experiences with testing this way, which they all seemed to love. Afterward, I got emails from at least half the group volunteering to do it again. They had all had an adventure, met a lot of new people, got some practice with skills, and helped the world be a become a better place through design.


Wilder than testing in the wild, but trust that it will work

On that Saturday in San Francisco the amazing happened: 16 people who were strangers to one another came together to learn from 40 users about how well a design worked for them. The researchers came out from behind their monitors and out of their labs to gather data in the wild. The planning and organizing that I did ahead of time let it feel like a flash mob event to the researchers, and it gave them room to improvise as long as they collected valid data. And it worked. (See the results.)


P.S. I did not originate this approach to usability testing. As far as I know, the first person to do it was Whitney Quesenbery in New York City in the autumn of 2010.

What to do with the data: Moving from observations to design direction

What is data but observation? Observations are what was seen and what was heard. As teams work on early designs, the data is often about obvious design flaws and higher order behaviors, and not necessarily tallying details. In this article, let's talk about tools for working with observations made in exploratory or formative user research.

Many teams have a sort of intuitive approach to analyzing observations that relies on anecdote and aggression. Whoever is the loudest gets their version accepted by the group. Over the years, I've learned a few techniques for getting past that dynamic and on to informed inferences that lead to smart design direction and creating solution theories that can then be tested.

Collaborative techniques give better designs

The idea is to collaborate. Let's start with the assumption that the whole design team is involved in the planning and doing of whatever the user research project is.

Now, let's talk about some ways to expedite analysis and consensus. Doing this has the side benefit of minimizing reporting – if everyone involved in the design direction decisions has been involved all along, what do you need reporting for? (See more about this in the last section of this article.)

Some collaborative analysis techniques I've seen work really well with teams are:

- Between-session debriefs

- Rolling issues lists

- K-J analysis

- Cross-matching rolling issues lists with K-Js

Between-session debriefs

Do you just grind through sessions until you're through them all, only to end up having an excruciatingly long meeting with the team where you're having to re-play every session because no one was there but you?

Schedule extra time between sessions

Try this: Schedule more time than usual between sessions. If you usually schedule 15 minutes between usability test sessions, for example, then next time, schedule 30 minutes. Use the additional time to debrief with observers.

If the team sees that there will be discussion in between the sessions that will help move the design forward, they're more engaged. If team members are already observing sessions, then this gives you a chance to manage the conversations that they're already having.

Knowing that you're going to want to debrief between sessions, the team is more likely to come to more sessions and to pay full attention. They'll learn that if they're at the sessions, they get more say in the design outcome, and the design outcomes will make more sense. If they don't attend, they don't get as much say, simply because they've observed less and have less evidence for their inferences.

All you have to do is get the team to talk about what they saw and what they heard, and what was most surprising about that. Save the design solutions for later, unless you're doing rapid iterative testing.

Play 'guess the reason'

To get teams in the practice of sticking to discussing observations rather than jumping to design conclusions, I've tried playing a game called "Guess the Reason" with them. It's easy. Show a user interface – just one page or screen or panel – and describe the behavior observed. Then ask the team to guess why that happened. It's a brainstorming activity. The first person to go to a design solution has to put money in the team's drink fund. You can use the same system during your own debriefs, which can make it fun (and profitable).

Rolling issues lists

I've written about these before. Simply put, this technique gets the team further engaged in the collecting of observations and takes the burden off the moderator/researcher.

Gather whiteboard and markers

The idea is that those observations that come out in the debrief get written down on a white board that all the observers can see. Each observation gets tracked to the participants who had the issues. As the moderator, you start the list, but as the sessions go on, you encourage the team to add their own.

Natural consensus through debrief disucssion

As team members add, and the team talks about the observations that go onto the list, there's a natural consensus building that goes on. Does everyone agree that this is something we want to track? Does everyone agree that this way of talking about it makes sense to everyone?

Draws out what is important to the team

When I moderate user research sessions, doing this often means that I don't have to take notes at all because the team is recording what is important to them. As they're doing that, I also get to see what is important to the different roles on the team.

K-J analysis

I admit that I stole this idea from User Interface Engineering (UIE). But it's one of the most powerful tools in the collaboration toolbox. Jared Spool has an excellent article (that doubles as a script) about this technique.

When I do K-Js in workshops, everyone gets really excited. It's an amazing tool that objectively, democratically identifies what the high priority items are from subjective data.

The technique was invented by Jiro Kawakita to help his co-workers quickly come to consensus on priorities by getting them to discuss only what was really important to the whole team. There are 8 steps:

1. Create a focus question. For a usability test, it might be, "What are the most important changes to make to improve the user experience for this design?" In workshops, I often choose a more philosophical question, like, "What obstacles to teams face in implementing user experience design practices in their organizations?"

2. Get the group together. When I use this technique with teams at the end of a user research project, I invite only people who observed at least one session.

3. Put data or opinions on sticky notes. For the user research focus question, I ask for specific, one-sentence observations that are clear enough for other people to understand. (Team members often bring their computers with them to go through the notes they took during sessions.)

4. Put the sticky notes on the wall. Everyone puts their sticky notes up, in random order on one wall, while reviewing what other people are also putting on the wall. Allow no discussion.

5. Group similar items. This step is like affinity diagramming. Pick up a sticky; find something that seems related to it. Move to another wall, and put those two stickies on the wall, one above the other to form a column. All team members do this step together. Keep going until all the stickies have moved from one wall to the other and all the stickies are in a column. No discussion.

6. Name each column. Using a different color of stickies now, everyone in the room writes down a name for each group and puts their name on the wall above the appropriate column. Everyone must name every column, except if someone else has already stuck up a name that was exactly what you had written down. No discussion.

7. Vote for the most important columns. Everyone writes down what they think are the 3 most important columns. Next, they vote by marking 3 Xs for their most important group, 2 Xs for the second most important, and 1 X for their third most important group. Again, no discussion.

8. Tally the votes, which ranks the columns. On a flip chart or a white board, number a list from 20 to 1. Pull all the column name stickies that have votes and stick them next to the number of votes that are on the sticky. Now the facilitator can read off to the team which groups had the highest votes and thus are the highest priority. Now is the opportunity for discussion as the team determines which stickies can be combined. The decision to combine stickies – and thus, what the most important topics are – must be unanimous.

You're done. Very cool. Now the team knows exactly what to focus on to discuss, resolve, and remedy. And, if you're doing a report, you now know what to bother to report on. (See what I meant about reporting?)

Cross-match the rolling issues with the K-J

If your team or your management is into validation, you can now go back to your desk and compare what came out of the rolling issues with what the K-J generated. My experience so far has been that they match up. And it isn't because everyone at the K-J was primed by being at all the debriefs between sessions. People who observed remotely often contribute to the K-Js live, so you'd think that their data might change the K-J results. Your mileage may vary, but so far, mine matches up.

Directed collaboration is fun and generates better design solutions

When you help the team review together what they saw and heard during user research sessions, there is more likely to be consensus, buy-in, and a shared vision of the design direction. In testing early designs especially, consensus, buy-in, and shared vision are crucial to ending up with great user experiences. Collaborative techniques for analyzing observations turn work into fun, and take the pressure off the researcher to generate results. Because everyone on the team was involved in generating observations and setting priorities, everyone can move on quickly to making informed decisions that lead to coordinated, smart designs.

P.S. To anyone who gets my email newsletter whose email address was in the CC field rather than the BCC field: I apologize.  The production manager (me) must still have been asleep, and the QA manager (me) didn't catch that. We'll try to get it right next time. 

Mastering the art of user resarch - a new seminar for UI 14

This month, we're back to school. No, I'm not doing a graduate degree. But I am doing a lot of teaching. Today it's UI 14 that I want to tell you about: a special seminar I've developed, and a discount on registration.

Mastering the Art of User Research: 
Stress-free, Advanced Techniques for Creating Great Designs

For folks who feel solid on basic methodology for usability testing and user research, I've developed "Mastering the Art of User Research" just in time for User Interface 14 (or UI 14 if you're on friendly terms with UIE, producers of the conference), being held in Boston, Massachusetts on November 1-3.

I developed this seminar because I was thinking of you.

You know there's an art to making research valuable to the team and integral to design. You know how to design and conduct a good study, but you're thinking there just have to be shortcuts to make the process smarter. I bet you're the kind of person who is always teasing out ways to get the most out of every opportunity you have to be with users -- without trading off rigor.

In this seminar, I'll step through advanced techniques that will help you master your craft. You'll learn ways seasoned pros speed up planning, data collecting, and reporting. These technques will lighten your load, net you more and better data than you're getting with your current practices, and -- the best part -- will get the team excited about the design direction the data suggests.

Because you're actually asking more of your participants and your team, you'll have better, richer relationships with them. Participants will help you learn more than ever. The team will help you know what they care about and what to do about it.

You'll learn how to:

 - Get participants to recruit and document data for you.
 - Get a head start on data analysis during sessions.
 - Get consensus on observations on the fly.
 - Get the design team to prioritize issues in record time.
 - Generate short, focused communications that will help the team internalize results.


Come having practiced the classic processes for planning, designing, carrying out, and analyzing data for usability tests and the more common user research methods and techniques. This is not learning to cook cheese soup in Home Ec. This is advanced knife skills and saucing at the Cordon Bleu. This is riffing on a classic recipe to create something that is easier, quicker, tastier. I'm thinking 30-minute cassoulet.

Registration discount: Use my name and get $50 off of each day of registration

When you go to www.uie.com/events/uiconf/2009/register to sign up for UI 14, in the Promotion code box, just type in CHISNELL. If you register for all three days of the conference, you get $150 off the full registration AND a set of Bose headphones. Very cool.

UI 14 has an awesome line-up of speakers. Check out the program and register with the CHISNELL promotion code here: http://www.uie.com/events/uiconf/2009/.


See you in Boston in November.

Thanks for letting me interrupt you briefly. Now, back to our regularly scheduled program.

Popping the big question(s): How well? How easily? How valuable?

When teams decide to do usability testing on a design, it is often because there’s some design challenge to overcome. Something isn’t working. Or, there’s disagreement among team members about how to implement a feature or a function. Or, the team is trying something risky. Going to the users is a good answer. Otherwise, even great teams can get bogged down. But how do you talk about what you want to find out? Testing with users is not binary – you probably are not going to get an up or down, yes or no answer. It’s a question of degree. Things will happen that were not expected. The team should be prepared to learn and adjust. That is what iterating is for (in spite of how Agile talks about iterations).

Ask: How well
Want to find out whether something fits into the user’s mental model? Think about questions like these:

• How well does the interaction/information information architecture support users’ tasks?
  
• How well do headings, links, and labels help users find what they’re looking for?
• How well does the design support the brand in users’ minds?
  

Ask: How easily
Want to learn whether users can quickly and easily use what you have designed? Here are some questions to consider:

• How easily and successfully do users reach their task goals?
  
• How easily do users recognize this design as belonging to this company?
  
• How easily and successfully do they find the information they’re looking for?
  
• How easily do users understand the content?
  
• How easy is it for users to understand that they have found what they were looking for?
  
• How easy or difficult is it for them to understand the content?

Ask: How valuable

• What do users find useful about the design?
  
• What about the design do they value and why?
  
• What comments do participants have about the usefulness of the feature?

Ask: What else?

• What questions do your users have that the content is not answering?
  
• What needs do they have that the design is not addressing?
• Where do users start the task?

Teams that think of their design issues this way find that their users show them what to do in the way they perform with a design. Rarely is the result of usability testing an absolute win or lose for a design. Instead, you get clues about what’s working – and what’s not – and why. From that, you can make a great design.

Getting ready for sessions: Don’t forget…

There are a bunch of things to do to get ready for any test besides designing the test and recruiting participants.

• make sure you know the design well enough to know what should happen as the participant uses it
  
  
• copy any materials you need for taking notes
  
  
• copy of all the forms and questionnaires for participants, including honorarium receipts
  
  
• organize the forms in some way that makes sense for you. (I like a stand-up accordion file folder, in which I sort a set of forms for each participant into each slot. I stand up the unused sets and then when they’ve been filled out, they go back in on their sides.)
  
  
• check in with Accounting or whoever on money for honoraria or goodies for give-aways
  
  
• get a status report from the recruiter
  
  
• double-check the participant mix
  
  
• make sure you have contact information for each participant
  
  
• check that you have all the equipment, software, or whatever that you need for the participant to be able to do tasks
  
  
• run through the test a couple of times yourself
  
  
• double-check the equipment you’re going to use (I use a digital audio recorder, so I need memory sticks for that, along with rechargeable batteries)
  
  
• charge all the batteries
  
  
• double-check the location
  

Which gets us to where you’re going to do the sessions. But let’s talk about that later.

Moderating tips and techniques

Getting the right information from the participant can be a difficult. As the moderator, you must attend to many things besides what the participant doing and saying. Focusing on a few specific behaviors of your own will help you have a better test.

Focus your attention on what’s happening now

• Quickly build rapport with the participant
• Listen attentively
• Be open to what might happen in a session – be ready to learn from the participant

Tips for being a better moderator

Be the neutral observer – avoid priming or teaching. If you’re too close to the product or the domain, you may train participants without realizing it by using keywords in your task scenarios or materials.

Observe at the expense of collecting data, if you must. It is difficult to take notes and to watch the participant at the same time. If things are happening quickly or you find yourself missing things the participant is saying or doing, just stop taking notes. Instead, listen and spend time between sessions making notes about what happened. Go through your recordings later if you need to, or ask observers to share their notes.

Play dumb – don’t answer questions. If participants perceive that you are an expert on the product, they may ask you questions about it or look for your approval on actions. Instead, let her know that you are learning too, and that you’ll note her questions but won’t always be able to answer them.

Flex the script and test plan. Even after you pilot test your test, you may have to adjust on-the-fly when participants do unpredictable things. That’s okay. You’re learning important things that fit into your aggregate patterns of use.

Practice and get feedback. Ask co-workers and observers to give you feedback about how you conduct sessions and how you ask questions.

Your own self-awareness is your best tool for moderating test sessions successfully. Following these guidelines should help you get valid, reliable data from your participants, even if your attention is slightly divided.

Why create a test design?

I get a lot of clients who are in a hurry. They get to a point in their product cycle that they’re supposed to have done some usability activity to exit the development phase they are in and now find they have to scramble to pull it together. How long can it take to arrange and execute a discount usability test, anyway?

Well, to do a usability test right, it does take a few steps. How much time those steps take depends on your situation. Every step in the process is useful.


The steps of a usability test
Jeff Rubin and I think there are these steps to the process for conducting a usability test:

1. Develop a test plan
2. Set up the testing environment and plan logistics
3. Find and select participants
4. Prepare test materials
5. Conduct the sessions
6. Debrief participants and observers
7. Analyze data and observations
8. Create findings and recommendations

Notice that “develop a test plan” and “prepare test materials” are different steps.

It might seem like a shortcut to go directly to scripting the test session without designing the test. But the test plan is a necessary step.


Test plan or test design?
There’s a planning aspect to this deliverable. Why are you testing? Where will you test? What are the basic characteristics of the participants? What’s the timing for the test? For the tasks? What other logistics are involved in making this particular test happen? Do you need bogus data to play with, userids, or other props?

To some of us, a test design would be about experiment design. Will you test a hypothesis or is this an exploratory test? What are your research questions? What task scenarios will get you to the answers? Will you compare anything? If so, is it between subjects or within subjects? Will the moderator sit in the testing room or not? What data will you collect and what are you measuring?

It all goes together.



Why not just script the session without writing a plan?
Having a plan that you’ve thought through is always useful. You can use the test plan to get buy-in from stakeholders, too. As a representation of what the study will be, it’s understanding the blueprints and renderings before you give the building contractor approval to start building.

With a test plan, you also have a tool for documenting requirements (a frozen test environment, anyone?) for the test and a set of unambiguous details that define the scope of the test. Here, in a test plan, you define the approach to the research questions. In a session script, you operationalize the research questions. Writing a test plan helps you know what you’re going to collect data about and what you’re going to report on, as well as what the general content of the report will be.



Writing a test plan (or design, or whatever you want to call it) will give you a framework for the test in which a session script will fit. All the other deliverables of a usability test stem from the test plan. If you don’t have a plan, you risk using inappropriate participants and getting unreliable data.