Consensus on observations in real time: Keeping a rolling list of issues

Design teams often need results from usability studies yesterday. Teams I work with always want to start working on observations right away. How to support them while giving good data and ensuring that the final findings are valid?

Teams that are fully engaged in getting feedback from users – teams that share a vision of the experience they want their users to have – have often helped me gather data and evaluate in the course of the test. In chatting with Livia Labate, I learned that the amazing team at Comcast Interactive Media (CIM) came to the same technique on their own. Crystal Kubitsky of CIM was good enough to share photos of CIM's progress through one study. Here’s how it works:


1. Start noting observations right away
After two or three participants have tried the design, we take a longer break to debrief about what we have observed so far. In that debrief, each team member talks about what he or she has observed. We write it down on a white board and note which participants had the issue. The team works together to articulate what the observation or issue was.

2. Add observations to the list between sessions
After each succeeding session, as the team generates observations, I add observations to that list, including the numbers for each participant who had the issue. We note any variations on each observation – they may end up all in one, or they may branch off, depending on what else we see.

Here’s an example observation from one study. We noted on the first day of testing that

Participants talked about location, but scrolled past the map without interacting with it to get to the search results (the map may not look clickable)

The team and I later added the participant numbers for those who we observed doing this:

Participants talked about location, but scrolled past the map without interacting with it to get to the search results (the map may not look clickable) PP, P1, P3

Each day of testing, the team and I add more observations and more participant numbers. The CIM team debriefs to review top observations, highlighting what they learned and color-coding participants or segments as they capture rolling observations:
















3. Invite team member observers to add observations to the list themselves
As the team gets better at articulating the issues they have seen in a test session, it is my experience that they start adding to the list on their own. Often one of the observers voluntarily takes over adding to the list. This helps generate even more buy-in from the team and means that I can concentrate on focusing the discussion on the issues we agreed to explore when we planned and designed the test.

At the end of the last session, it’s easy then to move to a design direction meeting because the team has observed sessions, articulated issues, and already analyzed that data together.
















The CIM team documents which participants had which behaviors in the table at the top right of the photo above.

What makes it a “rolling” list of observations and issues
There are three things that are “rolling” about the list. First, the team adds issues to the list as they see new things come up (or that you didn’t notice before, or seemed like a one-off problem). Second, the team adds participant numbers for each of the issues as the test goes along. Third, the team refines the descriptions of the issues as they learn more from each new participant.

Double-check the data, if there’s time
Unless the team is in a very rapid, iterative, experimental mode, I still go back and tally all of the official data that I collected during each session. I want to be sure that there are no major differences between the rolling issues list and the final report. Usually, the rolling issues list and the final data match pretty closely because the team took care in defining the research questions and issues to explore together when we planned and designed the test.

Doing the rolling list keeps teams engaged, informed, and invested. It helps the researcher cross-check data later, and it gives designers and developers something to work from right away.

Observers are your friends

Research that you do alone ends up in only your head. No matter how good the report, slide deck, or highlights video, not all the knowledge gets transferred to your teammates. This isn’t your fault. It just is.

So what to do? Enlist as many people on your team as possible to help you by observing your usability testing sessions. You can even give your observers jobs, such as time keeper if you’re measuring time on task. Or, if you are recording sessions, it could be an observer’s job to start and stop the recordings and to label and store them properly.

The key is to involve the other people on the team – even managers – so they can

• Help you
• Learn from participants
• Share insights with you and other observers
• Buy in
• Reach consensus on what the issues are and how to solve them

Who should observe: Everyone
Ideally, everyone on the design and development team should observe sessions. Every designer, every programmer, every manager on the project should watch as real people use their designs.

Each of the observers should watch as many sessions as possible. (Okay, you could settle for two sessions.) Seeing only one session is just a snapshot; what happens in that session may not be similar to what happens in the other sessions. If the observer who saw only one participant is in an influential position in the organization, that one set of observations may outweigh others when it comes time to get some consensus from observers about what the issues are. (Hint: You don’t really want that.)


Training observers: Ground rules are essential
Observers can watch from a separate space – most labs are set up this way – or they can be present in the testing room with you and the participant during each session. Either way, you should brief them on

• What to look for
• How they might see it
• How to behave with participants
• What to do (and not do) with their observations

It probably isn’t enough just to hand out a list of rules if this is the first usability test these people will watch. Consider holding a short training session to explain how the usability test will be conducted and what they can expect in watching sessions. Be plain (but diplomatic, of course) about your expectations of your observers.

Here’s one of my favorite sets of guidelines for observers who will be in the testing room with you and the participant.

If observers will not be in the testing room with you and the participant but rather will be watching from a separate viewing room, ask observers to resist redesigning or finding solutions until the end of the study (or until the day’s debrief, whichever works best in your situation). This is another good reason to give each observer a little job – it keeps them occupied at different things that demand the attention that might otherwise be given to premature redesign.


Working with observers: Reaching consensus
No small part of working with observers is understanding what their separate stakes in the test are. When you know this, you can facilitate discussion among observers effectively. If you don’t know, you’re probably working harder than you need to to gain consensus.

Consensus is important. Reaching consensus among observers means that

• There’s a shared vision of where the design should go. This is good for the team and the design.
• Everyone uses the same descriptions for issues. Having a common language for talking about the usability issues among departments or groups will help the diverse group visualize the user experience together.
• Nuances and exceptions have been discussed and agreed on. Anything that isn’t optimally feasible to fix can be negotiated among design and development team members in a daily debrief or final design direction meeting.
• There should be no surprises when you publish the report. Because team members observed sessions and discussed the issues together, insights and outcomes in the report should reflect the agreed direction from those observations and group discussions.

Feeling lonely? Gather up some observers for your next usability test from your design and development teams. Make them feel welcome, give them duies, and work with them to understand the issues. They’ll thank you for it.