tag:blogger.com,1999:blog-1245259064183064432.post7608383383295313265..comments2023-10-06T05:11:29.584-07:00Comments on Usability Testing: Usability testing is broken: Rethinking user research for social interaction designDana Chisnellhttp://www.blogger.com/profile/00714198669290460546noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-1245259064183064432.post-45442822234051122412011-01-17T10:48:09.378-08:002011-01-17T10:48:09.378-08:00Agree with Michelle, quite an interesting post and...Agree with Michelle, quite an interesting post and thread of responses. Indeed, users are constantly changing our UI. It's important to stay on top of how these changes affect us, and try to stay a step ahead with user research. Great post, look forward to more from you Dana.Kailahttp://www.userlytics.comnoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-52448845461778967152010-12-08T13:25:12.676-08:002010-12-08T13:25:12.676-08:00Fascinating thread...rather than either / or think...Fascinating thread...rather than either / or thinking in terms of both / and is useful. We won't throw the baby out with the bath water (since we are using metaphors loosely). But, the initiating question is awesome - as others have stated. Foremost - it acknowledges that no matter how individualistic we think we are - we are always reacting / responding / interacting to and with others in our environment. (Including the 'anti-social' comment.) It is an interesting question to ponder how our information gathering systems (aka - research) approaches can / should evolve to gather insights from this 'strange new world.'Unknownhttps://www.blogger.com/profile/12408098325189944851noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-32495470457211882782010-12-08T08:08:44.457-08:002010-12-08T08:08:44.457-08:00you know what, the simplest things often leave the...you know what, the simplest things often leave the most profound of impacts on my nerve cells .... when people say what is social, i usually look the other way, knowing fully well, that i am being an anti social, cause the answer is so damn simple .... next time, i am going to quote 'emails are social' .... simple you know, usually does the trick. thanks for writing this piece.Kunal Kapoorhttps://www.blogger.com/profile/18083586885559475792noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-14391541571542270692010-12-05T15:33:40.381-08:002010-12-05T15:33:40.381-08:00Nah, usability testing isn't broken. Gross ov...Nah, usability testing isn't broken. Gross overstatement. As other people have said here, what's broken is the misapplication of evaluative usability methods. To use Randolph's analogy, you've changed to a screw but you're still whacking at it with a hammer. The best solution is to get a screwdriver, not to ruin the current useful applications of the hammer by trying to turn it into a screw-driving tool.<br /><br />Usability testing is a tool to outline usability problems - i.e. points where someone using an interface can't accomplish a task because <i>something is wrong with the design of the interface</i>. This means a button is missing or mislocated, or information is poorly worded, or things are hard to read, and etc.<br /><br />Often we're stuck with UT, because we have a prototype that's already been built without a sufficient understanding of context or needs, and then we're brought in to see whether it works - from a usability perspective. But we're trained to be problem-spotters. What happens, often, is that we start to see gaps in implementation where specific needs or use cases are not addressed. As you've pointed out, usability tests are sorely lacking in context.<br /><br />The ability to draw out these types of insights is great - it's one of the cool things about UT - but it really shouldn't be the primary goal of usability testing as an exercise. If we're looking to understand social context of product use, there are better tools to use than usability testing - ethnographic methods, contextual inquiry, interviews, shadowing, diary studies, etc etc. Your last post to Randolph started to get at this, and I agree with course correction.<br /><br />So maybe your post should / could be a call for a new hybridized research method, or diversification of backgrounds in our field. The problem here isn't that usability testing is broken - the problem (which may be illustrated by your post) is that usability practitioners, especially those from a software engineering background, don't understand the range of tools available to them, and tend to view UT as a panacea. We need to be broadening our toolkits and knowing when to apply specific methods - or at least understand the limitations of the methods we're required to use given the constraints of a project or situation.Dan Sewardhttp://peakusability.com.aunoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-18653668391881995762010-12-04T12:51:28.116-08:002010-12-04T12:51:28.116-08:00Calum,
I'm surprised that anyone comes her...Calum, <br /><br /> I'm surprised that anyone comes here, and I write the stuff. I'm glad you came, whatever you might think. <br /><br /> Too bad you felt it necessary to say not-so-nice things. <br /><br /> I hope you did look around. If you have any interest in usability testing and user research techniques and methods, you might get some ideas here. Or, you might find that you know it all already. In that case, I'd like to learn from you. <br /><br />Best, <br />DanaDana Chisnellhttps://www.blogger.com/profile/00714198669290460546noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-5064847733216102010-12-04T11:23:29.987-08:002010-12-04T11:23:29.987-08:00So really ... usability testing is not broken, it ...So really ... usability testing is not broken, it just needs to be rethought and extended for certain situations. But I suppose the soundbite, albeit incorrect, title of the post gets more people to look. Certainly worked for me... athough I now have zero interst in seeing what other overstated claims are made on this blog.Calumnoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-34942784374988270842010-12-01T04:11:12.680-08:002010-12-01T04:11:12.680-08:00Great reading.
Got me thinking about:
* Stop thi...Great reading.<br /><br />Got me thinking about:<br /><br />* Stop thinking about the people we design for and with as "lab rats"<br />* Get more fluid and creative about how we run our research (improvisation is ok too)<br />* Break down research questions into components that allow us to learn from each part towards better design (as opposed to treating research as something where we are required to get all the answers to the universe in one sitting)<br />* Design the research to feel more relaxed in order to nurture the good stories and warm interactions from the people we design for and with <br /><br />In short, be human :)<br /><br />rgds,<br />DanDaniel Szuchttp://www.apogeehk.comnoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-65889043113341815152010-11-30T20:35:39.027-08:002010-11-30T20:35:39.027-08:00Last year I used your book. This year I didn'...Last year I used your book. This year I didn't assign a textbook -- rather I had everyone pick a different book, read it, and do a book review of it for others to read. So we had 19 textbooks. <br /><br />And yes, I've got some academiosity -- wear it proudly. But it follows 22 years of practice, and is geared towards helping students find and excel at jobs. Knowing rules is good. Understanding the theory and history of practice that undergird the rules is even better -- it enables one to adjust intelligently when the situation changes (say, when the nail turns into a screw). (What kinda weird alchemy are we dealin' with, here?) And so, as I said in my earlier post, we (at least at UT) ARE evolving our methods, to supplement, not replace, usability testing.<br /><br />Hope to see you at UPA or the like.<br /><br />R.Randolph Bias.http://www.ischool.utexas.edu/~rbias/website/noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-21975366782511067362010-11-30T15:47:02.652-08:002010-11-30T15:47:02.652-08:00Dear Randolph,
I'm honored that you've r...Dear Randolph, <br /><br />I'm honored that you've reacted to my article about rethinking usability testing for social interaction design. <br /><br />Knowing your love of analogy, let's go with the classic. I'm not saying that we shouldn't do "traditional usability testing." That tool is good for what it is. I'm saying that we're using that hammer so much we've worn off the head and we're banging on the nail with the handle. <br /><br />And now the nail has changed to a screw. <br /><br />I said that usability testing is broken. If usability testing is a hammer, we broke it not because it's not a good tool, but many of us are not thinking about other tools for the expanded carpentry we're facing with fluid, ambient use of technology. <br /><br />The screw is social interaction design. As a field, UXers are miserable at testing this because we also don't know how to research it. All of the senior researchers I talked to at all those social network companies admitted that. We're making it all up. Which is what we were all doing in the 1980s, when we borrowed heavily from psychology. <br /><br />But social interaction isn't just psychological. It's behavioral, cognitive, cultural, sociological, political, and bunch of other -ologies. And as a field, I don't think we're doing a good job of looking at those aspects or borrowing from those fields to learn anything about people as they integrate technology in their lives. <br /><br />By the way, your pushing the Learn-the-rules-first approach exposes your academic-osity. But you're right, everyone has to start somewhere. I don't know how you're teaching usability testing at UT-Austin, but I will say that the book you're using, Handbook of Usability Testing Second Edition, is pretty prescriptive. It doesn't even *suggest,* "Hey, if this doesn't answer your research question, try some other techniques!" I helped build that hammer, I'll cop to that. But now I'm saying that technology has evolved, users of technology have evolved, culture has evolved, but in general people working in user experience research have not evolved their methods, and we need to. <br /><br />As for not knowing what you think, I'm not surprised by the content of your reaction at all. And I appreciate it because it's helping me refine my argument. <br /><br />Thanks, Randolph. As always, a stimulating exchange.<br /><br />DanaDana Chisnellhttps://www.blogger.com/profile/00714198669290460546noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-59253330439029467472010-11-30T15:34:44.579-08:002010-11-30T15:34:44.579-08:00Thanks, everyone for your comments. It's good ...Thanks, everyone for your comments. It's good to hear from you all!Dana Chisnellhttps://www.blogger.com/profile/00714198669290460546noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-83350846530121918012010-11-30T13:31:11.192-08:002010-11-30T13:31:11.192-08:00I would also comment that it feels that we are bac...I would also comment that it feels that we are back loading too much work into the "usability" part of the process. <br />There's a stack of work that should be done upfront at the "design research" end of the process which would provide more complete tasks and context - and hopefully better designs that include the social dimensions - so that the issues don't have to be resolved in usability.<br />Simulating these situations in order to usability test the efficiencies and effectiveness is really best done in the real context of use where people have access to the resources they normally have available (and in an ideal world the time they would normally take to do these things).<br />But IMHO the reality is that the social stuff should have been figured out long before you hit testing.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-21778141047621153242010-11-30T11:10:52.067-08:002010-11-30T11:10:52.067-08:00Dana, argghhh! "But that model of usability ...Dana, argghhh! "But that model of usability testing is broken." No it is not broken, any more than a hammer used to hit someone in the head is "broken" -- it is just a tool that can be misapplied. I fear you've been reading too much from some guy named Spool who told a roomful of 1000 "beginners" at SXSW-I in 2009 that the era of user-centered design is dead, it proven ineffectual. No way - UCD is alive and well. Now, it can be misapplied, just as one-on-one usability testing of user interfaces can be (and often is, when we claim to be testing "usability" when we're actually testing "learnability"). It can be leaned on too much. There are many problems it can't solve -- how to julienne fries being just one. But the need for an expanded world view, and the concomitant need for new tools doesn't dismiss the need for traditional usability testing, just as the development of nail guns didn't preclude the need for . . . oh, hell, even I am tired of that analogy.<br /><br />"I also think that people who are new to usability testing and user research are not going to do a good job of testing social interaction design, because constructing a study cannot be very formal or controlled. Measuring what’s happening is much more complex."<br /><br />Absolutely. But I would bet for them to get good at the testing of social interaction design they'll have to be good at fundamental usability testing. (There's a great book on this topic I might point you towards!) (Hmm, which analogy should we lean on, here? The best expressionists being good, first, at portraiture? The best free-verse poets having first labored under the strictures of rhyme and metre?) Just yesterday the students in my Advanced Usability class were pitching their final projects, and a couple had discovered (pioneered?) some pretty creative ways to get good usability data about social networking systems. But they were all well prepared for this by having read your book, and others, about the fundamentals of testing human subjects.<br /><br />"We have to stop thinking about human-computer interaction. That model by default is too limiting." Because it is "limiting" doesn't mean we stop thinking about it. There are many questions yet to be answered about HCI (an individual H I-ing with a C). And indeed EVERY social interaction of which you speak that includes some piece of technology will STILL entail HCI -- you canNOT get the social interaction right if an individual cannot interact with his/her piece of technology. So say it is limiting. Say it doesn't cover the new, social waterfront. But we "stop thinking about it" at our peril<br /><br />“'Task' doesn’t mean what you think it means." I humbly submit that you have no idea what I think -- well, except for all the foregoing.<br /><br />Warmly.<br /><br />Randolph.Randolph Bias.http://www.ischool.utexas.edu/~rbias/website/noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-25610415467734902002010-11-30T08:30:37.357-08:002010-11-30T08:30:37.357-08:00Very interesting and thought-provoking. I think yo...Very interesting and thought-provoking. I think you're right; we need to test products and services in a context that matches where they'll be used - or actually *is* the context where they'll be used. <br /><br />I believe that there is still *huge* value in focused 1:1 testing to identify points of the experience where things aren't clear, satisfying - where we, as designers, have made assumptions.<br /><br />Very interesting :)Paul Mayhttp://paulmay.orgnoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-77765682836100989552010-11-30T07:27:58.096-08:002010-11-30T07:27:58.096-08:00Great post Dana!
Both the challenges and how much...Great post Dana! <br />Both the challenges and how much we could be learning from new ways of constructing research. Thanks so much for pulling this together here, I vote for hearing more from you about it at IA Summit '11! We have definitely seen challenges like this in researching collaborative applications.Amelia Bellowshttp://communicationdesign.wordpress.com/noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-28764172513838955452010-11-30T07:02:21.388-08:002010-11-30T07:02:21.388-08:00You're right, testing is a social context - an...You're right, testing is a social context - and <a href="http://www.slideshare.net/folletto/mode-motivational-design-social-usability" rel="nofollow">everything is social</a> - it's hard.<br /><br />For me the main problem is about scale, or as I prefer saying, it's that it's a <strong>complex system</strong>. And for that reason there's no deterministic, simple approach to the problem.<br /><br />I'm not an expert about complex systems by themselves, but I'm studying those mainly as social environments and the first thing you always see cited is about seeing the <strong>weak signals</strong> that are emerging. But the system needs to be live and running: that's because otherwise you'll work in a not-complex, not-social system. And that wouldn't be the same. It's like using Twitter alone.<br /><br />So even if I'm not sure, I'm toward thinking that there's not "testing" possible. You either roll-out or not. And that's exactly how Facebook does things for example (but as well as other companies, like Google, even if they aren't social).<br /><br />Still, testing shouldn't be thrown away, because it solves the layers existing even before social dynamics kicks in. For example, I found that using the <a href="http://intenseminimalism.com/2010/social-usability-checklist/" rel="nofollow">Social Usability Checklist</a> we discussed some time ago and doing tests with a user interacting with another person online (simulating another social network user) are indeed useful to check the "connecting tissue".<br /><br />But once the connecting tissue is there, you need to release and pay attention to the weak signals of the complex system you're working on.<br /><br />This topic connects perfectly to agile methodologies (small iterations) and management techniques (small teams, release often, release early, and test, test, test). But that's probably a whole other topic, I'm just saying that everything is connected and the change should happen on multiple layers, where "testing" is just one of those. :)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-6003536753205479342010-11-30T04:48:29.892-08:002010-11-30T04:48:29.892-08:00Crystal,
Well, I hope to get on the IA Summit ...Crystal, <br /><br /> Well, I hope to get on the IA Summit '11 program with this topic. It seems there's interest from organizers. But you never know what happens during the review and selection process. <br /><br /> In any case, I'll see you there. We can have a "breakout" over a drink or two. <br /><br /> In the meantime, your blog post here - http://www.kubitsky.net/archives/322 - is a wonderful addition to the conversation. Thanks for that excellent, thoughtful response. <br /><br />DanaDana Chisnellhttps://www.blogger.com/profile/00714198669290460546noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-50947580004622787452010-11-30T04:39:56.787-08:002010-11-30T04:39:56.787-08:00Hi Kyle!
I appreciate your comment - and the c...Hi Kyle! <br /><br /> I appreciate your comment - and the constraints that UX researchers face. I also think it is up to us to not sell out when the design team shrugs off the importance of observing users themselves. It's awesome that so many organizations are including testing in their design cycles. But we have to stop usability testing being treated like QA. This is very hard. I get that. <br /><br /> The teams I see succeeding at making evaluation more realistic manage to do it because they're doing research and testing continuously. It's not a one-off event that interrupts or is separate from "design". <br /><br /> I think the rest of us can do it in small steps. As you suggest, eliciting "tasks" from users rather than coming in with pre-fab ones is huge. But it is difficult. It takes practice to do very well. And it might mean getting involved in "pre-research" activities like recruiting (a thing I could go on about for another 2,000 words pretty easily). <br /><br /> By the way, affordability is a strategic thing as well as a tactical thing. What was the cost to Google rolling out Buzz the way it did? Regardless of the monetary count, that episode degraded the credibility and trust that many users had with Google. Over the long term, that's a much more costly problem than a $15,000 or $50,000 usability test. <br /><br />Thanks for reading. Keep up the good work! <br /><br />DanaDana Chisnellhttps://www.blogger.com/profile/00714198669290460546noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-34993280957488423422010-11-30T04:22:41.051-08:002010-11-30T04:22:41.051-08:00Jacob,
Thanks for saying those nice things. Th...Jacob, <br /><br /> Thanks for saying those nice things. These are thoughts that have been percolating in my brain for a long while. It was great to talk to smart people and to get reactions at User Research Friday. The reaction to this blog post has been surprising, though, I must say! <br /><br /> The striking of balances is how we've rationalized doing what we do for a long time. I think getting time with users in their own time in their own place is a terrific step toward building a better, more realistic situation for an evaluation. <br /><br /> However, I think one of the problems we've created for ourselves in striking those balances is what we are unknowingly leaving out that causes real problems for real people down the road -- especially as more people become more aware of protecting their privacy and security online and off. As researchers and testers, we're making trade offs for the convenience of the business or the design team, often without thinking of what the real life implications of those trade offs are. <br /><br />Thanks for writing. I'm glad you're out there. <br /><br />DanaDana Chisnellhttps://www.blogger.com/profile/00714198669290460546noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-28171638995621596812010-11-29T18:24:10.610-08:002010-11-29T18:24:10.610-08:00You can definitely count on me for attending that ...You can definitely count on me for attending that IA Summit talk in whatever form it takes ;) I love this topic!Crystal Kubitskyhttp://www.kubitsky.netnoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-15025022497052203392010-11-29T17:51:41.209-08:002010-11-29T17:51:41.209-08:00Great post! I agree completely. We absolutely need...Great post! I agree completely. We absolutely need to stop making participants pretend. The best tasks are "created" by the participants. <br /><br />It would be great if clients would dedicate more time and participants for research studies, as you stated, but I fear that we would become a bottleneck -- a very expensive one too. Finding a way to provide real value through REAL data and still be affordable (in time and $) is key.Kyle Soucyhttp://www.usableinterface.comnoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-80093320190069441712010-11-29T13:24:40.445-08:002010-11-29T13:24:40.445-08:00That's quite a thought provoking article.
The...That's quite a thought provoking article.<br /><br />These days we do mostly remote testing - so we can test people where and when it suits them, but getting face time is important too. I guess like with everything in life it's about striking a balance, and getting the right mix of information for whatever project you happen to be working on. <br /><br />We run our own usability tool (http://intuitionhq.com) and one of the more interesting things about that from my perspective is that I get to watch and learn how people interact, what kind of information they are looking for, and the ways they go about gathering information. <br /><br />Thanks again for sharing, really an interesting article.Jacob Creechhttp://intuitionhq.comnoreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-57280719937468700002010-11-29T10:46:40.700-08:002010-11-29T10:46:40.700-08:00Hi Crystal,
Did someone set you up to ask abou...Hi Crystal, <br /><br /> Did someone set you up to ask about a talk? In any case, thanks. <br /><br /> I did a brief talk about this at User Research Friday in San Francisco on November 19. Someday, there'll be video of it online. (Though I'm not sure that's a good thing.) <br /><br /> Inspired to expand the URF talk, and get some input from other smart people, I put in a proposal to the IA Summit for 2011. <br /><br /> There seems to be a lot of interest in the topic whenever I bring it up. After the day I had with the smart people I mentioned in the post, I also interviewed user researchers from Twitter and Google. They had some awesome ideas for approaching user research for SxD, as well. I've been keeping a little notebook with methodological goodies in it. <br /><br /> Maybe there's a book in this...Dana Chisnellhttps://www.blogger.com/profile/00714198669290460546noreply@blogger.comtag:blogger.com,1999:blog-1245259064183064432.post-80426416263832386382010-11-29T10:32:01.310-08:002010-11-29T10:32:01.310-08:00I agree with your assessment that research needs t...I agree with your assessment that research needs to evolve with how technology has evolved in our customer's lives. Traditional practices only get us part of the picture and that part may be inaccurate and ineffective.<br /><br />I believe this is also applies to personas. It's difficult for me to find a single-person persona useful in the areas that I research -- entertainment. Every time I encounter them I'm left with questions about how other people in their lives affect their decisions and their activities. So much so that I want to see the persona in the context of their influential circle.<br /><br />Our methods need to evolve and we should feel free to try new approaches to do so, but we may not always be able to get the extra time that is required to do them well. Experimenting with new techniques and finding ways to save time with them will be important in changing how this kind of research is adopted.<br /><br />I would love to hear more about the ideas shared with you and an opportunity to share my own. Are you planning a workshop or discussion at an event soon?Crystal Kubitskyhttp://www.kubitsky.netnoreply@blogger.com