Where do heuristics come from?

Recently I had the honor and pleasure of working on a project for the National Institute of Standards and Technology (NIST) to develop style guidelines for voting system documentation. Yawner, right? Not at all, it turns out. It made me think about where guidelines and heuristics come from for all kinds of design. Yes, if you live in the United States, you paid for me to find this out. Thank you.

What I learned in the process of developing style guidelines for voting system documentation (which, astonishingly took about a year) is that most heuristics -- accepted principles -- used in evaluating user interfaces come from three sources: Lore or folk wisdom, specialist experience, and research.

Though style guidelines for content are important, I’m going to talk about each of these sources of heuristics with various design examples. I’m sure you’ll see something that you’ve encountered before.


Lore or folk wisdom
First come guidance from “They,” as in “They say…,” for which no one knows the true source. For example: “Feed a cold, starve a fever.” “Never end a sentence with a preposition.” “Limit the number of items in the main navigation to seven plus or minus two.”

Where do these come from? Someone’s belief that this is a good practice. They may have heard something or done something that they think supports the practice, but there’s really not basis in fact for any of these.

A New York Times article published on February 13, 2007 by Anahad O’Connor says that recent research about whether to eat a lot when you have a cold and fast when you have a fever is inconclusive. No one seems to know how this one started. It may just feel like there’s some inherent logic to it.

Not ending sentences with prepositions was encoded by a British guy named Henry Fowler in 1926. He was a crotchety, proscriptionist pedant but his book was a best seller. People wanted guidance about how to speak and write “properly,” especially in class-conscious England. So, a rule to not use words like “to,” “in,” “for,” “with,” or “on” as the last word in a sentence became wildly popular as a marker of a well bred, well educated person. But it was really just Fowler’s personal preference, and today the usage seems affected.

My favorite example in the web design world is a guideline about limiting the number of items in a navigation menu or list to five to seven items. Most people don’t know where this came from – if they did, they’d know that this isn’t the best use of that “rule” and imposing it actually won’t make the design usable. This one does originate in research, specifically, an article published in 1956 by George Miller in the Psychological Review called “The magical number seven, plus or minus two: Some limits on our capacity for processing information.” (You can see a reproduction of the article here: http://www.musanim.com/miller1956/) The findings from the research Miller describes are about working memory. The lore passed down from that article is that humans can only hold about seven things in their short-term memory at a time. BUT, Miller heavily, heavily qualifies this as “suggestive” and an “estimate.” More importantly, what in the world does this have to do with designing web site navigation? Nothing. Navigation is persistent. We’re not asking people to remember from section to section where they can go. It’s right there for them to see and use. The number of items in navigation should be determined by the data from research about the users and their task goals.

If you catch yourself saying, “they say,” or “I’ve heard” when making an observation about a design issue when you’re inspection mode, you may be caught without a lot of support for your point. Basing an inspection on your own experiences observing users can hold more authority – but not as much as research-based guidelines or heuristics.


Specialist experience
Older adults who use the Web need high contrast and large targets. If they are not expert Web users, they can be easily distracted, so to ensure that they’re successful, we should design in smooth task paths and clear labeling that doesn’t use jargon.

One of my special interests since about 2003 has been Web design for older adults. I’ve internalized the design principles above (as well as many others) after watching dozens of people who are age 55 or older use a variety of web sites. I am confident that implementing a design that takes these design principles into account will make the design easier for older people to use than designs that use subtle colors layered on one another, small buttons and links, and cluttery page layout with trendoid headings and labels. Though I have observed many types of people using lots of different kinds of web sites, I have specialist experience from watching one audience try to do typical tasks on a variety of web sites.

Specialist experience means expertise in a particular domain or product. You get it only after hours and hours and hours of seeing the same kinds of things happen to the same types of users. Basing an inspection on specialist experience is definitely a step up from working from lore, but if you haven’t distilled what you have found in the many hours of observing a type of users using a site or type of site, then you may be working from hunches and opinion that could make it difficult for you to justify the evaluation recommendations.


Evidence from research
Some things that experienced designers have internalized do have data to support them: Eliminate horizontal scrolling. Design for working memory limitations. Facilitate scanning.

These all rate a 5 for strength of evidence in the guidelines on usability.gov. (I’ll get to the rating thing in a minute.) Usability.gov started as a project at the National Cancer Institute (NCI), which is part of the US Department of Health and Human Services. NCI needed help designing usable cancer information web sites. A simple goal.

On the way, the NCI team realized that not all guidelines were equal. Some guidelines were supported by a lot of data from multiple studies (like the high scoring heuristics above). Some guidelines might come from only one study. Still, evidence was evidence and NCI wanted to use “quantified, peer-reviewed web site design guidelines,” which they found simply didn’t exist. And as far as I know, there’s nothing like the resource NCI created at usability.gov.

To reach their goal, NCI put together panels of experts to review research. The panelists then who rated each guideline for strength of evidence (which among other considerations was “cumulative and compelling” for a 5 out of 5).

The idea was that teams could use the ratings to help them make design decisions. But the guidelines were not meant to be a substitute for usability testing. Why not? The main reason was that the guidelines at usability.gov were developed for information-rich web sites (versus e-commerce or transaction-based sites) with content about major illnesses. That’s fairly specialized. But when you read through the 500 guidelines that NCI identified, it is obvious that almost all could apply to many types of web sites or many types of pages within sites. Your mileage may vary.


The basis of the heuristic matters
As the folks at NCI learned in developing usability.gov and I learned in the work for NIST*, provenance is important. This is true of all implicit and explicit heuristics applied in design decisions.

Learning about where heuristics come from – lore or folk wisdom; specialist experience; or research – helped me understand better where some of the teams I’ve worked with were coming from as they developed design principles. Sometimes they based the principles on lore, sometimes on expertise. Rarely did they go to the research.

Expertise is good, but research is better. Research-based heuristics simply have more heft: credibility, specificity, and applicability.

Still, there’s no substitute for primary research. Firsthand observation of your users in their context reveals subtleties of behavior that even research-based heuristics can’t match. And if your research of your users in their context contradicts the known research, what do you do? (You don’t get two guesses to answer this question.) If you go with what your users do, then even the most deeply researched heuristics are at best a poor substitute for doing the right thing.


* I couldn't have made the discoveries I did on that project without Susan Becker, my project partner, who did most of the heavy lifting.


:: :: :: :: ::

Related links

You can pore through the evidence-based guidelines for usability developed by the National Cancer Institute (NCI) at www.usability.gov.

You might also want to check out other sets of research-based heuristics and guidelines. I've worked on a couple of them:

www.aarp.org/olderwiserwired holds a set of heuristics for designing web sites for older adults. The National Institute on Aging (NIA) has just published an update of its guidelines for designing senior-friendly websites at http://www.nia.nih.gov/HealthInformation/Publications/website.htm.

vote.nist.gov now links to style guidelines for voting system documentation, which are based on research in technical communication, information design, usability, and instructional design. Click on the Publications link to download the PDF of the guidelines and to view other design research related to voting systems.

Oh, and you might want to read Miller's article to judge for yourself: http://www.musanim.com/miller1956/