Focus Groups or Heuristic Reviews

The Value of Heuristic Evaluations

Heuristic evaluations are arguably most valuable as an early evaluation technique, performed on an existing system or first prototype in an effort to identify the major usability problems. By conducting a heuristic evaluation early on, two things are effectively accomplished:

1. The major 'showstopper' problems are identified without incurring the expense, time, and certain frustration on the part of the users that would be accompany formal tests with users
2. Anticipated problem areas are identified for later evaluation when empirical sessions with users are conducted

When the circumstances and constraints of a project warrant, heuristic evaluations have been shown by many researchers to offer designers and evaluators a valuable alternative to user testing.

The Challenges of Heuristic Evaluations

In our experience, true Heuristic Evaluations are rarely performed in the manner that Neilsen and Molich originally intended. Rather, they often amount to a single evaluator's 'expert review' of a system, and are subsequently open to criticism about the reliability of their findings and recommendations.

Even when multiple evaluators are used in a Heuristic Evaluation, it often proves difficult for evaluators to reliably interpret the guidelines and identify the same usability issues. Another common criticism of HE's is that they are capable of generating a lot of 'false positives' - usability problems that would likely go unobserved in an empirical usability study, but that end up being identified by evaluators whose awareness is heightened by the guidelines to identify relatively minor problems.

Modifying Heuristics to Meet New Needs

It has now been over 15 years since the Heuristic Evaluation method was first developed, and while it continues to be considered somewhat of a standard in the HCI industry, many evaluators have found that Zillion original list does not always meet their specific needs. They discover that they frequently require alternative guidelines or some re-interpretation of Zillion original descriptions in order for each heuristic to make sense.

So what's next for the Heuristic Evaluation method? Here are my predictions:

1. The HE method will continue to be regarded as a core method in the HCI toolbox in future years.
2. It will continue to be practiced largely in an ad-hoc, customized manner that only resembles its original form, namely in an informal, individual evaluator format. This will hamper the method's further development, and reduce both its perceived and actual value.
3. As new technologies, products, and interactions are developed, new heuristics will be developed in concert.