CS 3302 Introduction to Software Engineering
Winter 1995
Gregory Abowd

Performing a Cognitive Walkthrough

This explanation of the cognitive walkthrough is adapted from the following sources:

Alan Dix, Janet Finlay, Gregory Abowd and Russell Beale, Human-Computer Interaction, Prentice Hall, International, 1993. Chapter 11 contains information on evaluation techniques.

Clayton Lewis and John Rieman, Task-Centered User Interface Design: A practical introduction. A shareware book published by the authors, 1993. Original files for the book are available by FTP from ftp.cs.colorado.edu.

Cathleen Wharton, John Rieman, Clayton Lewis and Peter Polson, The Cognitive Walkthrough: A practitioner's guide. In Jakob Nielsen and Robert L. Mack, editors, Usability Inspection Methods. John Wiley and Sons, Inc. 1994.

The origin of the cognitive walkthrough approach to evaluation is the code walkthrough familiar in software engineering. Walkthroughs require a detailed review of a sequence of actions. In the code walkthrough, the sequence represents a segment of the program code that is stepped through by the reviewers to check certain characteristics (e.g., that coding style is adhered to, conventions for spelling variables versus procedure calls, and to check that system wide invariants are not violated).

In the cognitive walkthrough, the sequence of actions refers to the steps that an interface will require a user to perform in order to accomplish some task. The evaluators then step through that action sequence to check it for potential usability problems. Usually, the main focus of the cognitive walkthrough is to establish how easy a system is to learn. More specifically, the focus is on learning through exploration. Experience shows that many users prefer to learn how to use a system by exploring its functionality hands on, and not after sufficient training or examination of a user's manual. So the kinds of checks that are made during the walkthrough ask questions that address this exploratory kind of learning. To do this, the evaluators go through each step in the task and provide a story about why that step is or is not good for a new user.

To do a walkthrough (the term walkthrough from now on refers to the cognitive walkthrough, and not any other kinds of walkthroughs), you need four things.

  1. A description of the prototype of the system. It doesn't have to be complete, but it should be fairly detailed. Details such as the location and wording for a menu can make a big difference.

  2. A description of the task the user is to perform on the system. This should be a representative task that most users will want to do.

  3. A complete, written list of the actions needed to complete the task with the given prototype.

  4. An indication of who the users are and what kind of experience and knowledge the evaluators can assume about them.

Given this information, the evaluators step through the action sequence (item 3 above) to critique the system and tell a believable story about its usability. To do this, for each action, the evaluators try to answer the following four questions.

A. Will the users be trying to produce whatever effect the action has?

Are the assumptions about what task the action is supporting correct given the user's experience and knowledge up to this point in the interaction?

B. Will users be able to notice that the correct action is available?

Will users see the button or menu item, for example, that is how the next action is actually achieved by the system? This is not asking whether they will know that the button is the one they want. This is merely asking whether it is visible to them at the time when they will need to invoke it. An example of when this question gets a negative supporting story might be if a VCR remote control has a hidden panel of buttons that are not obvious to a new user.

C. Once users find the correct action at the interface, will they know that it is the right one for the effect they are trying to produce?

This complements the previous question. It is one thing for a button or menu item to be visible, but will the user's know that it is the one they are looking for to complete their task?

D. After the action is taken, will users understand the feedback they get?

Assuming the users did the correct action, will they know that. This is the completion of the execution/evaluation interaction cycle. In order to determine if they have accomplished their goal, the user needs appropriate feedback.

Important Note

It is vital to document the cognitive walkthrough to keep a record of what is good and what needs improvement in the design. It is therefore good to produce some standard evaluation forms for the walkthrough. The cover form would list the information in items 1-4 above, as well as identify the date and time of the walkthrough and the names of the evaluators. Then for each action (from item 3 on the cover form), a separate standard form is filled out that answers each of the questions A-D above. Any negative answer for any of the questions for any particular action should be documented on a separate usability problem report sheet. This problem report sheet should indicate the system being built (the version, if necessary), the date, the evaluators and a detailed description of the usability problem. It would also be useful to determine the severity of the problem, that is, whether the evaluators think this problem will occur often and an impression of how serious the problem will be for the users. This information will help the designers to decide priorities for correcting the design.