Comments on the Usability Testing Assignment
Prof. Van House
3/9/10
USABILITY TESTING
Most of you found setp was more complicated, took more time, and had more problems than you expected. Good up-front planning was needed. You also found you need to be very familiar with the application, the tasks, and the questions you have. Not as easy as it seems! This is why I stress pre-testing.
Testing your set-up: not only is everything working, but are you getting good recording etc. E.g., do some sample videoing and see how it plays back.
Testing your set-up, more: it’s really easy to get distracted by the logistics. Another reason you want everything to work seamlessly: so you can focus on YOUR tasks, and not have to worry about logistics, equipment.
Some of you commented that giving the user a printed list of tasks worked better than having to read them the tasks.
There’s a balance between specific, orchestrated tasks that give you good, comparable data, and enough flexibility for you to see what the user would really do. The latter might lead you in useful directions that you hadn’t anticipated.
It helps to know who the users are likely to be, why they would be using your site/product, what tasks they are likely to want to accomplish, and what they would already know. For example, if they are going to come with a very specific task, then you want to formulate a specific task – or ask them to describe a task they would likely be performing, and then have them perform that task. (This is also one reason why having you as the test subjects wasn’t ideal.
Useful to focus on the user’s complete task not (just) object/tech – e.g., the campus map project: people want to park as well as find buildings etc. But the campus and parking maps weren’t coordinated.
Value of trying to put yourself in someone else’s shoes, as test subject trying to see the situation as a user might.
Seeing yourself on tape as a user – often a disjunct between what you say and do, what you say and what you’re really thinking. Explaining away your mistakes so as not to appear stupid. Also a user who afterward says that “in real life” she would have done something different (i.e., given up). Of course you want to know what they’ll think/do/say in real life.
Decisions about what to write down in the moment (and how to go back and find key moments on the video).
A site that has user-generated (or at least other-generated) content is combining the work of many people, and so you have to at least try to parse these out. For example, bSpace shouldn’t take the rap if I as an instructor make a mess of my site (although bSpace should try to help me avoid making a mess)
Mobile usability is an added challenge – can’t see the screen as well, have to put the mobile device under a video camera and ask the user not to move it, etc.
You learned about your own actions as moderator (mostly) in reviewing the video. Another reason why it’s good to do video or at least audio recording.
Debriefing the user(s) is really important, as many of you commented. Users may be debriefed individually and/or collectively – ideally you would do both (so they could compare reactions and prod one another), but anything is better than nothing, and more is better than less.
Recurring theme: hard to take notes AND watch and listen to what you’re taking notes on! Could be valuable to have someone in the room (videographer?) who has less to do and is better able to just watch (and jump in when necessary). User body language, facial expression, tone of voice, all are important. Which the note-taker often misses when looking at screen or paper.
Moderator needs to be familiar with the application, to be able to ask questions and nudge the user when needed.
REPORTS
If you don’t report clearly and persuasively on your usability testing, you might as well have not done it! Pat attention to the “usability” of your report. Assume that your user is busy and that you have to capture their attention and to make sure that they understand your findings and recommendations. It’s up to you.
Recommendations are sometimes for design changes; sometimes for standards of user performance; sometimes for further testing (with details). All are reasonable and legitimate. But be careful not to unconsciously slide among these. In a client situation, which kind of recommendation you make will depend on what you’ve been asked to do, what stage of design the project is in, and other such considerations. It’s useful to be deliberate about this – decide with your client or within your group what kind of recs you’re going to give, and why.
I saw two different but effective report formats:
-step-by-step description of user actions, with problems clearly identified.
-summary of problems, with description/explanation for each. When user had no trouble, the report said, “User had little difficulty with doing x but did voice a minor complaint about y.”
How much does the audience for your report know about what you’re testing? How much do you need to tell them? This may vary. Use screenshots or something similar to illustrate your points, and link images closely to the text, rather than relying on your reader’s memory or risking losing the reader who doesn’t know the application you’re testing.
Use images, and link images to text! It’s often hard for the reader to understand when you describe (rather than show) things like what the user is seeing at a specific stage in the process.
Unless your text is unambiguously linked to specific images, number images (Figure x, table x) and then refer to them by # in the text. Give Figures titles, too, so that the reader knows what she’s seeing, what each figure is intended to illustrate.
Remember that a major task of your report is not just to communicate but to persuade the reader/client that (1) you’re worth listening to, and (2) your recommendations are worth following. If some of your recommendations seem unnecessarily picky or burdensome, you may undermine your credibility with your client and especially with the people who have to implement your recommendations. Ditto if your recommendations seem too “easy” – e.g., one group recommended that the designers limit the # of buttons on their interface – but if there are too few buttons, then does each button end up having too many functions? Might be more useful and credible to suggest that they test various prototypes with fewer buttons – to try to find the right balance.
Write succinctly, clearly. Avoid redundancy. Use headings, format, emphasis, to convey your meaning clearly and easily. But be careful that the hierarcy among headings is clear. Otherwise the reader can get confused as to what’s related to what, what’s subordinate to what. Use size, indentation, other such obvious indicators. (For example, if you use bold, italics, and underlining for headings, the hierarchy is not clear.)
When you’re doing comparisons (such as across sites or applications), tables can often be helpful for comparing elements, tasks, and the like, in parallel, succinctly.
Beware of long and convoluted sentences. Also long paragraphs. User wants to read fast and get the major points with least effort.
Use formatting to help the reader find most important info, and to emphasize your most important points. Don’t bury your most important points in the text. Consider highlighting evaluations, recommendations, key observations – e.g., bold, or call-outs (such as boxed text). Such as, “Both users experienced difficulty with xxx.”
Reporting user time on tasks: this can be useful info (especially when comparing designs that differ considerably) but be careful about comparing time when you’re asking them to think-aloud, or asking questions. You are likely slowing them down. And even if you aren’t, your client may think you are and not believe your reported findings.