Slides for several classes posted today:

Field Observation

Interviewing

Surveys

Diary Studies

Focus groups

Good summary of the literature and main advantages and disadvantages to online focus groups (for all purposes):

BMC Med Res Methodol. 2009 Mar 3;9:15.

Online focus groups as a tool to collect data in hard-to-include populations: examples from paediatric oncology.

Tates K, Zwaanswijk M, Otten R, van Dulmen S, Hoogerbrugge PM, Kamps WA, Bensing JM.

NIVEL (Netherlands Institute for Health Services Research), BN Utrecht, The Netherlands. k.tates@nivel.nl

BACKGROUND: The purpose of this article is to describe and evaluate the methodology of online focus group discussions within the setting of paediatric oncology. METHODS: Qualitative study consisting of separate moderated asynchronous online discussion groups with 7 paediatric cancer patients (aged 8-17), 11 parents, and 18 survivors of childhood cancer (aged 8-17 at diagnosis). RESULTS: All three participant groups could be actively engaged over a one-week period. Respondents highly valued the flexibility and convenience of logging in at their own time and place to join the discussion. Adolescent patients and survivors emphasized that the anonymity experienced made them feel comfortable to express their views in detail. The findings indicate a strong preference for online group discussions across all participant groups. CONCLUSION: The findings show that online focus group methodology is a feasible tool for collecting qualitative data within the setting of paediatric oncology, and may offer new opportunities to collect data in other hard-to-include populations. The evaluations seem to indicate that the online group discussions have given participants an opportunity to articulate their experiences and views in a way they might not have done in a traditional group discussion.

Please bring to class a dozen or more topics, observations, quotes, issues, etc. from your interviews, from usability tests, from the literature, or other sources related to your project.  If possible, bring on medium-sized post-its, preferably 3×3 inch size.

If you don’t have such evidence for your project now, make something up for the sake of the exercise. Or find something online.

You’re going to spend some time in class with your group combining the contributions from all of you to find some themes.

If you’re doing a project on your own, you can attach yourself to a group for the sake of this exercise — you don’t HAVE to have observations of your own to contribute.

We won’t spend the whole class time on this.

This page has links to dozens of reports!  To evaluate them, put yourself in the position of the client: does this tell you what you need to know?  Too little? Too much? Not the right info?  http://www.pdfgeni.com/book/usability-report-pdf.html

From Usability.gov —  http://www.usability.gov/templates/index.html#Usability — their other templates are useful, too.

Test report templates for writing the usability test report.

• Research style usability test report template (DOC – 144KB)
• Short (informal) usability test report template (DOC – 216KB)
• Long (formal) usability test report template (DOC – 536KB)

Just ran across this —

Preview of the Usability Body of Knowledge — Usability Professionals’ Association is creating this reference work.

I’d like to meet with every project group to see where you are — this week or next. The purpose is for us to discuss where you are and what you’re doing in time for my review to be of help to you in finishing your project.

I’ve posted a schedule on http://www.wejoinin.com/sheets/jhveb. Pls sign up your group.

Please group the meetings — that is, if you can, please don’t leave gaps in between my scheduled meetings.

Please send me at least a day ahead of time a 1-2 page summary before the meeting, covering the following — and be prepared to talk about these topics, with an emphasis on anywhere that you think I might be of help to you:

—what you’re doing (topic, methods)
—where you are
—what your plans are for the rest of the semester
—any specific problems, barriers, questions that you may have

If none of these times works for your group, first see if you can find a time on my schedule when most of your group can make it — maybe all but one person.

Then if none REALLY works, email me with some times when you can make it.

http://www.audiotranskription.de/english/f4.htm

f4 is our free transcription software for audio and video recordings

f4 – a huge support for you. Works with Mac, Windows and Linux systems and significantly alleviates typing.

f4 supports your transcription process, meaning your personal typing of video or audio recordings from interview situations. Applications like Windows Media Player or iTunes do not qualify for this kind of work. They lack essential features that simplify the transcription process. Automatic voice recognition software will cease operating if there is more than one voice involved (which is usually the case during an interview).

f4 alleviates your work by slowing down the playback speed, providing key operated control (also directly from „Word“), performing a short automatic rewind when hitting the Stop button and allowing the (automatic) insertion of text fragments and cue points through simple key combinations. f4 works with mp3, wma, wav, ogg, aif, mov, avi and mpg files (in some cases also DVD).

Express Scribe — free transcription software.  It has quick-keys for adding time code and stopping/starting the audio.  It has settings so that you can play the recording more slowly or faster than recorded.  Also, you can click on time code in your transcript and will automatically be taken to that point in your
audio.

http://www.nch.com.au/scribe/

Also, iTunes works pretty well.

Ran across this — I haven’t seen a site publish these figures.  I assume it’s because they’re a gov’t site — and want to show how useful they are.

Statistics Canada Website Traffic and Vistor Satisfaction

Comments on the Usability Testing Assignment

Prof. Van House

3/9/10

USABILITY TESTING

Most of you found setp was more complicated, took more time, and had more problems than you expected.  Good up-front planning was needed. You also found you need to be very familiar with the application, the tasks, and the questions you have.  Not as easy as it seems!  This is why I stress pre-testing.

Testing your set-up: not only is everything working, but are you getting good recording etc. E.g., do some sample videoing and see how it plays back.

Testing your set-up, more: it’s really easy to get distracted by the logistics. Another reason you want everything to work seamlessly: so you can focus on YOUR tasks, and not have to worry about logistics, equipment.

Some of you commented that giving the user a printed list of tasks worked better than having to read them the tasks.

There’s a balance between specific, orchestrated tasks that give you good, comparable data, and enough flexibility for you to see what the user would really do. The latter might lead you in useful directions that you hadn’t anticipated.

It helps to know who the users are likely to be, why they would be using your site/product, what tasks they are likely to want to accomplish, and what they would already know. For example, if they are going to come with a very specific task, then you want to formulate a specific task – or ask them to describe a task they would likely be performing, and then have them perform that task.  (This is also one reason why having you as the test subjects wasn’t ideal.

Useful to focus on the user’s complete task not (just) object/tech – e.g., the campus map project: people want to park as well as find buildings etc.  But the campus and parking maps weren’t coordinated.

Value of trying to put yourself in someone else’s shoes, as test subject trying to see the situation as a user might.

Seeing yourself on tape as a user – often a disjunct between what you say and do, what you say and what you’re really thinking.   Explaining away your mistakes so as not to appear stupid.  Also a user who afterward says that “in real life” she would have done something different (i.e., given up).  Of course you want to know what they’ll think/do/say in real life.

Decisions about what to write down in the moment (and how to go back and find key moments on the video).

A site that has user-generated (or at least other-generated) content is combining the work of many people, and so you have to at least try to parse these out.  For example,  bSpace shouldn’t take the rap if I as an instructor make a mess of my site (although bSpace should try to help me avoid making a mess)

Mobile usability is an added challenge – can’t see the screen as well, have to put the mobile device under a video camera and ask the user not to move it, etc.

You learned about your own actions as moderator (mostly) in reviewing the video.  Another reason why it’s good to do video or at least audio recording.

Debriefing the user(s) is really important, as many of you commented.  Users may be debriefed individually and/or collectively – ideally you would do both (so they could compare reactions and prod one another), but anything is better than nothing, and more is better than less.

Recurring theme: hard to take notes AND watch and listen to what you’re taking notes on!  Could be valuable to have someone in the room (videographer?) who has less to do and is better able to just watch (and jump in when necessary). User body language, facial expression, tone of voice, all are important.  Which the note-taker often misses when looking at screen or paper.

Moderator needs to be familiar with the application, to be able to ask questions and nudge the user when needed.

REPORTS

If you don’t report clearly and persuasively on your usability testing, you might as well have not done it!  Pat attention to the “usability” of your report. Assume that your user is busy and that you have to capture their attention and to make sure that they understand your findings and recommendations.  It’s up to you.

Recommendations are sometimes for design changes; sometimes for standards of user performance; sometimes for further testing (with details). All are reasonable and legitimate.  But be careful not to unconsciously slide among these.   In a client situation, which kind of recommendation you make will depend on what you’ve been asked to do, what stage of design the project is in, and other such considerations.   It’s useful to be deliberate about this – decide with your client or within your group what kind of recs you’re going to give, and why.

I saw two different but effective report formats:

-step-by-step description of user actions, with problems clearly identified.

-summary of problems, with description/explanation for each. When user had no trouble, the report said, “User had little difficulty with doing x but did voice a minor complaint about y.”

How much does the audience for your report know about what you’re testing?  How much do you need to tell them? This may vary.  Use screenshots or something similar to illustrate your points, and link images closely to the text, rather than relying on your reader’s memory or risking losing the reader who doesn’t know the application you’re testing.

Use images, and link images to text! It’s often hard for the reader to understand when you describe (rather than show) things like what the user is seeing at a specific stage in the process.

Unless your text is unambiguously linked to specific images, number images (Figure x, table x) and then refer to them by # in the text. Give Figures titles, too, so that the reader knows what she’s seeing, what each figure is intended to illustrate.

Remember that a major task of your report is not just to communicate but to persuade the reader/client that (1) you’re worth listening to, and (2) your recommendations are worth following. If some of your recommendations seem unnecessarily picky or burdensome, you may undermine your credibility with your client and especially with the people who have to implement your recommendations.   Ditto if your recommendations seem too “easy” – e.g., one group recommended that the designers limit the # of buttons on their interface – but if there are too few buttons, then does each button end up having too many functions?  Might be more useful and credible to suggest that they test various prototypes with fewer buttons – to try to find the right balance.

Write succinctly, clearly.  Avoid redundancy.  Use headings, format, emphasis, to convey your meaning clearly and easily.  But be careful that the hierarcy among headings is clear.  Otherwise the reader can get confused as to what’s related to what, what’s subordinate to what.  Use size, indentation, other such obvious indicators.  (For example, if you use bold, italics, and underlining for headings,  the hierarchy is not clear.)

When you’re doing comparisons (such as across sites or applications), tables can often be helpful for comparing elements, tasks, and the like, in parallel, succinctly.

Beware of long and convoluted sentences.   Also long paragraphs.  User wants to read fast and get the major points with least effort.

Use formatting to help the reader find most important info, and to emphasize your most important points.  Don’t bury your most important points in the text.  Consider highlighting evaluations, recommendations, key observations – e.g., bold, or call-outs (such as boxed text).  Such as, “Both users experienced difficulty with xxx.”

Reporting user time on tasks: this can be useful info (especially when comparing designs that differ considerably) but be careful about comparing time when you’re asking them to think-aloud, or asking questions. You are likely slowing them down.  And even if you aren’t, your client may think you are and not believe your reported findings.

Wolfinger, N.H., 2002. On Writing Fieldnotes: Collection Strategies and Background Expectancies. Qualitative Research 85-95.