maximizing your moderating_game

Maximizing Your Moderating Game!Susan Mercer!Senior Experience Researcher!smercer@madpow.com!@susanamercer!

What is Usability Testing?!

•  It is NOT:!•  Market Research!•  The same as a Focus Group!•  About user preferences!•  “Fluffy”!

•  It is:!•  A systematic way to observe user behavior!•  A method to discover usability issues!•  A ongoing field of scientific inquiry!

!2!

Topics!

•  What is think-aloud?!•  Your roles as a moderator!•  5 rules of great moderating!•  Keeping sharp!!

3!

What is think-aloud?!

Think-aloud Protocol!

•  Most commonly-used usability testing method!•  1 participant / 1 moderator!•  Participant does tasks and thinks aloud!•  Origins in Cognitive Psychology!

Original protocol:!•  To understand short-term memory!•  Moderator invisible observer!•  Only comments: “Keep talking”, etc.!•  Don’t interrupt short-term memory!

Source: Ericsson & Simon, 1980!5!

Think-aloud Protocol - Problems!

•  Many practitioners feel the strict no-comments moderation is not ideal:!•  Makes participants feel uncomfortable!•  Feels “unnatural” not to respond!•  Participants may get stuck and not be able to

continue!

6!

Think-aloud Protocols!

Short term memory!

…to understand this !

Study this…!

Original Think-aloud!

System usability

…to understand this !

Study this…!

Usability Testing!

7!

Modified Think-aloud Protocol!

•  Based on Conversational Theory!•  Set up a conversation!

•  Participant is main speaker!•  Use short, neutral response phrases (continuers) !

•  “Mhmm”, “Uh-huh”, “And now…?” !•  Utter phrases like questions !

•  Ask for clarifications by repeating a phrase!•  Participant: “That was odd…”!•  Moderator: “Odd?”!

•  Provide hints if necessary !•  Neutrality is key!

Source: Boren & Ramey, 2000!8!

The Reality of Think-aloud!

•  Many usability professionals get sloppy:!•  Ask leading questions!

•  Participant: “I would share this article with friends.”!•  Moderator: “By printing it?”!

•  Ask closed-ended questions!•  Moderator: “Would you print this?”!

•  Focus on known problems for “ammunition”!•  Use biasing response phrases “Good job”, “Yes,

that’s a problem.”!!

Source: Norgaard & Hornbaek, 2006!9!

The Reality of Think-aloud!

•  Current practice goes beyond neutrality and includes probing questions!•  Some evidence that probing may influence what

usability problems are discovered!•  Evidence is only suggestive; more research is

needed!

Source: Krahmer & Ummelen, 2004!10!

What does this mean?!

•  Current moderating practices are not consistent!

•  Moderating is often taught through mentorship; so bad habits can easily be perpetuated!

•  Moderators should be willing to examine their practices and adapt if needed!

•  Therefore, we need to pay attention to our moderation and remain neutral. !!

11!

Your roles as a moderator!

Your Roles (Yes, plural)!

•  Gracious Host!•  Guide!•  Neutral Observer!•  Curious Learner!•  Technical Support!!

13!

Structure of a Usability Session!

Intro! Pre-Test Questions!

Tasks! Post-Test Questions!

Wrap-Up!

Gracious Host!Guide!Curious Learner!Neutral Observer!Technical Support!

14!

Structure of a Usability Session!

Intro! Pre-Test Questions!

Tasks! Post-Test Questions!

Wrap-Up!

Gracious Host!Guide!Curious Learner!Neutral Observer!Technical Support! ?! ?! ?! ?! ?!

Challenge: Navigate the Transitions Smoothly!

15!

5 Rules of Great Moderating!

5 Rules of Great Moderating!

1.  Know your goals!2.  Stay neutral !3.  Be responsible!4.  Be confident!5.  Be genuine!!

17!

5 Rules of Great Moderating!

1.   Know your goals!2.  Stay neutral !3.  Be responsible!4.  Be confident!5.  Be genuine!!

18!

1. Know Your Goals!

•  What are you trying to achieve?!•  Formative !•  Summative !

•  What topics are highest priority?!!These will influence!

•  Where you focus!•  What you skim when time runs short!•  How you fine-tune your moderating style!

19!

5 Rules of Great Moderating!

1.  Know your goals!2.   Stay neutral !3.  Be responsible!4.  Be confident!5.  Be genuine!!

20!

2. Stay Neutral!

•  Be Quiet!!•  Don’t insinuate they gave a wrong answer!

✘ “Why did you do that?”!✘ “What made you think to click that link?”!

•  Don’t put ideas into their heads!✘ “Does entering your SSN make you feel uncomfortable?”!ü “What do you think about entering your SSN?”!

•  Use their words!•  If they call the dashboard a “chart area”, then call it a

“chart area”!•  Keep your tone naturally curious, not derisive!

•  “What did you expect here?”!

21!

2. Stay Neutral!

•  Avoid closed-ended (Yes/No) questions !

•  Don’t start with a verb!✘ Did you see that?!✘ Was that difficult?!✘ Is this frustrating?!

22!

2. Stay Neutral!

•  Use Open-ended questions!

•  Start with What, When, Where, Why, How!ü What would you change to make this easier?!ü When in this process would you be ready to buy?!ü Where did you expect to find that information?!ü Why did you say this was easy?!ü How does that error message make you feel?!

23!

2. Stay Neutral!

•  Use neutral, “continuing” response phrases!

•  Acknowledge that they gave feedback!•  Don’t imply whether it is good or bad!

✘ “Oh”, “Hmm”, “Interesting” - Assessing!✘ “OK”, “Yeah”, “That’s good” - Agreeing!ü “Uh huh”, “Mhmm”, “Tell me more” - Continuing!

Source: Boren & Ramey, 2000 24!

5 Rules of Great Moderating!

1.  Know your goals!2.  Stay neutral !3.   Be responsible!4.  Be confident!5.  Be genuine!!

25!

3. Be Responsible!

You have responsibilities to:!!•  Participant!•  Company!•  Future Users !•  Study Integrity!

26!

3. Be Responsible!

•  Informed Consent:!1.  Voluntary!2.  Comprehension!3.  Disclosure!

•  The purpose of the study!•  Any reasonably foreseeable risks!•  Potential benefits !•  Notification of recording and observers!•  Confidentiality protections !•  Ability to withdraw at any time without penalty!

27!

3. Be Responsible!

Situations happen during testing.!You may need to juggle your responsibilities.!!What are your priorities?!!•  Study Integrity!•  Company!•  Future Users!•  Participant!

28!

What to do?!

If…!•  Participant doesn’t want to be recorded!•  You don’t have time to ask every question!•  The participant rants about the website and

calls it “crap”!

29!

5 Rules of Great Moderating!

1.  Know your goals!2.  Stay neutral !3.  Be responsible!4.   Be confident!5.  Be genuine!!

30!

4. Be Confident!

•  Project confidence!•  If you make a mistake, keep going!•  Don’t over-apologize!•  Pause if you need to collect your thoughts!•  “Fake it until you make it”!

31!

5 Rules of Great Moderating!

1.  Know your goals!2.  Stay neutral !3.  Be responsible!4.  Be confident!5.   Be genuine!!

32!

5. Be Genuine!

•  Don’t play the notes on the page…!

…play the music!!•  Don’t just read the

questions from the page…!…have a conversation!

33!

5. Be Genuine!

•  LISTEN!•  Listen and respond!

•  Stay in the moment!•  Don’t worry about past !•  Or the future!•  Create a conversation!

!

34!

5 Rules of Great Moderating!

1.  Know your goals!2.  Stay neutral !3.  Be responsible!4.  Be confident!5.  Be genuine!!

35!

Keeping sharp!

Keep Improving!

1.  Identify your improvement areas!•  Watch your videos!•  Have colleagues give you feedback!

37!

Keep Improving!

2.  Learn from others!•  Watch others moderate!•  Be a participant!•  Listen to talk radio interviews!

38!

Keep Improving!

3.  Practice skills in everyday life!•  People watch (observe) in public!•  Be quiet and listen in everyday conversations!

!

39!

The most important thing is…!

•  No one is perfect the first time!•  Do your best and continually improve!•  You’ll be awesome in no time!!

PRACTICE!

40!

Questions?

41!

Thank You!

(Slides will be posted to Slideshare..follow me on twitter for link)

Susan Mercer!Senior Experience Researcher!smercer@madpow.com!@susanamercer!

42!

References / Photo Credits!Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE

Transactions on Professional Communication, 43 (3), 261-278.!Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.!Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.!Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3),

215-251.!Krahmer, E. and Ummelen, N. (2004) Thinking about thinking aloud: A comparison of two

verbal protocols for usability testing. IEEE Transactions on Professional Communication, 47 (2), 105-117.!

Norgaard, M. and Hornbaek, K. (2006) What do usability evaluators do in practice? An exploratory study of think-aloud testing. DIS 2006, 209-218.!

!!!Slide 4: Samantha Louras Photography !Slide 5: flickr: torgeaux!Slide 14: flickr: mkorcusa, flickr: Calsidyrose, flickr: marioanima, flickr: breity, flickr: Qole Pejorian!Slides 27, 29 and 30: flickr: Raphael Quinet, flickr: acme, flickr: betsyweber, flickr: KirkOls!Slide 32: flickr: bberburb!Slide 34: flickr: Tulane Public Relations!Slide 35: flickr: apdk, flickr: marktristan!Slide 38: flickr: Raphael Quinet!Slide 39: flickr: l-i-n-k!Slide 40: flickr: loppear!!!

43!