FIELD STUDIES

User studies

Ubicomp: people use technology Must conduct user studies Also:

Focus groups Ethnographic studies Heuristic evaulations Etc.

User studies

Laboratory studies: Controlled environment

Field (in-situ) studies Real world

Field studies

Appropriate for ubicomp: Abundant data Observe unexpected challenges Understand impact on lives

Trade-off: Loss of control Significant time and effort

Three common types

Current behavior Proof of concept Experience with prototype

How to think about user studies?

Formulate hypotheses

Research steps

1. State problem(s)2. State goal(s)3. Propose hypotheses4. Propose steps to test hypotheses5. Explain how problem(s), goal(s) and

hypotheses fit into existing knowledge6. Produce results of testing hypotheses7. Explain results8. Evaluate research9. State new problems

What is a hypothesis?

Proposing an explanation Theory or hypothesis? “This is just a theory.” Some theories we live by (“just” not

justified): Newton’s theory of motion Einstein’s theory of relativity Evolutionary theory

Hypothesis

Must be tentative Must predict

Hypothesis

Some criteria of scientificity Self-consistent Grounded (fits bulk of relevant knowledge) Accounts for empirical evidence Empirically testable by objective

procedures of science General in some respect and to some

extent

On proposing hypotheses

Anomalous phenomena: Strange and unfamiliar (Bermuda triangle) Familiar yet not fully understood (cognitive

load) Is there already an explanation?

Types of hypotheses

Incremental Fundamental shift:

Ptolemy (c. 90 – c. 168): geocentric cosmology

Copernicus (1473 – 1543): heliocentric cosmology

And then came…

Kepler (1571 – 1630): elliptical orbits

Fundamental shift example

Ulcer: Stress? Spicy food? Bacteria.

Types of proposed explanations Causes Correlation Causal mechanisms Underlying processes Laws Functions

Proposing causal explanations Studies show that using a cell phone

while driving increases the probability of getting into an accident. Why is that so? Pick up ringing phone Dial number See but don’t perceive

Effects not always there

Cell phone + driving: Usually no accident Only one of the factors

Remote and proximate causes Cell phone + driving:

Attention shift → missed signal → accident Remote cause → proximate cause → effect

Correlation

A and B are correlated if: A → B B → A C → A and C → B A combination of (some of) the above Coincidence

Correlation vs. causal relation: Correlation doesn’t imply causal relation Cannot determine cause direction (A → B or B

→ A)

Correlation

Positive, negative None found ≠ none exists Causal link → correlation:

May provide initial evidence for causal link Less explanatory value than facts about

causal links

Causal mechanisms

Mechanisms connecting remote causes and their effects.

E.g.: Damaged artery in heart → clotting Clotting → blocked artery Blocked artery → heart attack Aspirin inhibits clotting → lower risk of heart

attack

Underlying processes

Photoelectric effect

Photoelectric effect

Einstein: 1921 Nobel Prize in Physics

Laws

General regularities in nature Universal:

F = ma Non-universal:

Statistical laws

Functions

What is the purpose of the phenomenon?

FOR SALE A prime lot of serfs or SLAVES GYPSY (TZIGANY) Through an auction at noon at the St. Elias Monastery on 8 May 1852 consisting of 18 Men 10 Boys, 7 Women & 3 Girls in fine condition

Functions

William Harvey (1578 – 1657): Heart pumps blood through

circulatory system No modern instruments! Experiments with a number

of animals: Various fish, Snail, Pigeon, etc.

Multiple methods together

Function → → causal mechanism → → underlying processes

National Ignition Facility(Dennis O’Brien @ UNH):Ignition with lasers → → Laser, target chamber → → Physics of nuclear fusion

Multiple methods together

Law → underlying processes Isaac Newton (1643 – 1727),

second law of motion:F = ma → Graviton?

Ockham’s razor

Crop circles: pranksters or aliens?

Ockham’s razor

William of Ockham (c. 1288 – c. 1348)

http://en.wikipedia.org/wiki/File:William_of_Ockham.png

Do I have a hypothesis?

Yes. Do you realize you do?

How to think about user studies?

Formulate hypotheses

Three common types

Current behavior Proof of concept Experience with prototype

Research steps

1. State problem(s)2. State goal(s)3. Propose hypotheses4. Propose steps to test hypotheses5. Explain how problem(s), goal(s) and

hypotheses fit into existing knowledge6. Produce results of testing hypotheses7. Explain results8. Evaluate research9. State new problems

Current behavior

Insights and inspiration: State problem(s), goal(s) Propose hypotheses

Relatively long

Current behavior – example 1

AJ Brush and Kori Inkpen, “Yours, mine and ours?...” (pdf) (2005 movie inspiring title)

Home technology: users share, etc.

Current behavior – example 2 Schwetak Patel et al. “Farther Than You

May Think…” (pdf) Hypothesis: Mobile phone a proxy to

user location.

Three common types

Current behavior Proof of concept Experience with prototype

Research steps

1. State problem(s)2. State goal(s)3. Propose hypotheses4. Propose steps to test hypotheses5. Explain how problem(s), goal(s) and

hypotheses fit into existing knowledge6. Produce results of testing hypotheses7. Explain results8. Evaluate research9. State new problems

Proof of concept

Technological advance: Produce results: prototype Explain results: prototype

Relatively short

Proof of concept – example 1 J. Sherwani et al., “Speech vs. Touch-

tone: Telephone Interfaces for Information Access by Low Literate Users” (pdf) (video)

Hypothesis: Speech better telephony interface than touch-tone for low literate users.

Proof of concept – example 2

John Krumm and Eric Horvitz, “Predestination:…” (pdf)

Hypothesis: Destinations from partial trajectories.

Train/test algorithm on GPS tracks from 169 people

Used pre-existing data: Krumm and Horvitz, “The Microsoft Multiperson

Location Survey” Collecting original data a significant

contribution Leverage!

Three common types

Current behavior Proof of concept Experience with prototype

Research steps

1. State problem(s)2. State goal(s)3. Propose hypotheses4. Propose steps to test hypotheses5. Explain how problem(s), goal(s) and

hypotheses fit into existing knowledge6. Produce results of testing hypotheses7. Explain results8. Evaluate research9. State new problems

Experience with prototype

Users’ interaction with technology: Produce results: prototype Explain results: prototype

Relatively long

Prototype an example!

Others don’t care about: Raw usage information Usability problems Intricate implementation details Etc.

Generalize! Scientific and good technical work

Experience – example 1

C. Neustaedter, et al., “A Digital Family Calendar in the Home:…” (pdf) (video)

Hypothesis: At-a-glance awareness, remote access are significant benefits.

4 households, 4 weeks each (Best Student Paper, Graphics Interface

2007)

Experience – example 2

Rafael Ballagas et al., “Gaming Tourism:…” (pdf) (video)

Hypothesis: Learning through a game. 18 participants: 2 alone + 8 pairs (8 x 2

= 16)

Study design

Who is the consumer? Manager(s)

Industry, academic lab Professor(s)

E.g. thesis committee Researchers

E.g. advisor’s collaborators Reviewers

For paper, proposal, thesis Funding agency

Report on progress, proposal for funding Public

Friends, family, alumni, potential students, donors, potential employers

Study design

How can I explain this to a layperson? What is key? What can be omitted?

How will I write this up? Paper Thesis Report Blog post

Start writing paper/thesis/report/blog post at the beginning of the study.

Study design

Test hypothesis/hypotheses

Testing hypotheses via user studies

User studies: Laboratory studies

Good: Control, easier to evaluate results Bad: Constraints

Field studies Good: Fewer constraints Bad: Less control, more difficult to evaluate

results

Criteria

Falsifiability: Prediction fails = explanation isn’t correct Account for other factors!

Note: Criterion - singular Criteria - plural

Criteria

Verifiability: Prediction successful = explanation is

correct Account for other factors!

The meat of it…

Battleship Potemkin, 1925 film

Rotten meat scene

Why larvae in meat?

Francesco Redi (1626-1697)

Generation of insects, 1668

Causal explanation: fly droppings

Redi’s research

Hypothesis: Worms derived from fly droppings

Testing hypothesis: Two sets of flasks with meat: sealed and

open Prediction: worms only in open flask

Falsifiability criterion

Can anything cause a failed prediction even if explanation is correct?

Did the apparatus operate properly? Tight seal? Meat not initially spoiled? Other?

Verifiability criterion

Can anything result in successful prediction even if explanation is wrong?

What if “active principle” in the air is responsible for spontaneous generation?

Modify experiment: Replace seal with veil:

Flies cannot reach meat Air in contact with meat

Modification helps meet verifiability criterion

Verifiability criterion

Experimental vs. control group: Only difference in level of one independent

variable Redi’s experiment:

Control: Open flasks Experimental: Veil-covered flasks

Control: laboratory experiment Meat in veil-covered flasks? Creating control/experimental groups

often impossible without careful design/control

Study design

Test hypothesis/hypotheses Formulate in terms of:

Independent variables (multiple conditions) Dependent variables

Design: Within-subjects Between-subjects Mixed design

Within-subjects design: example

Police radio UI: hardware Speech

Blog post, video

Within-subjects design: example

Results in graphical form:

Within-subjects design: example

Results in graphical form:

Example: between-subjects design

Classical example: testing a drug

Mixed design: example 1

SUI characteristics study Secondary task: speech control of radio 2 x 2 x 2 design:

SR accuracy: high/low PTT button: yes/no – ambient recognition Dialog repair strategy: mis-/non-

understanding

Mixed design: example 2

Motivation: PTT vs. driving performance Secondary task: speech control of radio 2 x 3 x 3 design:

SR accuracy: high/low PTT activation:

push-hold-release/push-release/no push PTT button: ambient/fixed/glove

Push-hold-release Push-release No-push

Ambient Fixed Glove Ambient Fixed Glove Ambient Fixed Glove

High

Low

Control condition

Baseline: e.g. no technology vs. later introduced technology

Considerations

What will subjects do? Normal behavior – may take long Scenarios

Augment existing or brand new? Augment: taking advantage of familiarity New: more control (fewer inherited

constraints) Simulate or implement?

E.g. WoZ

Data to collect

Qualitative Insight into what participants did. How do participants compare? Did they do

what they thought they did? Use quantitative data.

Quantitative How did people behave? But why? Use qualitative data.

Data to collect

At least three types of data: Demographic Usage Reactions

Data to collect

Run pilot experiments!

Collecting data

Logging Surveys Experience sampling Diaries Interviews Unstructured observation – ethnography

Logging

Plan ahead, not after the fact! Testing hypotheses

Don’t leave important data out Don’t save data you don’t need

Leverage logging: Everything OK?

E.g. Mike Farrar’s MS research: files appearing on server indicates apps OK

Explicit communication with server: “I’m OK!”

Surveys

Open-ended Multiple-choice Likert-scale

Surveys

Questions should allow positive and negative feedback.

Text clear to others? Check! One question at a time!

“Fun and easy to use?” Length?

Don’t bore subjects to death. Standard questions (e.g. QUIS)?

Previously used questions?

Example: Mike Farrar’s study Hypotheses:

Initialize grammar (video): From previous tags From tags by users with similar interests

Voice commands convenient way to tag photos (video)

Keyboard users will use voice less Low task completion: give up on voice

Experience sampling (ESM)

Short questionnaire Timing:

Random Scheduled Event-based

Experience sampling (ESM)

How often? How many? Relate to quantitative data?

Diaries

Similar to ESM

Interviews

Semi-structured: List of specific questions + follow-up

questions Bring data

E.g. Nancy A. Van House: “Flickr and Public Image Sharing:…”

Interviews + photo elicitation

Interviews

Neutral questions Negative feedback is OK (this is hard):

Don’t argue!

Participants

Follow IRB rules

Participants

Who to recruit? Representative of intended users Not your friends, family, colleagues – bias! May need different types

Recruit sufficient numbers of each type

Participant profile

Age E.g. age significant for driving

Gender Technology use and experience Other

Eye tracker studies: no glasses

Number of participants

Between-subjects usually requires more than within-subjects

Proof-of-concept: typically fewer and many types

Longer study: may be able to use fewer Time commitment per participant is

significant! Recruit (Craigslist), organize, train, run,

transfer data, process data Participants will drop out – recruit extra

Counterbalancing may not work out

Compensation

Don’t try to save on this! Driving simulator lab study cost example

1 graduate student year at UNH ≈ $50k Software maintenance fees per year ≈

$20k Trip to conference ≈ $2k PC or laptop ≈ $2k $20 x 24 participants ≈ $0.5k (less than

1%)

Compensation

Must not affect data E.g. in image tagging study if we paid per

picture: More data Unrealistic as interactions are for money not for

value of prototype

Compensation

Leverage if you can: Latest driving simulator lab study in

collaboration with Microsoft Research: Use Microsoft software as compensation

Data analysis

Test hypotheses Use multiple data types Tell a story

Data analysis

Statistics: Descriptive Inferential

Descriptive statistics

Level of measurement: Nominal Ordinal Interval

Descriptive statistics

Level of measurement: Nominal Ordinal Interval

Level of measurement

Nominal: Unordered categories E.g. yes/no Valid to report :

Frequency

Level of measurement

Ordinal: Rank order preference without numeric

difference E.g. responses on Likert scale

Five of the eight participants strongly agreed or agreed with the following statement: “I prefer to have a GPS screen for navigation.”

Valid to report : Frequency Median Some people report means but what is the mean

of “strongly agree” and “strongly disagree”?

Level of measurement

Interval: Numerical differences significant E.g. age, number of times an action

occurred, etc. Valid to report:

Sum Mean Median Standard deviation (outliers?)

Outliers in interval data

Inferential statistics

Significance tests t-test ANOVA Many others

Which to use: depends on data

Significance test: example 1

To assess the effect of different navigation aids on visual attention, we performed a one-way ANOVA using PTD as the dependent variable. As expected, the time spent looking at the outside world was significantly higher when using spoken directions as compared to the standard PND directions, p<.01. Specifically, for spoken directions only, the average PDT was 96.9%, while it was 90.4% for the standard PND.

Significance test: example 2

-5

0

5

10

15

20

60-80 80-100 100-120 120-140 140-160

PDT

on st

anda

rd P

ND

[%]

distance from previous intersection [m]

… PDT on the PND screen changes with the distance from the previous intersection… significant main effect, p<.01…

Significance test: example 3

Randomization test Kun et al. (pdf) Idea from Veit et al. (pdf)

Significance test: example 3

0

5

10

15

20

25

30

35

0 1 2 3 4 5 6 7 8

Rstw

[deg

rees

^2 ]

lag [seconds]

standard

p = 0.05

spoken only