techniques and tools for field-based early-stage study and iteration of ubicomp applications
DESCRIPTION
Techniques and tools for field-based early-stage study and iteration of Ubicomp applications. Scott Carter EECS Department University of California, Berkeley [email protected]. Committee Jennifer Mankoff (chair), Anind Dey, John Canny, Peter Lyman. 1. - PowerPoint PPT PresentationTRANSCRIPT
Techniques and tools for field-based early-stage studyand iteration of Ubicomp applications
1
Scott CarterEECS DepartmentUniversity of California, [email protected]
CommitteeJennifer Mankoff (chair), Anind Dey, John Canny, Peter Lyman
2
what are ubicomp apps? examples…social proximity jabberwocky (paulos ’04)
location urban tapestry (lane ‘05)
activity
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
resource queries transit services tour guide (abowd ’97)
3
capture and access field data health (alivetech ’05) reminder apps
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
gaming cysmn (blast theory ’01)
location,proximity-based
what are ubicomp apps? examples…
4
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp evaluation is hard (Carter ’05)
5
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp evaluation is hard
rapid prototyping
6
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp evaluation is hard
rapid prototyping
scale
7
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp evaluation is hard
rapid prototyping
scale
unobtrusiveness
8
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp evaluation is hard
rapid prototyping
scale
unobtrusiveness
data sparsity
9
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
thesis
Leverage pre-existing infrastructures and the increasing power of mobile devices to address the challenges of early stage field evaluations of ubicomp applications.
10
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
why early stage and field?
early stage
field
>> potential to iterate>> could answer some research questions
>> realism especially important in ubicomp >> Sharp & Rehman ‘05: “Real-world deployment is the only way”>> Kjeldskov: might be able to recreate field in a lab
11
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
approach: WoZ
researcher designs app behavior in momento and provides instructions for wizards
wizards use momento to simulate app behavior
12
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
approach: report methods
researcher designs experiment, configures momento or reporter
researcher runs experiment, monitoring tool
13
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
limitations: when challenges do not applydata easy to get (no sparsity)
infrastructure already available (no scale issues)
target users early, early adopters (no unobtrusiveness)
tools off-the-shelf (no prototyping)
14
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
most relevant related work
rapid prototyping
Topiary (Li ’04)
scale
unobtrusiveness
data sparsity
rapid prototypingContextESM (Intille ’03)
scale
unobtrusiveness
data sparsity
WoZ Report methods
SUEDE (Klemmer ‘00)
DART (Dow ‘05)
Doorman (Makela ‘01)
Crossweaver (Sinha ‘03)
Active Capture (Chang ‘05)
other work
iESP (Intel ’03)
Mobile probes (Hulkko, ‘04)
other work
15
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
outline
ubicomp challenges case study
report methods
momento
validation plans
timeline
summary
16
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp challenges case study: dr. bluesocial mobile game
get points by collecting cures, healing the sick
17
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp challenges case study: dr. blue
18
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp challenges case study: dr. blue
19
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: dr. blue: rapid prototypingmajor difficulty: building a mobile application
communicationsensordisplay
also difficult: game engine
20
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: dr. blue: scale
everyone needs a device and recording method
21
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: dr. blue: unobtrusivenessnew device and interface
interruptions: evaluation and application behavior
22
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: dr. blue: sparsity
encounters may be limited
23
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
outline
ubicomp challenges case study
report methods
momento
validation plans
timeline
summary
24
momento
WoZ simulation using SMS/MMS infrastructure
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
25
momento
WoZ simulation using SMS/MMS infrastructure
… and J2ME on the mobile device
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
26
momento
input and output
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
27
momento: building dr. blue
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
28
momento: building dr. blue
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
29
momento: building dr. blue
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
30
momento: building dr. blue
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
31
momento: building dr. blue
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
32
momento: limitations
SMS/MMS, wizard latency
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
24-hour wizards?
synchronous and streaming data
gui creation (damask – Lin ’02)
momento: summary
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
scale
unobtrusivenessdata sparsity
rapid prototypingany device supporting sms/mms
esm feedback on critical events
easy to change application behavior
integrate context
more control
participants
rules to take advantage of external services
wizards
familiar devices, but interruptions
34
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
outline
ubicomp challenges case study
report methods
momento
validation plans
timeline
summary
35
report methods
diary study
experience-sampling method (esm)
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
36
diary study issues
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
unobtrusiveness
data sparsity
scale
neglect due to burden of study
generic responses
no feedback
data re-entry, transcribing
37
diary study technique (Carter & Mankoff ’05)
capture
annotate
upload
feedback
elicitation
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
38
diary study technique: reporter
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
39
reporter
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
scale
unobtrusivenessdata sparsity
rapid prototypingany digital capture device
rapid feedback cycle
-
familiar devices
no interruptions
no transcribing
responses tied to captured artifacts feedback not in situ
40
esm: momento
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
scheduled or context events
automatic question formatting
41
esm: momento
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
scale
unobtrusivenessdata sparsity
many devices
responses in situ
rapid prototypingknowledge feedback to application behavior
redirect questions
participants
integrate context
familiar devices, but interruptions
42
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
outline
ubicomp challenges case study
report methods
momento
validation plans
timeline
summary
43
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
validation: diary study pipeline
testbed: diary study at uc berkeley
interviews with researchers
compare to traditional diary studies (literature)
unobtrusiveness
data sparsity
scale
success?
fewer instances of neglect due to burden of study
more specific recall, fewer generic responses
reduce time to data analysis (no data re-entry)
44
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
validation: momento esm study
sensor data for awareness app
location, activity data
interviews with wizards, participants
unobtrusiveness
data sparsity
scale
success?
minimal neglect due to burden of study
useful feedback on critical events
wizards able to manage momento queries and complete other work
45
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
validation: momento scale study
implement dr. blue
wide deployment (>~30)
feedback on critical events, retainment
unobtrusiveness
data sparsity
scale
success?
minimal neglect due to burden of study
useful feedback on critical events
wizards able to manage momento queries and complete other work
rapid prototypingiteration time shorter than similar social mobile apps
46
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
contributions
a tool (momento) that improves evaluations by simplifying prototyping, making them easier to scale, reducing obtrusiveness, and facilitating data collection
a diary study technique that is easier to scale, less obtrusive, and yields better quality data than previous diary study techniques
identification of major challenges in ubicompevaluation: rapid prototyping, scale, unobtrusiveness, and data sparsity
validation of momento and diary study technique
47
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
timeline
48
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
timeline
49
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
timeline: diary study
50
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
timeline: momento
51
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
summary
Leverage pre-existing infrastructures and the increasing power of mobile devices to address the challenges of field evaluations of Ubicomp applications
Wizard-of-Oz and report methods for early stage, field experiments
Ubicomp evaluation challenges: rapid prototyping, scale, unobtrusiveness, data sparsity
52
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
acknowledgements
Scott Lederer, Christopher Beckmann, Jonathan Hey, Jeffrey Heer, Tara Matthews, Scott Klemmer, James Lin, Tye Rattenbury, Alan Newberger, David Nguyen, Judd Antin, Joe Hall, Nathan Good, Siobhan Hadley
Prof. Robert Kraut, Prof. Susan Fussell, Prof. Jason Hong, Laura Dabbish, Joe Tullio, Jeffrey Nichols, James Fogarty
Techniques and tools for field-based early-stage studyand iteration of Ubicomp applications
53
Scott CarterEECS DepartmentUniversity of California, [email protected]
CommitteeJennifer Mankoff (chair), Anind Dey, John Canny, Peter Lyman
54
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
other slides
55
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
worldwide txt stats
number of computer users: about 1 bil by 2006 (IDC)email users: 700 mil by 2006 (Radicati Group)
about 2 bil mobile subscribers by 2006 (In-Stat/MDR)70% use sms (text.it) (number much lower in us)
56
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
scope
formative and early stages
field methods
distributed, asynchronous activities
>> easier to iterate, could answer some research questions
>> realism especially important in ubicomp >> Sharp & Rehman ‘05: “Real-world deployment is the only way…”>> Kjeldskov: might be able to recreate field in a lab
>> distributed harder, concentrate on asynchronous
broad user base>> minimize programming
57
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
rapid prototyping
scale
unobtrusiveness
data sparsity
ambiguity
>> simplify interface iteration, reduce coding, reduce burden of reinstallations
>> minimizing infrastructure deployment and time, encourage uptake
>> use familiar devices and lower distraction costs
>> capture and encourage feedback from critical events
>> allow people to control error handling
thesis: addressing challenges
58
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
approach: wizard of oz
Kelley (1984)
59
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
approach: wizard of oz
Kelley (1984)
60
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
approach: wizard of oz
61
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
approach: report methods
diary
experience sampling
62
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
approach: report methods
diary
experience sampling>> researchers send questions
63
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
approach: report methods
diary
experience sampling
>> participants report on events
>> researchers send questions
64
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
approach: report methods
65
diary studies
feedback or elicitation
participants report on events
form completion (feedback)participant controlled capture (elicitation)
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
66
advances in recording devices
smaller
part of everyday infrastructure
easier to transfer files
annotation
new, more frequent uses for media
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
67
findings
+ photos best for who and where recognition
audio elicitation
tangible objects
raw location information
- poor recognition
+/- clandestine captures
- poor recall
+ attitudes and beliefs
- poor sequencing and recall
+ peripheral information often crucial
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
68
findings
tangible objects and attitudes and beliefs
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
69
findingsperipheral information often crucial
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
70
momento
desktop
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
71
applications
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
72
applications
interruptibility updates (social, activity)
Momento: How busy are you right now? 1 not busy at all, 5 is really busy.
…
P: Who is most available?
Momento: Joe
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
73
applications
txt adventure (social, gaming)
Momento: You're in a forest. There is a trail to the N and a cabin to the W. Which way would you like to go? N/W
P: N
Momento: You open the cabin but there was a troll and he squashed you with his fist. Game over. Bad luck
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
74
applications
near by (location, resource)
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
75
applications
idea tagger (capture and access)
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
76
diary study or esm?
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
77
diary study or esm?
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
>> auto-driven: participant-sensed events>> creativity, attitudes, beliefs, breadth
diary study
78
diary study or esm?
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
>> auto-driven: participant-sensed events>> creativity, attitudes, beliefs, breadth
diary study
>> researcher-driven: more control>> time use, depth
esm
79
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
validation: interface support
augment current application
add functional feature
success?
demonstrate it works
no deployment
novel approach
80
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
validation: interface support
81
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
validation: interface support
82
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
validation: interface support
83
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
other slides
When Participants Do the Capturing
84
Carter & Mankoff
diary studies and related methods
beeper studies
When Participants Do the Capturing
85
Carter & Mankoff
diary studies and related methods
cultural probes
86
momento: building dr. blue
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
87
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp evaluation is hard
rapid prototyping
scale
unobtrusiveness
data sparsity
ambiguity
88
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: dr. blue: ambiguity
89
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp evaluation challenges
rapid prototyping
scale
unobtrusiveness
data sparsity
ambiguity
Metrics: adoption, awareness, distraction, invisibility, (Scholtz and Consolvo)
90
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubicomp challenges case study: hebb
91
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: rapid prototyping
92
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: rapid prototyping>> simplify interface iteration, reduce coding, reduce burden of reinstallations
93
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: scale
94
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: scale>> minimize infrastructure deployment and time, encourage uptake
95
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: unobtrusiveness
96
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: unobtrusiveness>> use familiar devices and lower distraction costs
97
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: sparsity
98
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: sparsity
>> capture and encourage feedback from critical events
99
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: ambiguity
100
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
case study: hebb: ambiguity
>> allow people to control error handling
101
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubiquitous computing (weiser)
102
CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications
ubiquitous computing
off-the-desktop interfaces