Inventing Virtual Teachers and Therapists

Promises, Systems & Challenges

Ron Cole & the CSLR Virtual Human Research Group

September 2, 2005SPACE meeting; Leuven Belgium

Outline

• TALK ONE: Vision, Rationale & Systems– The promise of virtual humans– Scientific rationale– System overviews and demos

• TALK TWO: Inventing Virtual Humans– Technologies implemented to date– Technical challenges and missing science– Conclusions and recommendations

What is a Virtual Human?

A Believable Computer Character

• with personality and attitude

• that engages users in natural face-to-face conversation

• to produce great learning experiences

Marge

The Promise of Virtual Humans

• Effective teachers, therapists, assistants– A virtual human is patient and tireless: learning can be more

engaging, motivating, personal and effective

• Accessible to nearly everyone, anywhere, anytime– Via multilingual natural dialog interaction on networked

computers

• Awesome benefits to individuals and society– Humans are expensive and often inaccessible, Virtual Humans

are inexpensive

• Research tools for acquiring missing knowledge;– they can be programmed to behave in predictable ways;

whereas people are often guided by unconscious behaviors

Theoretical & Empirical FoundationsThe media equation: media = real life

Cliff Nass & Byron Reeves

“We have found that individuals’ interaction with computers are fundamentally social and natural, just like interactions in real life.”

“All these rules come from the world of interpersonal interaction, and from studies of how people interact with the real world. But all of them apply equally well to media.”

“The more a media technology is consistent with social and physical rules, the more enjoyable the technology will be to use. including feelings of accomplishment, competence, and empowerment.”

The Persona Effect(Lester et al., 1997)

• Hypothesis: Interfaces that use voice and/or face to foster social agency produce more satisfactory and effective experiences.

• Students form a social bond with virtual humans and are motivated to learn and succeed

• Better experiences reported in hundreds of experiments– Students give top ratings to: “Do you think Marni is a good

teacher?” “Does Marni act like a real human teacher?” and “How well does Marni help you learn to read?”

• Better outcomes supported by dozens of experiments– Several experiments show benefits of talking head compared to

voice alone

The power of one-on-one tutoring

• Benjamin Bloom (1982) posed the “two sigma challenge”: – Demonstrated a two sigma benefit of one-on-

one tutoring relative to classroom instruction

• Meta-analysis of 100s of tutoring studies confirms benefits of tutoring (Cohen, Kulik & Kulik, 2982)

Cognitive Theory of Multimedia LearningMayer (2001)

• Mayer examined how presentation of words and pictures affected learning in transfer tasks

• Best learning occurs when voice is used to explain phenomena displayed simultaneously in pictures or animation

• Atkinson demonstrated benefits of talking head relative to voice alone

Recipe for a Virtual Human System

– Develop a deep understanding of the task • Theory, research and practice

– Analyze and model the performance of human experts– Develop the system

• In collaboration with experts and users (participatory design)• Design and test, design and test, test, test, test

– Evaluate the system (formative evaluation / clinical trials)– Scale Up and Sustain the system

Virtual Teachers and Therapists

• Theoretical & Empirical Rationale• Demos

– Baldi teaches vocabulary to students with hearing loss

– Marni teaches children to read– Marni conducts speech therapy for individuals

with Parkinson disease and aphasia

Research funded by grants from NSF and NIH

A Virtual Teacher for Students with Hearing Loss

• 1998-2001: Baldi teaches vocabulary to students at Tucker Maxon School– Rapid acquisition of vocabulary– >50% retention several months later– Dramatic improvements in speech production– Featured on national TV and NSF Home page

Foundations to Literacy

A comprehensive, scientifically-based reading program designed to

• Teach children to read and comprehend text• Through conversational interaction with Marni, a

virtual teacher • That behaves like a sensitive and effective

reading teacher

Cognitive theory & scientifically- based reading research

Skilled reading is– Word recognition processes + comprehension processes– This is called the “Simple Model of Reading” (Gough et al.,

1996)

• Word recognition processes – Alphabet, Phonological awareness, Encoding, Decoding, Sight

words – Reading in context until fluent & automatic– Evidence-based pedagogy: SBRR (Rayner et al, 2001)

• Comprehension processes– Train fluent and expressive reading– Train comprehension through “thinking questions”

Main Components of FtL

• Foundational Skills Tutors– Teach underlying reading skills

• Interactive Books– Teach fluent reading & comprehension

• Managed Learning Environment– Enrolls students, tracks and displays progress,

manages individual study plans for each student

FtL Status

• Now in 50 Colorado classrooms– Formal assessment in 40 K-2 classrooms; 2

computers per classroom, 20 min per day per designated students

– About 1/3 ESL students (one school has ALL native Spanish speaking students)

– 10 Special Ed classrooms for remedial instruction of students with cognitive disabilities

• Learning gains in kindergarten and first grade classrooms; kids love the program

LSVT Lee Silverman Voice Treatment

““If only we could hear and understand her”If only we could hear and understand her”-- family of Lee Silverman-- family of Lee Silverman

Parkinson’s disease

1.5 Million individuals US aloneOver 6 million worldwide

89% have a speech or voice problem89% have a speech or voice problem(Logemann et al.,1978)(Logemann et al.,1978)

4% receive traditional speech therapy4% receive traditional speech therapy(Hartelius & Swenson, 1994; Oxtoby, 1982)(Hartelius & Swenson, 1994; Oxtoby, 1982)

19901990 Consensus: Speech treatment does not workConsensus: Speech treatment does not work(Sarno, 1968; Allan, 1970; Green,1980; Aronson, 1990; (Sarno, 1968; Allan, 1970; Green,1980; Aronson, 1990;

Weiner & Singer, 1989)Weiner & Singer, 1989)

Perceptual Characteristics of SpeechPerceptual Characteristics of Speech

Reduced loudnessReduced loudnessHoarse voice qualityHoarse voice quality

MonotoneMonotoneImprecise articulationImprecise articulation

Vocal tremorVocal tremor

Some patients report volume, hoarse voiceSome patients report volume, hoarse voiceor monotone as the or monotone as the firstfirst PD symptom PD symptom

(Aronson, 1990)(Aronson, 1990)

““If you don’t talk loud enough, If you don’t talk loud enough, people stop listening”people stop listening”

-Individual with Parkinson DiseaseBoston, May 1996

SOFTSOFT

LOUDLOUD

To a patient……major life impact

“My voice is alive again”

“I can talk to my grandchildren!”

“I feel like my old self”

“I am confident I can communicate!”

Ramig et al., 2001; J Neurol, Ramig et al., 2001; J Neurol, Neurosurgery, PsychiatryNeurosurgery, Psychiatry

60

65

70

75

-2 0 2 4 6 8 10 12 14 16 18 20 22 24

Months

SP

L R

ain

bo

w (

50 c

m)

LSVT R

Unexpected outcomes:System-wide spread of effect

Benefits to

Articulation Swallowing

Speech Rate Face

Speech Motor Stability PET

(Spielman, et al. 2002; El-Sharkawi, 2002; Spielman et al., 2003; Kleinow et al., 2001; Liotti et al., 2003)

To a patient……major life impact

“My voice is alive again”

“I can talk to my grandchildren!”

“I feel like my old self”

“I am confident I can communicate!”

+60

Pre-LSVTSMA

+60

Post-LSVT

L R

+34

z-score

-4 ±2.25 +4

+34

+4

DLPF9 R a Ins

+10+4

R Put

+4

+34

+34 +10

Phonation Task - PD N=5

LSVT® Applications

• Parkinson Plus (Countryman et al., 1994)

• Post Surgery, Fetal cell (Countryman, et al., 1993)

• Stroke (Fox et al, 2002; Will et al., 2002)

• Multiple Sclerosis (Sapir et al., 2001)

• Ataxia (Sapir et al., 2003)

• Cerebral palsy (Fox, 2002)

• Down Syndrome (Robinson et al., 2004)

• Aging (Ramig et al., 2001)

LSVTVT-Loud AH LSVTVT– Hi-Lo pitch

LSVTVT-Loud Phrases LSVTVT-Read Aloud

LSVT VT Status

• System being tested on three individuals with Parkinson disease

• Patients use system during 10 of 16 1-hour sessions using system

• Patients enjoy using the system

• STATUS: Clinical trials begin Sept 14 05

Virtual Therapists for Aphasia

The three systems described next were based on speech and language treatments developed for individuals with Aphasia

• C-COSTA (Developed by Leora Cherney, Chicago Rehabilitation Institute

• ORLA (Leora Cherney)

• TUF-T (Developed by Cynthia Thompson, Northwestern University

C COSTAComputerized Oral Scripts for Teach Aphasia

• Develop conversational scripts personally relevant to the individual with aphasia– A sequence of sentences that a person typically

speaks in routine communication situations • Ordering pizza over the phone• Making a doctor’s appointment

• Implement and evaluate the computerized intervention relative to human therapists

• STATUS: 5 subjects tested

Oral Reading for Language in AphasiaORLA

• ORLA is a speech-language treatment protocol for patients with aphasia (Cherney et al., 1986),

• A multi-modality stimulation approach that involves• listening to a sentence, tapping along with the rhythm of

the sentence, repeated practice saying the sentence in unison with the clinician and then independently.

• Studies indicate improvements in oral expression, auditory comprehension, and written expression (Cherney et al., 1986).

• STATUS: 5 subjects tested

Treatment of Underlying Forms (TUF) for Aphasia by Cynthia Thompson at Northwestern University

• Individuals with Broca’s aphasia have difficulty comprehending and producing sentences, particularly sentences with complex syntax.

• Training simple sentence types fails to generalize to untrained sentence types and contexts.

• Research with TUF demonstrated generalization occurs to sentences that involve similar movement properties. – For example, “It was the thief who chased the artist” results in

improved production and comprehension of wh-questions such as “Who did the thief chase?”

• STATUS: Testing Prototype