new creating a new communication system : gesture has the upper … a new... · 2015. 11. 24. · :...
TRANSCRIPT
Powerpoint Templates
Page 1Powerpoint Templates
Creating a New Communication System
: Gesture has the Upper Hand
인지과학 협동과정
방효석
Powerpoint Templates
Page 2
Table of Contents
Backgrounds
Overview of Present Research
Experiment 1
Experiment 2
Discussion
Powerpoint Templates
Page 3Powerpoint Templates
Backgrounds
Powerpoint Templates
Page 4
Backgrounds
Previous Research
: Creating a communication system from scratch: gesture beats vocalization
hands down (Fay et al., 2014)
<Research Question>
Does the modality of communication drives the creation of shared
inventory of sign-meaning mapping?
Powerpoint Templates
Page 5
Backgrounds
Previous Research
: Creating a communication system from scratch: gesture beats vocalization
hands down (Fay et al., 2014)
<Hypothesis>
1. Communication success will be higher for gesture than for non-
linguistic vocalization.
2. Communication success will be higher in the combined modality
compared to gesture-alone.
3. There will be greater alignment in the gestural modality than in
the vocalization modality.
Powerpoint Templates
Page 6
1. Language Origin
• Proto-speech account
: Rudimentary vocalization came to have communicative meaning
• Proto-sign account
: Language evolved first from manual gestures, before shifting to the
vocal modality
• Multimodal account
: The earliest forms of language were not restricted to a single modality
Basic Concepts
Backgrounds
Powerpoint Templates
Page 7
Backgrounds
Theories of Origin of
Languages
1. Proto-sign account
2. Proto-speech account
3. Multimodal account
Experimental Design
1. Gesture-only condition
2. Non-linguistic vocalization
condition
3. Multimodal condition
Powerpoint Templates
Page 8
2. Types of Signs(Pierece’s triadic model of sign)
1) Iconic: Signs where the signifier resembles the signified(e.g., a
portrait, a cartoon, sound effects, or a statue)
2) Indexical: Signs where the signifier is caused by the signified(e.g.,
smoke signifies fire)
3) Symbolic: The signifier is assigned arbitrarily or is accepted as
societal convention. The relationship must be learned (i.e. alphabet,
mathemetical signs, computer codes)
Basic Concepts
Backgrounds
Powerpoint Templates
Page 9
3. Referential Communication Task
Referential communication?
: Communication occured by making reference
Ex) You can refer one of the cars on the right,
by using
• names (the Corvette)
• noun modifiers, like adjectives
(the light green car)
• prepositional phrases
(the car in the upper right-hand corner)
• relative clauses
(the car that you just put on the shelf)
Basic Concepts
Backgrounds
Powerpoint Templates
Page 10
4. Measures
1) Effectiveness: measures how successful the signs were at identifying
their referent (communication success)
2) Alignment: : measures the similarity between the sign a participant
produced, and the sign their partner produced on the previous game
when communicating the same concept (basis of shared inventory)
Basic Concepts
Backgrounds
Powerpoint Templates
Page 11
5. Motivated signs & Arbitrary signs
1) Motivated signs: Signs that are linked to meaning by structural
resemblance or by natural association (i.e. Iconic/indexical signs)
2) Arbitrary signs: Signs where the relation between signifier and signified
is purely conventional and culturally specific (i.e. Symbolic signs; name,
language...)
Basic Concepts
Backgrounds
Powerpoint Templates
Page 12Powerpoint Templates
Research Overview
Powerpoint Templates
Page 13
Research Overview
Purpose of the present research
1. Limitation of previous referential task studies
: lack of the number of concepts used
2. Provide the comparison between the vocal and gestural
modalities to figure out the differences of affordances of sign
motivations of gesture and non-linguistic vocalization
Powerpoint Templates
Page 14Powerpoint Templates
Experiment 1
Powerpoint Templates
Page 15
Experiment 1
Hypothesis
1. Sign motivation would be higher for gestured signs than for
signs produced using non-linguistic vocalization
2. Communication success would be higher for gesture than for
non-linguistic vocalization
3. Alignment would be higher in the gesture-only condition than
in the vocal-only condition
Powerpoint Templates
Page 16
Experiment 1
Method
1)Participants:
106 undergraduate students(63 females)
Placed into unacquainted pairs
All were free from auditory, visual, speech and motor impairment
2)Materials:
18 target concepts were sampled without replacement
(Adjective, Noun, Verb)
Presented with six distractor concepts
A different set of concepts were used by each pair
Powerpoint Templates
Page 17
Experiment 1
Method
3) Task & Procedure:
• 2 task conditions (gesture/vocal) with 6 games in each condition
• Pairs 2 participants (matcher/director)
matcher: Select the same concept more than once within the
same game
director: Produce as many gestures and vocalizations as they
wished for each concepts
• Counterbalancing of modality across participants
gesture-only: face one another
vocal-only: face away from each other(eliminate gesture effects)
Powerpoint Templates
Page 18
Experiment 1
Result & Discussion
Three-way ANOVA (modality, game, concept)
1) Motivation
• 7-point likert scale: 0 (entirely symbolic) to 6 (highly motivated, either
iconic or indexical)
• Gesture outperformed vocalization [F(1, 52) = 700.33, p < .001]
• Increase in sign motivation across games 1-6 in both conditions [F(5,
260) = 39.89, p < .001]
Powerpoint Templates
Page 19
Experiment 1
Result & Discussion
2) Communication Success
Powerpoint Templates
Page 20
Experiment 1
Result & Discussion
3) Alignment
• 7-point likert scale, 0 (did not copy partner’s sign) to 6 (nearly identical
copy of partner’s sign)
• Gesture showed higher alignment than vocalization(p<.001)
• Alignment increased in both modalities across games(p<.001)
Powerpoint Templates
Page 21
Experiment 1
Result & Discussion
4) Limitation
• Concurrent feedback effect in gesture-only condition
: Participants in gesture-only condition may have and advantage
seeing their partner’s facial expressions of confusion or
comprehension
→ Experiment 2 addresses this issue
Powerpoint Templates
Page 22Powerpoint Templates
Experiment 2
Powerpoint Templates
Page 23
Experiment 2
Method
1)Participants:
60 undergraduate students(42 females)
2)Materials:
Same corpus of concepts used in experiment 1 was used
Fewer concepts were sampled(540)
3) Task & Procedure:
Participants communicated signs to a video camera
Only 2 games were played
Powerpoint Templates
Page 24
Experiment 2
Result & Discussion
1) Motivation
Powerpoint Templates
Page 25
Experiment 2
Result & Discussion
2) Communication Success
• Gestured signs were communicated more successfully than vocal
signs(p<.001)
3) Alignment
• Alignment was higher in gesture-only condition[F(1, 29)=283.41,
p<.001]
Powerpoint Templates
Page 26
Experiment 2
Result & Discussion
• Experiment 2 replicates the pattern of results observed in experiment 1
: Indicates that the benefit of gesture over non-linguistic vocalization is
due to the modality itself, rather than concurrent feedback
Powerpoint Templates
Page 27
Experiment 2
Result & Discussion
Based on interaction effect of 3 variables:
Powerpoint Templates
Page 28Powerpoint Templates
Discussion
Powerpoint Templates
Page 29
Discussion
If gesture is better than non-linguisitc vocalization in terms of sign
motivation, why do we use conventional spoken language instead of
conventional sign language?
There are several studies showing evidence that people also
spontaneously produce motivated vocal language in their speech. If
both gestural and vocal language have same effect of motivation,
which of these do you think the origin of conventionalized language?