![Page 1: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/1.jpg)
Martin J. Pickering, Simon Garrod, Behavioral and Brain Sciences, 2012.
An Integrated Theory of Language Production and Comprehension
Computer Science & Engineering2012-20835
Sang-Woo Lee
![Page 2: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/2.jpg)
Background - Aphasia
![Page 3: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/3.jpg)
3
Expressive aphasia Agrammatic aphasia Understand what other people say, but cannot speak the sen-
tence well. Caused by damage to, or developmental issues in the anterior
regions of the brain Including (but not limited to) the Broca’s area
Broca’s Aphasia
![Page 4: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/4.jpg)
4
Also known as Receptive Aphasia Fluent apahsia, or sensory aphasia Speak the sentence fluently, but not well-organized sense in
their speech Traditionally associated with neurological damage to Wernicke’s
area in the brain (Actually it is not just simply associated to Wernicke’s area in current exper-
imental result, but anyway…)
Wernicke’s Aphasia
![Page 5: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/5.jpg)
5
There are modules which specify some function
The “classical Lichtheim-Broca-Wernicke” Model
perceptionaction
![Page 6: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/6.jpg)
6
Background - Aphasia Traditional independence of production and comprehension Interweaving in action and action perception
Perception process in action Predict next action of other Joint Action
Interweaving in Production and Comprehension Comprehension process in Production Predict next speech of other Interactive Language
Professor’s Question
Contents
![Page 7: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/7.jpg)
Traditional independence of
production and comprehension
![Page 8: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/8.jpg)
8
Traditional model of communication Discrete stages
A produces, B comprehends B produces, A comprehends
![Page 9: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/9.jpg)
9
Assumes “horizontal split” between pro-duction and comprehension Arrows-within-arrows indicate feedback (in in-
teractive accounts) But this feedback is internal to production or
comprehension It may involve “general knowledge” But production does not involve comprehension pro-
cesses And comprehension does not involve production pro-
cesses
Horizontal Split
![Page 10: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/10.jpg)
10
Interlocutors are not static, as the traditional model as-sumes, but are “moving targets” performing a joint activitiy (Garrod & Pickering, 2009)
Example of predict in Behavioral Instance
![Page 11: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/11.jpg)
11
Example of predict in Neuroscience
Big(neuter)
Painting (neuter)
Big (common)Bookcase (com-
mon)
(Pickering & Garrod, 2007)
große Gemälde großen Bücherschrank
![Page 12: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/12.jpg)
12
Also, many experiments demonstrate ef -fects of one on the other Picture-word interference (Schriefers et al.,
1990) Word identification affected by externally con-
trolled cheek movement (Ito et al., 2009) And strongly overlapping neural circuits
for production and comprehension (e.g., Pulvermüller & Fadiga, 2010; Scott et al., 2009)
Other Counterexample
![Page 13: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/13.jpg)
13
Prediction process could be naturally understood with pro-duction module
When they comprehend the utterance, they also use pro-duction model internally.
Result
- There is forward modelTo predict perception caused by their own ut-terance- Fast alert when you say something wrong
![Page 14: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/14.jpg)
Interweaving in action and action perception
Perception process in action Predict next action of other Joint Action
![Page 15: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/15.jpg)
Interweaving in action and action perception
• Close links between action and action percep-tion, e.g.– participants’ arm movements affected by observing
another person’s arm movements (Kilner et al., 2003)– And making hand movements can facilitate concur-
rent visual discrimination of deviant hand postures (Miall et al., 2006)
• Such links could have various purposes – Supporting overt imitation– facilitating memory or understanding (“postdictively”)
• But authors propose that they aid prediction of own and others’ actions, by use of a forward model– Based on computational neuroscience (Wolpert, 1997;
see Grush, 2004)
![Page 16: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/16.jpg)
Forward modelling in action In our terms, the action command causes the action im-
plementer to move the hand and the perceptual imple-menter to construct the percept
And the efference copy causes the forward action model to generate the predicted hand movement and the forward perceptual model to construct the predicted percept
![Page 17: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/17.jpg)
Forward Modeling in Action- Just act- Feel involved per-
cepts of your own acte.g. Own coordination infoFeeling of wind blow-ing to your armGravity info …
![Page 18: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/18.jpg)
18
Efference Copy
![Page 19: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/19.jpg)
Forward Modeling in Action- Predict perception caused by their own action
e.g. Own coordination infoFeeling of wind blowing to your armGravity info …
![Page 20: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/20.jpg)
Prediction-by-Simulation
- Predict perception of other’s next actionby seeing other’s current action
![Page 21: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/21.jpg)
Joint Action• People are highly adept at joint activities
(Sebanz et al., 2006). – ballroom dancing, playing a duet, carrying a
large object together– Precise timing is crucial
• To succeed, A predicts B’s action and B predicts B’s action
![Page 22: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/22.jpg)
22
Joint Action
![Page 23: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/23.jpg)
Interweaving in Production and Comprehension
Comprehension process in Production Predict next speech of other Interactive Language
![Page 24: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/24.jpg)
Forward modeling in language production
• Action implementer production imple-menter
• Perceptual implementer comprehension implementer
• Action command production command– Drives the production implementer– Efference copy drives the forward models
• Comparator monitor – compares the utterance percept and the pre-
dicted utterance percept
![Page 25: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/25.jpg)
Unifying production and comprehension
• Production and comprehension are inter-woven– Tight coupling in dialogue (Clark, 1996; Picker-
ing & Garrod, 2004)– Behavioural experiments show effects of com-
prehension processes on production and vice versa (e.g., Schriefers et al., 1990)
– Overlap of brain circuits for production and comprehension (e.g., Pulvermuller & Fadiga, 2010)
• Such interweaving facilitates prediction of self and other’s utterances
![Page 26: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/26.jpg)
26
Classical modeling in language production
production command ti
Production implementer
Efference copy
Utterance
tphonsynsemp ,,
Forward production model
Comprehension implementer
Utterance percept tphonsynsemc ,,
Forward comprehension model
monitor
Predicted utterance
tphonsynsemp ,,ˆ
Predicted utterance percept tphonsynsemc ,,ˆ
- Just say utterance,- Listen what you say.
![Page 27: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/27.jpg)
27
Forward modeling in language production
production command ti
Production implementer
Efference copy
Utterance
tphonsynsemp ,,
Forward production model
Comprehension implementer
Utterance percept tphonsynsemc ,,
Forward comprehension model
monitor
Predicted utterance
tphonsynsemp ,,ˆ
Predicted utterance percept tphonsynsemc ,,ˆ
- Predict perception caused by their own ut-terance- Fast alert when you say something wrong
![Page 28: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/28.jpg)
Self-monitoring
Speaker wishes to say kite In the past, she has always constructed
the kite-concept and then uttered /k/ She therefore constructs forward model
p^[phon](t) = /k/ If she then incorrectly constructs p[phon]
= /g/, the monitor notices the mismatch If she believes the forward model, she will
detect an error (and perhaps reformulate) Otherwise, she will alter her forward model
![Page 29: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/29.jpg)
29
Prediction-by-simulation
Efference copy
Forward production model
Comprehension implementer
Inverse model + context
Comprehension implementer
Forward comprehension model
monitor
Covert imitation
Person B
Person A
B’s utterance 1,, tphonsynsemp B
Derived production command
tiB
Predicted utterance
1,,ˆ tphonsynsemp B
Predicted utterance percept 1,.ˆ tphonsynsemc B
Utterance percept tphonsynsemc B,,
Utterance percept 1,, tphonsynsemc B
Overt Responses
Derived production command
1tiB
B’s utterance tphonsynsemp B,,
Derived intentional act of communication iA(t+1)
- Predict perception of other’s next utteranceby listening other’s cur-rent utterance
Big(neuter)
Painting (neuter)
![Page 30: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/30.jpg)
30
Interactive Language
Joint action involves combining accounts of action and action perception
Similarly, interactive language involves combining ac-counts of production and comprehension Facilitates coordination (e.g., short intervals between speak-
ers; Wilson & Wilson, 2005) Facilitates alignment (developing same representations; Pick-
ering & Garrod, 2004) Alignment in turn facilitates comprehension (better prediction
of others)
![Page 31: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/31.jpg)
31
Interactive Language
![Page 32: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/32.jpg)
Conclusion We propose that language production and com-
prehension are interwoven It assumes a central role to prediction in produc-
tion, comprehension, and dialogue Speakers construct forward models to predict as-
pects of their upcoming utterances Listeners covertly imitate speakers and use for-
ward models to predict the speakers Our account helps explain the efficiency of pro-
duction and comprehension and the remarkable fluidity of dialogue
![Page 33: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/33.jpg)
Thank you
![Page 34: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/34.jpg)
Professor’s Question
Comprehension process in Production Predict next speech of other Interactive Language
![Page 35: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/35.jpg)
35
Q1: Give the evidence for how language production and comprehension are tightly interwoven. How does this relate to the perception-action cycle theory of cognitive systems?
Question 1
![Page 36: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/36.jpg)
36
Q2: Explain and give the evidence for how action, action perception, and joint action are interwoven. Explain how the authors use this to develop accounts of production, com-prehension, and interactive language.
Question 2
Action implementer pro-duction implementer
Perceptual implementer comprehension implementer
Action command produc-tion command
Comparator monitor
participants’ arm move-ments affected by ob-serving another person’s arm movements (Kilner et al., 2003)
And making hand move-ments can facilitate con-current visual discrimina-tion of deviant hand pos-tures (Miall et al., 2006)
![Page 37: Martin J. Pickering, Simon Garrod , Behavioral and Brain Sciences, 2012](https://reader035.vdocuments.pub/reader035/viewer/2022062521/568166d5550346895ddae8b2/html5/thumbnails/37.jpg)
37
Q3: Give examples of what behavioral and neuroscientific data on language processing can be explained by the integrated theory of language production and comprehension explains, while modular theory does not.
Question 3
- Behavioral data
- Neuroscientific data (Pick-ering & Garrod, 2007)