shuta's ma thesis 口頭試問プレゼン用
DESCRIPTION
TRANSCRIPT
Relationships between EFL Writing and Speaking Skills
- The Case of 16 Japanese EFL Students’ Performanceson the TOEFL CBT/iBT -
MA Thesis Oral ExaminationShuta Miyazaki
2012-2-14
Background of the Study
• Key words: EAP, TOEFL, Productive Skills, Holistic Assessment, Analytic Assessment
• Purpose: To find out…– Whether EFL Writing and Speaking skills are
correlated– If so, what kind of relationships?
• An experimental study on Japanese EFL learners’ performance on TOEFL
Earlier Studies (1)
• Lee, Gentile, and Kantor (2008)• Holistic and analytic scoring of 930 samples
for TOEFL CBT’s Writing section.• Holistic rubrics – ETS (1998)• Analytic rubrics – Developed for this study by
a panel of applied linguists• Pearson Correlation
Overall Development Organization Vocabulary Sentence Grammar Mechanics
Overall Correlation Coefficient 1 0.88 0.85 0.9 0.87 0.83 0.72
Development Correlation Coefficient 0.88 1 0.84 0.85 0.81 0.77 0.66
Organization Correlation Coefficient 0.85 0.84 1 0.81 0.78 0.74 0.66
Vocabulary Correlation Coefficient 0.9 0.85 0.81 1 0.87 0.83 0.70
Sentence Correlation Coefficient 0.87 0.81 0.78 0.87 1 0.88 0.72
Grammar Correlation Coefficient 0.83 0.77 0.74 0.83 0.88 1 0.73
Mechanics Correlation Coefficient 0.72 0.66 0.66 0.70 0.72 0.73 1
Prompt 1
Overall Development Organization Vocabulary Sentence Grammar Mechanics
Overall Correlation Coefficient 1 0.88 0.83 0.88 0.87 0.85 0.75
Development Correlation Coefficient 0.88 1 0.81 0.82 0.81 0.77 0.73
Organization Correlation Coefficient 0.83 0.81 1 0.78 0.77 0.71 0.71
Vocabulary Correlation Coefficient 0.88 0.82 0.78 1 0.83 0.81 0.72
Sentence Correlation Coefficient 0.87 0.81 0.77 0.83 1 0.89 0.72
Grammar Correlation Coefficient 0.85 0.77 0.71 0.81 0.89 1 0.73
Mechanics Correlation Coefficient 0.75 0.73 0.71 0.72 0.72 0.73 1
Prompt 2
Earlier Studies (2)
• Xi and Mollaun (2006)• Holistic and analytic scoring of 280 samples
for TOEFL iBT’s Speaking section.• Holistic rubrics – ETS (2004)• Analytic rubrics – developed for this study by a
panel of language teaching and testing specialists
• Pearson Correlation
Overall Delivery Language UseTopic
Development
Overall Correlation Coefficient 1 0.85 0.87 0.84
Delivery Correlation Coefficient 0.85 1 0.84 0.84
Language Use Correlation Coefficient 0.87 0.84 1 0.84
TopicDevelopment Correlation Coefficient 0.84 0.84 0.84 1
Prompt 1
Overall Delivery Language UseTopic
Development
Overall Correlation Coefficient 1 0.72 0.72 0.75
Delivery Correlation Coefficient 0.72 1 0.79 0.78
Language Use Correlation Coefficient 0.72 0.79 1 0.81
TopicDevelopment Correlation Coefficient 0.75 0.78 0.81 1
Prompt 2
Findings of Earlier Research
• Dependability of analytic scoring rubrics• Large-scale EAP (ESL, EFL) writing study– High correlations • Between holistic scores and analytic scores• Among each analytic score
• Large-scale EAP (ESL, EFL) speaking study– High correlations • Between holistic scores and analytic scores• Among each analytic score
So, How aboutRelationship between
Writing and Speaking??
Present Study
• Writing and Speaking as two aspects of productive skills – interface??
• Same scoring rubrics as earlier studies (both holistic and analytic)
• Same prompts (writing test only)• Case study of Japanese EFL learners
ParticipantsID# Sex Age Major
Mothertongue
Onset oflearningEnglish
Time spentabroad
(Place) Score(TOEFL) Score(TOEIC)Grade(STEP)
1 M 21German/ InternationalRelations
J apanese 10 1 month New Zealand 800
2 M 21 Philosophy J apanese 13 0 755
3 M 18EnglishLiterature
J apanese 12 0 445
4 M 20 Economics J apanese 12 0
5 F 19EnglishLiterature
J apanese 12 0
6 M 22 Economics J apanese 13 0 735
7 F 19EnglishLiterature
J apanese 13 0 2nd
8 F 19EnglishLiterature
J apanese 12 0
9 F 20 Biology J apanese 10 3 years United States
10 F 19English/ InternationalRelations
J apanese 10 0 Pre- 1st
11 F 20FrenchLiterature
J apanese 8 0 Pre- 1st
12 F 19English/ InternationalRelations
J apanese 12 0 800 Pre- 1st
13 M 20 Law J apanese 13 0 760
14 M 20English/ Linguistics
J apanese 10 1 year Sweden 580(PBT) Pre- 1st
15 F 21 Portuguese J apanese 12 016 F 22 Sociology J apanese 6 0 650
Writing Prompts
Prompt ID Prompt Topic Sample Size
A
Do you agree or disagree with the following topic?
It is more important for students to study history and literaturethan it is for them to study science and mathematics.
Use specific reasons and examples to support your opinion.
9
BSome young children spend a great amount of time practicing sports.Discuss the advantages and disadvantages of this.Use specific reasons and examples to support your opinion.
7
Speaking Prompts
Prompt ID Prompt Topic Sample Size
AWhere in your home country would you most like to go on vacationand why?Include details and examples in your explanation.
8
BSome students like to study in the morning.Others feel they study best in the evening.Which time do you think is better for you to study and why?Include details and examples in your explanation.
7
Procedure
• Test administration – 30 mins. Writing & 45 secs. Speaking (after 15 secs. preparation)
• Rater training and scoring – 1 NS, 2 NNS, ESL/EFL teachers
• Culculations with SPSS– Inter-rater reliability– Non-parametric analysis (Spearmann’s rho) among
writing and scoring scores
Results
• Inter-rater reliability
Rater A Rater B Rater C
Spearman's rho Rater A Correlation Coefficient 1 0.387 .702**
Sig. (2- tailed) . 0.138 0.002
Rater B Correlation Coefficient 0.387 1 .675**
Sig. (2- tailed) 0.138 . 0.004
Rater C Correlation Coefficient .702** .675** 1
Sig. (2- tailed) 0.002 0.004 .
Results (Con’d)
• WritingOverall Development Organization Vocabulary Sentence Grammar Mechanics
1 3.67 4.00 2.67 3.00 2.33 4.00 4.002 3.00 3.00 3.33 3.33 2.33 3.67 3.673 2.00 2.67 3.00 3.00 2.33 3.00 3.004 2.00 1.67 1.67 1.67 2.67 1.00 1.337 2.67 2.67 3.33 2.67 2.33 3.67 3.67
11 3.67 4.00 3.33 3.33 3.00 3.67 3.0012 4.00 4.00 4.00 3.33 3.33 3.67 4.3313 3.33 3.67 3.67 3.00 3.33 3.67 3.6716 2.67 3.00 2.67 2.67 2.67 3.33 2.33
Mean 3.00 3.19 3.07 2.89 2.70 3.30 3.22SD 0.73 0.80 0.68 0.53 0.42 0.90 0.93
Writing Prompt A
Overall Development Organization Vocabulary Sentence Grammar Mechanics
5 4.00 4.33 4.67 3.67 4.00 4.00 4.006 2.33 2.67 3.00 3.00 2.67 3.00 3.008 2.33 2.33 3.00 2.67 2.33 3.33 3.339 2.33 2.33 2.67 2.00 2.33 2.67 3.33
10 3.00 3.33 3.67 2.67 2.67 3.67 3.6714 3.00 3.33 3.33 3.00 3.00 3.67 3.6715 3.33 3.00 3.33 2.67 2.67 3.67 3.67
Mean 2.90 3.05 3.38 2.81 2.81 3.43 3.52SD 0.63 0.71 0.65 0.50 0.57 0.46 0.33
Writing Prompt B
Results (Con’d)
• Writing (Con’d) n=16Overall Development Organization Vocabulary Sentence Grammar Mechanics
Overall Correlation Coefficient 1 .926** .646** .626** .697** .710** .611*
Sig. (2- tailed) 0 0.007 0.009 0.003 0.002 0.012
Development Correlation Coefficient .926** 1 .624** .729** .743** .644** 0.488
Sig. (2- tailed) 0 0.01 0.001 0.001 0.007 0.055
Organization Correlation Coefficient .646** .624** 1 .649** .835** 0.487 .581*
Sig. (2- tailed) 0.007 0.01 0.007 0 0.056 0.018
Vocabulary Correlation Coefficient .626** .729** .649** 1 .626** .507* 0.33
Sig. (2- tailed) 0.009 0.001 0.007 0.009 0.045 0.212
Sentence Correlation Coefficient .697** .743** .835** .626** 1 0.423 0.329
Sig. (2- tailed) 0.003 0.001 0 0.009 0.103 0.213
Grammar Correlation Coefficient .710** .644** 0.487 .507* 0.423 1 0.379
Sig. (2- tailed) 0.002 0.007 0.056 0.045 0.103 0.148
Mechanics Correlation Coefficient .611* 0.488 .581* 0.33 0.329 0.379 1
Sig. (2- tailed) 0.012 0.055 0.018 0.212 0.213 0.148
Results (Con’d)
• Speaking n=15
Overall DeliveryLanguage
UseTopic
Development1 4.00 4.00 3.33 3.67
5 3.00 3.00 3.00 2.67
6 2.67 3.00 2.00 2.67
8 2.00 2.33 2.00 2.00
12 2.67 3.00 2.33 2.33
13 2.00 2.33 2.33 2.00
14 3.00 3.33 3.00 2.67
16 2.67 2.67 2.67 2.00
Mean 2.75 2.96 2.58 2.50
SD 0.64 0.55 0.50 0.56
Speaking Prompt A
Overall DeliveryLanguage
UseTopic
Development
2 3.00 2.67 3.00 2.67
3 2.33 2.67 2.33 2.00
7 2.00 2.33 2.33 2.00
9 1.00 1.67 1.00 1.0010 3.33 3.67 3.00 3.3311 2.67 3.00 2.33 2.3315 2.00 2.33 2.00 1.67
Mean 2.33 2.62 2.29 2.14SD 0.77 0.62 0.68 0.74
Speaking Prompt B
Results (Con’d)
• Speaking (Con’d)
Overall DeliveryLanguage
UseTopic
Development
Correlation Coefficient 1 .919** .876** .940**
Sig. (2- tailed) . 0 0 0
Correlation Coefficient .919** 1 .715** .907**
Sig. (2- tailed) 0 . 0.003 0
Correlation Coefficient .876** .715** 1 .765**
Sig. (2- tailed) 0 0.003 . 0.001
Correlation Coefficient .940** .907** .765** 1
Sig. (2- tailed) 0 0 0.001 .Topic
Development
Overall
Delivery
Language Use
Results (Con’d)
• Writing and Speaking n=15Overall Delivery
LanguageUse
TopicDevelopment
Overall Correlation Coefficient 0.441 0.437 0.463 0.393
Sig. (2- tailed) 0.10 0.10 0.08 0.15
Development Correlation Coefficient .622* .642** .640* .559*
Sig. (2- tailed) 0.01 0.01 0.01 0.03
Organization Correlation Coefficient 0.196 0.194 0.243 0.245
Sig. (2- tailed) 0.48 0.49 0.38 0.38
Vocabulary Correlation Coefficient 0.513 0.492 0.443 .536*
Sig. (2- tailed) 0.05 0.06 0.10 0.04
Sentence Correlation Coefficient 0.208 0.319 0.163 0.191
Sig. (2- tailed) 0.46 0.25 0.56 0.50
Grammar Correlation Coefficient .576* 0.493 .698** .554*
Sig. (2- tailed) 0.03 0.06 0.00 0.03
Mechanics Correlation Coefficient 0.360 0.304 0.448 0.396
Sig. (2- tailed) 0.19 0.27 0.09 0.14
Writing
Speaking
Discussion
• Weak to moderate correlation among W&S holistic and analytic scores.
• Writing and speaking abilities could be correlated, although the correlation coefficients were not always statistically significant.
• Strengths of relationships among W&S sub-skills seem to be different.
Limitations
• Small-scale (not big enough for parametric tests)
• Relatively high-proficiency participants• Participants were not quite prepared to take
the tests.
Pedagogical implications
• W&S skill teaching– is ought to be implemented in a systematic
manner to build each sub-skill of both aspects of ESL/EFL productive skills
– Could be well combined, which could help the learners to build both abilities.