shuta's ma thesis 口頭試問プレゼン用

Relationships between EFL Writing and Speaking Skills

- The Case of 16 Japanese EFL Students’ Performanceson the TOEFL CBT/iBT -

MA Thesis Oral ExaminationShuta Miyazaki

2012-2-14

Background of the Study

• Key words: EAP, TOEFL, Productive Skills, Holistic Assessment, Analytic Assessment

• Purpose: To find out…– Whether EFL Writing and Speaking skills are

correlated– If so, what kind of relationships?

• An experimental study on Japanese EFL learners’ performance on TOEFL

Earlier Studies (1)

• Lee, Gentile, and Kantor (2008)• Holistic and analytic scoring of 930 samples

for TOEFL CBT’s Writing section.• Holistic rubrics – ETS (1998)• Analytic rubrics – Developed for this study by

a panel of applied linguists• Pearson Correlation

Overall Development Organization Vocabulary Sentence Grammar Mechanics

Overall Correlation Coefficient 1 0.88 0.85 0.9 0.87 0.83 0.72

Development Correlation Coefficient 0.88 1 0.84 0.85 0.81 0.77 0.66

Organization Correlation Coefficient 0.85 0.84 1 0.81 0.78 0.74 0.66

Vocabulary Correlation Coefficient 0.9 0.85 0.81 1 0.87 0.83 0.70

Sentence Correlation Coefficient 0.87 0.81 0.78 0.87 1 0.88 0.72

Grammar Correlation Coefficient 0.83 0.77 0.74 0.83 0.88 1 0.73

Mechanics Correlation Coefficient 0.72 0.66 0.66 0.70 0.72 0.73 1

Prompt 1

Overall Development Organization Vocabulary Sentence Grammar Mechanics

Overall Correlation Coefficient 1 0.88 0.83 0.88 0.87 0.85 0.75

Development Correlation Coefficient 0.88 1 0.81 0.82 0.81 0.77 0.73

Organization Correlation Coefficient 0.83 0.81 1 0.78 0.77 0.71 0.71

Vocabulary Correlation Coefficient 0.88 0.82 0.78 1 0.83 0.81 0.72

Sentence Correlation Coefficient 0.87 0.81 0.77 0.83 1 0.89 0.72

Grammar Correlation Coefficient 0.85 0.77 0.71 0.81 0.89 1 0.73

Mechanics Correlation Coefficient 0.75 0.73 0.71 0.72 0.72 0.73 1

Prompt 2

Earlier Studies (2)

• Xi and Mollaun (2006)• Holistic and analytic scoring of 280 samples

for TOEFL iBT’s Speaking section.• Holistic rubrics – ETS (2004)• Analytic rubrics – developed for this study by a

panel of language teaching and testing specialists

• Pearson Correlation

Overall Delivery Language UseTopic

Development

Overall Correlation Coefficient 1 0.85 0.87 0.84

Delivery Correlation Coefficient 0.85 1 0.84 0.84

Language Use Correlation Coefficient 0.87 0.84 1 0.84

TopicDevelopment Correlation Coefficient 0.84 0.84 0.84 1

Prompt 1

Overall Delivery Language UseTopic

Development

Overall Correlation Coefficient 1 0.72 0.72 0.75

Delivery Correlation Coefficient 0.72 1 0.79 0.78

Language Use Correlation Coefficient 0.72 0.79 1 0.81

TopicDevelopment Correlation Coefficient 0.75 0.78 0.81 1

Prompt 2

Findings of Earlier Research

• Dependability of analytic scoring rubrics• Large-scale EAP (ESL, EFL) writing study– High correlations • Between holistic scores and analytic scores• Among each analytic score

• Large-scale EAP (ESL, EFL) speaking study– High correlations • Between holistic scores and analytic scores• Among each analytic score

So, How aboutRelationship between

Writing and Speaking??

Present Study

• Writing and Speaking as two aspects of productive skills – interface??

• Same scoring rubrics as earlier studies (both holistic and analytic)

• Same prompts (writing test only)• Case study of Japanese EFL learners

ParticipantsID# Sex Age Major

Mothertongue

Onset oflearningEnglish

Time spentabroad

(Place) Score(TOEFL) Score(TOEIC)Grade(STEP)

1 M 21German/ InternationalRelations

J apanese 10 1 month New Zealand 800

2 M 21 Philosophy J apanese 13 0 755

3 M 18EnglishLiterature

J apanese 12 0 445

4 M 20 Economics J apanese 12 0

5 F 19EnglishLiterature

J apanese 12 0

6 M 22 Economics J apanese 13 0 735

7 F 19EnglishLiterature

J apanese 13 0 2nd

8 F 19EnglishLiterature

J apanese 12 0

9 F 20 Biology J apanese 10 3 years United States

10 F 19English/ InternationalRelations

J apanese 10 0 Pre- 1st

11 F 20FrenchLiterature

J apanese 8 0 Pre- 1st

12 F 19English/ InternationalRelations

J apanese 12 0 800 Pre- 1st

13 M 20 Law J apanese 13 0 760

14 M 20English/ Linguistics

J apanese 10 1 year Sweden 580(PBT) Pre- 1st

15 F 21 Portuguese J apanese 12 016 F 22 Sociology J apanese 6 0 650

Writing Prompts

Prompt ID Prompt Topic Sample Size

A

Do you agree or disagree with the following topic?

It is more important for students to study history and literaturethan it is for them to study science and mathematics.

Use specific reasons and examples to support your opinion.

9

BSome young children spend a great amount of time practicing sports.Discuss the advantages and disadvantages of this.Use specific reasons and examples to support your opinion.

7

Speaking Prompts

Prompt ID Prompt Topic Sample Size

AWhere in your home country would you most like to go on vacationand why?Include details and examples in your explanation.

8

BSome students like to study in the morning.Others feel they study best in the evening.Which time do you think is better for you to study and why?Include details and examples in your explanation.

7

Procedure

• Test administration – 30 mins. Writing & 45 secs. Speaking (after 15 secs. preparation)

• Rater training and scoring – 1 NS, 2 NNS, ESL/EFL teachers

• Culculations with SPSS– Inter-rater reliability– Non-parametric analysis (Spearmann’s rho) among

writing and scoring scores

Results

• Inter-rater reliability

Rater A Rater B Rater C

Spearman's rho Rater A Correlation Coefficient 1 0.387 .702**

Sig. (2- tailed) . 0.138 0.002

Rater B Correlation Coefficient 0.387 1 .675**

Sig. (2- tailed) 0.138 . 0.004

Rater C Correlation Coefficient .702** .675** 1

Sig. (2- tailed) 0.002 0.004 .

Results (Con’d)

• WritingOverall Development Organization Vocabulary Sentence Grammar Mechanics

1 3.67 4.00 2.67 3.00 2.33 4.00 4.002 3.00 3.00 3.33 3.33 2.33 3.67 3.673 2.00 2.67 3.00 3.00 2.33 3.00 3.004 2.00 1.67 1.67 1.67 2.67 1.00 1.337 2.67 2.67 3.33 2.67 2.33 3.67 3.67

11 3.67 4.00 3.33 3.33 3.00 3.67 3.0012 4.00 4.00 4.00 3.33 3.33 3.67 4.3313 3.33 3.67 3.67 3.00 3.33 3.67 3.6716 2.67 3.00 2.67 2.67 2.67 3.33 2.33

Mean 3.00 3.19 3.07 2.89 2.70 3.30 3.22SD 0.73 0.80 0.68 0.53 0.42 0.90 0.93

Writing Prompt A

Overall Development Organization Vocabulary Sentence Grammar Mechanics

5 4.00 4.33 4.67 3.67 4.00 4.00 4.006 2.33 2.67 3.00 3.00 2.67 3.00 3.008 2.33 2.33 3.00 2.67 2.33 3.33 3.339 2.33 2.33 2.67 2.00 2.33 2.67 3.33

10 3.00 3.33 3.67 2.67 2.67 3.67 3.6714 3.00 3.33 3.33 3.00 3.00 3.67 3.6715 3.33 3.00 3.33 2.67 2.67 3.67 3.67

Mean 2.90 3.05 3.38 2.81 2.81 3.43 3.52SD 0.63 0.71 0.65 0.50 0.57 0.46 0.33

Writing Prompt B

Results (Con’d)

• Writing (Con’d) n=16Overall Development Organization Vocabulary Sentence Grammar Mechanics

Overall Correlation Coefficient 1 .926** .646** .626** .697** .710** .611*

Sig. (2- tailed) 0 0.007 0.009 0.003 0.002 0.012

Development Correlation Coefficient .926** 1 .624** .729** .743** .644** 0.488

Sig. (2- tailed) 0 0.01 0.001 0.001 0.007 0.055

Organization Correlation Coefficient .646** .624** 1 .649** .835** 0.487 .581*

Sig. (2- tailed) 0.007 0.01 0.007 0 0.056 0.018

Vocabulary Correlation Coefficient .626** .729** .649** 1 .626** .507* 0.33

Sig. (2- tailed) 0.009 0.001 0.007 0.009 0.045 0.212

Sentence Correlation Coefficient .697** .743** .835** .626** 1 0.423 0.329

Sig. (2- tailed) 0.003 0.001 0 0.009 0.103 0.213

Grammar Correlation Coefficient .710** .644** 0.487 .507* 0.423 1 0.379

Sig. (2- tailed) 0.002 0.007 0.056 0.045 0.103 0.148

Mechanics Correlation Coefficient .611* 0.488 .581* 0.33 0.329 0.379 1

Sig. (2- tailed) 0.012 0.055 0.018 0.212 0.213 0.148

Results (Con’d)

• Speaking n=15

Overall DeliveryLanguage

UseTopic

Development1 4.00 4.00 3.33 3.67

5 3.00 3.00 3.00 2.67

6 2.67 3.00 2.00 2.67

8 2.00 2.33 2.00 2.00

12 2.67 3.00 2.33 2.33

13 2.00 2.33 2.33 2.00

14 3.00 3.33 3.00 2.67

16 2.67 2.67 2.67 2.00

Mean 2.75 2.96 2.58 2.50

SD 0.64 0.55 0.50 0.56

Speaking Prompt A

Overall DeliveryLanguage

UseTopic

Development

2 3.00 2.67 3.00 2.67

3 2.33 2.67 2.33 2.00

7 2.00 2.33 2.33 2.00

9 1.00 1.67 1.00 1.0010 3.33 3.67 3.00 3.3311 2.67 3.00 2.33 2.3315 2.00 2.33 2.00 1.67

Mean 2.33 2.62 2.29 2.14SD 0.77 0.62 0.68 0.74

Speaking Prompt B

Results (Con’d)

• Speaking (Con’d)

Overall DeliveryLanguage

UseTopic

Development

Correlation Coefficient 1 .919** .876** .940**

Sig. (2- tailed) . 0 0 0

Correlation Coefficient .919** 1 .715** .907**

Sig. (2- tailed) 0 . 0.003 0

Correlation Coefficient .876** .715** 1 .765**

Sig. (2- tailed) 0 0.003 . 0.001

Correlation Coefficient .940** .907** .765** 1

Sig. (2- tailed) 0 0 0.001 .Topic

Development

Overall

Delivery

Language Use

Results (Con’d)

• Writing and Speaking n=15Overall Delivery

LanguageUse

TopicDevelopment

Overall Correlation Coefficient 0.441 0.437 0.463 0.393

Sig. (2- tailed) 0.10 0.10 0.08 0.15

Development Correlation Coefficient .622* .642** .640* .559*

Sig. (2- tailed) 0.01 0.01 0.01 0.03

Organization Correlation Coefficient 0.196 0.194 0.243 0.245

Sig. (2- tailed) 0.48 0.49 0.38 0.38

Vocabulary Correlation Coefficient 0.513 0.492 0.443 .536*

Sig. (2- tailed) 0.05 0.06 0.10 0.04

Sentence Correlation Coefficient 0.208 0.319 0.163 0.191

Sig. (2- tailed) 0.46 0.25 0.56 0.50

Grammar Correlation Coefficient .576* 0.493 .698** .554*

Sig. (2- tailed) 0.03 0.06 0.00 0.03

Mechanics Correlation Coefficient 0.360 0.304 0.448 0.396

Sig. (2- tailed) 0.19 0.27 0.09 0.14

Writing

Speaking

Discussion

• Weak to moderate correlation among W&S holistic and analytic scores.

• Writing and speaking abilities could be correlated, although the correlation coefficients were not always statistically significant.

• Strengths of relationships among W&S sub-skills seem to be different.

Limitations

• Small-scale (not big enough for parametric tests)

• Relatively high-proficiency participants• Participants were not quite prepared to take

the tests.

Pedagogical implications

• W&S skill teaching– is ought to be implemented in a systematic

manner to build each sub-skill of both aspects of ESL/EFL productive skills

– Could be well combined, which could help the learners to build both abilities.