2015/6/3

1

复旦大学

复旦大学Assessing English for Academic Purposes (EAP):

Construct Definition, Current Practice, and Future Development

Presenter: Jinsong FAN, Ph.D. Fudan University

Email: jinsongfan@fudan.edu.cnBeijing, May, 2015

复旦大学

Outline of Presentation

1. Language Assessment as “the Art of the Possible”

2. Understanding the Construct: What is EAP?

3. The Application of Socio-Cognitive Validity Framework in EAP Assessment: The Case of Academic Listening

4. Assessing EAP in the Classroom: English Translation Course (an EGAP Course)

5. The Future of EAP Assessment

2

复旦大学

Language Assessment: The Art of the Possible

Deciding the purpose of assessment

Understanding the construct

Operationalizing the construct

Giving feedback

3

Placement Achievement Proficiency Diagnostic

Theory in linguistics & AL

Pedagogical experience

MCQ SAQ Essay writing …

Numeric score Grade Description …

复旦大学

Fundamental Considerations in Language Assessment

4

Reliability

Construct validity

Authenticity

Interactiveness

Impact

Practicality

TEST USEFULNESS

Test Usefulness Framework, from Bachman & Palmer, 1996

复旦大学

Assessing EAP: From Large-Scale Testing to Classroom Assessment

Classroom-based EAP assessment: Essay writing, presentation, portfolio, quiz…

Localized EAP assessments: TEPT, FET, TOPE…

Large-scale national EAP assessments: CET-6, TEM-4 & 8, PETS-4 & 5…

Large-scale international EAP assessments: IELTS, TOEFL, and PTE…

5 复旦大学

Understanding the Construct: What is EAP?

EAP has emerged out of the broader field of ESP, a theoretically and eclectic parent, but one committed to tailoring instruction to specific rather than general purposes (Hyland & Hamp-Lyons, 2002, p. 2).

EAP is defined as teaching English with the aim of facilitating learners’ study or research in that language (Flowerdew & Peacock, 2001; Jordan, 1997).

6

2015/6/3

2

复旦大学

ESP

EOP EAP

Airlines Medicine EGAP ESAP

PilotsAir traffic

controllers

EST Liberal Arts

Science Technology Humanities

Doctors Nurses

Social Sciences

The ESP hierarchy with examples of courses (from Clapham, 2000, p. 513)7 复旦大学

Academic English is a variety or register of English used in professional books and characterized by the specific linguistic features associated with academic disciplines… Academic English tasks include reading abstracts, getting down the key ideas from lectures, and writing critiques, summaries, annotated bibliographies, reports, case studies, research projects, expository essays. It includes a wide range of genres (Scarcella, 2003, p. 9).

8

复旦大学

EAP Vs. EGP: Linguistic and Cognitive Features

PROFICIENCY PROFICIENCY

Higher Order Thinking

Higher Order Thinking

Metalinguistic Abilities

Metalinguistic Abilities

Phonological Features

Phonological Features

Grammatical Features

Grammatical Features

Vocabulary Features

Vocabulary Features

Sociolinguistic Features

Sociolinguistic Features

Strategies Strategies

Background Knowledge

Discourse FeaturesDiscourse Features

Background Knowledge

Writing an expository essay Vs. Participating in everyday conversation,

from Scarcella, 2003, p. 26-27 9 复旦大学

The Case of Academic Listening

1. Taxonomies of listening skills, sub-skills, and strategies (e.g., Buck, 2001; Richards, 1983; Weir, 1993)

2. Conduct needs analysis/assessment to understand the TLU domain (e.g., Weir, 1983, 1990)

3. Basic and higher level skills in communication and literacy: The distinction between basic interpersonal communication skills (BICS) and cognitive academic language proficiency (CALP) (Cummins, 1980)

4. Can do statements / descriptors (e.g., Common European Framework of Reference – Council of Europe, 2001)

10

复旦大学

Taxonomies of Listening Skills

11

Conversational ListeningEGP

LISTENING ABILITY & SKILLS

Academic ListeningESP - EAP

Ability to discriminate between the distinctive sounds of the target language

Ability to recognize stress patterns of words

Ability to process speech at different rates

…

Ability to identify purpose and scope of lecture

Ability to identify topic of lecture and follow topic development

Familiarity with different registers

Knowledge of classroom conventions

…

复旦大学

Taxonomies of Academic Listening

1. Ability to identify purpose and scope of lecture

2. Ability to identify topic of lecture and follow topic development

3. Ability to identify relationship among units within discourse (e.g., major ideas, generalizations, hypotheses, supporting ideas, examples)

4. Ability to identify role of discourse markers in signaling structure of a lecture (e.g., conjunctions, adverbs, gambits, routines)

5. Ability to infer relationships (e.g., cause, effect, conclusion)

6. Ability to recognize key lexical items related to subject/topic

7. Ability to deduce meanings of words from context

8. Ability to recognize markers of cohesion

9. Ability to recognize function of intonation to signal information structure (e.g., pitch, volume, pace, key)

10. Ability to detect attitude of speaker toward subject matter

11. Ability to follow different modes of lecturing: spoken, audio, audio-visual 12

2015/6/3

3

复旦大学

12. Ability to follow lecture despite differences in accent and speed

13. Familiarity with different styles of lecturing: formal, conversational, read, unplanned

14. Familiarity with different registers: written versus colloquial

15. Ability to recognize relevant matter: jokes, digressions, meanderings

16. Ability to recognize function of non-verbal cues as markers of emphasis and attitude

17. Knowledge of classroom conventions (e.g., turn-taking, classification requests)

18. Ability to recognize instructional/learner tasks (e.g., warnings, suggestions, recommendations, advice, instructions)

13

Source: Richards, 1983, see also Buck, 2001

复旦大学

1. Ability to identify purpose and scope of lecture

2. Ability to identify topic of lecture and follow topic development

3. Ability to identify relationship among units within discourse (e.g., major ideas, generalizations, hypotheses, supporting ideas, examples)

4. Ability to identify role of discourse markers in signaling structure of a lecture (e.g., conjunctions, adverbs, gambits, routines)

5. Ability to infer relationships (e.g., cause, effect, conclusion)

14

复旦大学

12.Ability to follow lecture despite differences in accent and speed

13.Familiarity with different styles of lecturing: formal, conversational, read, unplanned

14.Familiarity with different registers: written versus colloquial

15 复旦大学

Examples of CEFR Descriptors

16

复旦大学17 复旦大学

TEST TAKER CHARACTERISTICS Physical / Physiological Psychological Experiential

TASK SETTING Response method Weighting Knowledge of criteria Order of items Channel of presentation Text length Time constraints, incl.

number of listenings

SETTINGS: ADMIN Physical conditions Uniformity of admin Security

LINGUISTIC DEMANDS: TASK INPUT AND OUTPUT

Overall text purpose Speaker-listener

relationship Discourse mode Functional resources Grammatical resources Lexical resources Nature of information Content knowledge

INTERLOCUTOR Speech rate Variety of accent Number + gender

CONTEXT VALIDITY

COGNITIVE PROCESSES Goal setting Decoding acoustic/visual

input Syntactic parsing Establish propositional

meaning Inferencing Building a mental model Creating a text-level

representation Monitoring comprehension

COGNITIVE VALIDITY

RESPONSE

SCORE/GRADE

SCORING VALIDITY Item difficulty Item discrimination Internal consistency Error of measurement Marker reliability Grading and awarding

Using the Socio-Cognitive Model (Weir, 2005)

for Academic Listening Assessment

Development & Validation

18

2015/6/3

4

复旦大学

In assessing academic listening, “at least some of test tasks should require test takers to engage not just lower level comprehension processes (lexical access, syntactic parsing, propositional meaning at clause/sentence level), but also high-level processes that force them to integrate and reconcile incoming information, possibly from different sources, with their own knowledge sources (topical, genre, rhetorical, etc.) in order to create a text-level or inter-textual presentation or mental model (Taylor & Geranpayeh, 2011, p. 97).

19 复旦大学

EAP Classroom Assessment in Practice: English Translation Course

1. English Translation (ET) Course as an EGAP course

2. ET course delivery: Teaching content and method

3. Learner characteristics

4. ET course assessment

20

Assessment Context

Assessment Method

复旦大学21

The position of ET course in the teaching system

复旦大学

The overarching goal of this courseTo further improve students’ EGAP ability and skills, especially in terms of reading and writing, through the study of translation

1. Develop some basic understanding about the nature, process, and product of translation

2. Learn some basic skills and strategies in conducting C-E and E-C translation

3. Learn how to critique a translation based on what they have learned in class

4. Learn how to conduct a translation mini-project and present the relevant research findings

Course objectives at the general and specific levels

复旦大学

1. Lecture (Translation theory & practice)

2. Review of students’ translation assignments (Translation skills & strategies)

3. Translation critique (Critical thinking, seminar discussion)

4. Translation mini-projects (Academic writing, oral presentation)

The four components in ET course delivery

复旦大学

Overall Design of ET Course Assessment

Component Method Participant Target Construct

FormativeAssessment (FA)

Translation assignment

Teacher Translation skills

Translation project (written)

TeacherStudent

Academic writing

Translation project (oral)

TeacherStudent

Oral presentation

Translation critique (written)

TeacherStudent

Academic writing Critical thinking

Translation critique (oral)

TeacherStudent

Oral presentation

Classroom quiz Teacher Translation skills

SummativeAssessment (SA)

Final examination Teacher Academic writing Critical thinking Translation skills

24

2015/6/3

5

复旦大学

Course evaluation

Translation critique 10%Students read a translation work out of class, and write a short reading report. Students need to discuss their reading experience in the seminar session.

Translation mini-projects

20%

Students work in groups on a translation mini-project(mainly out of class), and then present the relevant findings, both orally and in the written form, in class. Peer review constitutes part of the evaluation.

Translation assignments

20%

Each student is required to complete the translation assignments of 8 texts, 4 from English into Chinese and the other 4 from Chinese into English after class. Peer review constitutes part of the evaluation.

Final examination 50%

Students are required to comment on a given translation, and complete the translation of two texts, one from English into Chinese and the other from Chinese into English.

复旦大学

Examples of translation assignment

E-C translation

Example 1: The Difficulty of Translation

Example 2: How I Escaped the Reform School

C-E translation

Example 1: Language and Social Identity

Example 2: Flowers and Air-Conditioners

复旦大学

Translation Mini-Projects (FA)

Students are required to work in groups (3-5 per group) on two translation mini-projects based on the project description;

After working for two weeks on the project, students are required to submit a research report (written in English around 1,500 words) to describe how the project is undertaken and to summarize the major findings;

Students need to present orally how they have conducted the project and the major research findings.

复旦大学

Examples of students’ written reports

Example 1: Poetry translation (from the 2013 spring cohort)

Marking criteria

Example 2: The role of purpose in translation (from the 2014 spring cohort, TT & Report)

Example 3: Yan Fu’s translation (from the 2013 autumn cohort)

Teacher’s feedback

复旦大学

An example of reading assignment (C-E)

Chapter 1 of Fortress Besieged, written by Ch’ien Chung-shu and translated by Kelly & Mao 复旦大学

An example of reading assignment (E-C)

Animal Farm, written by George Orwell and translated by Fu Weichi

2015/6/3

6

复旦大学

Is it possible to adopt a combination of teacher-, self-, and peer-assessment in ETC: Students’ views

Items Min Max Mean SD

1. I can assess my own translation work responsibly. 3.00 6.00 5.16 0.79

2. I can assess my others’ work responsibly. 4.00 6.00 5.35 0.63

3. I think it necessary for me to assess my own work. 1.00 6.00 4.37 1.29

4. I think it necessary for me to assess others’ work. 1.00 6.00 4.95 1.14

5. I can better understand the rating criteria better through assessing my own translation work. 1.00 6.00 4.61 1.17

6. I can better understand the weaknesses of my translation through assessing my own translation work. 1.00 6.00 4.88 1.08

7. I can better understand the rating criteria better through assessing others’ translation work. 3.00 6.00 5.36 0.70

8. I can better understand the weaknesses of my translation through assessing others’ translation work. 3.00 6.00 5.53 0.65

复旦大学

Peer-assessment form for oral presentation

32

复旦大学

Rationales for further study

1. Doubts over the reliability and validity of self- and peer-assessment;

2. Most rating scales are designed for standardized translation tests, but very few scales are available for teacher-, self-, and peer-assessment in translation teaching;

3. Previous studies are either qualitative or adopting the true score theory. Many-Facet Rasch Analysis (MFRA) can address the limitations of statistical analyses in previous studies.

复旦大学

Two Recent Projects

1. The development and validation of analytic rating scale for translation teaching (2013ZD05, Fudan University Key Research Project)

2. Teacher-, self-, and peer-assessment in translation teaching: A many-facet Rasch modeling approach (2014SH008, National Higher Education English Teaching Research Project)

34

复旦大学

A new rating scale?

An example of the rating scale

Reliability

Construct validity

Practicality

Diagnostic feedback

Teachers’ and students’ views

复旦大学

The Future of EAP Assessment

1. A clearer understanding of the EAP construct being assessed: Construct underrepresentation (CU) and construct-irrelevant variance (CIV) as two major threats to test validity (see Messick, 1989, 1996)

2. The increasing role of technology in assessing EAP: Assessing EAP listening (input mode); assessing EAP writing and speaking (task delivery, automated scoring, e.g., Xi, et al., 2008; Xi, 2010); assessing integrated EAP ability and skills (e.g., Yu, 2013)

36

2015/6/3

7

复旦大学

Connecting EAP Teaching, Learning, and Assessment

1. Developing localized EAP assessments, and use them in combination with large-scale national/international EAP assessments (e.g., Fan, Ji, & Song, 2014; Fan & Song, 2015): Content & construct validity, relevance, and washback

2. The development of EAP/ESP ability descriptors which tailor to the needs of EAP/ESP teaching and learning in China (e.g., Jin, Zhang, & Zou, 2015)

3. The development of diagnostic EAP assessments, using the state-of-the-art Cognitive Diagnostic Assessment (CDA) models in language assessment (e.g., Read, 2008; Jang, 2009; Lee & Sawaki, 2009)

37 复旦大学

References

38

复旦大学

复旦大学

39复旦大学

复旦大学

Jinsong (Jason) FAN, Ph.D

Language Testing Centre, Fudan University

jinsongfan@fudan.edu.cn40