A Locally Appropriate English Language Test Locality, Globality & Validity

Dr Jessica Wu(吳若蕙)

R&D Program Director

The Language Training & Testing Center (LTTC)

Taipei, Taiwan

jw@lttc.ntu.edu.tw

Some definitions

• International tests: TOEFL, TOEIC, IELTS, Aptis, Cambridge English, etc.

• Locally-produced tests

• Which is more locally-appropriate? (Implicit assumption: Local tests

are more appropriate in assessing local learners because it is tailored to the specific educational systems and the contexts of test use)

• Test Localization (Adapting tests to the local context – presented by

O’Sullivan at 2014 New Directions)

“Localization is required when we are testing a well defined population within a well defined context in order to make decisions that will apply only to that context.” (O’Sullivan, 2013)

• Both local and international tests could serve as locally-appropriate tests.

2

NEAT/ TEPS

National English Ability Test/ Test of English Proficiency (Korea)

National Foreign Languages 2020 Project (Vietnam)

2020 Project

The Development of EFL tests in Asia

EIKEN － Test in Practical English Proficiency (Japan)

GEPT－General English Proficiency Test (Taiwan)

College English Test (China)

An Emerging Trend

3

English Education in Taiwan

An EFL context

• A compulsory subject starting Grade 1-3 up to University 1st year.

• To enhance the global competitiveness, English ability is demanded by schools and employers.

• Standardized English language tests are popular.

• Over one million test-takers per year take various kinds of standardized English language tests. (Taiwan population: 23 million)

4

Changing Landscape: Standardized English Language Tests

Test Description Test-takers in Taiwan/yr

GEPT A five-level EFL testing system designed to assess the general English proficiency of Taiwanese learners at all levels; developed by LTTC.

550,000

TOEIC Designed to measure the everyday English skills of people working in an international environment; developed by ETS.

300,000

IELTS Designed to assess the English language proficiency of people who want to study or work in an English speaking country; developed by British Council, Cambridge, IDP.

25,000

TOEFL Designed to measure the ability to use and understand English at the university level; developed by ETS.

23,000

5

Two major challenges

The GEPT is a locally-appropriate test? – Wishful thinking unless it can be supported by evidence through validation. Challenges

• Evidence supporting local relevance

• Evidence supporting international recognition

• Tension between locality and globality

6

Challenge 1

Locality: Reflecting macro and micro contextual features in test construction and implementation

• Culture

• Curriculum

7

Contextual mediation in test construction/revision

Educational and social features

Teaching & learning

objectives

(local curriculum)

Test construct,

content and format

WHY Communicative needs in social & educational systems

WHAT Knowledge, skills and abilities to be taught and learnt

HOW Test construction to positively impact teaching and learning

Impact by design (Saville, 2009; 2010)

8

Impact by design: Contextual mediation in the construction of the GEPT

Important features:

• Local curriculum

• Local cultural/social references

• Appropriate for local learners linguistically, visually, and conceptually (O’Sullivan, 2014)

• Increasing learners’ motivation and autonomy

• Helping learners to demonstrate language ability at their best

• Introducing positive washback

9

Measured Construct

“With an increased public expectation of transparent and explicit test, …., test providers need to satisfy the expectations of stakeholders (learners, employers, receiving institutions, professional bodies) concerning the comparability of the constructs measured by each test version in terms of both cognitive and contextual validity, and scoring validity.” (Weir, 2013: 3-4)

10

• Test content • Task requirement Evaluative criteria

11

Washback

The socio-cognitive framework for test development and validation (Weir, 2005; O’Sullivan, 2000; O’Sullivan & Weir, 2011)

Relationships with CEFR levels

12

GEPT Kids

GEPT HI

GEPT I

GEPT E

GEPT A

GEPT A

Grade 1-9 Curriculum Guidelines

National Curriculum Guidelines for Senior High Schools

13

A1 A1

Individualized Diagnostic Feedback

14

Based on a survey of 8,000 GEPT test takers who are currently enrolled in colleges or universities in Taiwan

Benefit Learning and Teaching

• When you were in high school, did you take the GEPT for the purpose of applying to universities?

• Also, 60% reported that what the GEPT was testing was very similar to what they had learned in high school.

15

Increase the Globality of the GEPT

• Following international quality standards (e.g., AERA/APA/NCME Standards, ILTA Code of Ethics, ALTE Principles of Good Practice, ISO-9001)

• Aligning the GEPT to internationally-recognized frameworks, e.g., the CEFR

• Carrying out validation studies

• Promoting exchange with the global community through research, academic, and educational networks

16

Recognized by UK General Medical Council

• The UK GMC reviewed a number of English language tests in terms of their suitability for use within the Council’s registration process for internationally trained doctors wishing to practice in the UK.

• IELTS Equivalence Research Project (GMC 133) Final Report: six alternatives can be broadly predictive of overall IELTS scores.

– Cambridge English: Advanced, CAE

– Cambridge English: Proficiency , CPE

– GEPT Advanced

– TOEFL iBT

– Occupational English Test, OET

– Pearson Test of English – Academic , PTE-A

http://www.gmc-uk.org/about/research/28134.asp

17

International Recognition of the GEPT

100+ universities around the world have accepted GEPT scores to admit Taiwanese students to various types of programs.

18

Challenge 2

Globality: responding to the changing context

• Free market society

• New assessment needs

19

All in One: Total Solution

• TOEIC/TOEFL (ibt & ITP) for high school students

• IELTS/Cambridge English exams

Applications for study abroad

programs

University entry &

graduation benchmark

Certification

• Achievement

assessment • Assessing learning

outcome • Monitoring teaching

effectiveness 20

TOEIC growth

208,867 230,173

114,133 122,673

2014 2015

TOEIC Numbers

Students Non-students

21

1. Market needs 2. Commercial promotion and operation 3. Easier to achieve at a certain CEFR level 4. Recognized globally

+15%

Which Test to Use

‘…to understand the working of political, institutional, economic, and practical factors in effecting the development of a test.’ (Measured Words – Spolsky, 1995) The decision of which standardized English test to use is affected by

• personal preference

• stakeholders’ dynamic views towards the exams 22

Attitudes and perceptions of teachers and students towards

the GEPT and international standardized tests

• Perceived values

– Assessing English ability

– Reflecting learning outcome

– Resembling learning/working contexts

– Culturally friendly

• Factors

– Group (students vs teachers)

– Educational level (JH, SH, U/C, AL)

Are the US/UK-made English language tests preferred due to country image? Country image is viewed as a halo that consumers use to evaluate product

attributes (Han, 1989). - e.g. the country image for French wine vs Japanese

wine; German cars vs Chinese cars

23

GEPT

IELTS TOEFL iBT

TOEIC

Resembling Learning or Working Contexts

Junior and senior high school students consider the GEPT to be more relevant to their learning/working contexts.

24

Assessing English Ability

More mature learners are more positive about TOEFL iBT and IELTS.

25

Most Recommended tests by Instructors/learners

26

Instructors Learners

Attitudes towards Exams

Overall, GEPT is well-known and the most cost-friendly exam in Taiwan, while international tests are considered to be of higher quality than the locally-produced one.

27

2

3

4

5

Brand namerecognition

Test qualityTest fee

GEPT IELTS TOEFL iBT TOEIC

Opportunity for international tests: Enhancing the usefulness of international tests in local contexts

How can the usefulness of international tests be enhanced in local contexts?

28

TOEFL/IELTS

(EAP-oriented)

TOEIC

(Workplace -oriented)

Support the internationalization of education in Taiwan; promote for use with

younger learners

Need to demonstrate:

Link to local context ( T/L in classroom;

content relevance to curriculum);

promoting learning (washback/impact)

International Cooperation Cases: Cambridge YLE Impact Study; British Council Aptis-GEPT Comparison Study Collaboration between test provider and local researchers – • Capture of relevant data • Deeper and more accurate understanding of the educational context • Fuller consideration of macro- and micro-level contextual parameters

Localization & Validation (O’Sullivan, 2014)

29

Demonstrating appropriate localization is a critical aspect of validation

Localization can only happen when we are aware of and take account of the test-taker in the context of the social domain

Localization is therefore a recognition of test consequence and social values as a critical aspect of the test development and validation process

Opportunity for locally-produced tests: ongoing validation efforts

• Promoting assessment literacy and communicating with

stakeholders. – Communication is essential for advancement (Fulcher, 2015; O’Sullivan, 2014).

• Increasing transparency; Being more open-minded (welcome and support external research)

• Cooperating with policy makers, teachers, test developers, and researchers – responding to the emerging needs for assessment

30

New assessment needs

• One-size-fits-all

• Understanding macro and micro contexts within the society where tests are being used

• Determining which elements facilitate/hinder the desired outcomes

Dynamical interplay between many sub-systems

(Saville, 2009 & 2012):

– social systems

– educational systems

– school systems

– individuals

31

Assessment in support of Internationalization of Higher Education

English is essential to the Internationalization of Higher Education.

• Graduation requirement

• EAP/ESAP courses

• EMI programs

Assessment should be better integrated with subject-specific teaching and learning – ESAP tests in business, engineering, medicine

32

Assessment in support of better learning – self assessment & autonomous learning

33

Assessments

Instruction Learning

Collecting information

Interpreting information

Making decisions about

Instruction or students

Using assessment as an instructional tool (involving students in assessment processes –

self-assessment)

34

Dr. Writing: Blended Learning

Student

Use the GEPT Online Marking System(OMS)

-Choose a writing task -Complete the task -Submit the essay

Teacher Use the GEPT OMS

-Mark the essay based on the GEPT criteria -Provide individualized feedback

Student Receive feedback via the GEPT OMS

-Rewrite [Submit a revised essay]

Teacher -Diagnose writing problems -Provide consultancy to facilitate learning

Online submission

Online instruction

Face-to-face instruction

Further instruction?

Teachers and learners dialogue around the assessment process and the criteria they apply to evaluate performance

Major types of errors

35

1. The six types of mistakes make up of over 93-96% of the errors.

2. Most of the errors were caused by the difference between Chinese, their first language, and English.

Types of tag

Word choice

Collocation errors

Incorrect use of words

Typos

Verb form Tense

Voice

Gerund/infinitive

Pluralising Incorrect use of plural or singular forms of nouns or verbs

Format errors

Wrong punctuation Lower-case/upper-case errors

Semantically unclear

Wrong prepositions

Final Note

Assessment is evidence-based practice.

International tests and locally-produced tests should consider both locality and globality as necessary components of validation rather than merely marketing claims.

36

Thank you!

37