1 classroom-based diagnostic assessment: practices … · getting background info about their...
TRANSCRIPT
Classroom-based Diagnostic Assessment: Practices of EFL Instructors
Hossein Farhady ELT Department, Yeditepe University, Istanbul, Turkey
Merve Selcuk School of Foreign Languages, Istanbul Kemerburgaz University
EALTA SIG 2016 Valencia SPAIN
23.05.2016
1
Diagnostic Assessment (DA)
“The ability to interpret students' foreign language growth, to skillfully deal with the assessment material and to provide students with appropriate help in response to this diagnosis" (Edelenbos&Kubanek-German, 2004, p.260).
“Informal diagnostic assessment occurs on a regular basis in the form of student questioning, explanation, and the provision of written feedback on quizzes, tests, and written work" (Blood; 2011)
"Diagnosis is the teacher’s concern of what has gone wrong" (Kunnan & Jang, 2009, p.6) to identify a learner’s strengths and weaknesses (ALTE, 1998; Lee, 2015)
23.05.2016
2
DA
23.05.2016
3
Alderson et al., (2014) have examined diagnostic assessment process by collecting data from diverse fields of professions and vocations by semi-structured interviews and have mapped out a theory of diagnosis in S/FL assessment.
Their framework attempts to identify essential
behavioral patterns that teachers exhibit whilst implementation of DA stages with the hope of providing guidance for instructors to ensure effective implementation of DA.
Principles of DA
23.05.2016
4
1. it is not the test that diagnoses, it is the user of the test. (…)
2 (…) instruments themselves should be designed to be user-friendly, targeted, discrete and efficient in order to assist the teacher in making a diagnosis. (…) designed or assembled (…) by a trained classroom teacher (or other experienced language professional), and should generate rich and detailed feedback for the test-taker (…) with a specific diagnostic purpose in mind. (…)
DA Framework
3. (…) DA process should include diverse stakeholder views, including learners’ self-assessments. (…)
4. (…) DA should ideally be embedded within a system that allows for four diagnostic stages: (a) listening/observation, (b) initial assessment, (c) use of tools, tests, expert help, and (d) decision-making (…)
5. (…) DA should relate, if at all possible, to some future treatment. (Alderson et al., 2014, p.20-1)
23.05.2016
5
Research Question
23.05.2016
6
This study focuses on DA practices of
experienced EFL instructors with the framework suggested by Alderson et al., (2014) with specific reference to reading skill at the tertiary level EFL classroom context.
1. What are the implementation procedures of expert EFL instructors in DA process regarding reading skill?
Characteristics of Participants
Gender F 12 M 5
Degree PhD 8 MA 15 Certificates CELTA/DELTA 2
Place of teaching
University Prep 16
Others 1
Years of Experience
5-7 years 7
8-14 years 3
More than 15 years 7
Major ELT 12 Translation &ELL 4
Others 1
23.05.2016
7
Instruments
23.05.2016
8
3 scenarios
7 questions for each scenario
Questions addressed the stages of DA
For stage 1 1 question For stage 2 1 question For stage 3 1 question For stage 4 4 questions
Questions for Each Stage
23.05.2016
9
DA Stages
Questions
1. Listening/ Observation
1. What would you do to obtain more in-depth information about the student’s detected problem?
2. Initial Assessment
2. What would be your initial hypothesis about the errors or mistakes?
3. Hypothesis checking/use
3. How would you test your hypothesis? (What specific ways tools, tests, expert help) would you follow/what specific tasks would you use to make sure that your hypothesis is correct?)
4. Decision making
4. What would you do after diagnosing the problem? (Assuming that your initial diagnosis is correct, note down all possible ways you could follow) 5. If you have decided to make an intervention, what kind of intervention would you employ? 6. How would you check the effect of intervention? 7. How would you approach the students who did not benefit from intervention?
Instrument
23.05.2016
10
Scenario 2 Problem Represented You are teaching reading skills in an advanced level EFL class at the tertiary level. You observe that one of your students seems to be reading word by word and appears to look up every word in the dictionary. He understands single sentences but he cannot answer questions related to finding the main idea and supporting details. Nor can he perceive the author's attitude type of questions.
The student cannot create a text level representation
Data Analysis
23.05.2016
11
Open coding Bottop-up approach Coded by one researcher
Findings
23.05.2016
12
Almost all participants consistently follow the 5 principles of DA. For stage 1 and 3, which are allocated to data collection, teachers utilize multiple sources of information to develop a more comprehensive picture of the problem. They come up with their own innovative tasks, procedures and solutions by paying attention to contextual and affective factors. They offer more than one solution to the same problem. This refers to the creativity of teachers. If their plan does not work at the end of DA cycle, they digress to one of the earlier stages of the cycle to readdress the problem.
Results
23.05.2016
13 Stage 1. Listening/Observation Q.1. What would you do to obtain more in-depth information about the student’s detected problem?
S P
1 Check the material appropriacy (level, vocab, content) Needs analysis Raise awareness on the issue Class survey Open ended questions about how they read the text Teacher made questionnaire based on predicted reasons & relevant literature (e.g. I find reading enjoyable/boring; I know many of the words/do not know many of the words; about the techniques they used; What would you like to read about, etc.,) Anonymous notes about their problems Do some trial and error with some different exercises, e.g., Find passages related to their interests, see if the problem persists. Teacher made diagnostic test Test and think aloud protocols Interview/ Talk to the students individually Translation A reading exercise and whole class discussion based on the exercise A test and retrospective recall
4 1 1 1 1 3 2 2 2 1 7 3 3 1
Results
23.05.2016
14 Stage 1. Listening/Observation Q.1.What would you do to obtain more in-depth information about the student’s detected problem?
S P P.
2 Check the material appropriacy (level, vocab, content, culturally appropriate) Diagnostic reading test Interview Needs analysis A similar task with think aloud Translation Give sts different texts, observe them, take notes to see whether they exhibit the same behaviour consistently Think aloud Design tasks (more texts and similar tasks) Check other top-down skills Can-do statement checklist Do trial and error with some tasks Check reading speed Getting background info about their experience in learning English Questionnaire
4 2 1 1 1 1 1 1 3 1 1 7 1 1 1 1
3 Check the material appropriacy (level, vocab, content, culturally appropriate) Diagnostic reading test Interview Needs analysis Read aloud Use more literacy-focused teaching materials to understand what the level of difficulty Developing a diagnostic questionnaire based on observed weaknesses Check students papers to decide on strong and weak points Professionally prepared RC test (placement) to understand student's level Reading habits questionnaire Getting background info about their experience in learning English (learning style, their experience in learning English, which English courses they took, which skills they focused on)
1 4 11 1 1 1 2 2 1 1 3
Findings
23.05.2016
15 Stage 2.Initial assessment Q2.What would be your initial hypothesis about the errors or mistakes?
S P
1 Motivation Cannot fulfill higher level processing Insufficient BICS knowledge (Vocabulary+grammar knowledge) Lack of reading strategies Content Being exam oriented (reading for the questions vs. reading for the main idea)
8 1 4 9 3 1
2 Lack of content vocabulary Problems with lower level skills Limited surface level reading Lack of reading strategies (guessing word meaning, main idea) No critical thinking Insufficient practice Lack of guessing word meaning They are Analytic learners
2 7 6 1 1 1 1
3 Mistakes/errors on orthographic control (letter sound correspondence) Open-ended questions may cause students to make more mistakes Lack of Reading skills Need advanced gr practice Educational background Langauge proficiency L1 intervention Lack of reading practice
12 1 5 1 1 1 7 1
Findings
23.05.2016
16 Stage 3. Hypothesis checking/ use Q3. How would you test your hypothesis? (What specific ways of tools, tests, expert help,
would you follow/what specific tasks would you use to make sure that your hypothesis is correct?)
S P
1 Task and class discussion /activity observation Think aloud protocols One-to-one interviews An experimental study Make a survey Give another similar task/texts and observe if the problem persists
3 3 3 3 1 4
2 Diagnostic Test (Reading speed test 2, Eye tracking 1, Vocab 1) Action research Interview Awareness raising activity More tasks/activities/questions regarding top-down skills Give him easier level of passages with easier tasks Give a similar task and try to observe difficulties A reading text but with some technical unknown words+ Qs measuring main idea Experimental study (Assuming they have vocab problem, give them an easy and hard text and compare the results + A reading text but with some technical unknown words and another easy task with the same questions to see the difference) Test-retest Think aloud
4 1 5 1 5 1 1 1 2 1 1
Results
23.05.2016
17 Stage 3. Hypothesis checking/ use Q3. How would you test your hypothesis? (What specific ways of tools, tests, expert help, would you follow/what specific tasks would you use to make sure that your hypothesis is correct?)
S P.
3 Think aloud The same test in L1 RC test Timed reading exercise One to one interviews Give a text and observe students on task. Give the same task as a recording and compare the results Action research Observe students in small groups of 5-6 Read aloud Slower reading (Giving students a standardized test to understand their level in the previous stage ), analyze the results of each item to understand what type of questions they are struggling with Compare these students' performance with the other same proficiency level students' performance Reading vocab and gr tests Observe to elicit their lang. prof. (in other skills as well)
1 1 1 1 1 1 1 1 1 1 1 1 1 1 3
Findings
23.05.2016
18 Stage 4.Decision-making Q4.What would you do after diagnosing the problem? (Assume that your first
diagnosis is correct and note down all possible ways you could follow)
Sce. part.s
1 Instruction (private tutoring) Practice &Tasks assignments Instruction and guidance One-to-one meeting Design VAKOG-relevant instruction Make changes, modify the syllabus
8 9 6 3 1 4
2 Instruction (private tutoring) Practice &Tasks assignments One-to-one meeting Experimental design (two classes) Extensive reading Strategy training
4 5 5 1 4 5
3 Instruction (private tutoring) Practice &Tasks assignments Practice &Tasks assignments with guidance Extensive reading Spelling exercise Passages supported with audio recordings Awareness raising activities for reading skills Apply Kolb's experiential learning cycle
5 11 3 2 5 1 1 1
Findings
23.05.2016
19 Stage 4. Q5. (If you have decided to make an intervention) What kind of intervention would you employ?
S P
1 Adapt&design tasks & assignments and guide ss during/after the exercises (observe) Extensive reading (periodical, based on ss’ interests, go over them together for strategy training) Strategy training Grammar reinforcement Talk to the student, convince him (for the importance of learning strategies) and make a tailored study plan for the student Give students a reading text in Spanish or Italian and have them find specific info (numbers, names) in a text written in a language they do not know just to show that they can do it! Give students a degree of choice for the materials and activities Scaffolding: Similar tasks with easier passages/easier tasks/easier shorter readings to help sts gain motivation and confidence and then increase the level of difficulty, gradually difficult texts
6 3 4 2 4 1 4 3
2 Tasks & assignments and guide sts during/after the exercises Extensive reading Strategy training Give an easy passage in which some words are replaced by pseudo words and show the student she can still grasp the meaning without knowing the words Peer teaching (activities & strong ones help share their strategies with the weaker ones) Working on students’ writing to integrate other skills in the comprehension process Scaffolding: Similar tasks with easier passages/shorter texts and tasks
5 2 3 1 1 1 3
3 Tasks&assignments and guide sts during the exercises Extensive reading Read aloud exercises practice (seeing hearing together) Grammar teaching Strategy training Sub Skills training
2 2 1 3 4 3
Findings
23.05.2016
20 Stage 4. Q6. How would you check the effect of intervention?
Sce. P
1 Informal assessment (group discussions, observation, asking questions etc., periodically) Post-test Feedback from analysis of assignments, homework, classwork (reading text and activities) Self assessment checklists Peer assessment checklists A new text and focus groups on how they arrived at the answers Focused interviews on the effect of the intervention
10 6 1 1 1 1 1
2 Informal assessment (group discussions, observation, asking questions etc., periodically) Post-test/re-test Feedback from analysis of assignments, homework, classwork (reading text and activities) Ask student to write a reflection Design productive tasks (writing & speaking) with specific criteria and rubrics The same test in L1 Interview
7 9 4 1 3 1 1
3 Informal assessment (group discussions, observation, asking questions etc., periodically) Post-test Feedback from analysis of assignments, homework, classwork (reading text and activities) Keep a teachers diary Compare post-test results with other international students' results Think aloud Professionally, internationally recognized tests Aptitude tests Reflection from learners
13 7 5 1 1 1 1 1 1
Finidngs
23.05.2016
21 Stage 4. Q7. How would you approach the students who did not
benefit from intervention? Sce. p
1 Tutorials remedial classes Extensive reading Interview/negotiate with the sts Interview+a diagnostic reading test More extra materials, more practice Reconsider my hypothesis, look at the issue from a different perspective, retstart the cycle, Analyze their learning preferences Dig more on their background experience Questionnaire Survey Give more time to the sts
1 1 6 1 5 3 1 1 1 1 1
2 Tutorials remedial classes Extensive reading Interview/negotiate with the sts / ask if they they have other weaknesses I have not observed More extra materials, more practice Find another way other intervention method (go back to stage 3) Adopt humanistic approach (Give more time to the st., ask him to be patient and keep working with him) Awareness raising activities, tasks Lower level materials scaffolding
3 1 4 4 4 3 1 1
3 Tutorials, remedial classes Extensive reading Interview/negotiate with the sts More extra materials, more practice Reconsider my hypothesis (go back to the first stage) Adopt humanistic approach (encouragement, talk to the student, motivate to study
)
1 2 8 2 4 6
Discussion
23.05.2016
22
Principle 1. Alderson (2005) suggests that all tests could be used for diagnostic purposes, but requires a skilled diagnostician.
Q3. “Like I said, I would give them professionally written evaluations. These exams are proven on an international standard. They’ve had millions of dollars invested into their accurate assessing abilities, so they can be trusted to give me an accurate read on my students’ levels. Also, I would assess the students in a detailed way. I wouldn’t just get results on ‘reading comprehension’. Rather, I would get detailed analysis of what types of questions they struggled with. Can they read for purpose, main idea, details and facts, work out the meaning of words they don’t know, paraphrase etc.” Neville
So the instructor uses a standardized test but he does not use the final score, he utilizes or exploits the test for diagnostic purposes and uses the results in a problem solving way.
Discussion
23.05.2016
23
Principle 2: As suggested by Alderson et al., (2014) instructors design targeted, purposebuilt diagnostic tools, selected from a bank according to purpose .
All teachers in our study assert that they design specific tools and instruction tailored for individual students.
Principle 3: Alderson et al., (2014) the diagnostic assessment process should include diverse stakeholder views, including learners’ self-assessments. Our findings reveal that the instructors use can-do statement self assesment forms and interviews to identify learner weaknesses and involve them in DA process. "Often, students are aware of the reasons for the breakdown that is occurring in their reading comprehension." (Neville)
Principle 5: All participants face several suggestions for future treatment as envisaged by Alderson et al. (2014)
Conclusion & Implications
23.05.2016
24
Teachers spend little amount of class time on diagnostic assessment (Edelenbous and Kubanek-German, 2004), the results could also be used to raise awareness. Workshops on helping teachers develop skills on implementing processes of DA can be beneficial. Participants I this study were mainly at MA and PhD levels. It would be interesting to investigate how high school teachers deal with DA. Teacher training programs could benefit from such studies and implement modules on DA.
References
23.05.2016
25 Alderson, J. C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. A&C Black. Alderson, J. C., Brunfaut, T., & Harding, L. (2014). Towards a theory of diagnosis in second and foreign language assessment: Insights from professional practice across diverse fields. Applied Linguistics, amt046. ALTE (1998). Multilingual Glossary of Language Testing Terms. Cambridge: Cambridge University Press. Blood, I. (2011). Diagnostic Second Language Assessment in the Classroom. Teacher’s College, Columbia University Working Papers in TESOL and Applied Linguistics, 11(1). Edelenbos, P., & Kubanek-German, A. (2004). Teacher assessment: The concept of 'diagnostic competence'. Language Testing, 21(3), 259-283. Harding, L., Alderson, J. C., & Brunfaut, T. (2015). Diagnostic assessment of reading and listening in a second or foreign language: Elaborating on diagnostic principles. Language Testing, 0265532214564505. Kunnan, A. J., & Jang, E. E. (2009). 32 Diagnostic Feedback in Language Assessment. The handbook of language teaching, 610.
THANK YOU!
Appreciate your comments
23.05.2016
26