1

全国高等学校英语教师教育与发展系列研修班

Statistics in action: A listening strategies project

25 July 2010

Victoria University of Wellington中国外语教育研究中心

顾永琦peter.gu@vuw.ac.nz

In this session…

Part 1: Feedback to your journals Part 2: Exercises 1 and 2 demo Part 3: Statistics in a listening

strategy project

Part 1

Your journals 收获 问题 建议

问题

运用实例，看什么研究问题用什么统计方法，生成什么样的图表，如何解读

一些关键术语还是模糊，如 nominal, ordinal, interval scales, p值，相关正负值

Comment ： realistic expectations

建议

加强练习与实际操作 SPSS操练手册？ 提供 references 提前把材料放在网上

Part 2

Exercises 1 and 2 demo

If you run Frequencies on the data, you get the descriptive statistics for all 44 cases, i.e., for both classes.

What you need is mean scores and SD for both classes.

You can get the descriptive statistics you want from the Explore command under Descriptive statistics.

Reporting your descriptive statistics

Class N 均值 标准差

Professor's Class 21 55.29 15.922

Graduate Student's Class 23 58.13 7.156

Interpreting your findings The mean score of the graduate student’s

class (Mean=58.13) was indeed higher than that of the professor’s class (Mean=55.29). In general, the graduate student’s class outperformed the professor’s class.

The professor’s class also had a much larger standard deviation than the student’s class, indicating that the professor’s class showed a wider spread of scores than the other class.

NB: you can also show the spread visually on a histogram.

Exercises 2

Correlation

Choose your options to include descriptive statistics

Report the descriptive statistics first

Report the correlation table

Interpreting your correlation findings

The findings showed that candidates’ abilities were positively correlated with the salary package offered to them. All four ability measures were found to be significantly correlated to the candidates’ annual salary.

The highest correlation existed between the candidates’ IQ scores and their salary (r=.844, p<.001). In other words, someone’s IQ score explains about 65% of the salary s/he earns. Communication score revealed the second highest correlation with salary (r=.813, p<.001)…

This is encouraging news for business school students, in that hard work, good marks, and an intelligent mind are indeed highly related to the salary they get after they graduate.

23

Part 3: Stats in action

3.1. Developing and validating the Strategy Battery for Young Learners of English (SBYLE)

3.2. Statistics in a listening strategy project

24

Strategy Battery for Young Learners of English (SBYLE)

Purpose: Profiling English language learner strategies used by upper primary school pupils in Singapore

1. Listening Strategy Questionnaire 2. Reading Strategy Questionnaire3. Writing Strategy Questionnaire

25

Designing the SBYLE The listening strategies questionnaire

as an example

26

Version numbers for the Strategy Battery for Young Learners of English

Version Purpose

1.0 Creation of questionnaire items

1.1 Validation by experts

1.2 Validation by school teachers

1.3 Validation by target sample (N=6)

1.4 •Piloting among large sample (N=293)•Ensuring internal consistency reliability

2.0 Final administration (N=3,618)

27

Version 1.0: Creation of questionnaire items

What was done Criteria1. Top-down: metacognitive,

cognitive, and social/affective

2. Bottom-up: Turn Phase I codings into questionnaire items.

3. List items in categories4. Find items in existing

literature, integrate them into what has been created, under existing or new categories.

1. Minimum 3, maximum 10 in each category, no single item variables. (strategies are latent variables: A strategy is not a single instance of behaviour)

2. Closed, 5-point Likert scale: Never to Always

3. Include a purpose, if possible, in each item. (strategies are purposeful)

28

Where do Questionnaire items come from? Phase I studies

Participants: 4 schools, all 6 grades in each school, one

high, one medium, and one low EL proficiency pupil from each grade (4x6x3=72)

Materials: Listening, reading, writing tasks Think-aloud protocols

Analysis: Coding and pattern exploration

29

Version 1.0 ExampleStrategy type Strategies Items

Metacognitive

Self-initiation

1. I look for opportunities to listen in English.

2. To improve my listening in English, I watch English TV programmes.

3. When I’m free, I find interesting things to listen to in English (for example, TV, radio, etc).

4. I try to find out how to improve my listening in English.

30

Validating the SBYLE

31

Validity checks Content validity

Consult experts, teachers, target group on 1) relevance 2) coverage 3) representativeness, and 4) exactness of wording

Construct validity Comparison with theory Factor analysis

Response validity Participating pretest: paired think-aloud

32

Reliability checksType of

reliabilityDecision

Internal consistency

reliability

yes

Test-retest reliability

no

33

Version 1.1: Validation by experts

What was done Guiding questions

Item-by-item examination and change

(What do we intend to cover?)

Conceptual coverage: •Do items under each category belong there? •Should there be other items?•Should there be other categories?

Wording: •Is wording for each item clear?•Is wording for each item simple enough to be understood by upper primary school pupils?

34

Changes made at this stage: Examples

Original Revised

When I listen to something, I try hard to understand every word before I can put these words together to understand the whole thing.

When I listen, I pay attention to every word.

When I listen, I try to form images in my mind for better understanding.

When I listen, I try to form pictures in my mind in order to understand better.

I try to relate what I hear to my own experiences in order to learn something from what I hear.

I try to see connections between what I hear and my experiences.

35

Version 1.2: Validation by teachers

What was done Guiding questions

Consultation with teachers

(intended meaning = perceived meaning?)

1. Is wording for each item clear? 2. Is wording for each item

simple enough to be understood by upper primary school pupils?

3. If the wording of an item is not clear enough, what alternative suggestions would you make?

36

Version 1.3: Validation by target sample (intended meaning = perceived meaning?)

What was done Guiding questions

1. Randomize items in each questionnaire

2. The formal version for small-scale piloting among 6 pupils item by item

3. Design answer sheets

1. Do pupils understand each item?

2. Are there items that are easily misunderstood?

3. Do pupils have problems with the answer sheet?

4. Are there any other snags?5. How much time is needed to

answer each questionnaire?

37

Version 1.4: Construct validity and internal consistency reliability

Formal piloting among 293 pupils (P4-P6)

Elicit student and teacher comments Revise, after factor analysis and item

analysis: delete, add, change, collapse categories, re-categorize items

Fine-tune survey administration procedures and detect problems (Guidelines for Questionnaire Administration)

38

Version 1.4: Reliability analysisVariable Item #

in pilot Q’naire

Item in original categoriesItem-Total correlation

Alpha if item deleted

Scale alpha

Inferencing 2

delete

When I don’t understand something, I use my knowledge about the English language to guess.

.371 .672 .696

6 When I don’t understand something, I use what I have already heard to guess.

.519 .623

22 When I don’t understand something, I make several guesses.

.395 .666

33 When I don’t understand something, I use my general knowledge to make a guess.

.538 .618

38

delete

When I don’t understand something, I use my knowledge about text structure to help me understand.

.218 .721

41 When I don’t understand something, I use my knowledge about the topic to guess.

.539 .618

39

Deleted item: #38Item in original categories Item-Total

correlation

Alpha if item deleted

When I don’t understand something, I use my knowledge about text structure to help me understand.

.218 .721

40

Deleted item #2

Item in original categories

Item-Total correlation

Alpha if item deleted

When I don’t understand something, I use my knowledge about the English language to guess.

.371 .672

Obtaining variables from individual items

COMPUTE LSELFINI=(L11+L12+L14 +L18)/4. COMPUTE LPLAN=(L3+L15+L23)/3. COMPUTE LMONITOR

=(L21+L32+L5+L9+L36)/5. COMPUTE LPERCEPT =(L8+L24 +L34+L37)/4. COMPUTE LINFER =(L1+L4+L17+L26

+L31+L27+L33)/7. COMPUTE LPREDICT

=(L6+L7+L13+L29+L35)/5. COMPUTE LUTILISE

=(L16+L19+L20+L22+L30)/5. COMPUTE

LSOCIAL=(L2+L10+L28+L25+L38)/5.

Running item analysis and reliability in SPSS

What we are doing:1. Checking if items that make up a variable

are consistent among themselves.2. Doing item-total correlation to see how

each item in the variable is correlated to the total variable score.

3. Determining which item should be removed from the variable (alpha is item deleted)

Example: the monitoring variable LMONITOR was made up of 5 items L21, L32, L5, L9, and L36.

Getting internal-consistency reliability

Locate all items in the variable

Specifying the statistics you want

Deciding if any item does not belong to the variable

48

Version 1.4: Problems and solutions

Administration problems (N=293)

Solutions

1.Pupils don’t remember ID numbers

2.Wrong understanding of “Grade”

3.Wrong answer sheets used

1.Ask schools to remind students in advance

2.Provide P4/P5/P6 choices3.Tell Teachers in advance

that different questionnaires require different answer sheets

49

Version 2.0: Final Administration

Number of Pupils

Percent

School 1 629 17.4School 2 403 11.1School 3 856 23.7School 4 762 21.1School 5 440 12.2School 6 528 14.6Total 3618 100.0

Statistics in the listening strategy project:

Deciding what stats to use Research question Statistics1. What is the listening strategy

use pattern among Singapore upper primary school pupils?

Descriptive statistics

2. How are listening strategies related to English language results?

Correlation, Regression (strength of associations between the two vars) orANOVA (locating differences between top, middle, & bottom groups)

3. Do strategies change from grade to grade?

ANOVA (differences among three grades)

Reporting descriptive statistics:

The table format

Interpretation and textual report

Listening strategies and English grades

Textual interpretation

Run your own item analysis and Chronbach’s alpha in SPSS Items 15, 17, 14, 25, 5, 8, 11, 34, 26 Items 1, 2, 3, 6, 18 Items 24, 29, 16, 19 Items 28, 27, 21 Items 30, 31, 32 Items 7, 10, 4, 33, 20 Items 23, 12, 22