chapter 11+12faculty.ecnu.edu.cn/picture/article/220/0a/d4/9eb... · ecnu educational research...

ECNU Educational Research Methodology

Sampling design Measurement and data collection

Chapter 11+12

2016/12/7 1 陈芳华东师大英语系


Plan

We have discussed many different research designs and their advantages and disadvantages.

Now, we will assume that you know which

design you will be using and we will learn how to actually go about getting our subjects, collecting data, analyzing it, and making conclusions.



Chapter 11

Sampling Design



Good Sampling

In general good sampling is

1. Goal Oriented – tailored to design 2. Measurable – provides needed data 3. Practical – it is feasible 4. Economical – have resources

Often, some of these are compromised. The

important overall criterion is that the design be feasible and adequately accommodate the research problem.



1. Random sample 随机抽样（suiji chouyang） This was typically in experimental or quantitative

research ①Simple random sampling ②Stratified random sampling ③Cluster sampling ④Systematic sampling

2. Purposeful Sampling 目的抽样 (mudi chouyang) We presented this idea for qualitative research

mainly ethnographic.

Introduction

2016/12/7 陈芳华东师大英语系 5


1. Random Sampling

Random Sampling or Probability Sampling In this case all members if the population have

some probability of being selected • These probabilities are not necessarily all equal but they

are non-zero

Computer software makes the sampling

process rather easy



SPSS function for simple random sampling



Random selection vs. random assignment We could randomly select people but not randomly

assign them to groups We could also have a sample (not random) and

randomly assign them to tasks. (think of a classroom situation)

Some terminologies



More terminologies

After we have a sample we must differentiate the difference between Sampling Error: Random

Sampling Bias: systematic



② Stratified Random Sampling 分层随机抽样

Imagine that we have a sample that consists of: 50% Subpopulation A 40% Subpopulation B 5% Subpopulation C 5% Subpopulation D



Two types of stratified sampling

We want a sample of 100. 1. Non-proportional allocation

a) randomly select 25 people from each strata --- equal allocation

b) select more persons from the stratum where there is more variability: maybe 30, 30, 20, 20

2. Proportional allocation: 50 people from strata A 40 people from strata B 5 people from strata C 5 people from strata D



Example of proportional allocation

Unless a large survey is being conducted, stratification seldom involves more than two stratifying variables, and the total number of strata would seldom exceed 20 and usually is considerably less.



③ Cluster Sampling

Clusters / strata. In cluster sampling we first divide the population

into clusters and randomly sample from the clusters. However, once the cluster is selected, all the members in it are selected. This is the difference between cluster sampling and stratified sampling.

Another difference is that the sample size is usually unknown until after the sample is selected because clusters are usually not of the same size. 2016/12/7 15 陈芳华东师大英语系


Book example

Clusters may be City

blocks Schools Counties



④ Systematic Sampling

This last method of sampling requires that the individual already has a list of people Imagine an alphabetical list or something like that; Compute the sampling fraction n/N=1/k Once we know k, we randomly select someone

from the first k people Starting with that person, we select every kth

person Once the first person has been selected the whole

sample is determined



Periodicity: Every kth member of the population has some

unique characteristics that are related to, or have an effect on, the dependent variable. In that case, the sample becomes biased.

Student name list ordered by ability within each

class.

Pro: this method requires less work.

Systematic Sampling and periodicity



Sample size

1. It is related to costs and time involved. Practicality

2. It is related to statistical precision and statistical power.



Statistical Precision and Statistical Power

Statistical Precision Larger sample size, better precision. Confidence interval.

Statistical Power Larger sample size enables us to be more likely to

find a difference when there is difference.



1. Sudman 1976: for regional or special surveys (thesis or dissertation) with few subgroups, 200-500 is typical.

2. The more variable the dependent variable the greater the number required in each group.

3. e.g. 8, or 10, 15-30 per group may be adequate. For two groups, maybe totally need 100.

4. Factorial, 10 or more in each cell. 5. …… 6. Kraemer & Thiemann 1987: How many

subjects? Statistical power analysis in research.

Guidelines?



2. Purposeful Sampling

The other kind of sampling we discussed was purposeful sampling when we want to find a sample that will be most informative for our purpose and a random sample may not make sense or may be infeasible. ① E.g. Impossible logistically to select a random

sample from all graduating high school. ② E.g. Unethical to assign randomly patients for

treatment or nontreatment. Ethnographic research, Generalizability and external validity?



Purposeful Sampling

There are 6 basic kinds of purposeful sampling discussed in the book Comprehensive： e.g. all handicapped Maximum Variation: Use a set of schools that are all quite

different (low SES, Medium SES, High SES) Extreme Case Typical Case Homogeneous Other

Examples?



Extreme Case Sampling

Make selections based on unique or special cases of a particular characteristic.

The logic of using extreme case sampling is

that what is learned from the extreme cases may be applied to typical cases. E.g. Studying a new method for school

• Could use a traditionally low achievement school



Typical Case Sampling

This is the opposite of Extreme case sampling. So an example of the average case is used Going back to the school example

• One with average performance is used



Homogeneous Sampling

A sample is selected to represent a specific subgroup of people In studying teachers, only beginning teachers are

used



Sample Size

Unlike random sampling, we are not as concerned with sample sizes

Typically, they are small and are selected for their specific purpose



Summary

So we have specified 4 types of random sampling (simple, stratified, cluster, systematic)

And we specified 6 types of purposeful sampling (Comprehensive, Maximum Variation, Extreme Case, Typical Case, Homogeneous, Other



Chapter 12 Measurement and Data Collection



Introduction

What is measurement? Concept of measurement and measurement scales Reliability of Measurement Validity of Measurement Tests and Inventories used for measurement



Measurement

Kerlinger (1986, p.391): Measurement is a process of assigning

numerals to events or objects according to rules.

Examples: Achievement Attitudes Intelligence



Measurement Scales

Depending on the rules and the properties of those numbers our measurement scale can be one of four types Nominal: just a name Ordinal: there is rank Interval: equal distance Ratio: true zero


Nominal Ordinal Interval Ratio

ECNU Educational Research Methodology Absolute zero means absence of electron movement, thus no heat

2016/12/7 33 陈芳华东师大 FANG CHEN ECNU


Measurement Characteristics

There are two basic characteristics that we use to characterize the quality of our measurement Reliability – Consistency of Measurement

Validity – Quality of Measurement (i.e., are we

really measuring what we mean to measure)



Reliability

We have many different ways to measure reliability empirically

Parallel forms Test-Retest Split-Half Kuder-Richardson Cronbach alpha



Reliability

While reliability varies depending on what is being measured, most of the time we aim for about .85 although Achievement is a little higher Psychology is a little lower

Correlation analysis; Factor analysis…



Validity

Are we measuring what we want to measure?

Two parts: Internal – Can we make the conclusions

that our experiment is intended to show External – Generalizability



Validity

Unitary concept + a matter of degree Content-Related: content sampling Criterion-Related Evidence

• Concurrent • Predictive

Construct-Related Face validity Consequential



Variables Measured in Education

There are several types of variables, typically quantitative, that are measured frequently in education. ① Achievement Tests in Academic and Skills Areas ② Attitude Inventories ③ Aptitude Tests ④ Rating Scales ⑤ Observation Systems



① Achievement tests

Achievement is one of the principal outcomes of schooling.

They come in three forms Norm-referenced (compared to others in the

normative group) Criterion-Referenced (compared to the standards) Proficiency (only testing a certain level of

proficiency)



② Attitude

Achievement tests generally are designed to measure an individual’s best or maximum performance, whereas an attitude inventory is intended to measure typical performance.

Attitude is toward something.

Measurement of the attitude is intended to place individuals on this continuum.



Attitude

Because attitude is usually thought to exist on a continuum these tests usually are Likert Scales Semantic Differential

• http://psc.dss.ucdavis.edu/faculty_sites/sommerb/sommerdemo/

In either case, if writing your own you should

pilot the test and evaluate the items and its reliability



1. Using the right response option The ABC institute had a very positive impact on my

teaching. • Strongly disagree --- Strongly agree

Rate the extent to which the ABC institute has had

a positive effect on your teaching. • None --- Some --- Much

Issues in Likert scale item writing



2. Using the right scoring SA A U D SD It is fun to work with math puzzles. I do not like to work fraction problems.

Issues in Likert scale item writing continued



3. Items with little variance will not differentiate between individuals or groups; may also prevent programs from running.

4. Items that are frequently omitted may be unclear or offensive to the respondents.

5. Fake responses.

6. The middle-of-the-roader.



The semantic differential provides a series of bipolar adjective scales relative to a word or concept; the respondents indicate their feelings on the continuum of each scale.

Format Scoring: Remember your scale

Semantic differential



③ Aptitude

These tests are designed to measure the potential for achievement. Intelligence tests General scholastic ability General mental ability



Other Measures

④ Rating Scales

⑤ Observation Scales • High Inference: • Low Inference: • Low inference observations tend to be

somewhat more reliable than high-inference observations because less judgment is involved.



Where to Find These Tests

Mental Measures Yearbooks Tests In Print Buros Desk Reference: Psychological

Assessment in the School ETS Disertations Abstracts International And many others…



Variables Measured in Education

Published tests are usually well constructed and should come with a reliability measure and other details Mental Measurements Yearbooks (MMY) Tests in Print (TIP)



Scoring and Data Prep

Once we have collected the data we will need to code it or score it before any analysis.

Codebook is an important part of quantitative research and analyses. It is the notebook to record the type of numerals you have assigned to each variable in your data. They have to be interpreted back into words for your conclusions.



Weighting http://applied-survey-methods.com/weight.html


chapter 11+12faculty.ecnu.edu.cn/picture/article/220/0a/d4/9eb... · ecnu educational research...

Documents