experiments types of experiments laboratory experiments experiments of short term duration and...
TRANSCRIPT
Experiments
• Types of Experiments
• Laboratory Experiments• Experiments of short term duration and usually conducted in a lab
under very controlled conditions.• Commonly used designs
• Classical Experimental Design• Solomon’s Four Group Design• One-shot pre-test/post-test• One shot post-test
• Field Trials or Field Experiments• Experimental methods used in the “field” or in the “real world”• Can use any of the designs listed above
Laboratory Experiments• Purpose
• For a Relationship to be “causal” then• The independent variable (X) has to come before the dependent
variable (Y) in time (X before Y)• X and Y have to be related• There is no “Z” variable that explains the relationship between X
and Y
• Laboratory Experiments help us establish a causal relationship because we can• Control the time order of X and Y• Control for any “Z” variables (other influences)
Laboratory Experiments• Example
• Question – Is there a relationship between a new fertilizer and crop yield?
• For a Relationship to be “causal” then• Introduction of the fertilizer has to come before increase in crop
yield. (X must come before Y)• Fields where the fertilizer is used must yield a greater crop yield than
those fields using no fertilizer. (X is related to Y)• Relationship between fertilizer and crop yield is not explained by
more rain in some fields than others. (No “Z” variable)• Plant corn in a controlled laboratory environment. Apply fertilizer to
some sections and not to others. Put equal amounts of rainwater on all sections.
• Measure corn yield in different sections and compare across sections.
Experiments and Evaluation Research • Steps
• Step 1• Identify independent and dependent variables.
• In evaluation research, the independent variable would be the intervention (i.e., the workshop, program, etc.)
• In evaluation research, the dependent variable would measure the objective/impact you want to achieve.
• Step 2• Develop indicators to measure these variables
• Independent variable – Construct questions to measure participation in the program (i.e., ask participants if they participated in program and/or how many hours they participated).
• Dependent variable – Construct questions to ask participants about their behavior/attitudes before/after participation.
Experiments and Evaluation Research
• Steps• Step 3
• If possible, recruit a control group (similar group that does not participate in program)
• Step 4• Administer a survey to experimental and control groups
BEFORE the Experimental group participates in the program.
• Step 5• Implement the program.
Experiments and Evaluation Research
• Steps• Step 6
• Administer a survey to experimental and control groups
AFTER the experimental group participates in the program.
• Step 7• Input data and using statistical analysis compare across groups
• Step 8• Alternate Plan
• If plan works
Classical Design• Experimental Group
• Pre-test• Measure the dependent variable
• Introduce intervention • Post-test
• Measure the dependent variable again
• Use a Control Group
• Pre-test• Measure dependent variable
• Post-test• Measure dependent variable again
• Compare post-test for control group and experimental group
Classical Experimental Design
Experimental Procedure
Pre-test Intervention Post-test
Experimental Group Y X Y
Control Group Y Y
Classical Experimental Design and Evaluation Research
• Step 1 - Divide participants into experimental and control groups.
• Step 2 - State objectives/impacts and develop intervention (i.e., program, workshop, etc.) to meet those objectives/impacts. (Please Note – Objective should have something to do with increasing/decreasing the level of your Y variable.)
• Step 3 - Measure Y for both groups BEFORE intervention
• Step 4 - Expose experimental group to intervention
• Step 5 - Measure Y for both groups after experimental group has been exposed to intervention
• Step 6 - Compare Y for both the experimental and the control group – before and after intervention. Then compare Y before and after intervention for the experimental group.
Classical Experimental DesignCAUTION
A MAJOR limitation of the Classical Experimental Design is that you have only established the impact of X on Y in a laboratory setting. You don’t know if X will have the same impact on Y in the “real world.”
ANOTHER Modification of Experimental Design
Solomon Four Group Design has an important advantage
over the Classical Experimental Design.
Advantages
It can help you determine if measuring the pre-test can actually have an impact on the post-test.
For instance - IF you take the SAT a second
time, will already having taken the test impact
your score the second time?
HOWEVER, there are important disadvantages, namely
Takes more Time
Takes more Money
A Quick Visual Comparison of the Two Designs
Experimental designs have HIGH internal validity….in other words, they tend to yield accurate results under the specified conditions. HOWEVER, to ensure high internal validity, you should remember the following:
HistoryAnything that happens between the pre and post-test can
potentially influence the post-test (i.e., the room gets warm!)
MaturationParticipants get tired or may actually get older and even die
between the time of the pre and post-tests.
TestingTaking the pre-test can actually influence how participants do on
the post-test – even in the laboratory.
Instrumentation In a lab, you must ask accurate questions if you are going to get
accurate answers.
Experimental designs have HIGH internal validity….in other words, they tend to yield accurate results under the specified conditions. HOWEVER, to ensure high internal validity, you should remember the following:
Causal Time OrderYou have to control the introduction of X and Y in the right order,
they won’t just “happen” in the right order.
Diffusion or Limitation of Treatment If the experimental and control groups interact during this time, they
can influence the results of the other.
Compensation It is important that you do not “feel sorry” for the control group,
because they don’t receive the intervention, and thus inadvertently treat them better/different than the experimental group.
Compensatory Rivalry If the control group discovers they are NOT the experimental group
and thus feel deprived, they may “try harder” to do better than the experimental group.
Experimental designs have HIGH internal validity….in other words, they tend to yield accurate results under the specified conditions. HOWEVER, to ensure high internal validity, you should remember the following:
Statistical Regression to the MeansExtreme answers on the pre-test will tend to yield less extreme
answers on the post-test regardless of what happens between.For instance, individuals who score a perfect score (1600) on the SAT are likely to score slightly less if they take it again – even if they study in the meantime.
DemoralizationThe control group may become demoralized when they realize they
aren’t receiving the “treatment,” and give up.
Experimental BiasThe experimentalist may want to obtain certain results and
consciously or unconsciously act in ways that will increase the probability of receiving the desired results.
Field Trials OR Field experiments are Similar to Laboratory Experiments EXCEPT:
They are conducted in the “real” world – not in laboratories.
They are generally used when, for practical reasons, laboratory experiments cannot be used. For instance:
When the program or intervention lasts longer than a few hours and/or cannot occur within a laboratory.
When participants cannot/won’t come into a laboratory.
They generally take considerably longer than laboratory experiments, and in fact, it may take months or years to conduct laboratory experiments.
Field Trials or Experiments
Example of a FAMOUS Field Experiment
The Perry Preschool Field Experiment
In the 1960s, low income children in the US went to Head Start.
Studies were done to evaluate the program. First researchers compared the IQ scores of the Head Start group to a control group. See the following slide for the results.
Parents and researchers felt IQ was not a good and valid indicators of the impact of Head Start. A consortium of researchers selected small subgroup of Head Start students and compared them to a control group over 19 years. See slide 19 for results.
Once they evaluated the program, they calculated the savings, to the government, of Head Start. The results are on slide 20.
Field Trials or Experiments
IQ
The Head Start Program - A Famous Field Experiment
Schooling Success
High School Graduation or GED
College or Vocation
Functional Competence
Classified as Mentally Retarded
Years in Special Education
School Responsibility
Detained or Arrested
Teen Pregnancy
Socioeconomic Success
Employed
Received Welfare
Slide 19
Power Point Created by:
Dr. Carol Albrecht
979-777-2421
For More Information
Contact:
Assessment Specialist
USU Extension