Learning Objectives
- Describe the strengths and weaknesses of Experiments.
- Differentiate between external and internal validity.
- List and define the different threats to internal validity.
Strengths & Weaknesses of Experiments Related to Validity Types
As sociologists, who are especially attentive to how social context shapes social life, are likely to point out, a disadvantage of experiments is that they are rather artificial. How often do real-world social interactions occur in the same way that they do in a lab? Experiments that are conducted in applied settings may not be as subject to artificiality, though then their conditions are less easily controlled. Experiments also present a few unique concerns regarding validity. Problems of external validity might arise when the conditions of an experiment don’t adequately represent those of the world outside the boundaries of the experiment. In the case of McCoy and Major’s (2003)McCoy, S. K., & Major, B. (2003). Group identification moderates emotional response to perceived prejudice. Personality and Social Psychology Bulletin, 29, 1005–1017. research on prejudice described earlier in this section, for example, the questions to ask with regard to external validity are these: Can we say with certainty that the stimulus applied to the experimental group resembles the stimuli that people are likely to encounter in their real lives outside of the lab? Will reading an article on prejudice against one’s race in a lab have the same impact that it would outside of the lab? This is not to suggest that experimental research is not or cannot be valid, but experimental researchers must always be aware that external validity problems can occur and be forthcoming in their reports of findings about this potential weakness. Concerns about internal validity also arise in experimental designs. These have to do with our level of confidence about whether the stimulus actually produced the observed effect or whether some other factor, such as other conditions of the experiment or changes in participants over time, may have produced the effect.
In sum, the potential strengths and weaknesses of experiments as a method of data collection in social scientific research include the following:
Table 7.3 Strengths and Weaknesses of Experimental Research
| Strengths |
Weaknesses |
| Researcher control |
Artificiality |
| Reliability |
Unique concerns about internal and external validity |
Threats to Internal Validity
Although experimental designs are considered more rigorous than other research methods in terms of the internal validity of their inferences (by virtue of their ability to control causes through treatment manipulation), they are not immune to internal validity threats. Some of these threats to internal validity are described below, within the context of a study of the impact of a special remedial math tutoring program for improving the math abilities of high school students.
- History threat is the possibility that the observed effects (dependent variables) are caused by extraneous or historical events rather than by the experimental treatment. For instance, students’ post-remedial math score improvement may have been caused by their preparation for a math exam at their school, rather than the remedial math program.
- Maturation threat refers to the possibility that observed effects are caused by natural maturation of subjects (e.g., a general improvement in their intellectual ability to understand complex concepts) rather than the experimental treatment.
- Testing threat is a threat in pre-post designs where subjects’ posttest responses are conditioned by their pretest responses. For instance, if students remember their answers from the pretest evaluation, they may tend to repeat them in the posttest exam. Not conducting a pretest can help avoid this threat.
- Instrumentation threat , which also occurs in pre-post designs, refers to the possibility that the difference between pretest and posttest scores is not due to the remedial math program, but due to changes in the administered test, such as the posttest having a higher or lower degree of difficulty than the pretest.
- Mortality threat refers to the possibility that subjects may be dropping out of the study at differential rates between the treatment and control groups due to a systematic reason, such that the dropouts were mostly students who scored low on the pretest. If the low-performing students drop out, the results of the posttest will be artificially inflated by the preponderance of high-performing students.
- Regression threat , also called a regression to the mean, refers to the statistical tendency of a group’s overall performance on a measure during a posttest to regress toward the mean of that measure rather than in the anticipated direction. For instance, if subjects scored high on a pretest, they will have a tendency to score lower on the posttest (closer to the mean) because their high scores (away from the mean) during the pretest was possibly a statistical aberration. This problem tends to be more prevalent in non-random samples and when the two measures are imperfectly correlated.
KEY TAKEAWAY
- Experiments tend to be weak in external validity and stronger in internal validity.
