7.7: Strengths, Weaknesses, and Validity

Learning Objectives

• Describe the strengths and weaknesses of experiments.

• Differentiate between external and internal validity.

• List and define the different threats to internal validity.

Strengths & Weaknesses of Experiments Related to Validity Types

As sociologists, who are especially attentive to how social context shapes social life, are likely to point out, a major disadvantage of experiments is that they are rather artificial. How often do real-world social interactions occur in the exact same way that they do in a sterile lab? Experiments that are conducted in applied settings (field experiments) may not be as subject to artificiality, though their conditions are less easily controlled.

Experiments also present a few unique concerns regarding validity. Problems of external validity might arise when the conditions of an experiment do not adequately represent those of the world outside the boundaries of the experiment. In the case of the Hunt et al. (2018) research on social media and well-being described earlier in this chapter, for example, the questions to ask with regard to external validity are these: Can we say with certainty that the stimulus applied to the experimental group resembles the stimuli that people are likely to encounter in their real lives outside of the lab? Will restricting social media to exactly ten minutes per platform in a controlled study accurately reflect the messy, interconnected way people engage with their phones in the real world?

This is not to suggest that experimental research is not or cannot be valid, but experimental researchers must always be aware that external validity problems can occur and be forthcoming in their reports of findings about this potential weakness.

Concerns about internal validity also arise in experimental designs. These have to do with our level of confidence about whether the stimulus actually produced the observed effect or whether some other factor, such as other conditions of the experiment or changes in participants over time, may have produced the effect instead.

In sum, the potential strengths and weaknesses of experiments as a method of data collection in social scientific research include the following:

Table 7.2 Strengths and Weaknesses of Experimental Research

Strengths	Weaknesses
Researcher control	Artificiality
Reliability	Unique concerns about internal and external validity

Threats to Internal Validity

Although experimental designs are considered more rigorous than other research methods in terms of the internal validity of their inferences (by virtue of their ability to control causes through treatment manipulation), they are not immune to internal validity threats. Some of these threats to internal validity are described below, within the context of a study of the impact of a special remedial math tutoring program for improving the math abilities of high school students.

• History threat: The possibility that the observed effects (dependent variables) are caused by extraneous or historical events rather than by the experimental treatment. For instance, students’ post-remedial math score improvement may have been caused by their intensive preparation for an upcoming statewide standardized test, rather than the remedial math program itself.
• Maturation threat: The possibility that observed effects are caused by the natural maturation of subjects (e.g., a general, age-related improvement in their intellectual ability to understand complex concepts) rather than the experimental treatment.
• Testing threat: A threat specific to pre-/posttest designs where subjects’ posttest responses are conditioned by their pretest responses. For instance, if students remember the specific questions from the pretest evaluation, they may simply repeat their answers or study those specific items for the posttest exam.
• Instrumentation threat: Also occurring in pre-/posttest designs, this refers to the possibility that the difference between pretest and posttest scores is not due to the remedial math program but due to changes in the administered test itself (e.g., the posttest being accidentally designed with a much lower degree of difficulty than the pretest).
• Mortality threat: The possibility that subjects may drop out of the study at differential rates between the treatment and control groups due to a systematic reason. For example, if the dropouts were mostly students who scored very low on the pretest and got frustrated, the results of the posttest will be artificially inflated by the remaining preponderance of high-performing students.
• Regression to the mean threat: This refers to the statistical tendency of a group’s overall performance on a measure during a posttest to regress toward the average (mean) of that measure rather than moving in the anticipated direction. For instance, if subjects scored abnormally high on a pretest due to a lucky statistical aberration, they will have a natural tendency to score lower on the posttest (closer to their true mean).

Key Takeaways

• Validity Trade-offs: Experiments tend to be weak in external validity (how well they apply to the real world) but stronger in internal validity (how confidently they isolate cause and effect).
• Vigilance against Threats: Researchers must actively design their experiments to minimize internal validity threats such as history, maturation, testing bias, instrumentation errors, mortality, and regression to the mean.