Learning Objectives
By the end of this section, you will be able to:
- Identify components of experimental designs
- Read and interpret research design notation
- Differentiate between experimental and nonexperimental designs
- Understand why nonexperimental designs are used
In political science, the “gold standard” is an experimental design. An experimental design can help determine the effect of the independent variable or the treatment on the dependent variable or the outcome because the treatment can be isolated as the likely cause. Comparisons are made between the experimental group and the control group to see if the outcomes are different. Because random assignment ensures that the two groups are the same and the only difference is the treatment, researchers can make the conclusion that the difference in the outcomes of the groups is likely due to the treatment. Because of these factors, an experimental design is best suited for the purposes of explanatory research to establish causality.
To help with understanding research design, it is common to utilize notation to provide a visual depiction of the design. We will utilize the following notation as borrowed from Trochim and Donnelly (2005).
Figure \(\PageIndex{1}\): Notation is useful to present a visual representation of research design. The figure displays the notation for an experimental design
The selection of individuals into groups is denoted by R (random assignment) or NR (nonrandom assignment). Observations are denoted as “O” and “X” is the treatment. One line of notation will refer to a single group. Two lines denote two different groups, three lines denote three groups, and so on. The notation from left to right denotes the passage of time. Using this notation, we can generally classify designs into experimental and nonexperimental. We will talk about the experimental design first and then nonexperimental designs.
There are three crucial components: random assignment, manipulation of the treatment, and the existence of a control group. Generally, there are two groups, an experimental group and a control group. The experimental group will be administered a treatment while the control group will not be administered a treatment. The control group is supposed to be what the experimental group would look like if the experimental group was not given the treatment.
Comparisons are then made between the two groups using pretests and posttests to determine the effect of the treatment on outcomes. Random assignment refers to the placement of cases into control and experimental groups in an unbiased manner such that the likelihood of any case being placed into groups is exactly the same. With random assignment, we can be assured the groups are equal to each other, or any reason that we might think they are different is removed. If there are differences, it is due to chance.
The pretest establishes a baseline, allowing us to understand how things are before the treatment is implemented and the posttest provides us with information about outcomes after the treatment. If this sounds familiar, it is because an experiment in political science is similar to an experiment performed in a science lab!
Figure \(\PageIndex{2}\): A variation on the classic experiment, this is an experimental design that does not contain a pretest.
Experimental designs can vary in relation to the classic example presented. One variation of the classic example is not administering a pretest. This can be due to fears that taking a pretest can affect the results or it just may be that a researcher is unable to administer a pretest. This makes it difficult to attribute the varying outcomes to the treatment but can still allow conclusions about causality because a control group does exist.
Figure \(\PageIndex{3}\): The Solomon 4-Group Design is an experimental design that combines the classic experiment with the posttest only design.
Another variation to the classic experiment is the introduction of groups beyond the traditional two groups that tries to address the effect of a pretest on outcomes. This variation is known as a Solomon 4-Group Design. As the name indicates, there are four groups. Two are experimental groups and two are control groups. One experimental group and one control will have a pretest and posttest and the remaining groups will not be pretested. In this way, comparisons can be made between the two pairs and it can be determined as to whether the pretest had an effect on results.
As we move further away from the classic experimental design, the ability of researchers to establish causality diminishes. Nonexperimental designs may lack random assignment into groups, the ability of the researcher to control the treatment, a control group, or all of these characteristics of an experiment. Ethical concerns may lead to the implausibility of implementing an experiment. For instance, to determine the effect of a treatment, a researcher may decide to randomly assign individuals into a control group and an experimental group. The experimental group receives a treatment that could cure a serious illness but those in the control group who could benefit from the same treatment are denied it. In a case such as this, ethical concerns may prevent the random assignment of the treatment and instead provide the treatment to all who are willing to be treated.
Figure \(\PageIndex{4}\): Quasi-experiments may attempt to be similar to an experiment but, in this particular case, lacks random assignment into groups.
Quasi-experimental designs try to approximate experiments but lack a key component, random assignment. For instance, in a nonequivalent 2-group comparative design, cases are divided into two groups, one an experimental group and a comparison group that is meant to be like a control group. Unlike an experiment, assignment into groups is not random. It is possible that individuals self-selected into groups. In a design known as matching, cases are matched together on multiple variables with the only variation being that of the treatment variable. While it may be possible to match on identified variables, it is difficult to discern whether variables that are not observed are also evenly distributed. Because the formation of the two groups was not through random assignment, we do not know if the groups are equivalent to one another. Variables that are unaccounted for could potentially be what is truly affecting the outcome rather than the treatment; thus, it is difficult to establish the effect of the treatment on the outcome we are trying to explain.
In the example above, it is possible that individuals who chose to be part of the experimental group to receive the life saving treatment were more likely to be individuals who exhausted all other types of available treatment and are now utilizing the remaining treatment as a last resort. It could also be possible that those in the treatment group wanted to take part in the study because they have a greater zest for life. Because such characteristics might not have been apparent in the matching phase of the study, this might have an added effect that was not accounted for.
Figure \(\PageIndex{5}\): A nonexperimental design with pre-test and a post-test, but no control group
Difficulty in establishing causality can also result when there lacks a comparison group. Researchers might be able to administer the treatment but are unable to have a control group for multiple considerations. In this case, the same group acts as a control for itself. Because the pretest is administered before the treatment, it provides us with the results of the outcomes before a treatment. It is then compared with the posttest to see if there were any changes between the two.
Comparison is made within-group. If there are differences, it may be attributed to the treatment. This design can be problematic because threats to the validity of the design exist. Without a control group, it can be difficult to attribute the outcome to the treatment because it could simply be due to maturation or normal growth. In other words, the results would have been the same without the treatment. It could also be due to the administration of a pretest that primed cases to be better prepared for the next test.
While there are multiple designs that exist, the purpose of your research will often dictate the best design to use for your study. If you are trying to establish causality, experimental designs will likely be the design of choice. Experimental designs have internal validity, thus ensuring that you can provide causal conclusions about an independent variable’s effect on an outcome. When it is not feasible and the reasons as to your research study are not establishing causality, but instead gathering information, nonexperimental studies that do not necessarily require random assignment or a control group will serve your research goal just as well.
