Learning Objectives
- Describe the different types of nonequivalent groups quasi-experimental designs.
- Describe the strengths of the different types of nonequivalent groups quasi-experimental designs.
Recall that when participants in a between groups design experiment are randomly assigned to conditions, the resulting groups are likely to be quite similar (initially equivalent). When participants are not randomly assigned to conditions, however, the resulting groups are likely to be dissimilar in some ways. For this reason, researchers consider them to be nonequivalent. A nonequivalent groups design, then, is a design that compares groups of participants who have not been randomly assigned to conditions. There are several types of nonequivalent groups designs we will consider here.
Nonequivalent Group Designs
Natural Groups Designs (AKA Posttest-Only Nonequivalent Groups Design)
In the chapter on experimental designs, there was a quick sidetrack about natural groups designs to help you see the strengths.
Exercise \(\PageIndex{1}\)
Without looking it up, try to describe what you can remember about natural groups designs.
- Answer
-
The definition of natural groups design is "Research study that compares groups that were not randomly assigned into the IV conditions."
Natural groups designs is the posttest-only nonequivalent groups design. Just like in natural groups designs, there is a group who is exposed to a treatment or experiences an event and another (nonequivalent) group is not exposed to the treatment or event, and then the two groups are compared. Some researchers may argue that natural groups designs are only when the research is comparing participant variables, and posttest-only nonequivalent groups design is only for research in which a treatment was implemented, but this distinction is not needed for our purposes.
Let's discuss an example to reinforce the issues when there's neither equivelent groups (created through random assignment) nor is there a comparison of a pretest to a posttest. Imagine, for example, a researcher who wants to evaluate a new method of teaching fractions to third graders. One way would be to conduct a study with a treatment group consisting of one class of third-grade students and a control group consisting of another class of third-grade students. This design would be a posttest-only nonequivalent groups design because the students are not randomly assigned to classes by the researcher, and there was no pretest to test for equivalency between the classes. This means that there could be important differences between them. For example, the parents of higher achieving or more motivated students might have been more likely to request that their children be assigned to Ms. Williams’s class. Or the principal might have assigned the “troublemakers” to Mr. Jones’s class because he is a stronger disciplinarian. In addition to the teachers’ styles and the students in the class, the classroom buildings might be very different and might cause different levels of achievement or motivation among the students. Maybe Ms. Williams has a smart board but Mr. Jones's classroom has a regular white board. If at the end of the study there was a difference in the two classes’ knowledge of fractions, it might have been caused by the difference between the teaching methods—but it might have been caused by any of these (or other!) confounding variables.
Thus, posttest-only nonequivalent groups designs are quasi-experimental designs because there is no initial equivalence between the groups; without true random assignment of the students to conditions, there remains the possibility that the DV was influenced by confounding variables that the researcher was not able to control. And without a pretest, its both impossible to know if the groups start out equivalent on the outcome, but it's also impossible to know if the scores on the outcome changed at all throughout the study.
Pretest-Posttest Nonequivalent Groups Design
Another way to improve upon the posttest-only nonequivalent groups design is to add a pretest. This is like the one-group pretest-posttest design, except now we do have a comparison group. In the pretest-posttest nonequivalent groups design, there is a treatment group that is given a pretest, receives a treatment, and then is given a posttest, and at the same time there is a nonequivalent control group that is given a pretest, does not receive the treatment, and then is given a posttest. The question, then, is not simply whether participants who receive the treatment improve, but whether they improve more than participants who do not receive the treatment.
Imagine, for example, that students in one school are given a pretest on their attitudes toward drugs, then are exposed to an anti-drug program, and finally, are given a posttest. Students in a similar school are given the pretest, not exposed to an anti-drug program, and finally, are given a posttest. Again, if students in the treatment condition become more negative toward drugs, this change in attitude could be an effect of the treatment, but it could also be a matter of history or maturation. If it really is an effect of the treatment, then students in the treatment condition should become more negative than students in the control condition. But if it is a matter of history (e.g., news of a celebrity drug overdose) or maturation (e.g., improved reasoning), then students in the two conditions would be likely to show similar amounts of change. This type of design does not completely eliminate the possibility of confounding variables, however. Something could occur at one of the schools but not the other (e.g., a student drug overdose), so students at the first school would be affected by it while students at the other school would not.
You might be thinking that there's no way that Hatzenbuehler et al. (2012) could also be an example here, but there was some additional groups in the study that haven't been mentioned yet. Not only did Hatzenbuehler et al. (2012) compare the same sexual minority men before and after same-gender marriage was legal in Massachusets, but Hatzenbuehler et al. (2012) also knew if these men were were long-term romantic relationship or not. This enabled a posttest-only nonequivalent groups design because partnered participants (one group) completed their health visits before (pretest) the legal recognition of same-gender marriage (treatment) and after (posttest), while nonpartnered men (nonequivalent group) also completed their health visits before (pretest) the legal recognition of same-gender marriage (treatment) and after (posttest). Hatzenbuehler et al. (2012) may not be a true example of a pretest-posttest nonequivalent groups design because the outcomes (dependent variables) are not directly related to the comparison groups or treatment like the example about the anti-drug program above, but it is interesting that one study can be similar to many different types of design, depending on how much detail you can see.
Interrupted Time-Series Design with Nonequivalent Groups
Just like the pretest-posttest nonequivalent groups design is an improvement on the one-group pretest-posttest design, the interrupted time series design with nonequivalent groups is an improvement on the the interrupted time-series design by to adding a comparison group. The interrupted time-series design with nonequivalent groups involves taking a set of measurements at intervals over a period of time both before and after an intervention of interest in two or more nonequivalent groups. Once again consider the manufacturing company that measures its workers’ productivity each week for a year before and after reducing work shifts from 10 hours to 8 hours. This design could be improved by locating another manufacturing company who does not plan to change their shift length and using them as a nonequivalent control group. If productivity increased rather quickly after the shortening of the work shifts in the treatment group but productivity remained consistent in the control group, then this provides better evidence for the effectiveness of the treatment.
Pretest-Posttest Design With Switching Replication
Some of these nonequivalent control group designs can be further improved by adding a switching replication. Using a pretest-posttest design with switching replication design, nonequivalent groups are administered a pretest of the dependent variable, then one group receives a treatment while a nonequivalent control group does not receive a treatment, the dependent variable is assessed again, and then the treatment is added to the control group, and finally the dependent variable is assessed one last time.
As a concrete example, let’s say we wanted to introduce an exercise intervention for the treatment of depression. We recruit one group of patients experiencing depression and a nonequivalent control group of students experiencing depression. We first measure depression levels in both groups, and then we introduce the exercise intervention to the patients experiencing depression, but we hold off on introducing the treatment to the students. We then measure depression levels in both groups. If the treatment is effective we should see a reduction in the depression levels of the patients (who received the treatment) but not in the students (who have not yet received the treatment). Finally, while the group of patients continues to engage in the treatment, we would introduce the treatment to the students with depression. Now and only now should we see the students’ levels of depression decrease.
A strength of this design is that it includes a built in replication. In the example given, we would get evidence for the efficacy of the treatment in two different samples (patients and students). Because all participants are not in all conditions, there is also not the concern over order effects like there is in a repeated measures designe.
Switching Replication with Treatment Removal Design
In a basic pretest-posttest design with switching replication, the first group receives a treatment and the second group receives the same treatment a little bit later on (while the initial group continues to receive the treatment). In contrast, in a switching replication with treatment removal design, the treatment is removed from the first group when it is added to the second group. Once again, let’s assume we first measure the depression levels of patients with depression and students with depression. Then we introduce the exercise intervention to only the patients. After they have been exposed to the exercise intervention for a week we assess depression levels again in both groups. If the intervention is effective then we should see depression levels decrease in the patient group but not the student group (because the students haven’t received the treatment yet). Next, we would remove the treatment from the group of patients with depression. So we would tell them to stop exercising. At the same time, we would tell the student group to start exercising. After a week of the students exercising and the patients not exercising, we would reassess depression levels. Now if the intervention is effective we should see that the depression levels have decreased in the student group but that they have increased in the patient group (because they are no longer exercising).
Demonstrating a treatment effect in two groups staggered over time and demonstrating the reversal of the treatment effect after the treatment has been removed can provide strong evidence for the efficacy of the treatment. In addition to providing evidence for the replicability of the findings, this design can also provide evidence for whether the treatment continues to show effects after it has been withdrawn. However, not all treatments can be easily removed, or the results of the treatments will not fade. This is an issue in repeated measures designs, as well in the switching replication with treatment removal design.
Good enough?
You may have noticed that each type of quasi-experimental design seemed a little better. Nonequivalent designs are better than one-group designs because there's a comparison group. Pretest-posttest designs are better than posttest-only designs because there's a comparison between before and after the treatment. Interrupted time series designs may be better with more measurements of the outcome. And replication, and then replication with removal, are better because there is strong evidence that nothing else other than the treatment could affect the the outcome. But, researchers still have this nagging feeling that maybe there is something else; is there another variable that affects both the treatment group and outcome, but not the comparison group (What about Z?). If you really want to show that the treatment caused changes in the outcome, then you really need initial equivalence and ongoing equivalence of the groups. We need to know that nothing could have affected the outcome other than our intervention. The last quasi-experimental designs do get at this, but the other quasi-experimental designs fall short of being able to show cause and effect.
References
Hatzenbuehler, M. L, O'Cleirigh, C., Grasso, C., Mayer, K., Safren, S., & Bradford, J. (2012). Effect of same-sex marriage laws on health care use and expenditures in sexual minority men: A quasi-natural experiment. American Journal of Public Health, 102(2), 285-291.
