4.3: Experimental Designs and Research Settings
- Page ID
- 10343
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)
( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\id}{\mathrm{id}}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\)
\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\)
\( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)
\( \newcommand{\vectorA}[1]{\vec{#1}} % arrow\)
\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow\)
\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vectorC}[1]{\textbf{#1}} \)
\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)
\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)
\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\(\newcommand{\avec}{\mathbf a}\) \(\newcommand{\bvec}{\mathbf b}\) \(\newcommand{\cvec}{\mathbf c}\) \(\newcommand{\dvec}{\mathbf d}\) \(\newcommand{\dtil}{\widetilde{\mathbf d}}\) \(\newcommand{\evec}{\mathbf e}\) \(\newcommand{\fvec}{\mathbf f}\) \(\newcommand{\nvec}{\mathbf n}\) \(\newcommand{\pvec}{\mathbf p}\) \(\newcommand{\qvec}{\mathbf q}\) \(\newcommand{\svec}{\mathbf s}\) \(\newcommand{\tvec}{\mathbf t}\) \(\newcommand{\uvec}{\mathbf u}\) \(\newcommand{\vvec}{\mathbf v}\) \(\newcommand{\wvec}{\mathbf w}\) \(\newcommand{\xvec}{\mathbf x}\) \(\newcommand{\yvec}{\mathbf y}\) \(\newcommand{\zvec}{\mathbf z}\) \(\newcommand{\rvec}{\mathbf r}\) \(\newcommand{\mvec}{\mathbf m}\) \(\newcommand{\zerovec}{\mathbf 0}\) \(\newcommand{\onevec}{\mathbf 1}\) \(\newcommand{\real}{\mathbb R}\) \(\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}\) \(\newcommand{\laspan}[1]{\text{Span}\{#1\}}\) \(\newcommand{\bcal}{\cal B}\) \(\newcommand{\ccal}{\cal C}\) \(\newcommand{\scal}{\cal S}\) \(\newcommand{\wcal}{\cal W}\) \(\newcommand{\ecal}{\cal E}\) \(\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}\) \(\newcommand{\gray}[1]{\color{gray}{#1}}\) \(\newcommand{\lgray}[1]{\color{lightgray}{#1}}\) \(\newcommand{\rank}{\operatorname{rank}}\) \(\newcommand{\row}{\text{Row}}\) \(\newcommand{\col}{\text{Col}}\) \(\renewcommand{\row}{\text{Row}}\) \(\newcommand{\nul}{\text{Nul}}\) \(\newcommand{\var}{\text{Var}}\) \(\newcommand{\corr}{\text{corr}}\) \(\newcommand{\len}[1]{\left|#1\right|}\) \(\newcommand{\bbar}{\overline{\bvec}}\) \(\newcommand{\bhat}{\widehat{\bvec}}\) \(\newcommand{\bperp}{\bvec^\perp}\) \(\newcommand{\xhat}{\widehat{\xvec}}\) \(\newcommand{\vhat}{\widehat{\vvec}}\) \(\newcommand{\uhat}{\widehat{\uvec}}\) \(\newcommand{\what}{\widehat{\wvec}}\) \(\newcommand{\Sighat}{\widehat{\Sigma}}\) \(\newcommand{\lt}{<}\) \(\newcommand{\gt}{>}\) \(\newcommand{\amp}{&}\) \(\definecolor{fillinmathshade}{gray}{0.9}\)Of the many features of research designs, the ones relevant to experiments and laboratories refer to the “where” and “how” of collecting data. In order to answer the causal questions of interest to relational meta-theorists, we want to create designs that allow us to make valid inferences about causes and effects as they unfold in the actual contexts of daily life. As usual, we will discover the tensions in our goals, the balances we can strike among them, and the multiple strategies that can be used to create clear lines of sight.
Didn’t we get rid of experiments and labs in 1977 when Bronfenbrenner basically demolished experimental child psychology?
In a way. At the very least it introduced a healthy dose of skepticism about lab settings. Instead of thinking about the lab as a place where the researcher could get more pristine information about his target phenomenon (i.e., the child and his or her behavior), the lab came to be regarded as one context with its own attributes (e.g., novelty) and set of social partners (i.e., the experimenter) that were exerting their own effects on the child. Moreover, contextualists like Bronfenbrenner argued that by removing the child from his or her familiar surroundings and interaction partners, researchers have inadvertently left much of the phenomena of interest behind.
Relational meta-theorists would likely go further. To us, contexts are not just geographic and architectural “settings” in the sense that you can pick people up and “set” them down in new places. Contexts have tentacles that reach out and wind themselves around people, and people have roots that reach down into places. They are connected, interpenetrated even, so that our most likely causal forces, our proximal processes, cannot even be constituted when we look at only one without the other. When researchers split the child from his or her context, it destroys the phenomenon itself, like removing the heart from the body in order to see how it works. You can’t. Once you remove it, it doesn't work any more.
So developmentalists don't conduct research in laboratory settings any more?
Not at all. Contextualists are just very wary about the idea of the “setting” and very aware of what is lost by leaving the “scene of the crime,” that is, the contexts of daily life.
Well, when would relational meta-theorists bring participants to the lab?
One important reason is to measure a construct that you can’t capture outside of the lab. There are some phenomena of great interest that are not visible without specialized instrumentation or procedures that can only be administered in the lab setting. All manner of neurophysiological constructs can only be measured in the lab setting using complex equipment, like fMRI, as well as the assessment of internal states and capacities, like IQ or executive function or delay of gratification. In fact, precisely because people and their contexts are so intertwined, we sometimes bring our participants into the lab to see what they can do without the scaffolds or interference of social partners.
A second important reason is to get more detailed information about proximal processes themselves. In this case, researchers have the task of re-creating the relevant proximal processes under more controlled conditions. They bring both the target person and their social partners into the lab setting, help a proximal process get started, and then are in a position to collect more information they could access in the field. Many studies of relationships include lab components, in which both partners (for examples, spouses, parents and adolescents, or children and their friends) are brought in to participate jointly in (what are hopefully) interesting activities, such as to discuss marital issues, work jointly on teaching and learning tasks, play competitive games, and so on. These exchanges are often videotaped or observed closely, and in some cases, simultaneous physiological measures are collected, such as heart rate or blood pressure.
A third reason researchers might turn to lab settings is to create conditions where they can trigger and then observe interactions that are relatively rare in field settings. For example, research on learned helplessness often brings children into controlled settings where researchers can watch them work with solvable and then with unsolvable puzzles, mazes, and concept tasks, while monitoring their strategies, efforts, and actions over time. (And, of course they always end with success experiences.) Another example is the Strange Situation in which researchers trigger the attachment system in the living-room-like lab setting, by sending in a stranger and asking the caregiver to leave, and then observe the child’s actions.
In all these cases, naturalistic observation may seem preferable, but because of assortativeness and the responsiveness of contexts, social processes can be impossible to tease apart. For example, mastery-oriented kids run into fewer tasks that they cannot solve than helpless-prone kids and so it is harder to catch them in failure situations, and in schools teachers do not assign impossible tasks, and so observers could go for weeks without seeing their phenomena. And, by the way, after about five years of age, kids are busy trying to hide their true reactions to negative events (a phenomenon called “masking”), which makes it harder for observers to actually detect undesired states (like anxiety or boredom).
Setting |
Design | Lab | Field |
Experimental | Lab experiment | Field experiment |
Naturalistic | Observation in lab | Field observation |
These sound like access or measurement issues. Where is the causality?
Part of causality is a measurement issue—where you can get the best view of your potential causal processes or your potential effects, and when you get there how deeply you can see into the steps of the process you are trying to understand. So the lab, and all its lovely paraphernalia, often offers the best strategies we have for how to measure our target causes and effects.
Cool inventions: Neurophysiological vests? |
Are labs good for other parts of detecting causality?
Indeed they are. They are handy locations for experiments. They cannot be beat for settings in which the researcher has more or less complete control over two key features of the design: (1) the random assignment of participants as to whether they will receive the causal treatment or not; and (2) the administration of the hypothesized causal variable.
Do contextualists care about random assignment?
Do we ever. Remember all those selection effects and assortativeness issues we talked about in previous chapters? Those are shorthand for the huge problems created by the fact that in the contexts of daily life people are not randomly assigned to causal conditions—there are particular personal characteristics that go with people who get in the way of particular causal factors, or who participate in them directly. And so, if we are going to distinguish pre-existing conditions that launched someone on a particular developmental trajectory from the causal factors that we are interested in deciphering, we have to create groups that are “the same on everything” before we start our causal show. Randomized assignment is one strategy to accomplish this, as well as its more systematic options, such as block randomization (randomly assigning different categories of people), matching, propensity score matching, and so on.
Aren’t there better strategies?
Okay, here’s what we would really like to do. We would really like to take our complete sample and expose them to the treatment (the potential causal factor) and see what happens to them, for however long we are interested in detecting effects. Then we would like to load them all into a time machine and take them back to a point in time before the treatment occurred and leave them alone, and see how they would have changed without the treatment. That’s what we are always trying to approximate, a time machine: Let’s see what this groups of people’s development would have been like with this factor and then compare it to the development of the same people without that factor. Pesky time, again. So we have to try to create groups of different people who are the same on everything we can imagine (matching) as well as everything we can’t (random assignment).
Why are we so excited about exact control of the causal factor?
Well, that’s the cool feature of the experimental design. The researcher is like the fairy godmother who waves her magic wand and introduces the potentially new future for the treatment group. So the researcher knows that the treatment group got the potential causal factor, and how much of the factor, and so on (like in a drug trial—the doctor administers the drug and its dosage). And then the researcher has approximately a bazgillion control groups, who got shades of everything but the hypothesized active ingredient (and these can be so creative, the control group with nothing, with only attention, with a visit to the lab but no causal factor, with a causal factor that looks like the actual causal factor but really isn’t, and so on).
Is it easier to control the administration of the causal factor in the lab? So much easier. Once researchers get out in the field, and especially if they decide that the treatment (often as an intervention program) will be administered through intermediaries (like teachers or parents or social workers), it can be a giant headache. There is a whole area of research called “implementation research,” and a focus on “implementation fidelity”—or how the heck you would know and could measure whether the participants actually made contact with the causal factor that you are studying. It’s like doctors who send the treatment pills home with their patients and then hope for the best, but never get to count the pills that are left in the bottle at the end of the trial, and if patients do not improve, they can’t really say whether the drug didn't work or whether the patients just didn't take their pills. Very unsatisfactory from a causal inference perspective.
So we are starting to warm up to labs, right?
Yes, we are regarding them at arm’s length but with respect and appreciation. They can be our ally in measurement and they can give us a leg-up on our simulated time machine for creating groups who are the same, so we can send the different groups on their separate (and with many control groups—their varieties) of carefully calibrated and dosed causal experiences.
And what about experimental designs-- are we starting to warm up to them, too?
Yes, we respect and appreciate them, too. But both lab and experimental studies have serious limitations when it comes to the kinds of questions that contextualists and developmentalists want to answer.
What are those limitations?
Let’s think about three big limitations. First, we already mentioned that labs and fields are not just settings to us. The “field” is an intrinsic and crucial part of the target we are trying to understand, and if we are going to bring our whole phenomenon into the lab, we have to know all the relevant elements of the context and effectively simulate them in the lab. For us, it is an issue of internal validity.
Second, we assume that all our causal factors, that is, our proximal processes, are embedded in contexts and shaped by them. So if we are looking at the functioning of proximal processes in the lab, we can be sure that the lab context is shaping then, which means we can’t be sure that they will operate the same way in the contexts of daily life. So we always have to admit that any causal links we may have watched operating in the lab have to be couched as “can cause” our target and not as “does cause” our target. We have to wait and see if these same processes are operating in the actual contexts that form the natural microsystems for our participants. This is a problem of external validity.
Third, the time span over which developmentalists assume that causal effects accumulate cannot be easily simulated in the lab. The causal processes of interest to developmentalists unfold over months and years and decades, across multiple contexts, so although we can use the lab to measure the long-term effects of causal factors by bringing our participants back to the lab as many times as we want to, if we want to actually look at the causal processes having their effects over months or years, it will be difficult to achieve that in the lab setting.
Please say that these problems are not just for developmentalists.
You are right. They apply to everyone. But there is one problem with typical lab research that in general does not apply to developmentalists.
What is that?
Much of the lab research that is conducted by university researchers uses convenience samples. And who could be more convenient to university researchers than college students? So a great deal of research, for example, in social psychology or on cognition or decision making or perception or education relies on samples of college sophomores—psychology majors, no less. If researchers take their populations seriously and worry about selection effects, then this is a big problem. However, most developmentalists dodge this particular bullet—they do not imagine that the average college sophomore could be considered a reasonable facsimile for an 8-year-old or an 80-year-old or a parent with three children or a person who has experienced the Great Depression. So developmentalists who work in the lab typically import participants from their actual target populations to the laboratory setting.
Then what is the fatal flaw with experimental research? As noted by many methodologists, the seemingly insurmountable problem with experimental designs is that it is not possible to randomly assign or manipulate the causal forces that are of biggest interest to developmentalists. No one can randomly assign their participants to a particular age group (“I have flipped a coin and you will be in the five-year-old group” “Oh no, I wanted to be 10!”) or to a particular cohort or developmental history.
In fact, most of the causal factors that are of interest to us can’t ethically be manipulated at all—the happy single-parent family or the unhappily married parents, the delinquent or theatre-obsessed friends, school failure or indifference, peer rejection or popularity, high stress reactivity, dangerous neighborhoods, perfect pitch, or height. Before you ask, we will just add that this same issue applies to all areas of psychology. Many applied problems cannot be manipulated—divorce, PTSD, dangerous job conditions, psychopathology, work-family conflict, serious medical diagnosis, intimate partner violence, and so on. So there are limits to how much experimental designs can help applied researchers study the conditions and causes that most interest us.
Rutter, Pickles, Murray, & Eaves (2001) on the interplay of risk and protective factors in designs for testing complex hypotheses about the causal effects of environmental risk: It is evident from numerous reviews that causal processes usually involve a complex interplay among risk and protective mechanisms, with indirect chain reactions, bidirectional influences, gene-environment interactions, and synergism between chronic and acute risk factors the rule rather than the exception…[T]he interplay concept means that there are certain further design implications, of which we emphasize four as especially important. First, putative risk variables must we conceptualized and measured in sufficiently broad terms to encompass the risks that may rely on a combination of factors. The extent to which that is the case, plus the delineation of which elements carry the main risk, is better done by subtraction techniques than by the addition of micorelements, each of which on its own might carry no significant risk. Second, Designs, samples, and analytic techniques must be chosen on the basis that they can test for the possibility of both gene- environment interactions and personenvironment interactions based on the effects on the person of prior experiences or of maturational features or gender… Third, appropriate designs must be used to examine the ways in which different forms of gene-environment interactions and person- environment correlations play a part in the causal processes associated with environmental risk mediation… Fourth, attention must be paid to the phenomenon of resilience, meaning a degree of resistance to psychosocial adversities, operationally defined as relatively good outcomes despite experiencing major environmental risks… The reality of the phenomenon has been well demonstrated, but the protective factors have been little explored as yet despite their potential implications for prevention and intervention” (Rutter et al, 2001, p. 297-298. |
Wait! What about optimization studies?
Yes, indeed. Those are rightly considered field experiments, and they can even be conducted as randomized control trials (the gold standard!). And it is correct that we can ethically study any old target we please as long as we are trying to optimize development—to remediate unfavorable developmental trajectories, to maintain resilient ones, and in general to prevent adverse and promote healthy development. So we can learn a great deal and do a great deal of good by trying to create and study interventions designed to optimize development.
At the same time, such optimization studies have two important limitations for developmentalists. First, one thing that such studies cannot tell us is what caused these unhealthy pathways of development in the first place, any more than studying aspirin can tell us what causes headaches or how to prevent them. So additional work will always be needed to fill in the causal puzzle of the factors that contribute to and maintain non-optimal development or lead to psychopathology. It seems that such studies would be essential to prevention efforts.
Second, we have a bone to pick with randomized control trials (RCT) as the ideal methodology for studying causal relationships. As you may know, this methodology was borrowed from clinical trials of medical treatments, and it is cool in many ways. It has time in its design, which is always welcome news to developmentalists. RCTs compare (at least) two groups who should be equivalent to each other (based on random assignment), one of which has received the drug and the other probably a placebo, so that researchers can examine the effects of the drug over and above the effects of knowing that one is being treated. Then after a sufficient amount of time for the drug to do its work, changes in the treatment and control group can be compared over however many time points the design includes.
This sounds very time-machine-esque. What is the problem?
The problem is that, at the end of the day, the only thing that this design can tell you is “yes” or “no,” that is, the only information it yields is whether the two groups are different. You can add many features, for example, many indicators of disease or health, you can measure dosage and its effects, over several time periods, and so on. However, developmentalists would say that, after all this work, the only thing we have in our hands is a causal description but not the thing that we most want, that is, a causal explanation. For the drug companies, everything they want to know about causal explanations is contained in the drug itself; to the extent that they care about how the drug works, its mechanisms of effects have already been studied (and of course, we take many drugs that are effective, but whose mechanisms of effects are unknown).
Limitations of Randomized Control Trails for Studying Causal Processes |
But as developmentalists, our interventions contain hundreds of potential active ingredients. And so we want to poke our heads in under the hood and look all around, watching the cogs engage and the wheels turn. (Whoops! Wrong metaphor for relational meta-theorists!) We want to watch the tennis game or the dance, and see who is hitting the ball the hardest and how the players adapt to each other’s style over time and who is playing the music. In other words, we are on the trail of causal explanation and so we can’t really be satisfied with “yes” or “no.” We will forever be asking “why?” or “why not?” and especially “how did that work?”. So we will always be supplementing experimental and lab studies, and even RCT studies, with studies using designs that can provide us with more complex process-oriented accounts of the multiple causes of differential developmental trajectories and transformations.
Laboratory Experiment | Advantages |
Control and precision Unambiguous causal inference. |
Precise control of hypothesized causal factor. |
||
Precise measure of hypothesized effect. | ||
Disadvantages | Artificiality | |
May change phenomena. | ||
Limited to “can cause” versus “does cause” causal conclusions. | ||
May or may not work in actual contexts. | ||
Most potential causal factors cannot be manipulated. | ||
Naturalistic Laboratory Study | Advantages |
Precision Measure constructs that are “below the surface” (e.g., neurophysiology, capacities, knowledge). |
Disadvantages | Distortion | |
Splitting of person from context may have destroyed causal factors. | ||
Hard to locate “active ingredient” of causal packages. | ||
Artificiality and novelty of context, instrument, or trigger distorts causal phenomena. | ||
Field Experiment | Advantages |
Control and Actual context Potential for causal inference. |
Potential to see how causes operate in situ. | ||
Potential to see effects in situ. | ||
Disadvantages | Messiness | |
Hard to precisely control the implementation of the potential causal factor. | ||
Especially if delivery agents are also naturalistic (i.e., parents, teachers, social workers) | ||
Limited to “can cause” versus “does cause” causal conclusions. | ||
Most potential causal factors cannot be manipulated. | ||
Limited account of causal process. | ||
Naturalistic Field Study | Advantages |
Authenticity Whole phenomena is intact. |
Can discover causes that were not expected. | ||
Disdvantages | Murkiness | |
Hard to specify “active ingredient” of causal packages. | ||
Impossible to control all selection effects. | ||
Limited to “may cause” versus “does cause” causal conclusions. |