Chapter 4: Studying Behavior
Learning objectives
After completing this chapter, you will be able to:
• Define construct validity, internal validity, external validity, and statistical conclusion validity, and compare the four.
• Define a variable and write an operational definition of one.
• Identify the four basic relationships between variables: positive, negative, curvilinear, and none.
• Compare and contrast nonexperimental and experimental research methods.
• Distinguish an independent variable from a dependent variable.
• Summarize the strengths and limitations of laboratory experiments, and explain the advantage of using multiple methods.
Warm-up: build an athleticism test
Imagine your instructor asks your class to create a test that measures athleticism. You have one week and a gym. What would you have people do?
Take two minutes and write down your plan before reading on. Most students start with something like "time them running a mile." Then they stop and ask: but what about strength? Or agility? Or the person who is a gifted swimmer but a slow runner?
That moment of doubt is what this chapter is about. Before researchers can study anything, they have to decide what to measure and how, and then ask whether their choice actually captures the thing they care about. We will return to your athleticism test throughout the chapter.
4.1 Validity: an introduction
In everyday speech, "valid" means sound or justified: a valid ID, a valid point, a valid argument. In research the word has a more precise meaning, and one of the clearest statements of it comes from Thomas Cook and Donald Campbell. In their 1979 book on research in field settings, they described validity as the "best available approximation to the truth or falsity of propositions," including propositions about cause (Cook & Campbell, 1979, p. 37). They added that we should always attach a qualifier such as "approximately" or "tentatively" to the words valid and invalid, because we can never know what is true; at best we know what has not yet been ruled out as false (p. 37). Later, with William Shadish, they put the same idea more briefly: validity is the approximate truth of an inference (Shadish, Cook, & Campbell, 2002).
Each part of that definition is worth a closer look.
• "Approximation." Science does not deliver certainty. A conclusion can be closer to or further from the truth, and it stays open to revision when better evidence arrives.
• "Best available." We judge a conclusion against the evidence and methods we have right now, not against a perfect study no one could run.
• "Propositions." Validity is a property of the claims we make, not of a study, a test, or a data set as such. We do not ask "Is this study valid?" as a yes-or-no question. We ask how valid a particular conclusion drawn from it is.
You already make validity judgments in daily life. Your fitness tracker says you slept eight hours last night: is that true? A headline says a new app "boosts memory": should you believe it? A friend decides a first date went well: is that impression accurate? In each case you are weighing how close a claim is to the truth, and the evidence is what you weigh it with. Researchers do the same thing, but they do it systematically and they make their reasoning public.
A research conclusion can go wrong in more than one way, so Cook and Campbell sorted the ways into types of validity. Campbell and Stanley (1963) had earlier distinguished internal and external validity; Cook and Campbell (1979) then subdivided each of those two, separating statistical conclusion validity from internal validity and construct validity from external validity. That gives the four-part system psychologists still use. Cook and Campbell described the four types as practical rather than logically required: each matches one of four decisions a researcher faces about a causal claim (p. 39). Each type asks about a different kind of claim:
Table 4.1. The four types of validity
|
Type of validity
|
The question it asks
|
The claim it protects
|
|
Construct validity
|
Did we measure or manipulate the concepts we say we did?
|
"This measure or procedure stands for that construct."
|
|
Internal validity
|
Did one variable really cause the change in the other, in this study?
|
"X caused Y here."
|
|
External validity
|
Would the finding hold for other people, settings, and times?
|
"This result generalizes."
|
|
Statistical conclusion validity
|
Did we reach the right conclusions about whether the variables are related, and how strongly?
|
"The variables covary, by about this much."
|
Keep this table handy. The rest of the chapter builds the vocabulary you need to apply it: variables and operational definitions (construct validity), relationships between variables, the logic of experiments (internal validity), and the limits of laboratory research (external validity). In Section 4.7 we return to the four questions as a checklist for evaluating any study.
4.2 Variables
A variable is anything that varies and can be measured. It can be a behavior (hours of sleep), a thought (belief in free will), a feeling (anxiety), a situation (noisy versus quiet room), a characteristic (height), or an event (receiving a failing grade).
Two features define a variable:
1. It must have at least two levels or values. Age varies because people have different ages. If every person in a study were 20 years old, age would be a constant in that study, not a variable.
2. Its values may be numeric or categorical. Some variables have true numeric properties: reaction time in milliseconds, number of friends, score on a depression inventory. Others simply sort people into categories: eye color, major, marital status, or which condition of an experiment a participant was in. Categorical values have no numeric size to them (a distinction Stanley S. Stevens, 1946, built into his four levels of measurement: nominal, ordinal, interval, and ratio), so we cannot say that one category is "more" than another.
This distinction matters later, because the kind of variable you have determines how you can describe it and which statistics fit it.
Try it: Identify the variable and two possible values for each: (a) a person's favorite music genre, (b) the number of hours someone studied, (c) whether a rat was given a food reward.
4.3 Operational definitions of variables
Psychologists often study abstract ideas such as stress, intelligence, attention, or happiness. You cannot hold stress in your hand. To study an abstract idea, a researcher has to translate it into something observable. That translation is called an operational definition: the specific set of procedures used to measure or manipulate a variable.
The idea of defining a concept by its operations comes from the physicist Percy Bridgman (1927), who argued that a concept is synonymous with the set of operations used to measure it. Stanley S. Stevens (1935) brought the idea into psychology, where it gave researchers a way to discuss concepts such as anxiety or intelligence in terms of observable procedures.
Here are some examples:
Table 4.2. Examples of operational definitions
|
Abstract variable
|
A possible operational definition
|
|
Stress
|
Score on a 10-item self-report stress questionnaire; or level of cortisol in saliva
|
|
Aggression
|
Number of times a child hits or pushes a peer during a 20-minute play session
|
|
Exam anxiety
|
Heart rate measured in the five minutes before an exam begins
|
|
Being rejected (a manipulation)
|
Being told that no other participant chose you as a partner for the next task
|
Why operational definitions matter:
• They force concrete thinking. To define a concept as procedures, you must decide exactly what you mean by it.
• They make research communicable. Another scientist can read your procedures and understand exactly what you did.
• They make replication possible. Without them, nobody could repeat your study to check the result.
• They are required. A variable cannot be studied empirically until it has an operational definition.
Notice that a single concept can have many different operational definitions. Stress could be measured with a questionnaire, a hormone, a heart rate, or even how many days someone has missed work. No one definition is automatically "the right one." Each captures a different slice of the idea.
For that reason, modern researchers do not treat any single operation as the whole meaning of a concept. Cook and Campbell (1979, p. 93) criticized the older "definitional operationalism" for encouraging researchers to treat their measures as if they were the construct itself. They also warn against mono-operation bias: a study that represents a construct with only one measure or one manipulation both leaves out parts of the construct and carries along irrelevant features of that single operation (p. 65). Their remedy is multiple operationalism, testing the same idea with several overlapping operations (p. 63). They recommend defining the construct carefully before choosing operations, for example by checking how it is defined in dictionaries and earlier research (p. 64). And the operations should differ in method, not just in content: if every measure is a paper-and-pencil questionnaire, the method itself may be driving the results, which they call mono-method bias (p. 66).
Back to the athleticism test: Could "how fast someone runs a mile" serve as an operational definition of athleticism? It is concrete, easy to score, and easy for someone else to repeat. But does it capture everything we mean by athleticism? That question leads to construct validity.
Construct validity: do our operational definitions work?
A construct is the abstract concept a researcher wants to study (athleticism, spatial memory, stress). Construct validity is the accuracy of our operational definitions: does the way we measured or manipulated the variable truly reflect the theoretical meaning of the construct? The term comes from Cronbach and Meehl (1955). Cook and Campbell (1979, p. 38) extended it from measured traits to manipulated independent variables, and defined it as the approximate validity with which we can generalize from research operations to the higher-order construct they are meant to represent.
Consider a classic example from animal research. A researcher wants to study spatial memory in mice and uses a maze: the faster a mouse reaches the food, the better its spatial memory. The question of construct validity is whether maze performance actually reflects spatial memory. A mouse might run quickly through the maze because it is hungrier, less anxious, or better at following smells, not because it remembers the layout. If those other factors drive the results, the maze has weak construct validity as a measure of spatial memory.
Applied to the mile run: it does a decent job of capturing endurance. But if athleticism also includes strength, agility, coordination, and power, a mile run alone has weak construct validity as a measure of athleticism. A better test might combine several tasks (e.g., a mile run, a sprint, a standing jump, and an agility course). The more fully an operational definition matches the construct, the stronger its construct validity.
Researchers look for two kinds of evidence about construct validity. Convergent evidence shows that different measures or manipulations of the same construct agree with each other; discriminant evidence shows that measures of related but different constructs do not simply track each other (Campbell & Fiske, 1959; Cook & Campbell, 1979, p. 61). You will study these tools in later chapter covering measurement concepts.
4.4 Relationships between variables
Most research asks whether two variables are related: when one changes, does the other change in a predictable way? For variables that have numeric values, researchers look at four common patterns.
Table 4.3. Four common relationships between two numeric variables
|
Relationship
|
What it looks like
|
Example
|
|
Positive linear
|
As one variable goes up, the other goes up.
|
Weekly hours of aerobic activity and cardiovascular fitness
|
|
Negative linear
|
As one variable goes up, the other goes down.
|
Hours of practice on a typing task and number of typing errors
|
|
Curvilinear
|
The direction of the relationship changes. The second variable rises and then falls, or falls and then rises.
|
Dose of a medication and symptom relief: relief rises with dose up to a point, then drops as side effects increase (a drug response curve shaped like an inverted U)
|
|
No relationship
|
Knowing one variable tells you nothing about the other.
|
Number of siblings and reaction time
|
Two important cautions:
• These are general patterns, not rules for every individual. In a positive relationship between study time and exam score, there will still be someone who studied twice as long and scored lower. Real data are messy.
• Strength matters, not just direction. A correlation coefficient is a number that describes the strength (and, for linear relationships, the direction) of a relationship. It ranges from −1.00 to +1.00. Values near zero mean a weak or absent relationship, and values near −1.00 or +1.00 mean a strong one. You will learn to calculate and interpret correlations in a later chapter.
Relationships reduce uncertainty
Why do researchers search for relationships at all? Because the world looks random until we find patterns. Every unexplained difference among people is uncertainty (also called random variability). Finding a systematic relationship lets us explain part of that variability and make better predictions.
Here is a simple example. Suppose you wanted to predict whether a student will pass the first exam in a research methods course. With no other information, you would be guessing. Now suppose you also learn how many practice quizzes each student completed before the exam. The numbers below are hypothetical and are for illustration only:
Table 4.4. Hypothetical first-exam pass rates by number of practice quizzes completed
|
Practice quizzes completed
|
Percent who passed the first exam
|
|
None
|
35%
|
|
1 or 2
|
58%
|
|
3 or more
|
82%
|
The number of practice quizzes now helps us predict exam results, even though plenty of exceptions remain: some students who skip every quiz still pass, and some who complete them all do not. Quiz completion explains some of the variability and reduces our uncertainty. That is what a relationship does, and discovering such relationships is a major goal of research.
4.5 Nonexperimental versus experimental methods
There are two broad ways to study whether variables are related.
• In the nonexperimental method, the researcher observes the variables of interest as they naturally occur and checks whether they vary together. A relationship is established when the two variables change together.
• In the experimental method, the researcher manipulates one variable and controls everything else, then checks whether the other variable changes as a result. Here the variables do not just vary together: one is introduced first, so we can test whether it affects the other.
The difference is not about how fancy the study is. It is about the researcher's role: observing what happens versus making something happen.
The nonexperimental method
Because it examines how variables go together, the nonexperimental method is also called the correlational method. It is useful for describing behavior and for making predictions. Its weakness appears when we ask why. Suppose a study finds that people who spend more time on social media report more loneliness. There are three possible explanations, and the data alone cannot tell us which one is true:
Table 4.5. Three possible explanations for a relationship found in a nonexperimental study
|
Possible explanation
|
What it says
|
Name of the problem
|
|
Social media use → loneliness
|
Using social media makes people lonelier.
|
|
|
Loneliness → social media use
|
Lonely people turn to social media more.
|
Directionality problem: we cannot tell which variable is the cause and which is the effect.
|
|
A third variable → both
|
Something else (for example, a recent move to a new city or a stressful job) increases both loneliness and social media use.
|
Third-variable problem
|
The third-variable problem is the possibility that some extraneous variable is producing the relationship. When we know a particular uncontrolled third variable is operating, we call it a confounding variable (or confound).
A classic example: ice cream sales and drowning deaths rise and fall together across the year. Ice cream does not cause drowning. Hot weather (the third variable) increases both.
This is the origin of the famous warning that correlation does not prove causation. A nonexperimental study can show that two variables are related, but, on its own, it cannot show that one causes the other.
The experimental method
The experimental method is designed to remove those ambiguities. Researchers do two things:
1. Manipulate the independent variable (IV): they decide which participants receive which level of it.
2. Measure the dependent variable (DV), to see whether it changes in response.
To make sure any difference in the DV is really due to the IV, experimenters use two forms of control:
• Experimental control: hold every other feature of the situation constant. Everyone gets the same instructions, the same room, the same time of day, and so on. If only one thing differs between the groups, only that thing can explain a difference in the results.
• Randomization: some things cannot be held constant (people's age, mood, intelligence, or sleep last night). For these, researchers use random assignment, in which chance decides who goes into which group. Over many participants, random assignment spreads these differences about equally across the groups, so they are unlikely to explain the results.
A worked example. Does caffeine affect reaction time?
• Independent variable: caffeine dose, manipulated at three levels: 0 mg (placebo), 100 mg, and 200 mg.
• Dependent variable: reaction time on a computerized task, measured in milliseconds.
• Experimental control: every participant tests in the same quiet room, at the same time of day, with identical instructions, and tests exactly 30 minutes after drinking.
• Randomization: a random process assigns each participant to one of the three doses. People who are naturally fast or slow, rested or tired, heavy or light coffee drinkers are spread about equally across groups.
If the groups differ in reaction time at the end, the experimenter can be fairly confident that caffeine caused the difference, because the groups were treated the same in every other way.
Independent and dependent variables
When researchers study the relationship between two variables, they typically think of one as the presumed cause and the other as the presumed effect.
• The independent variable is the presumed cause. In an experiment, the researcher manipulates it.
• The dependent variable is the presumed effect. It is what gets measured, and it "depends" on the independent variable.
A helpful memory trick: the researcher independently decides which level of the IV each participant receives, and the participant's behavior (the DV) depends on what happened to them.
Internal validity and the experimental method
Recall that internal validity is our ability to draw conclusions about cause and effect. A study has strong internal validity when all three of these conditions are met:
1. Temporal precedence: the cause comes before the effect. In the caffeine study, participants drank the caffeine before their reaction times were measured.
2. Covariation of cause and effect: the variables are related. Groups with different doses actually differ in reaction time.
3. Elimination of plausible alternative explanations: no other reasonable explanation accounts for the results. Experimental control and random assignment exist to rule these out.
The experimental method is the best tool researchers have for meeting all three. Nonexperimental studies can show covariation, but they struggle with the other two.
4.6 Experimental methods: additional considerations
Experiments are powerful, but they are not perfect, and they are not always possible.
Laboratory versus field experiments
In a laboratory, researchers can exert tight control. But the real world is not tightly controlled, and that gap leads to a question about external validity: can the results be generalized to other populations and settings?
The artificiality of a laboratory can limit external validity. People in a lab know they are being watched, face unusual tasks, and are often college students, a group that may not represent everyone.
One response is the field experiment, an experiment conducted in a natural setting. For example, a researcher might test whether a sign encouraging handwashing changes behavior by posting it in some public restrooms and not others, and then counting how many people wash. Field experiments feel more realistic, but the researcher has less control over extraneous variables. This is a trade-off between internal and external validity, and researchers must decide which matters more for their question.
When we cannot manipulate a variable
Some variables cannot be manipulated, and some should not be.
• Participant variables (also called subject variables) are characteristics of individuals that a researcher cannot assign: age, sex, personality, family background, a medical diagnosis. You cannot randomly assign a person to be 20 or 60.
• Ethical constraints. It would be unethical to randomly assign people to be abused, to take up smoking for 20 years, or to experience a severe trauma.
• Practical constraints. Some manipulations are simply impossible or too expensive.
When a variable cannot be manipulated, researchers rely on nonexperimental methods. These studies cannot offer the same causal conclusions, but they can still teach us a great deal.
Questions an experiment cannot answer
Some research goals do not require testing cause and effect at all. If a researcher wants to describe how often teenagers experience loneliness, an experiment is not the right tool. Likewise, successful prediction of future behavior is often possible without knowing the cause. An insurance company can predict that teen drivers are riskier without proving why.
The advantage of multiple methods
No single method is perfect, so the strongest understanding comes from using both experimental and nonexperimental studies on the same topic. Consider the link between smoking and lung cancer. We cannot ethically randomly assign people to smoke for decades. Instead, scientists combined large nonexperimental studies of people who chose to smoke or not, laboratory experiments on animals that were exposed to smoke, and studies of biological mechanisms. Each had limits, but together they produced convincing evidence.
When different methods, with different weaknesses, point to the same conclusion, our confidence in the conclusion rises.
4.7 Evaluating research: the four validities revisited
Validity is the idea that, given everything we know, a conclusion is reasonably accurate. When you read a study (or design your own), you can evaluate it by asking four questions:
Table 4.6. Using the four validities to evaluate a study
|
Validity
|
Ask yourself
|
Example of a problem
|
|
Construct validity
|
Do the measures and manipulations truly represent the constructs?
|
Using a mile run to measure "athleticism" overall
|
|
Internal validity
|
Can we conclude that the IV caused the change in the DV?
|
Participants who chose their own group differ in motivation, which could explain the results
|
|
External validity
|
Would the findings hold for other people, places, and times?
|
A result found only in first-year psychology majors at one university
|
|
Statistical conclusion validity
|
Were the statistical conclusions accurate?
|
Too few participants to detect a real effect, or an inappropriate statistical test
|
Statistical conclusion validity is the one we have not discussed in depth. It asks whether the researcher used appropriate statistical procedures and reached conclusions the data support, for example, avoiding claiming a relationship that is really just chance. You will learn how this works in later chapters.
No study is perfect on all four. Improving one can come at a cost to another (more lab control raises internal validity but may lower external validity). Good researchers know the trade-offs and are honest about them.
Key terms
Variable: anything that varies and can be measured.
Level (value): one of the possible values of a variable.
Operational definition: the procedures used to measure or manipulate a variable.
Construct: the abstract concept a researcher wants to study.
Validity: the extent to which a conclusion is reasonably accurate.
Construct validity: the accuracy of operational definitions.
Internal validity: the accuracy of conclusions about cause and effect.
External validity: the extent to which findings generalize.
Statistical conclusion validity: the accuracy of conclusions drawn from statistics.
Positive linear / negative linear / curvilinear relationship: patterns relating two variables.
Correlation coefficient: a number describing the strength of a relationship.
Nonexperimental (correlational) method: observing whether variables vary together.
Experimental method: manipulating one variable and controlling others.
Independent variable: the manipulated, presumed cause.
Dependent variable: the measured, presumed effect.
Experimental control: holding extraneous variables constant.
Random assignment (randomization): using chance to place participants in conditions.
Directionality problem: uncertainty about which variable causes the other.
Third-variable problem: an extraneous variable may explain the relationship.
Confounding variable: a known uncontrolled third variable.
Temporal precedence: the cause comes before the effect.
Field experiment: an experiment in a natural setting.
Participant variable: a characteristic of individuals that cannot be manipulated.
Check your understanding
1. In your own words, explain the difference between a variable and an operational definition. Give an example of each for the construct happiness.
2. A study measures "intelligence" using only a person's shoe size. Which type of validity is the main problem? Explain.
3. Classify each as positive, negative, curvilinear, or no relationship: (a) outside temperature and number of winter coats worn, (b) caffeine and alertness (alertness rises, then drops at very high doses), (c) height and the last digit of a phone number.
4. A news story reports that children who eat breakfast get better grades. Name one directionality explanation and one third-variable explanation. Could this study support the claim that breakfast causes better grades? Why or why not?
5. A researcher randomly assigns 60 students to either listen to classical music or sit in silence while solving puzzles. Identify the IV, the DV, and two features of the study that provide experimental control.
6. Why does random assignment strengthen internal validity?
7. Name the three requirements for internal validity and describe how each is met in the caffeine experiment.
8. Why might a field experiment have higher external validity but lower internal validity than a lab experiment?
9. Give one reason a researcher would use a nonexperimental method even though an experiment offers stronger evidence for causation.
10. Explain why using multiple methods increases confidence in a conclusion.
Activity: revisit the athleticism test
Go back to the test you outlined at the start of this chapter.
1. List the constructs you think make up athleticism (for example, speed, strength, endurance, agility).
2. Write an operational definition for each one: the exact task and how it will be scored.
3. Evaluate your own test for construct validity. Which parts of athleticism are well captured, and which are missing?
4. Imagine you want to test whether a six-week training program improves athleticism. Sketch a simple experiment: What is the IV? The DV? How would you use random assignment and experimental control?
5. Now name one external validity concern about your plan (for example, who would your participants be?).
Be ready to share your test with a classmate, and notice how different the tests are. That variety is exactly why operational definitions must be spelled out.
References
Bridgman, P. W. (1927). The logic of modern physics. Macmillan.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Rand McNally.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Houghton Mifflin.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Stevens, S. S. (1935). The operational definition of psychological concepts. Psychological Review, 42(6), 517–527.
Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680.