10.3: Measuring Public Opinion

Learning Objectives

By the end of this section, you will be able to:

• Describe how to measure public opinion.
• Discuss common polling problems.

Public Opinion Polls

Researchers use a variety of techniques to gauge public opinion, and the most common tool that researchers use is a poll. To conduct a poll, a researcher selects a group of subjects, called a sample, from a broader pool of citizens, called the population. The subjects in the sample respond to polling questions, and researchers then use their answers to make inferences about the broader population. Why not simply poll the entire population? Imagine that a researcher is interested in comparing responses across India and China, the two most populous states in the world, with a combined population nearing three billion people. It would be impossible to poll all Indian and all Chinese citizens, so researchers must rely on representative samples of the Indian and Chinese populations. A representative sample is a sample from a population that accurately represents the characteristics of that population, which researchers achieve through the process of randomization. Randomization is when every individual in the population has an equal chance to be chosen as part of the sample. By analyzing the results from a random, representative sample, a researcher aims to make reasonable inferences about what the larger population believes.

As a simple example, imagine you are making a large pot of spaghetti sauce that you plan to serve at a party. Before serving it to others, you will taste-test the sauce. You will use a spoon to take a sample from the pot, then make a reasonable inference about the taste of the entire batch of sauce. In order for this sampling process to work accurately, the small spoon you use for your taste-test must contain all the ingredients and seasonings at the same proportions as they exist in the larger pot of sauce. This will give you a representative sample of sauce. A similar dynamic exists in polling (NBC News Learn, 2020); if a researcher wants to gauge what a given public (the population of interest) believes on a set of political issues, then their sample must include all the combinations of demographics and regional influences that exist in the larger body.

Even with a random sample, all polls have a "margin of error." The margin of error is a statistical estimation of the accuracy of the results from a sample. That is, the margin of error quantifies uncertainty; it offers a range of how much the results from a poll, based on a sample, may differ from the true values for the whole population. A larger random sample (all else equal) will have a smaller margin of error (it will be more accurate) than a smaller one.

Returning to the spaghetti sauce example, imagine the difference between taste-testing with a large versus a small spoon. If you use a large spoon, you are likely to get more of the ingredients and seasonings than if you use a small spoon. The sample from a large spoon will have a lower margin of error than a sample from a small spoon.

When reading poll results, the margin of error appears as a "+/-" classification. If the result of a poll (based on a random, representative sample) shows that 35 percent of respondents are satisfied with their government and the margin of error is "+/- 5 percent," this means that pollsters believe anywhere between 30 percent (35-5) and 40 percent (35+5) of the population is satisfied with their government. How does a researcher know if the actual number is 30, 32, 35, 37, or 40 percent? They simply do not know. If a researcher collects a larger sample (all else equal), they will be able to reduce the margin of error and provide a more precise estimate, but uncertainty will always exist when making inferences from a sample to a population. Polling is a tool that estimates public opinion.

To create these estimates, researchers design and field surveys in different ways. Some polls are conducted in person; some polls are conducted online; and some polls are conducted over the phone, including through a process called random digit dialing (RDD). RDD polls allow researchers to use computers to contact large numbers of people within a short period of time. By gathering a large sample, researchers can produce more accurate results. Yet RDD polls also have problems. They exhibit bias against individuals who do not own phones (and therefore cannot be included in the sample), and they also exhibit bias because some individuals will be more likely to answer their phones than others (e.g., an older, retired person may be more likely to answer than a younger, working person).

No matter the chosen survey design, in polling, concerns about bias, randomization, and sample size are ubiquitous.

Additional Concerns in the Comparative Context

Even if a researcher believes their poll has little bias, good randomization, and a sufficiently large sample, there remain well-known issues related to survey question design, and these can be especially difficult to overcome in the comparative context.

The first issue is called "priming." Priming influences a respondent's answer to a survey question by pushing a respondent to think about a certain subject matter that they otherwise may not have been thinking about (Lenz, 2019). As an example, imagine that a survey researcher wishes to study attitudes toward far-right political parties. If the researcher asks questions about national identity or immigration concerns before they ask about the respondent's feelings toward the far-right, that researcher has primed the respondent through the earlier questions. The respondent may not have been thinking about attitudes toward identity and immigration on their own, and priming in this manner can skew the survey results related to attitudes about particular far-right political parties. In the comparative context, the way respondents assess questions will vary significantly by state; in the prior example, respondents in states with a history of far-right political parties may respond differently to nationalistic priming than states without such a history. Trying to minimize priming effects is therefore especially difficult (yet crucial) in the comparative context due to the complexity of cases involved.

The second issue, which can also skew results, is called "framing" (Nelson and Oxley, 1999). Framing influences a respondent's answer to a survey question based on how the question is presented. This may occur if a researcher uses emotionally-charged language in a survey question. Suppose that, in the same study about far-right political parties, a researcher wants to measure the amount of support for a far-right political party within a state. A researcher could word the survey question as follows: "Some people argue that far-right political parties pose an existential threat to democracy. To what extent do you support the far right?" Or, a researcher could alternatively word the survey question in this manner: "Some people argue that far-right political parties give a voice to citizens who have otherwise been left behind by traditional parties. To what extent do you support the far right?" Both of these questions employ clear frames; the first frames far-right political parties as a threat while the second frames far-right political parties as centering the experience of the average person. Just as with priming, the way individuals respond to frames may vary significantly across states, making it especially tough (yet important) to minimize framing effects.

The third issue is social desirability bias. Social desirability bias occurs when respondents give answers to questions that they think the broader society (or an interviewer) sees as "right" or answers that they think will make them look "good" in some way. If a state has a history of violent far-right political parties, for example, a respondent may not feel comfortable admitting that they support a far-right political party's anti-immigration stance. Because the respondent worries about external judgment, they will not accurately report their views. In the comparative context, what respondents see as socially desirable can vary considerably across states, and this depends on a complex slew of factors from aspects of a state's history to a state's socioeconomic status to a state's cultural identity. And, if even a relatively small number of people in the sample misreport due to social desirability bias (for whatever underlying reasons), it will skew the survey results (Streb, et al., 2008).

While there are many well-founded concerns about public opinion polls, there are also many excellent researchers trained in techniques to minimize these problems. In the next section, we will examine a series of polls that aim to conduct analysis of comparative public opinion.