7.6: Problems in Public Opinion Measurement

Polling is an inexact science. Often two polls purport to describe the same population over the same time period but contradict one another in such a way that both cannot be correct. Such differences could arise simply by luck of the draw, but there are also several types of error which can distort poll results.

Coverage error occurs when not everyone in the population is equally likely to be sampled. Some of the earliest American polls were conducted in taverns and on trains, resulting in samples that were easily accessible but poor matches for the overall public. In the early 2000s, telephone pollsters often did not call cell phones, which led them to undersample young people and non-homeowners, who tended to not own landline phones. (The Telephone Consumer Protection Act of 1991 banned the use of computer-based random-digit-dialing to call cell phones, making them more expensive to survey than landlines.) These biases can change the composition of the sample in numerous ways, making it an inaccurate representation of the population.

Nonresponse error occurs when not everyone who was sampled responds to a poll. Over time, nonresponse rates in polling have increased: today, a telephone pollster who calls ten people can expect nine of them (on average) to hang up without completing the poll. If nonresponse is nonrandom, it can make the poll less accurate, even if the sample itself is representative. The young, the employed, and the politically disengaged are less likely to complete a public opinion poll about politics than the old, the unemployed, and the politically engaged, all of whom tend to have more time for and interest in sharing their political views.

Measurement error occurs when the questions asked on a poll fail to adequately capture the opinions being sought. This can occur because the questions are worded in a confusing or biased manner, because the answer choices given to respondents do not cover all plausible options, or because the respondents are unable or unwilling to answer the questions accurately. In the United States, for example, self-reported voter turnout is typically much higher than actual voter turnout, because nonvoters often lie to pollsters by falsely claiming to have performed their civic duty.

A poll about marijuana legalization could easily produce each of these types of error. It could frame marijuana use in unflattering terms, thereby artificially depressing support. It could also force respondents to choose only “support” or “oppose,” without including an option for people who believe marijuana should be legal for medical but not recreational purposes. Finally, respondents might lie when asked whether they smoke marijuana to avoid the social stigma associated with it. In each case, the results would be distorted.

Even an error-free poll would at best be a measurement of the way things are at a particular moment in time. Attitudes and especially opinions change with new events and circumstances. There is no guarantee that a poll conducted today will still accurately reflect public opinion a year, month, week, or day later. If an election poll taken in October does not match the results of the election held in November, that does not prove the October poll was wrong, only that it was inaccurate by the time the votes were cast.