7.5: Problems in Public Opinion Measurement

Polling is a science, but an inexact one. Often two polls purport to describe the same population over the same time period but contradict one another in such a way that both cannot be correct. There are several potential sources of error which can bias the polling process and therefore the accuracy of poll results.

Coverage error occurs when not everyone in the population is equally likely to be sampled. Some of the earliest (unscientific) polls in America were conducted in taverns and on trains, resulting in samples that were easily accessible but poor matches for the overall public. Until relatively recently, modern telephone pollsters often did not call cell phones, which led to them undersampling young people and non-homeowners (who tended to not own landline phones). These biases can change the composition of the sample in numerous ways, making it an inaccurate representation of the population.

Nonresponse error occurs when not everyone who was sampled responds to a poll. Over time, nonresponse rates in polling have increased: today, a telephone pollster who calls ten people can expect nine of them (on average) to hang up without completing the poll. If this pattern of nonresponse is nonrandom, it can make the poll less accurate, even if the sample itself is representative. The young, the employed, and the politically disengaged are less likely to respond to an invitation to take a public opinion poll about politics than the old, the unemployed, and the politically engaged, all of whom tend to have more time for and interest in sharing their political views.

A jubliant Harry S. Truman holds up a newspaper erroneously proclaiming his defeat in the 1948 presidential election, based on early polls that failed to capture Truman's eleventh-hour comeback.

Measurement error occurs when the questions asked on a poll fail to adequately capture the opinions being sought. This failure can occur because the questions are worded in a confusing or biased manner, because the answer choices given to respondents do not cover all plausible options, or because the respondents themselves are unable or unwilling to answer the questions accurately. In the United States, for example, self-reported voter turnout is typically much higher than actual voter turnout, because nonvoters have a tendency to lie to pollsters and claim that they performed their civic duty.

A poll about marijuana legalization could easily produce each of these types of error. The poll could frame marijuana use in unflattering terms, thereby artificially depressing support. It could also force respondents to choose only “support” or “oppose,” without including an option for people who believe marijuana should be legal for medical but not recreational purposes. Finally, respondents might lie when asked whether they smoke marijuana to avoid the social stigma attached to marijuana use. In each case, public opinion would be distorted by the error.

No poll is completely without error, and even an error-free poll would at best be a measurement of the way things are at a particular moment in time. Attitudes and especially opinions change with new events and circumstances, and there is no guarantee that a poll conducted today will still accurately reflect public opinion a year, month, week, or day later. If an election poll taken in October does not match the results of the election held in November, that does not mean the October poll was wrong, only that it was no longer accurate by the time the votes were cast.