2.5A: Surveys

The goal of a survey is to collect data from a representative sample of a population to draw conclusions about that larger population.

Learning Objectives

• Assess the various types of surveys and sampling methods used in sociological research, appealing to the concepts of reliability and validity

Key Points

• The sample of people surveyed is chosen from the entire population of interest. The goal of a survey is to describe not the smaller sample but the larger population.
• To be able to generalize about a population from a smaller sample, that sample must be representative; proportionally the same in all relevant aspects (e.g., percent of women vs. men).
• Surveys can be distributed by mail, email, telephone, or in-person interview.
• Surveys can be used in cross-sectional, successive-independent-samples, and longitudinal study designs.
• Effective surveys are both reliable and valid. A reliable instrument produces consistent results every time it is administered; a valid instrument does in fact measure what it intends to measure.

Key Terms

• cross-sectional study: A research method that involves observation of a representative sample of a population at one specific point in time.
• successive-independent-samples design: A research method that involves observation of multiple random samples from a population over multiple time points.
• longitudinal design: A research method that involves observation of the same representative sample of a population over multiple time points, generally over a period of years or decades.
• sample: A subset of a population selected for measurement, observation or questioning, to provide statistical information about the population.

Selecting a Sample to Survey

The sample of people surveyed is chosen from the entire population of interest. The goal of a survey is to describe not the smaller sample but the larger population. This generalizing ability is dependent on the representativeness of the sample.

Nuclear Energy Support in the U.S.: This pie chart shows the results of a survey of people in the United States (February 2005, Bisconti Research Inc.). According to the poll, 67 percent of Americans favor nuclear energy (blue), while 26 percent oppose it (yellow).

There are frequent difficulties one encounters while choosing a representative sample. One common error that results is selection bias —when the procedures used to select a sample result in over- or under-representation of some significant aspect of the population. For instance, if the population of interest consists of 75% females, and 25% males, and the sample consists of 40% females and 60% males, females are under represented while males are overrepresented. In order to minimize selection biases, stratified random sampling is often used. This is when the population is divided into sub-populations called strata, and random samples are drawn from each of the strata, or elements are drawn for the sample on a proportional basis.

For instance, a Gallup Poll, if conducted as a truly representative nationwide random sampling, should be able to provide an accurate estimate of public opinion whether it contacts 2,000 or 10,000 people.

Modes of Administering a Survey

There are several ways of administering a survey. The choice between administration modes is influenced by several factors, including

• costs,
• coverage of the target population,
• flexibility of asking questions,
• respondents’ willingness to participate and
• response accuracy

Different methods create mode effects that change how respondents answer, and different methods have different advantages. The most common modes of administration can be summarized as:

• Telephone
• Mail (post)
• Online surveys
• Personal in-home surveys
• Personal mall or street intercept survey
• Hybrids of the above

Participants willing to take the time to respond will convey personal information about religious beliefs, political views, and morals. Some topics that reflect internal thought are impossible to observe directly and are difficult to discuss honestly in a public forum. People are more likely to share honest answers if they can respond to questions anonymously.

Questionnaire: Questionnaires are a common research method; the U.S. Census is a well-known example.

Types of Studies

Cross-Sectional Design

In a cross-sectional study, a sample (or samples) is drawn from the relevant population and studied once. A cross-sectional study describes characteristics of that population at one time, but cannot give any insight as to causes of population characteristics.

Successive-Independent-Samples Design

A successive-independent-samples design draws multiple random samples from a population at one or more times. This design can study changes within a population, but not changes within individuals because the same individuals are not surveyed more than once. Such studies cannot, therefore, identify the causes of change over time necessarily.

For successive independent samples designs to be effective, the samples must be drawn from the same population, and must be equally representative of it. If the samples are not comparable, the changes between samples may be due to demographic characteristics rather than time. In addition, the questions must be asked in the same way so that responses can be compared directly.

Longitudinal Design

A study following a longitudinal design takes measure of the same random sample at multiple time points. Unlike with a successive independent samples design, this design measures the differences in individual participants’ responses over time. This means that a researcher can potentially assess the reasons for response changes by assessing the differences in respondents’ experiences. However, longitudinal studies are both expensive and difficult to do. It’s harder to find a sample that will commit to a months- or years-long study than a 15-minute interview, and participants frequently leave the study before the final assessment. This attrition of participants is not random, so samples can become less representative with successive assessments.

Writing survey questions: Researchers must carefully design survey questions to ensure they receive accurate and unbiased results.

Reliability and Validity

Reliable measures of self-report are defined by their consistency. Thus, a reliable self-report measure produces consistent results every time it is executed.

A test’s reliability can be measured a few ways. First, one can calculate a test-retest reliability. A test-retest reliability entails conducting the same questionnaire to a large sample at two different times. For the questionnaire to be considered reliable, people in the sample do not have to score identically on each test, but rather their position in the score distribution should be similar for both the test and the retest.

Self-report measures will generally be more reliable when they have many items measuring a construct. Furthermore, measurements will be more reliable when the factor being measured has greater variability among the individuals in the sample that are being tested. Finally, there will be greater reliability when instructions for the completion of the questionnaire are clear and when there are limited distractions in the testing environment.

Contrastingly, a questionnaire is valid if what it measures is what it had originally planned to measure. Construct validity of a measure is the degree to which it measures the theoretical construct that it was originally supposed to measure.