Learning Objectives
- Explain the difference between major types of sampling.
- List some techniques that can be used to increase the response rate and reduce non-response bias, and why this is important.
Now that you have a sense of a how a sample is the subset of the population of interest whom researchers actually collect data from, let's discuss different methods for recruiting those participants; this is sampling.
Sampling
Essentially all research involves sampling—selecting a sample to study from the population of interest. Sampling falls into two broad categories: Probability Sampling or Non-Probability Sampling. The first category, probability sampling, occurs when the researcher can specify the probability that each member of the population will be selected for the sample. The second category, non-probability sampling, occurs when the researcher cannot specify probability that each member of the population will be selected for the sample.
Probability Sampling
When conducting quantitative research are large samples, probability sampling is the preferred method because the goal of most research is to make accurate estimates about what is true in a particular population, and these estimates are most accurate when based on a probability sample. For example, it is important for survey researchers to base their estimates of election outcomes—which are often decided by only a few percentage points—on probability samples of likely registered voters.
Probability sampling requires a very clear specification of the population; the population of interest depends on the research questions to be answered. The population might be all registered voters in California, all American consumers who have used a ride share app in the past year, people in Seattle over 40 years old who have received a mammogram in the past decade, or all the alumni of a particular college. Once the population has been specified, probability sampling requires a sampling frame. A sampling frame is essentially a list of all the members of the population from which to select the respondents. Sampling frames can come from a variety of sources, including enrollment records, lists of registered voters, or hospital records. In some cases, a map can serve as a sampling frame, allowing for the selection of cities, streets, or households.
There are a variety of different probability sampling methods. Simple random sampling, often just called random sampling, is done in such a way that each individual in the population has an equal probability of being selected for the sample. This type of sampling could involve putting the names of all individuals in the sampling frame into a hat, mixing them up, and then drawing out the number needed for the sample. Given that most sampling frames take the form of computer files, random sampling is more likely to involve computerized sorting or selection of respondents. A common approach in telephone surveys is random-digit dialing, in which a computer randomly generates phone numbers from among the possible phone numbers within a given geographic area. Random sampling is sometimes also called random selection since researchers are selecting participants out of the population (through a sampling frame) randomly.
A common alternative to simple random sampling is stratified random sampling, in which the population is divided into different subgroups or “strata” (usually based on demographic characteristics) and then a random sample is taken from each “stratum.” Slightly more complicated, proportionate stratified random sampling can be used to select a sample in which the proportion of respondents in each of various subgroups matches the proportion in the population. For example, because about 12.6 % of the American population is African American , stratified random sampling can be used to ensure that a survey of 1,000 American adults includes about 126 African-American respondents. Disproportionate stratified random sampling can also be used to sample extra respondents from particularly small subgroups—allowing valid conclusions to be drawn about those subgroups. For example, because Asian Americans make up a relatively small percentage of the American population (about 5.6%), a simple random sample of 1,000 American adults might include too few Asian Americans to draw any conclusions about them as distinct from any other subgroup. If representation is important to the research question, however, then disproportionate stratified random sampling could be used to ensure that enough Asian-American respondents are included in the sample to draw valid conclusions about Asian American s a whole.
Non-Probability Sampling
Despite the benefit of probability samples to create samples that better represent the population of interest, most social science research involves non-probability sampling. For example, convenience sampling (studying individuals who happen to be nearby and willing to participate) is a very common form of non-probability sampling used in social science research. Other forms of non-probability sampling include snowball sampling (existing research participants help recruit additional participants for the study) and self-selection sampling (in which individuals choose to take part in the research on their own accord, without being approached by the researcher directly). Snowball sampling is often used with populations that are small or rare, such as transgender People of Color in rural areas, as current participants can connect researchers to other group members. An example of self-selection sampling could be when a friend posts their results of an online survey, and then you decide to also take the survey.
Sample Size
When you are designing a study and choosing how to recruit your participants, it is necessary to decide how many participants you want to collect data from; how large does a sample need to be? In general, this estimate depends on two factors. One is the level of confidence in the result that the researcher wants. The larger the sample, the closer any statistic based on that sample will tend to be to the corresponding value in the population. The other factor is a practical constraint in the form of the budget of the study. Larger samples provide greater confidence, but they take more time, effort, and money to obtain. Taking these two factors into account, most survey research uses sample sizes that range from about 100 to about 1,000. Conducting a power analysis prior to launching the survey helps to guide the researcher in making this trade-off.
Sample Size and Population Size
Why is a sample of about 1,000 considered to be adequate for most survey research—even when the population is much larger than that? Consider, for example, that a sample of only 1,000 American adults is generally considered a good sample of the roughly 252 million adults in the American population—even though it includes only about 0.000004% of the population! The answer is a bit surprising.
One part of the answer is that a statistic based on a larger sample will tend to be closer to the population value and that this can be characterized mathematically. Imagine, for example, that in a sample of registered voters, exactly 50% say they intend to vote for the incumbent. If there are 100 voters in this sample, then there is a 95% chance that the true percentage in the population is between 40 and 60. But if there are 1,000 voters in the sample, then there is a 95% chance that the true percentage in the population is between 47 and 53. Although this “95% confidence interval” continues to shrink as the sample size increases, it does so at a slower rate. For example, if there are 2,000 voters in the sample, then this reduction only reduces the 95% confidence interval to 48 to 52. In many situations, the small increase in confidence beyond a sample size of 1,000 is not considered to be worth the additional time, effort, and money.
Another part of the answer—and perhaps the more surprising part—is that confidence intervals depend only on the size of the sample and not on the size of the population. So a sample of 1,000 would produce a 95% confidence interval of 47 to 53 regardless of whether the population size was a hundred thousand, a million, or a hundred million.
Sampling Bias
Probability sampling was developed in large part to address the issue of sampling bias. Sampling bias occurs when a sample is selected in such a way that it is not representative of the entire population and therefore produces inaccurate results. For example, if you sample from all Californians with driver's licenses, you are missing teens younger than 16, many teens in cities who choose not to get their license as soon as legally possible, and drivers who have lost their license due to traffic violations.
There is one form of sampling bias that even careful random sampling is subject to. It is almost never the case that everyone selected for the sample actually responds to the survey. Some may have died or moved away, and others may decline to participate because they are too busy, are not interested in the survey topic, or do not participate in surveys on principle. If these survey non-responders differ from survey responders in systematic ways, then this difference can produce non-response bias. For example, in a mail survey on alcohol consumption, researcher Vivienne Lahaut and colleagues found that only about half the sample responded after the initial contact and two follow-up reminders (Lahaut et al., 2002). The danger here is that the half who responded might have different patterns of alcohol consumption than the half who did not, which could lead to inaccurate conclusions on the part of the researchers. So to test for non-response bias, Lahaut et al. (2002) later made unannounced visits to the homes of a subset of the non-responders; they came back up to five times if they did not find the participants at home! Lahaut et al. (2002) found that the original non-responders included an especially high proportion of abstainers (nondrinkers), which meant that their estimates of alcohol consumption based only on the original responders was too high.
Although there are methods for statistically correcting for non-response bias, they are based on assumptions about the non-responders; for example, that they are more similar to late responders than to early responders—which may not be correct. For this reason, the best approach to minimizing non-response bias is to minimize the number of non-responders—that is, to maximize the response rate. There is a large research literature on the factors that affect survey response rates (Groves et al., 2004). In general, in-person interviews have the highest response rates, followed by telephone surveys, and then mail and Internet surveys. Among the other factors that increase response rates are sending potential respondents a short pre-notification message informing them that they will be asked to participate in a survey in the near future and sending simple follow-up reminders to non-responders after a few weeks. The perceived length and complexity of the study can also make a difference, which is why it is important to keep survey questionnaires as short, simple, and on topic as possible. Finally, offering an incentive—especially cash—is a reliable way to increase response rates. However, ethically, there are limits to offering incentives that may be so large as to be considered coercive.
You've now gotten an overview of the scientific method. Let's summarize, and then learn about ethics in research.
References
