3.7: Bias

By writing scientific articles we communicate science among colleagues and peers. By doing this, it is our responsibility to adhere to some basic principles like transparency and accuracy. Authors, journal editors and reviewers need to be concerned about the quality of the work submitted for publication and ensure that only studies which have been designed, conducted and reported in a transparent way, honestly and without any deviation from the truth get to be published. Any such trend or deviation from the truth in data collection, analysis, interpretation and publication is called bias. Bias in research can occur either intentionally or unintentionally. Bias causes false conclusions and is potentially misleading. Therefore, it is immoral and unethical to conduct biased research. Every scientist should thus be aware of all potential sources of bias and undertake all possible actions to reduce or minimize the deviation from the truth. This article describes some basic issues related to bias in research.

Definition of bias

Bias in data collection – volunteer bias, admission bias (like when only hospitalized patients are in study), survivor bias (like in the Seattle longitudinal study, or worse in cross sectional research where patients have died), and misclassification bias (where people may be put in the control group erroneously).

Bias in data analysis

Data analysis bias can occur in a number of different ways -

• reporting non-existing data from experiments which were never done (data fabrication);
• eliminating data which do not support your hypothesis (outliers, or even whole subgroups);
• using inappropriate statistical tests to test your data;
• performing multiple testing (“fishing for p”) by pair-wise comparisons (4), testing multiple endpoints and performing secondary or subgroup analyses, which were not part of the original plan in order to “find” statistically significant difference regardless to hypothesis.

For example, if the study aim is to show that one biomarker is associated with another in a group of patients, and this association does not prove significant in a total cohort, researchers may start “torturing the data” by trying to divide their data into various subgroups until this association becomes statistically significant. If this sub-classification of a study population was not part of the original research hypothesis, such behavior is considered data manipulation and is neither acceptable nor ethical. Such studies quite often provide meaningless conclusions such as with very specific groups that then become meaningless.

There is a very often quoted saying (attributed to Ronald Coase, but unpublished to the best of my knowledge), which says: “If you torture the data long enough, it will confess to anything”. Actually, it is well known that if 20 tests are performed on the same data set, at least one Type 1 error (α) is to be expected. Therefore, the number of hypotheses to be tested in a certain study needs to determined in advance. If multiple hypotheses are tested, correction for multiple testing should be applied or the study should be declared as exploratory.

Bias in data interpretation

There are also biases in interpretation that occur -

• discussing observed differences and associations even if they are not statistically significant (the often used expression is “borderline significance”);
• discussing differences which are statistically significant but are not clinically meaningful;
• drawing conclusions about the causality, even if the study was not designed as an experiment;
• drawing conclusions about the values outside the range of observed data (extrapolation);
• overgeneralization of the study conclusions to the entire general population, even if a study was confined to the population subset;
• Type I (the expected effect is found significant, when actually there is none) and Type II (the expected effect is not found significant, when it is actually present) errors (6).

Publication bias

The problem is that mostly positive results are most likely to be published whereas that should not be the case. So to enable publication of studies reporting negative findings, several journals have already been launched, such as Journal of Pharmaceutical Negative Results, Journal of Negative Results in Biomedicine, Journal of Interesting Negative Results and some other. The aim of such journals is to counterbalance the ever-increasing pressure in the scientific literature to publish only positive results.

The issue of culture

One of the big issues that psychology has been rightly accused of is that much of our research has been conducted with WEIRD (Western, educated, individualistic, rich, democratic) participants. In fact WEIRD participants constitute the data in 95% of studies, but they constitute only 12% of the world’s population. This limitation is important to consider because we might assume things like the Muller Lyer illusion is universal, when in fact research (Segall’s research with 17 cultures’ children and adults) has shown that it is not. Obviously, research needs to be done with people from other cultures to ensure that those “facts” we consider unassailable are actually universal. Also, it is hard to use our tools in other cultures. The most obvious example is that our tests are written in English so a verbal test that measures self esteem will not be understood by Chinese students in China. But even more seemingly universal (because they are biological) tests like EEG are not necessarily foolproof. If infants are put to bed in positions different from Western children, the shape of their heads would preclude our knowing where to place electrodes on their head for research using EEG. Since development is influenced by interacting systems within us and in our environment there is likely no universal standard that can be used. So, a grand average would likely be descriptive of no one.

Child development is a fascinating field of study – but care must be taken to ensure that researchers use appropriate methods to examine infant and child behavior, use the correct experimental design to answer their questions, and be aware of the special challenges that are part-and-parcel of developmental research. Hopefully, this information helped you develop an understanding of these various issues and to be ready to think more critically about research questions that interest you. There are so many interesting questions that remain to be examined by future generations of developmental scientists – maybe you will make one of the next big discoveries![2]

For More Information

Got Bias - Exploring Implicit bias in early childhood education by Tina Sykes has information about reducing implicit bias when working with children and families.

Attributions:

Bias in research from Simundić AM. Bias in research. Biochem Med (Zagreb). 2013;23(1):12-5. doi: 10.11613/bm.2013.003. PMID: 23457761; PMCID: PMC3900086. Licensed CC-BY 4.0

[1] Confidentiality and Informed Consent: Issues for Consideration in the Preservation of and Provision of Access to Qualitative Data Archives by Louise Corti, Annette Day & Gill BackhouseSource is licensed under CC BY 4.0 (modified by Jennifer Paris); "No thank you, not today": Supporting Ethical and Professional Relationships in Large Qualitative Studies by Lisa J. Blodgett, Wanda Boyer & Emily TurkSource is licensed under CC BY 4.0 (modified by Jennifer Paris)

[2] Research Methods in Developmental Psychology by Angela Lukowski and Helen Milojevich is licensed under a CC BY-NC-SA 4.0

Got Bias - Exploring Implicit bias in early childhood education by Tina Sykes