Are open evidence sites unbiased?

Are Open Evidence Sites Unbiased?

No evidence source is completely unbiased, and open evidence sites face the same systematic biases that plague all medical research—including publication bias, selective outcome reporting, and methodological limitations that cannot be eliminated simply by making data openly accessible.

Understanding Bias in Evidence Synthesis

The fundamental challenge is that bias exists at multiple levels in the research ecosystem, regardless of whether evidence is openly accessible or behind paywalls 1. The PRISMA guidelines explicitly require assessment of "risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies)" because these biases are pervasive 1.

Publication and Selective Reporting Bias

• Only 36% of systematic reviews published in 2004 even assessed for publication bias, and only a quarter intended formal assessment 1
• A review of antidepressant trials found that meta-analysis of only published trials gave effect estimates 32% larger on average than when all trials submitted to drug agencies were analyzed 1
• In only half of 196 trial reports comparing drugs in arthritis were all effect variables in the methods and results sections the same, indicating widespread selective outcome reporting 1

Methodological Quality Issues

• Most meta-analyses are based on fewer than ten studies, display statistically significant heterogeneity, and reviewers frequently misinterpret the test results 1
• The quality of underlying data matters enormously—for example, self-reported dietary data showed measurement 'noise' (error) more than nine times greater than the 'signal' (valid information), rendering conclusions suspect regardless of open access 1

Specific Concerns About Evidence Quality

Adherence to Reporting Standards

• Only 40% of journals adequately endorse CONSORT guidelines, only 12.5% adequately endorse PRISMA, and only 37.5% require trial registration 1
• Lack of adherence to reporting guidelines allows biased studies to be published, doing a disservice to readers and preventing proper scientific evaluation 1

Transparency Does Not Equal Validity

• Making protocols available can help identify when authors deviate from pre-specified outcomes, but empirical evidence shows authors often ignore what they stated in protocols 1
• Statistical machinations, however sophisticated, cannot overcome systematic recall bias that renders inferences suspect 1

The Reality of Research Bias

Financial and Institutional Conflicts

• Funding source is sometimes associated with estimated treatment effects, necessitating transparent reporting 1
• The potential for bias due to financial conflicts remains in many guideline groups 2

Incomplete Evidence Base

• High-quality evidence from RCTs is often incomplete, contradictory, or absent even in areas that have been most exhaustively studied 2
• Bad reporting of RCTs is common, which has serious consequences for clinical practice, research, policy making, and ultimately for patients 3

Critical Evaluation Framework

When using any evidence source, open or otherwise:

• Assess whether publication bias was formally evaluated using funnel plots and statistical tests 1
• Compare protocols (when available) with published outcomes to identify selective reporting 1
• Evaluate adherence to reporting standards like CONSORT for trials and PRISMA for systematic reviews 1, 3
• Consider residual confounding and exposure ascertainment as particular areas of concern 1
• Recognize that observational studies have inherent weaknesses and specific methodological pitfalls 1

The Path Forward

Three achievable actions could deliver significantly improved transparency and mitigation of bias: (1) mandatory registration of interests by those involved in research; (2) journals supporting 'registered reports' publication format; and (3) comprehensive study documentation for all publicly funded research made available on a WHO research repository 4.

Independent verification by a community of scientists remains the time-honored process for arriving at truth, with journals serving critical evaluative and editorial functions to ensure transparent and objective research descriptions 5.