Source-linked AI summary
Reproducible Research Can Still Be Wrong: Adopting a Prevention Approach
Jeffrey T. Leek, Roger D. Peng
TL;DR
Scientific confidence is challenged by analyses that may be reproducible yet invalid, while peer review alone may not reliably detect poor analysis or address its source. The paper proposes prevention through expanded data-analysis education coupled with empirical identification of reproducible and replicable methods and tools.
Problem
Reproducibility and replicability are threatened by flawed analyses, including confounding, poor study design, and missing data, while peer review faces growing difficulty identifying such problems.
Method
The paper proposes primary prevention by increasing trained data analysts and empirically identifying statistical methods, protocols, and software that improve reproducibility and replicability for basic or intermediate users.
Results
The paper argues that combining large-scale education with evidence-based data analysis can quickly test analytic practices among users most at risk of data-analytic mistakes.
Takeaways & Limitations
Maintaining scientific integrity and public trust requires routinely using software tools alongside substantially expanded data-analysis education.
Takeaways & Limitations
Peer reviewers and editors often lack the training and time for proper data-analysis evaluation, and the medication approach does not address problems at their source.
Abstract
from arXiv · showhide
Reproducibility, the ability to recompute results, and replicability, the chances other experimenters will achieve a consistent result, are two foundational characteristics of successful scientific research. Consistent findings from independent investigators are the primary means by which scientific evidence accumulates for or against an hypothesis. And yet, of late there has been a crisis of confidence among researchers worried about the rate at which studies are either reproducible or replicable. In order to maintain the integrity of science research and maintain the public's trust in science, the scientific community must ensure reproducibility and replicability by engaging in a more preventative approach that greatly expands data analysis education and routinely employs software tools.