Source-linked AI summary
Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché-Buc, Emily Fox, Hugo Larochelle
TL;DR
Machine learning research needs stronger evidence that published results are sound and reproducible, amid difficulties involving experimental variation, reporting, and access to research artifacts. The paper examines NeurIPS 2019’s three-part reproducibility program—code submission, a community challenge, and a checklist—and reports what was learned from its deployment. The initiative provides a framework for evaluating and improving reproducibility practices across machine learning research.
Problem
Machine learning findings can be difficult to reproduce because experimental variables, reporting details, code, and statistical analyses are often insufficiently controlled or documented.
Method
The paper describes the deployment and evaluation of NeurIPS 2019’s code submission policy, community-wide reproducibility challenge, and machine learning reproducibility checklist.
Results
The program generated independent verification reports and an archival record, while code availability at submission was positively associated with reviewer score (p < 1e −08).
Takeaways & Limitations
Reproducibility initiatives can incorporate artifact sharing, independent verification, and reporting requirements into machine learning research processes.
Takeaways & Limitations
Reproduction can remain impractical when experiments require substantial computation, including training very large neural models.
Abstract
from arXiv · showhide
One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable. Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentially reduce unintentional errors. In 2019, the Neural Information Processing Systems (NeurIPS) conference, the premier international conference for research in machine learning, introduced a reproducibility program, designed to improve the standards across the community for how we conduct, communicate, and evaluate machine learning research. The program contained three components: a code submission policy, a community-wide reproducibility challenge, and the inclusion of the Machine Learning Reproducibility checklist as part of the paper submission process. In this paper, we describe each of these components, how it was deployed, as well as what we were able to learn from this initiative.
1. Introduction
Scientific findings are reliable only when experiments can be repeated with consistent results, yet reproducibility remains difficult in machine learning. NeurIPS 2019 introduced a three-part program to improve how machine learning research is conducted, communicated, and evaluated.
- 1. Introduction: More than 70% of researchers failed to reproduce another researcher’s experiments, while over 50% failed to reproduce their own.These figures come from a 2016 Nature survey.
- 1. Introduction: Machine learning reproducibility is hindered by differences in data, underspecified models and training procedures, unavailable or erroneous code, and insufficiently specified metrics.Additional problems include improper statistical testing and selective reporting with adaptive overfitting.
- 1. Introduction: Improper statistical analysis and selective reporting can further undermine the reliability of machine learning findings.Examples include claiming significance without proper testing and ignoring adaptive overfitting.
- 1. Introduction: Over-claiming occurs when conclusions extend beyond the evidence, such as when too few experiments support a claim or the hypothesis does not match it.
- 1. Introduction: NeurIPS introduced a reproducibility program comprising a code submission policy, a community-wide challenge, and a machine learning reproducibility checklist.The program was designed to improve standards for conducting, communicating, and evaluating machine learning research.
- 1. Introduction: The report describes how the program’s components were deployed and what the initiative revealed about community perceptions, scientific quality, and finding reliability.
2. Background
Reproducibility concerns in machine learning include insufficient exploration of experimental variables, incomplete reporting, and limited statistical analysis. The paper situates NeurIPS’s approach within broader open-science and reproducibility efforts, while noting that reproducibility programs address only one aspect of research quality.
- 2. Background: Machine learning studies may insufficiently explore experimental conditions and hyperparameters, making it difficult to identify what drives benchmark improvements.Prior studies found proposed methods were sometimes no better than earlier implementations after broader hyperparameter searches or different random initializations.
- 2. Background: 63.5% of results in 255 manuscripts were successfully replicated; success reached 85% with author assistance versus 4% without author response.The authors note that selection bias could contribute to this contrast, leaving open whether the central issue is reproduction or reporting.
- 2. Background: Reproducible work redoes an experiment with the same data and analytical tools, whereas replicable work uses different data and robust work uses the same data with different analysis.Generalisable work leads to the same conclusions under broader conditions.
- 2.2 The Open Science movement: Open science emphasizes transparent, accessible knowledge shared through collaborative networks, including long-term access to code, data, and other research artifacts.The NeurIPS program incorporated artifact sharing alongside manuscripts to improve transparency and reporting quality.
- 2. Background: Reproducibility programs have precedents in database and machine learning conferences, including accepted-paper tracks, community discussion platforms, and the 2018 ICLR challenge.
- 2. Background: Reporting checklists specify minimum information for manuscripts, while the NeurIPS machine learning checklist focuses on experiment-specific details such as data splits and hyperparameter ranges.
- 2. Background: A reproducibility program is only one factor among the many steps and standards affecting how scientific research is conducted, communicated, and evaluated.
3. The NeurIPS 2019 code submission policy
The NeurIPS 2019 code submission policy encouraged authors to provide code while allowing submission by the camera-ready deadline. Code availability increased over submission stages, was positively associated with reviewer scores, and faced practical constraints including confidentiality and computational cost.
- Practical constraints: Dataset confidentiality can prevent code or data release, especially in healthcare and finance, although open benchmark results are offered as one mitigation.The paper presents confidentiality as a legitimate constraint on releasing datasets.
- Affiliation patterns: Industry-affiliated authors often supplied code later, while most industry submissions reported code by the camera-ready deadline and retained a higher acceptance rate than academia-affiliated submissions.Figure 2 compares code availability and acceptance rate by first- and last-author affiliation.
- Practical constraints: Large computational requirements can make reproducing experiments impractical for reviewers and most researchers, even when code and data are available.The paper gives very large neural models as examples, while noting that both AlphaGo and BERT were later reproduced.
- Practical constraints: Code can contain mistakes, so access to the original implementation does not guarantee correctness or replace reimplementation from the paper’s description.The paper links this concern to robustness rather than reproducibility alone.
- Policy design: The policy expected code only for accepted papers by the camera-ready deadline, so code submission was not mandatory during review.Authors could still provide code with their initial submission, but reviewers were not expected to use it to assess soundness.
- Initial availability: About 40% of authors reported providing code initially, and reviewers confirmed code for 71.5% of those submissions.The paper reports a growing trend toward adding code links even with soft encouragement and no coercive measures.
- Reviewer response: Availability of code at submission was positively associated with reviewer score (p < 1e −08).Reviewers were surveyed about their use and perception of code during the review period.
4. The NeurIPS 2019 Reproducibility Challenge
The NeurIPS 2019 reproducibility challenge independently examined empirical claims in accepted papers and created a public record of the findings. It attracted broad participation and produced detailed reports, while the authors cautioned that most reproducibility studies cannot be reduced to a single score.
- Purpose and deployment: The challenge aimed to independently verify empirical claims in accepted NeurIPS papers and leave a public trace of secondary analyses.It began after the final paper submission deadline so participants could benefit from author code submissions.
- Participation: 173 papers were claimed for reproduction, a 92% increase over the previous ICLR 2019 reproducibility challenge.Participants came from 73 institutions, including 63 universities and 10 industrial labs.
- Public record: Challenge reports were publicly available on OpenReview, often with reimplementation code, to provide examples and help researchers avoid reproduction pitfalls.Most reports gave detailed and nuanced accounts of reproduction efforts and methodological fidelity.
- Interpretation: The authors did not find that most reproducibility studies support a coarse single “reproducibility score.”They instead emphasize the detailed findings and claims of individual papers.
- Archival process: A review cycle selected a small number of high-quality reports for publication in ReScience, creating a lasting archival record for this research artefact.The reports were reviewed by NeurIPS conference reviewers before selection.
5. The NeurIPS 2019 ML reproducibility checklist
The checklist was introduced to address recurrent gaps in machine-learning experimental methodology and gathered author and reviewer assessments of reproducibility practices. Responses showed strong reporting on some items, mixed views on statistical reporting, and associations between checklist usefulness and review outcomes.
- Checklist deployment: The checklist was developed in response to recurrent gaps in experimental methodology identified in recent machine-learning papers.An earlier version was trialed with NeurIPS 2018 camera-ready submissions before the NeurIPS 2019 deployment described here.
- Author responses: 97% of submissions reported a clear description of the mathematical setting, algorithm, and/or model, while 89% reported describing how experiments were run.The authors regarded the small share of no/NA responses to the first item as potentially reflecting interpretation margin of error.
- Statistical reporting: 87% of papers valued clearly defined metrics and statistics, whereas 36% judged error bars not applicable to their results.The authors identify this combination of responses as raising questions about statistical reporting practices.
- Reviewer assessment: 34% of reviewers said the checklist answers were useful for evaluating submissions.This measure captures reviewers’ direct assessment of the checklist’s usefulness in judging paper soundness.
- Review outcomes: Reviewers who found the checklist useful gave higher paper scores, showed greater assessment confidence than reviewers who had not read it, and reviewed papers were more likely to be accepted.The reported acceptance-rate association disappeared for NA responses on some figures-and-tables questions when analyses were restricted to reviews where the checklist was useful.
6. Discussion
The NeurIPS 2019 reproducibility program showed encouraging voluntary participation in code sharing, reproducibility analysis, and checklist use, but did not yet establish an impact on paper quality. The initiative also identified open evaluation questions and possible extensions through tooling and adoption by other venues.
- Program assessment: The program is best viewed as a case study of incorporating code submission, a reproducibility challenge, and a checklist into a conference.The report states that concluding evidence of impact on the quality of submitted or published work is not yet available.
- Participation indicators: Nearly 40% more submissions to NeurIPS suggest the new processes did not significantly reduce author interest.
- Participation indicators: Voluntary code submissions rose from less than 50% to nearly 75%, although complete compliance was not the goal because some materials may be confidential.
- Participation indicators: Thousands of reviewers reported consulting or wanting to consult submitted code, indicating that code can be useful during review.
- Participation indicators: Participation, reproducibility reports, and report reviewers all increased, suggesting greater voluntary engagement in secondary analysis.
- Checklist assessment: One-third of reviewers found checklist answers useful, while reviewers who valued them assigned higher paper scores.
- Open questions: Open questions include incentive effects, checklist benefits and answer accuracy, effects on paper quality and review, and the code’s long-term value.
- Future directions: Future directions include standardizing tools such as Docker, ReproZip, and WholeTale and adapting reproducibility practices across conferences and journals.