Source-linked AI summary
A Step Toward Quantifying Independently Reproducible Machine Learning Research
Edward Raff
TL;DR
The paper asks what makes research independently reproducible, addressing a lack of empirical evidence behind competing claims about paper features and code release. It manually attempted to reproduce 255 papers, quantified their features, and found that reproducibility patterns have remained statistically unchanged over 35 years while implementation details and empirical content matter.
Problem
The paper asks what makes a paper independently reproducible, because reproducibility debates lack quantitative evidence and code release alone is insufficient.
Method
The authors manually attempted to implement 255 papers, recorded quantified paper features, and analyzed their relationship with independent reproducibility without using authors’ code.
Results
Paper reproduction rates did not change statistically significantly over 35 years; more empirical papers and papers including implementation details tended to be more reproducible, while pseudocode alone was insufficient.
Takeaways & Limitations
Independent reproducibility is associated with paper content and implementation detail, not simply whether authors release code.
Takeaways & Limitations
All papers were attempted by one author, and paper selection reflected the authors’ interests and prior source-code exposure.
Abstract
from arXiv · showhide
What makes a paper independently reproducible? Debates on reproducibility center around intuition or assumptions but lack empirical results. Our field focuses on releasing code, which is important, but is not sufficient for determining reproducibility. We take the first step toward a quantifiable answer by manually attempting to implement 255 papers published from 1984 until 2017, recording features of each paper, and performing statistical analysis of the results. For each paper, we did not look at the authors code, if released, in order to prevent bias toward discrepancies between code and paper.
1 Introduction
The paper defines independent reproducibility as reproducing results without the authors’ code and argues that code release alone is insufficient. It studies which paper properties are associated with independent reproducibility through empirical analysis rather than assumptions.
- Independent reproducibility means reproducing a paper’s results without using code from its authors.
- Code and data availability are valuable, but released code alone may not establish reproducibility.The paper identifies insufficient explanations, omitted implementation details, and discrepancies between code and description as possible problems.
- The study asks which paper properties make independent reproduction more likely and argues that this question requires quantifiable evidence.Prior feature-based definitions classified reproducibility without testing whether selected features actually affected reproduction.
- The authors’ first step is an empirical study of 255 papers that they attempted to implement independently.
2 Procedure and Features
The study selected 255 papers under independence-focused criteria, manually recorded paper features, and compared those features with reproduction outcomes. Features included objective publication characteristics, implementation details, resource requirements, and explicitly acknowledged subjective judgments.
- Selection criteria: The sample contains 255 papers proposing at least one new algorithm or method, with initial reproduction attempts occurring from 2012 through 2017.Papers were excluded if the reproducers had previously seen the authors’ source code or had a significant relationship with the authors.
- Reproduction outcome: 162 of 255 papers (63.5%) were successfully replicated, while 93 were not.The authors compare this outcome with a prior feature-based estimate of 26% reproducibility.
- Feature collection: The study recorded 26 attributes per paper using policies intended to minimize subjectivity.Feature collection took approximately 20 minutes per paper, while preparing the study data took approximately six months.
- Feature selection: Features were chosen for plausible relationships with reproduction and practical availability, while feature recording generally used the main paper body excluding appendices.
- Quantified features: Recorded features included authorship, publication details, appendices, pages, references, venue, author replies, tables, graphs, equations, compute needs, data availability, and pseudo-code.Compute requirements were categorized as Desktop, consumer GPU, Server, or Cluster using contemporary resource levels.
- Subjective features: The authors separated unambiguous features from mildly subjective and significantly subjective features, documenting protocols for the latter.Subjective judgments included algorithm difficulty, estimated from implementation size or experience when implementation was unavailable.
3 Results
The study found several paper and interaction features associated with independent reproducibility, while publication timing and venue showed no significant relationship. Results also highlight readability, communication with authors, and compute requirements as important empirical dimensions, with topic-specific findings requiring caution.
- Non-significant Relationships: Publication year, reproduction-attempt year, venue, venue type, appendices, conceptualization figures, exemplar problems, and released code showed no significant relationship with reproducibility.Topic-specific results found no successful reproductions for Bayesian or Fairness papers, but the authors caution against extrapolating because the reproducers were not experts in every topic.
- Significant Relationships: Ten variables were significantly correlated with reproducibility, including tables, equations, compute needed, pseudo-code, and specified hyper-parameters.The study used non-parametric tests for numeric features and marked significance at α ≤0.05.
- Significant Relationships: Readability had the strongest empirical relationship with reproducibility, and every paper rated “Excellent” was reproduced.The authors measured readability by the number of readings needed to reach a mostly complete implementation.
- Significant Relationships: Tables and specified hyper-parameters were positively correlated with reproducibility, whereas equations per page were negatively correlated.The readability explanation for equation density was significant (p = 0.001), while the algorithmic-difficulty explanation was not (p = 0.239).
- Significant Relationships: Author replies were associated with 22 successful reproductions among 26 replies, compared with one among 24 non-replies.The authors note that replies may reflect living-paper updates, increased reproducer motivation, or other confounding factors.
4 Study Deficiencies
The study’s deficiencies include selection and implementer biases, incomplete historical records, omitted author effects, and subjectivity in key factors. The authors propose broader communal studies and question binary reproducibility labels.
- Study biases: All papers were attempted by one author, so different implementers might find different papers easy or difficult to reproduce.The sample contains many papers selected through the reproducer’s interests and prior work on the JSAT library.
- Study biases: The study’s selection process was shaped by the reproducers’ interests and excluded papers whose released source code they had previously examined.
- Data limitations: Historical records limited the analysis to recorded notes and information that could be re-derived from the papers.Paper-cataloging software made the study possible but constrained what information remained available.
- Future study: The authors recommend communal studies that standardize paper features, reproduction resources, time, and implementer backgrounds.
- Framing reproduction: Reproduction may be better modeled as survival over time and resources than as a binary property of a paper.The longest reproduction effort took 4.5 years of non-continuous effort.
- Unmeasured factors: Author backgrounds and writing styles were not analyzed, despite potentially affecting reproduction outcomes.The authors identify interactions between authors and implementers as a topic for future study.
5 A Subjective Recall of Non-Reproduction
The authors recall several recurring reasons for non-reproduction, while emphasizing that these experiences do not imply individual papers are false or seriously flawed.
- Study limitation: The study did not record the believed reason for each reproduction failure.
- Reasons for non-reproduction: Unclear notation or language can make an algorithm component ambiguous or difficult for reproducers to understand.
- Reasons for non-reproduction: Missing algorithm steps or details can prevent reproducers from implementing the described method accurately.
- Reasons for non-reproduction: Undescribed gradients can require non-trivial re-derivation and produce results that do not match.
- Reasons for non-reproduction: Missing hyper-parameters or other minor implementation details can substantially change results.
- Interpretive boundary: The authors rarely suspected that unreproduced results were false or caused by seriously flawed implementations.
6 Conclusions
The study presents an empirical analysis of factors associated with independent reproducibility. Reproduction rates did not significantly change over 35 years, while empirical papers and implementation-focused details tended to align with higher reproducibility.
- The study is presented as the first empirical analysis of what impacts a paper’s reproducibility.
- Reproduction rates did not change significantly over the past 35 years.
- Papers with a more empirical nature tended to be more reproducible.
- Including implementation-relevant details was associated with higher reproducibility, whereas simply including pseudo-code was insufficient.
- Papers with fewer equations and more tables tended to be more reproducible, while cluster computing may create a latent reproduction issue.
A Some Preemptive Responses to Questions
The author preemptively addresses recurring questions about the study’s context and potential personal biases. Some related themes are kept outside the paper’s main content.
- The author received advice and feedback from colleagues while preparing the manuscript.
- The manuscript lists common questions about related themes that do not necessarily belong in the main academic content.
- The responses aim to clarify the work’s context and potential biases arising from the author’s personal background.
A.1 Why Where you Recording Information About Papers?
The author’s earlier arbitrary-precision math project led to reading and implementing papers for the techniques involved.
- The author worked on an arbitrary-precision math library as a side project for four years.
- That project involved implementing algorithms for decompositions, mathematical constants, common functions, and complex numbers.
- The author began reading and implementing papers related to these techniques.
A.2 Why Where you Implementing so many Papers?
The author entered machine learning after initially emphasizing computer science coursework and taking little advanced mathematics, aside from numerical analysis.
- The author had taken only the minimum advanced mathematics courses required for a computer science degree.
- The author focused instead on taking many more computer science courses.
- A numerical analysis course was an exception because it seemed relevant to the arbitrary-precision math library.
- The author became enamoured with machine learning after taking a machine learning class.
B Statistical Test assumptions
The analysis uses non-parametric tests because the numeric features violate normality assumptions; variance equality generally holds except for publication year and page count, without changing the page-count conclusion.
- All numeric features deviated significantly from a normal distribution, making a standard Student’s t-test inappropriate.
- The Mann-Whitney test’s equal-variance assumption held for all numeric features except publication year and total page count.
- Year (p = .554) and Pages (p = 0.134) still showed no significant relationship with reproducibility under a Welch test.
Number of References
Figure 2 presents histograms of the page-normalized numeric variables considered, with the displayed scale including 0, 0.5, 1, and 1.5.
- Figure 2 contains histograms of the page-normalized numeric variables considered.
- The figure is presented as a distributional view of numeric variables normalized by page count.
D Contingency Tables for Nominal Features
These contingency tables report χ2 tests relating paper features to readability and the ability to reproduce results. Reported associations vary substantially across features, with several very small p-values and several nonsignificant results.
- Venue type, author code availability, appendices, data availability, toy problems, exact compute specification, and readability-related measures have larger reported p-values.The reported values include p = 0.502 for Venue Type, p = 0.184 for Author’s Code, p = 0.330 for an Appendix, p = .558 for Data Available, p = 0.720 for an Exemplar Toy Problem, and p = 0.257 for Exact Compute Used.
- p = 6.016 × 10−8 for whether an author replied to email questions, showing a very small reported test probability.
- p = 1.545 × 10−9 for Rigor vs Empirical, indicating the strongest reported test association in this set.
- p = 2.939 × 10−5 for Algorithm’s Difficulty and p = 2.308 × 10−4 for whether a paper has pseudo-code.
- Hyperparameters being specified and the level of compute needed also have small reported p-values: 8.450 × 10−6 and 2.788 × 10−5, respectively.