The Fundamental Principles of Reproducibility

1. Introduction

• Reproducibility has no single commonly agreed definition, despite its central role in science.The paper compares this definitional problem to descriptions of an elephant from people who each perceive only part of it.Source: §1 · p. 1
• The lack of a definition poses an issue for automating scientific discovery, including efforts to develop AI that can conduct research.The paper cites a challenge to develop an AI co-author capable of formulating research questions and generating novel contributions by 2040.Source: §1 · p. 1
• The paper analyzes reproducibility through a process model of the scientific method, using machine learning as an example while arguing that the resulting definition applies across sciences.It draws on task modeling to decompose the scientific method into tasks and problem-solving methods.Source: §1 · p. 2

2. The Scientific Method and Machine Learning

• The scientific method uses observations and hypothesis tests to acquire knowledge, with experiments producing analyses and interpretations that can update beliefs.Exploratory and assessment studies suggest hypotheses; manipulation and observational studies test explicit hypotheses.Source: §2 · p. 2
• The process model comprises ten subprocesses, from observing and theorizing through experiment design, analysis, interpretation, belief updating, and systematic observation.The figure's arrows indicate subprocesses, and the nouns they point to are research-artifact outputs.Source: §2 · pp. 2–3
• The machine-learning example configures a target system, preprocessing, hyperparameters, workflow, datasets, and hardware before running the experiment.Its example uses data compiled from six computer-vision benchmark datasets and a GPU.Source: §2 · p. 3
• Deep convolutional neural networks are described as competitive with humans on widely used computer-vision benchmark tasks, although the super-human-performance claim remains debated.The cited analysis of low errors by deep-learning methods on visual benchmark tasks has been reproduced many times.Source: §2 · pp. 3–4
• Systematic observation across several datasets can reduce bias because datasets may themselves be biased.The example also illustrates forming beliefs from observation and using exploratory or assessment studies.Source: §2 · pp. 2, 4

3. A Survey of Definitions

• The survey uses shared concepts—including investigators, procedure, implementation, laboratory conditions, data, outcomes, and analysis—to compare definitions.For machine-learning experiments, the procedure is encoded in source code, while hardware and ancillary software capture dimensions of a laboratory.Source: §3 · p. 4
• Repeatability means the same investigators conduct the same experiment under the same conditions again.The survey treats it as the least general term among the definitions compared.Source: §3 · p. 6
• The paper interprets “same laboratory” as the same hardware and ancillary software, and “same experiment” as the same setup, preprocessing workflow, target system, and data.It distinguishes outcome, analysis, and interpretation, reserving “result” for the conclusion drawn from interpreting the analysis.Source: §3 · pp. 6–7
• Most surveyed definitions treat replication as the more general term, but proposed degrees of replication or reproducibility do not align well.The survey notes that several authors define both terms, while others use only one term for different degrees.Source: §3 · p. 7
• Most definitions concern repeating an experiment with the same or similar procedure under varying conditions; indirect replication and Drummond’s reproducibility instead resemble corroboration.The exceptions focus on strengthening belief in hypotheses or theories rather than copying or repeating an experiment.Source: §3 · p. 7

4. On the Difference between Corroboration and Reproducibility

• A theory is corroborated by evidence from tested hypotheses, not reproduced by rerunning one of its experiments.The paper argues that when all interpretations already support a theory, rerunning an experiment adds no new evidence for it.Source: §4 · p. 8
• A hypothesis is corroborated when a new and different experiment supports it; repeating the same experiment strengthens belief in that experiment’s result instead.The paper says hypotheses cannot themselves be reproduced.Source: §4 · p. 8
• The same experiment may yield different outcomes across hardware, software, randomness, or analysis choices, and different interpretations can lead investigators to different conclusions.Source: §4 · p. 8
• Repeated runs can increase trust in an experiment’s conclusion, but do not strengthen belief in the hypothesis; that requires new and different experiments.Source: §4 · p. 8

5. Experiments as Tasks

• Explicit task and problem-solving-method representations offer a way to analyze reproducibility, alongside earlier approaches that represented expertise explicitly.Source: §5 · p. 8
• Tasks have goals, inputs, outputs, and completion methods; problem-solving methods decompose them into subtasks, order those tasks, and ultimately specify executable actions.The resulting structure is a directed acyclic graph; actions are explicit algorithms implemented in code.Source: §5 · p. 9
• The MNIST classifier-comparison example decomposes data collection and preprocessing from algorithm selection, training, hyperparameter optimization, and testing.The test error is analyzed and interpreted by comparing classifiers; other metrics such as accuracy and F1-score are also possible.Source: §5 · pp. 9–10, Figure 4 · p. 9
• Detailed protocols may be impractical for complex experiments, and implementation differences or task order can affect results.The paper cites missing details that hindered AlphaGo and AlphaZero reproduction and reports that algorithm implementations can produce significantly different results.Source: §5 · p. 10

• Reproducibility is independent investigators drawing the same conclusions by following documentation shared by the original investigators.Source: §6 · p. 10
• The three degrees allow different outcomes and analyses when interpretation still leads to the same conclusion.Outcome reproducibility requires the same outcome; analysis reproducibility allows different outcomes if the same analysis and interpretation follow; interpretation reproducibility requires only the same conclusion.Source: §6 · pp. 10–11
• The four types distinguish experiments by shared documentation: text only, code plus text, data plus text, or complete documentation including code and data.Source: §6 · p. 11
• The degree of reproducibility depends on how well the original experiment was documented and how much documentation is shared.Source: §6 · p. 11
• More shared documentation increases transparency and eases reproduction, while less documentation can increase the generality of results by requiring more variation.The paper contrasts reusing the same code and data with a new implementation on different data.Source: §6 · p. 11
• Full transparency lets researchers reimplement or change one task, action, or problem-solving method instead of rebuilding the whole experiment.The paper notes that written language is an imprecise and space-limited substitute for code and higher-level abstractions.Source: §6 · p. 12

• Reproducibility requires both sameness and difference: at minimum, the independent experiment follows the same methodology, while variation is also needed.Source: §7 · p. 13
• The paper separates outcome, analysis, and interpretation to clarify what counts as the same result; reproducibility turns on reaching the same conclusion.Source: §7 · pp. 12–13
• The analysis proposes a definition, three degrees, and four documentation-based types, and concludes that transparency and openness support faster knowledge discovery.The paper says sharing more documentation makes reproduction easier, which it links to faster discovery.Source: §7 · p. 13

• Electronic documents give reproducible research a new meaning.Source: §8 · p. 16

• The evaluation distinguishes assumptions about investigators, theory and hypothesis, procedure, implementation, ancillary software, hardware, data, outcome, and analysis.Source: Variable Value Evaluation · p. 16
• If software or hardware differences change recomputed figures enough to alter their apparent meaning, the written interpretation may no longer fit those figures.Source: Variable Value Evaluation · p. 16
• An earlier computational-research proposal attached a button or tag to each figure caption to recalculate it from its data, parameters, and programs.Source: Variable Value Evaluation · p. 16

Variable Value Evaluation

• The shared WaveLab package supplies the experiment's code and data, which are used to rerun it.The account says anyone with access to the package can reproduce the work.Source: Variable Value Evaluation · p. 17
• The experiment, theory, hypothesis, procedure, implementation, data, outcome, and analysis are treated as the same across reproductions.The account assumes the rerun produces the same outcome.Source: Variable Value Evaluation · p. 17
• Reproduction does not require the same hardware, operating-system version, or ancillary software, although Matlab is required.The account leaves the investigator's interpretation open.Source: Variable Value Evaluation · p. 17
• The cited software-environment principle is to publish the complete environment used to generate computer-produced figures.The passage says this is intended to ease others' reproduction of the figures.Source: Variable Value Evaluation · p. 17

Variable Value Evaluation

• In the surveyed account, replication can use a different study procedure and implementation while retaining the same theory and hypothesis.The selection of subjects and software or hardware can differ, and the data may come from different subjects.Source: Variable Value Evaluation · p. 18
• The surveyed definition says reproducible research cannot address study-design and subject-selection questions that arise before data collection.It also notes that some modeling choices involving uncollected variables may remain unexamined.Source: Variable Value Evaluation · p. 18
• The surveyed definition says fully replicable studies allow all aspects of the original study to be adjusted or modified.It places this broader scope closer to full replication.Source: Variable Value Evaluation · p. 18

Paper title

• The paper title frames replicability as distinct from reproducibility and says it is not, by itself, good science.This is a claim in the title, not a conclusion supported by the supplied section text.Source: Paper title · p. 20

Variable Value Evaluation

• The section argues that Bradley’s substantially different experiment was not a replication, and that greater difference from the first experiment can give the second greater evidential power.The example distinguishes reproducibility, which requires changes, from replicability, which avoids them.Source: Variable Value Evaluation · p. 20
• Direct replication repeats an experimental procedure and anticipated conditions to confirm facts, whereas conceptual replication tests a hypothesis or prior result with different methods to develop understanding.The account characterizes direct replication as low risk and conceptual replication as high risk.Source: Variable Value Evaluation · pp. 22, 23, 37
• Independent reproduction inevitably involves deviations, so judging success requires distinguishing core claims from report details while preserving the interpretation that supports the hypothesis.The passages note that not all experimental conditions can be controlled and that some accounts leave result-comparison methods unspecified.Source: Variable Value Evaluation · pp. 32–34
• Computational internal replication reruns original code, while external replication lets readers rerun it using shared tools and records, with outcomes potentially sensitive to hardware and software differences.Internal reproducibility may require version control and electronic lab journals; external reproduction also requires permanent records of simulation details.Source: Variable Value Evaluation · pp. 32–33
• Cross-replication runs or reimplements the same model on a different simulation platform, exposing dependence on simulator details but raising questions about how to compare results.It may use a different platform or language based on original code, or a simulator-independent model format.Source: Variable Value Evaluation · p. 34
• Independent implementation rebuilds a model from the paper without downloading its scripts, though the boundary between this approach and cross-replicability is not always clear.A structured declarative model description can blur the distinction between implementing from a publication and using the original code.Source: Variable Value Evaluation · p. 35
• Recomputation preserves a computational experiment in a virtual machine for future reruns, but does not promise exact reproduction of results, which may vary even on an identical machine.The account distinguishes enabling code execution again from repeatedly obtaining the same result.Source: Variable Value Evaluation · p. 36
• Direct replication attempts to reproduce conditions and procedures theory and evidence deem necessary; it adds data for meta-analysis, tests generalizability, and negative high-powered results expose boundary conditions.Additional data can improve effect-size precision and help identify false positives; there is no exact replication, so presumed irrelevant differences may reveal constraints on an effect. Scientific claims are evaluated through evidence rather than trust in a source.Source: Variable Value Evaluation · p. 37
• Conceptual replications abstract phenomena from their original operationalization and aid theoretical understanding, but design changes make nonsignificant findings poor tests of a particular effect.Such findings rarely lead researchers to question the phenomenon itself.Source: Variable Value Evaluation · p. 38
• Repeatability means the same team can obtain a measurement at stated precision using the same procedure, system, operating conditions, and location across trials.For computational experiments, this means reliably repeating one’s own computation; artifacts may include software, scripts, datasets, raw data, and analysis scripts.Source: Variable Value Evaluation · p. 42