Source-linked AI summary
Sustainable computational science: the ReScience initiative
Nicolas P. Rougier, Konrad Hinsen, Frédéric Alexandre, Thomas Arildsen, Lorena Barba, Fabien C. Y. Benureau, C. Titus Brown, Pierre de Buyl, Ozan Caglayan, Andrew P. Davison, Marc André Delsuc, Georgios Detorakis, Alexandra K. Diem, Damien Drix, Pierre Enel, Benoît Girard, Olivia Guest, Matt G. Hall, Rafael Neto Henriques, Xavier Hinaut, Kamil S Jaron, Mehdi Khamassi, Almar Klein, Tiina Manninen, Pietro Marchesi, Dan McGlinn, Christoph Metzner, Owen L. Petchey, Hans Ekkehard Plesser, Timothée Poisot, Karthik Ram, Yoav Ram, Etienne Roesch, Cyrille Rossant, Vahid Rostami, Aaron Shifman, Joseph Stachelek, Marcel Stimberg, Frank Stollmeier, Federico Vaggi, Guillaume Viejo, Julien Vitay, Anya Vostinar, Roman Yurchak, Tiziano Zito
TL;DR
Computational research faces replication problems, and publication systems often do not include software in peer review. ReScience provides an openly reviewed venue for reproducible, open-source replications, with 17 articles published in its first two years and a reported bias toward successful replications.
Problem
Computational research remains vulnerable to replication problems, while software and data are often excluded from peer review and missing code, data, dependencies, or precise descriptions characterize non-reproducible work.
Source: Introduction · p. 2, Short-term and long-term reproducibility · p. 5, Outlook · p. 7Method
ReScience uses open peer review to evaluate reproducible, open-source replications of published computational research, with submissions and discussion hosted on GitHub.
Source: Te ReScience initiative · p. 4Results
17 articles had been published in ReScience’s first two years and four were under review; reported replications were 100% successful, a bias the authors link to reluctance to publish failures.
Source: Te ReScience initiative · p. 4Takeaways & Limitations
Software Heritage identifiers promise permanent references to projects and snapshots, supporting reuse, computational reproducibility, and fairer credit for research infrastructure.
Source: Outlook · pp. 7–8Docker provides an executable environment but not a human-readable description, and the long-term stability of its container format remains unevaluated.
Source: Outlook · p. 7
Abstract
from arXiv · showhide
Computer science offers a large set of tools for prototyping, writing, running, testing, validating, sharing and reproducing results, however computational science lags behind. In the best case, authors may provide their source code as a compressed archive and they may feel confident their research is reproducible. But this is not exactly true. James Buckheit and David Donoho proposed more than two decades ago that an article about computational results is advertising, not scholarship. The actual scholarship is the full software environment, code, and data that produced the result. This implies new workflows, in particular in peer-reviews. Existing journals have been slow to adapt: source codes are rarely requested, hardly ever actually executed to check that they produce the results advertised in the article. ReScience is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research can be replicated from its description. To achieve this goal, the whole publishing chain is radically different from other traditional scientific journals. ReScience resides on GitHub where each new implementation of a computational study is made available together with comments, explanations, and software tests.
Introduction
Computational science faces replication problems despite relying on code and data, and access to source code alone does not establish that results are correct or reproducible. Studies found that code often could not be run, while incentives for re-implementing prior work remain limited.
Introduction
- Computational and computer sciences are not exempt from the replication crisis, despite relying on code and data rather than experimental observations.The passage notes that non-replicability has also been reported in other scientific fields.Source: Introduction · p. 2
- 102 of 515 potentially reproducible papers (less than 20%) were ultimately run, without checks of code correctness or result reproducibility.The study assessed whether code could be run, not whether it implemented the paper correctly or reproduced its results.Source: Introduction · p. 2
- Journal policies can improve access to code and data, but access alone does not guarantee correctness or reproducibility.The passage cites data and software sharing policies as examples.Source: Introduction · p. 2
- Re-implementing published software can improve research practice, but it is time-consuming, weakly incentivized, and may reproduce the original authors’ mistakes.The passage notes that the incentives for this work are low or nonexistent.Source: Introduction · p. 2
Replication and reproduction
Reproduction reruns the same software on the same data, whereas replication uses new software based on the published description to seek equivalent results. Reproducibility supports inspection and reuse, but computational environments and incomplete descriptions make both goals difficult.
Replication and reproduction
- Reproducibility means rerunning the same software on the same input data and obtaining the same results.The passage frames this as checking whether the computational protocol was recorded correctly.Source: Replication and reproduction · p. 2
- Lost software or data, mismatched environments, inadequate instructions, and floating-point differences can obstruct exact reproduction.Complex software stacks and recording every scientist-computer interaction also make exact reproduction difficult.Source: Replication and reproduction · p. 2
- Publishing software and input data supports understanding and reuse, while results that depend on hard-to-control implementation details can obscure their relation to the underlying models and methods.The passage also links publication of computational materials with excluding common computer-work mistakes.Source: Replication and reproduction · p. 3
- Replication uses new software based on a published model or method description and seeks results similar enough to be considered equivalent.What counts as sufficiently similar depends on the kind of computation and requires field expertise.Source: Replication and reproduction · p. 3
- Replicability is a stronger quality indicator than reproducibility because independent implementations yielding equivalent results make serious implementation errors less likely and test whether the description was understandable.Reproducibility documents computational ingredients but does not establish that they are correct or appropriate.Source: Replication and reproduction · p. 3
Te ReScience initiative
ReScience was created to provide a peer-reviewed venue for computational replications, with submissions built around open code and public discussion. Its process evaluates whether implementations can be run and whether their results match the submission and the original work.
Te ReScience initiative
- ReScience was founded because computational replications lacked a major journal venue, despite researchers’ varied reasons for undertaking them.Reasons include learning through replication and needing a model for collaboration; some researchers also wanted a place to share practical knowledge about models.Source: Te ReScience initiative · pp. 3–4
- 17 articles had been published in ReScience’s first two years, four were under review, and reported replications were 100% successful amid acknowledged reluctance to publish failures.The passage describes this success rate as a strong bias and notes the social and professional risks of publishing failed replications.Source: Te ReScience initiative · p. 4
- ReScience conducts submission, review, and publication on GitHub, where the process is open for anyone to see and comment on.The journal treats submissions similarly to contributions to an open-source software project.Source: Te ReScience initiative · p. 4
- A submission includes a reference to the replicated work, an explanation of success or failure, difficulties encountered, open-source code, and an explanation of that code.Partial replications are accepted when justified, and submissions are made as GitHub pull requests.Source: Te ReScience initiative · p. 4
- Two reviewers assess whether they can run the implementation, reproduce the submission’s results, and judge them sufficiently close to the original; novelty or importance is not the criterion.Reviewers discuss the work with authors until both consider it acceptable.Source: Te ReScience initiative · p. 4
- A failed-replication submission requires further scrutiny, contact with the original authors, and a community challenge; if no errors are found, the original work is declared non-replicable.Source: Te ReScience initiative · p. 4
- Replicators cannot submit their own or close collaborators’ work, and should rely on the original publication unless documented exceptions apply.The passage connects independence to avoiding biases that may carry over from the original implementation.Source: Te ReScience initiative · p. 4
Lessons learned
Although ReScience is young, its handled submissions already provide experience concerning reproducibility and replicability in computational research.
Lessons learned
- ReScience is described as a young project.Source: Lessons learned · p. 5
- Submissions handled so far already provide valuable experience.Source: Lessons learned · p. 5
- That experience concerns reproducibility and replicability of computational research.Source: Lessons learned · p. 5
Short-term and long-term reproducibility
ReScience aims to balance reproducibility, robustness, and implementation effort while using GitHub for code review and Zenodo for long-term archiving. Because software may stop working as computing environments change, the accompanying replication article is central to a publication’s long-term value.
Short-term and long-term reproducibility
- The recommendations aim for a workable compromise between reproducibility, robustness, and implementation effort, while acknowledging common problems such as missing code, undocumented dependencies, and imprecise descriptions.The authors say these problems can escape attentive reviewers; they expect recommendations to evolve over time.Source: Short-term and long-term reproducibility · p. 5
- Code written for ReScience may eventually stop functioning as computing technology changes, so the original and replication articles provide the lasting description that enables later replication.The authors argue that a competent scientist should be able to replicate the work again even 5, 10, or 20 years later.Source: Short-term and long-term reproducibility · p. 5
- ReScience’s open review has been constructive and courteous in the authors’ experience, and it can include original-study authors, such as when one granted permission to reuse an image.Editors mainly keep reviews moving and answer process questions.Source: Open reviewing · p. 5
- ReScience adopted a software-development platform because, to the authors’ knowledge, scientific publishing platforms did not support code submission and review; GitHub offers free use for public projects.The authors describe ReScience’s activities as organized around open-source projects hosted on GitHub.Source: Publishing on the GitHub platform · pp. 5–6
- GitHub supports code review, but authors need basic Git and collaborative-development skills; editors provide technical assistance, and no author feedback has indicated that this requirement deters submissions.The authors note that potential authors could be discouraged, although they have received no feedback indicating this has happened.Source: Publishing on the GitHub platform · p. 6
- GitHub does not handle the desired publication outputs, so accepted work is archived on Zenodo with a DOI while the managing editor prepares the PDF.A submission on GitHub consists of Markdown article text and code and data files; the desired archival form is a PDF plus a permanent archive.Source: Publishing on the GitHub platform · p. 6
- GitHub’s review workflow uses pull requests to the journal-wide project, and it does not permit code review on a newly created repository, preventing a separate repository for each article.A separate repository would make final publishing steps easier, but GitHub’s review feature applies only to contributions to an existing repository.Source: Publishing on the GitHub platform · p. 6
- Because GitHub is a commercial service whose terms or availability may change and offers no contractual recourse, ReScience deposits accepted articles on Zenodo for archiving.The authors caution against relying on GitHub for long-term preservation.Source: Publishing on the GitHub platform · p. 6
Outlook
ReScience’s outlook calls for software-development platforms to become part of scientific publishing, with computational environments archived in inspectable and executable forms. It also considers prepublication replication, a model that could strengthen verification but faces practical limits.
Outlook
- Research publications should include a human-readable environment description and an executable form to inspect package versions, rerun analyses, and inspect intermediate results.Ideally, the description would permit rebuilding the executable; Guix pursues this approach.Source: Outlook · p. 6
- A conda environment file describes and reconstructs an environment, but its packages are not themselves reproducible and the file omits their compiler versions.The paper contrasts this limitation with Guix.Source: Outlook · p. 7
- Current publishing platforms often relegate software and data to supplements and exclude them from peer review; ReScience aims to verify scientific software.The authors say integrating automated validation frameworks into software-development platforms seems feasible and could enhance verification.Source: Outlook · p. 7
- Scientific publishing should embrace software-development technology and shift focus from isolated papers to incremental, interdependent research.ReScience’s experience suggests that integrating article narratives into software-development frameworks is feasible.Source: Outlook · p. 7
- Software Heritage identifiers can reference ongoing projects, time-specific snapshots, whole projects, and individual files, promising better reuse, reproducibility, and credit for infrastructure contributors.The paper presents these benefits as promises of such references.Source: Outlook · pp. 7–8
- CoScience proposes independent replication before publication, with successful work jointly endorsed and certified; failures would prevent publication, and the model would slow publishing.The authors consider this approach unlikely under current publication pressure and weak replication incentives.Source: Figure 1 · p. 7, Outlook · p. 8