PaperlayerView exact revision on arXiv

Source-linked AI summary

Computational reproducibility of Jupyter notebooks from biomedical publications

Sheeba Samuel, Daniel Mietchen

arXiv:2308.07333v1cs.DLcs.CE

TL;DR

Evidence on computational reproducibility of Jupyter notebooks associated with biomedical publications has been limited at scale. This study automatically reran notebooks and examined reproducibility; it found low but improving reproducibility and identified dependencies as a main issue.

  • Problem

    The study addresses the limited assessment at scale of computational reproducibility for Jupyter notebooks associated with biomedical publications.

    Source: Implications · p. 18

  • Method

    The authors reran Python notebooks associated with PubMed Central publications using a pipeline with Conda environments and notebook-output comparison tools.

    Source: Computational environment setup · p. 6, Reproducibility pipeline and analysis… · p. 6, Conclusions · p. 19

  • Results

    Reproducibility was low but improving across 15,817 notebooks associated with 3,467 PubMed Central publications.

    Source: Conclusions · p. 19

  • Takeaways & Limitations

    The authors identify code and data dependencies as main issues and suggest better documentation and systematic automated reproducibility checks as potential improvements.

    Source: Conclusions · p. 19

    The workflow considered only the first error per notebook, did not use Docker images, and did not systematically attempt manual fixes.

    Source: Limitations · p. 18

Abstract

from arXiv · show

Jupyter notebooks facilitate the bundling of executable code with its documentation and output in one interactive environment, and they represent a popular mechanism to document and share computational workflows. The reproducibility of computational aspects of research is a key component of scientific reproducibility but has not yet been assessed at scale for Jupyter notebooks associated with biomedical publications. We address computational reproducibility at two levels: First, using fully automated workflows, we analyzed the computational reproducibility of Jupyter notebooks related to publications indexed in PubMed Central. We identified such notebooks by mining the articles full text, locating them on GitHub and re-running them in an environment as close to the original as possible. We documented reproduction success and exceptions and explored relationships between notebook reproducibility and variables related to the notebooks or publications. Second, this study represents a reproducibility attempt in and of itself, using essentially the same methodology twice on PubMed Central over two years. Out of 27271 notebooks from 2660 GitHub repositories associated with 3467 articles, 22578 notebooks were written in Python, including 15817 that had their dependencies declared in standard requirement files and that we attempted to re-run automatically. For 10388 of these, all declared dependencies could be installed successfully, and we re-ran them to assess reproducibility. Of these, 1203 notebooks ran through without any errors, including 879 that produced results identical to those reported in the original notebook and 324 for which our results differed from the originally reported ones. Running the other notebooks resulted in exceptions. We zoom in on common problems, highlight trends and discuss potential improvements to Jupyter-related workflows associated with biomedical publications.

Introduction

Reproducibility is one proxy for the trustworthiness of research, but computational notebooks do not ensure reproducibility merely by being shared. This study examines notebook reproducibility in biomedical publications and situates it within broader reproducibility and environmental-footprint concerns.

Reproducibility issues in contemporary research

Terminology

  • Methods, Results, and Inferential reproducibility distinguish repeating procedures and data, matching original results, and reaching the same conclusions, respectively.Source: Terminology · p. 3
  • Here, “Computational reproducibility” means using the same code on the same data source, while recognizing that execution environments and dynamic data can affect what that entails.Source: Terminology · p. 3

PubMed Central

  • PubMed Central provides full-text biomedical articles, and GitHub combines version control with collaboration and automation for sharing software.Source: PubMed Central · p. 3, GitHub · p. 3

Computational reproducibility in biomedical research

  • The study automatically analyzes Jupyter notebooks in GitHub repositories associated with full-text biomedical publications in PubMed Central.It addresses the gap in large-scale assessment of these notebooks’ reproducibility.Source: Computational reproducibility in… · p. 3, Jupyter · p. 4

Jupyter and reproducibility

  • Jupyter notebooks are widely used to share computational workflows, but prior studies have examined them at different scales and in different contexts.One study reproduced 3 of 22 notebooks associated with five PMC publications; other work examined large GitHub collections and notebook practices.Source: Jupyter · p. 4, Jupyter and reproducibility · p. 4

Environmental footprint

  • The study also considers the environmental footprint of Jupyter-based biomedical workflows, focusing on carbon dioxide and excluding other greenhouse gases and ecological-footprint components.Source: Environmental footprint · p. 4

Methods

The pipeline mined PubMed Central articles for GitHub-hosted notebooks, gathered notebook and publication metadata, and attempted execution in prepared environments. The authors repeated the pipeline in 2023 using essentially the same methodology as in their 2021 run.

Methods

  • The 2023 re-run used essentially the same methodology as the 2021 run, with results reported for the re-run and earlier values included where feasible.Source: Methods · p. 4

PMC search

  • The PMC search combined “ipynb,” “jupyter,” or “ipython” with “github,” then retrieved full-text XML and publication metadata.Source: PMC search · p. 4

Metadata extraction

  • The pipeline extracted article, journal, author, and repository information, then collected notebook metadata, cell sources and outputs, and code features using Python AST.Repository links were normalized and linked to articles; article MeSH terms were grouped into 108 top-level subject areas.Source: Metadata extraction · p. 5

Computational environment setup

  • Execution environments were prepared with conda using the notebook’s declared Python version and dependencies from repository files, or Anaconda dependencies when none were declared.Source: Computational environment setup · p. 6

Reproducibility pipeline and analysis tools

Code structure and visualization

Initial run (2021)

  • The initial pipeline run took 117 hours and 52 minutes in February 2021; the 2023 re-run took 43 days and experienced infrastructure interruptions and long-running errors.Some notebooks producing repeated RequestException errors were excluded.Source: Initial run (2021) · p. 7, Re-run (2023) · p. 7

Environmental footprint estimation

Results

The corpus includes 27,271 Jupyter notebooks from 2,660 repositories associated with 3,467 PubMed Central publications, spanning a broad mix of research fields and journals.

General statistics of our study

Research fields

Journals

Programming languages

Python dominates the notebooks, while the corpus also shows variation in Python versions and in notebook structure and Markdown content.

Programming languages

Python versions

  • Python 3.7 appeared in 7,667 notebooks and Python 3.6 in 5,211; overall, 19,508 notebooks used major version 3.Source: Python versions · p. 9

Notebook structure

  • Notebooks had a median of 23 cells and 15 code cells; 22,733 (83.58%) had Markdown cells, while 4,467 (16.42%) had none.Source: Notebook structure · pp. 9–10
  • Among Markdown cells, 36.77% of notebooks used only English, and language detection failed for 8,660 notebooks (38.09%).Source: Notebook structure · p. 10

Notebook naming

Notebook titles include common names such as “talktorial,” “Untitled,” and “demo.” Python notebooks frequently use external modules, while several reproducibility-related extensions and dependency declaration files appear in the corpus.

Notebook naming

  • Notebook titles averaged 19 characters (minimum 1, maximum 123), and talktorial, Untitled, and demo were the three most frequently encountered notebook titles.114 (0.42%) notebooks had titles that were or began with “Untitled”; 2,454 (9%) had names not recommended by the POSIX fully portable filenames guide.Source: Notebook naming · pp. 10–11

Notebook dependencies

  • 19,944 (96.35%) of valid Python notebooks used external modules, while 2,174 (10.5%) used local imports.The most declared modules were numpy (13,521), pandas (11,870), and matplotlib.pyplot (10,539).Source: Notebook dependencies · p. 11, Figure 18 · p. 11, Figure 16 · p. 11
  • Autoreload and watermark were among the popular load extensions, alongside rpy2 for using R code within Python-kernel notebooks.Autoreload reloads modules before dependent code runs; watermark saves metadata about the notebook’s execution environment.Source: Notebook dependencies · pp. 11–12, Figure 17 · p. 11
  • 11,818 notebooks belonged to repositories declaring dependencies through setup.py, requirements.txt, or pipfile.Across notebooks, 9,419 (34.54%) used setup.py, 8,231 (30.18%) used requirements.txt, and 353 (1.29%) used pipfile.Source: Figure 18 · p. 11, Notebook dependencies · p. 12

Notebook reproducibility

Only some Python notebooks with declared dependencies could be installed and run successfully. Exceptions were common among attempted runs, with patterns reported across publication years, fields, journals, article types, and notebook characteristics.

Notebook reproducibility

  • 5,429 (34.32%) of 15,817 Python notebooks could not have their declared dependencies installed, leaving 10,388 for execution.The authors found no malformed requirement files, conflicting dependencies, missing requirement files, or required external tools to explain the installation failures.Source: Notebook reproducibility · p. 12

Exceptions

  • Across articles’ publication years, ModuleNotFoundError increased after its introduction with Python 3.6 in 2016 and overtook ImportError by 2019.Exception counts normalized by notebook number peaked in 2021; if that trend holds, the numbers for 2023, where data were incomplete, would be expected to be lower than for 2022.Source: Exceptions · p. 12
  • Exceptions occurred in large numbers across research areas where Jupyter is popular, while some journals had rates above 50% and others below that mark.Journal rates were normalized by notebook count; the authors also reported that technically oriented article types performed better than average, while biological articles published by Oxford University Press underperformed.Source: Exceptions · p. 13, Figure 23 · p. 13, Figure 22 · p. 13
  • Talktorials tended to have fewer exceptions than figure-related notebooks, while unknown exceptions were frequent in tutorial notebooks.Exception patterns also varied with filename length and notebook cell count, but not uniformly across exception types.Source: Exceptions · p. 13
  • Low Markdown-to-code cell ratios correlated with exceptions; 34% of FileNotFoundError cases came from notebooks with no Markdown cells.This percentage describes the share of FileNotFoundError cases, not the error rate among notebooks without Markdown.Source: Exceptions · p. 13
  • The dataset and methodology could be adapted beyond biomedical research, including for education and training in computational research practices.Filtering notebooks by error type and criteria such as Python version, MeSH terms, journal, or article type could help learners find relevant examples.Source: Exceptions · p. 13

Successful reproductions

Among successfully executed notebooks, some matched the original results and others did not. The paper compares these outcomes across notebook and publication characteristics and notes that several aspects of result differences are difficult to interpret.

Successful reproductions

  • 879 notebooks (5.56%) reproduced the original results, while 324 (2.05%) ran without errors but produced different results.The different-to-identical ratio among successful runs fell from 0.38 in the initial run to 0.27 in the rerun.Source: Successful reproductions · p. 13, Table 2 · p. 15
  • Newer repositories did not generally produce notebooks with better reproduction rates than older repositories.Repository age was measured relative to 2023, and notebooks may be younger than the repositories hosting them.Source: Successful reproductions · p. 14, Figure 27 · p. 14
  • Identical results were more frequent than different results in some MeSH fields, while the reverse held in others.The figure compares successful executions by research-field proxy and distinguishes notebooks with identical versus different results.Source: Successful reproductions · pp. 14–15, Figure 28 · p. 15
  • Notebooks with declared dependencies were more common in the identical-results group than in the different-results group.For notebooks without declarations, the authors used default conda dependencies; original dependency versions were undocumented and might have differed from the conda versions.Source: Successful reproductions · p. 14, Table 2 · p. 15
  • Notebooks in the identical-results group had more Markdown relative to code, while those in the different-results group tended to have more code cells.The authors note that Markdown-to-code ratio has been suggested as a proxy for notebook quality, but describe the group differences as tendencies.Source: Successful reproductions · p. 15
  • Identical-result notebooks had fewer code cells but longer total execution times, with execution time per code cell about 2.7 times that of the different-results group.The authors had no good explanation for this rerun pattern and suggested further research.Source: Successful reproductions · p. 15
  • Result-difference counts are difficult to interpret because they include output-cell, cell-counter, and output-file differences, and early differences can lead to later ones.Other possible sources of divergent results include dynamic data or different cell execution order; the rerun executed each code cell once, top to bottom.Source: Successful reproductions · p. 15
  • Python 3.6 and 2.7 appeared among the five most frequent versions in both result groups, ranking higher for different than identical results.Other top-five versions for different results were Python versions older than 3.6; the other top-five versions for identical results were newer than 3.6.Source: Successful reproductions · p. 15, Table 3 · p. 15

Notebook styling

The notebook corpus contains recurring Python style, definition, and deprecated-key warnings, with some smaller differences between initial and rerun results.

Notebook styling

  • E231 was the most common coding style error, followed by E225 and E265.F403 and F405 relate to variable or module definition errors; W601 and W606 concern deprecated and reserved keys.Source: Notebook styling · p. 16, Table 4 · p. 16
  • The initial and rerun warning profiles were similar overall, but several individual warning types shifted slightly in prevalence.E231 and E262 decreased slightly; E265, E402, E127, F405, and F401 increased.Source: Notebook styling · p. 16
  • Table 5 catalogs common exception types in Python-based Jupyter notebooks and briefly outlines their nature and possible remedies.Source: Table 5 · p. 16

Environmental footprint

The rerun consumed more energy and had a larger estimated carbon footprint than the initial pipeline run.

Environmental footprint

  • 373.78 kWh for the rerun produced an estimated 126.58 kg CO2e, compared with 47.38 kWh and 16.05 kg CO2e initially.The corresponding estimates were 11.51 tree years for the rerun and 17.51 tree months for the initial run, using default values for Germany.Source: Environmental footprint · p. 17

Discussion

The discussion places notebook reproducibility within a broader research ecosystem and considers uptake, methodological boundaries, and practical implications. It identifies dependency handling and automated checks as potential routes to improvement while noting that the study covers only a defined subset of notebooks and reproducibility concerns.

Discussion

  • The study examined method reproducibility for Python notebooks on GitHub mentioned in PubMed Central articles available when the pipeline began.The authors state that inferential reproducibility was not addressed and results reproducibility was only touched on briefly.Source: Discussion · p. 17, Limitations · p. 18

Exploring interactions between Jupyter and research via Wikidata

  • Scholia’s Jupyter use profile links to research resources used alongside notebooks, spanning software and non-software items; Wikidata coverage of scholarship remains incomplete.The broader ecosystem includes resources such as data, instruments, and models that the study’s pipeline captured only partly.Source: Exploring interactions between Jupyter… · p. 17, Figure 29 · p. 17

Limitations

  • The pipeline excluded notebooks outside its scope, including non-Python notebooks, non-GitHub repositories, and branches other than each repository’s base branch.It also assessed only the first error per notebook and did not use Docker images or systematically attempt manual fixes.Source: Limitations · p. 18

Implications

  • The authors report low reproducibility for notebooks associated with biomedical publications and argue that automated basic checks could be included in review procedures.They caution against relying on reviewers alone and suggest systemic approaches across publishers, technology stacks, and languages.Source: Implications · p. 18
  • Dependencies, especially code and data dependencies, are among the common error sources; the authors point to improved dependency handling and standardized publishing checks as potential improvements.They also recommend wider use of Python dependency workflows such as requirements.txt, conda environment files, or Poetry.Source: Implications · pp. 18–19
  • About four thousand notebooks in repositories last committed in 2023 still used earlier Python versions, including versions past their support windows, which the authors say contributes to reproducibility issues.Unspecified library versions may also invoke newer versions incompatible with the original notebook; similar issues can involve APIs, datasets, ontologies, and standards.Source: Implications · p. 19
  • The authors suggest that notebooks and their documented errors could support education, with filtering by research topic or exception type helping users find relevant examples.Using them effectively would require mapping notebook strengths and weaknesses to learning objectives or curriculum requirements.Source: Implications · p. 19
  • The study documents 879 notebooks with results matching the originals, but the authors note that communicating reproducibility status across thousands of notebooks remains unresolved.They also raise the question of how to represent notebooks with differing results, exceptions, installation problems, or missing repositories.Source: Implications · p. 19

Conclusions

The study concludes that biomedical-publication notebooks are increasingly popular and diverse, but their reproducibility remains low despite improvement. Dependency documentation and automated checks are identified as possible avenues for improvement.

Conclusions

  • Low but improving reproducibility was found across 15,817 notebooks associated with 3,467 PubMed Central publications, consistent with earlier studies in other contexts.Source: Conclusions · p. 19
  • The authors identify code and data dependencies as main issues and suggest better documentation and systematic automated checks in peer review as ways reproducibility could improve.They also mention combining notebooks with approaches such as registered reports.Source: Conclusions · p. 19

Data availability

The study’s initial and rerun data and analysis code are available publicly. The authors report no competing interests, no requirement for ethical review, and identify their funding and contributions.

Data availability

  • The initial-study data and rerun data are available through separate Zenodo records, and the analysis code and notebooks are available on GitHub.The records are identified as and.Source: Data availability · p. 20

Ethical Approval (optional)

  • Although the data contain personally identifiable information sourced directly from PMC, the authors report that no research facet triggered ethical review.They describe addressing ethical implications by emphasizing systemic aspects, avoiding focus on individual notebooks or authors, and including environmental-footprint information.Source: Ethical Approval (optional) · p. 20

Competing Interests

Funding

  • Support came from the Carl Zeiss Foundation, Alfred P. Sloan Foundation, MaRDI, and DFG grants; computing resources were supported in part by DFG grants.The passage gives grant and project identifiers for the support.Source: Funding · p. 20

Author’s Contributions

  • Both S.S. and D.M. are credited with conceiving and designing the experiments, performing them, analyzing data, preparing figures or tables, contributing to drafts, and approving the final draft.Source: Author’s Contributions · p. 20

Authors’ information (optional)

S.S. works in computer science and e-Science, while D.M. is a biophysicist; both describe research connected to research workflows and data.

Authors’ information (optional)

  • S.S. works in e-Science, Semantic Web, and Machine Learning, focusing on reproducibility through provenance, linked data, and knowledge graphs.Her stated aim is to foster transparency and facilitate sharing and reuse of research data.Source: Authors’ information (optional) · p. 20
  • D.M. is a biophysicist whose work integrates open research and education workflows with the web across the research cycle.His research spans spatial and temporal scales of life and focuses on data in the life sciences and adjacent fields.Source: Authors’ information (optional) · p. 20
Loading 2308.07333v1…