A large-scale study on research code quality and execution

• Research code reuse and reproduction are often difficult because published materials may lack adequate documentation, contain code errors, or omit files.Source: §1 · pp. 1–2
• Literate programming intersperses source code with natural-language explanations, but researchers often lack incentives to structure code for reuse.Source: §1 · p. 2
• Repositories support dissemination but cannot fully prevent irreproducibility and code-rot across diverse languages and complex computing processes.Source: §1 · p. 2
• The study analyzes shared datasets and code, evaluates external re-execution and common errors, compares re-execution across contexts, and develops recommendations for researchers, journals, and repositories.Source: §1 · p. 2

• Harvard Dataverse is an open-source repository platform for sharing, archiving, and citing research data, with instances hosted by more than 60 institutions worldwide.Source: §2 · pp. 2–3
• A replication package is a bundle of data, code, documentation, or related files associated with a published scientific result, typically deposited by researchers themselves.Source: §2 · p. 3
• Stata and R are the most popular programming languages among Dataverse users, reflecting the repository’s origins in quantitative social science research.Source: §2 · p. 3
• R is chosen for analysis because it is free, open source, popular in scientific computing, and widely used by statisticians and data analysts.Source: §2 · pp. 3–4

3 Implementation and methods

• The study analyzes 2,109 publicly available replication packages containing 9,078 R code files retrieved from Harvard Dataverse.Source: §3 · p. 4
• AWS Batch parallelizes package retrieval and re-execution by submitting DOI-defined jobs to a queue and running each package in a pre-installed Docker image.Source: §3 · p. 4, Figure 2 · p. 5
• Each package is executed on an EC2 instance with 16 vCPUs and 1024 GB of memory, with results stored in DynamoDB.Source: §3 · p. 4

3.1 Data collection workflow

• The workflow downloads packages, checks retrieval integrity, measures package and code properties, and records execution outcomes with and without automatic cleaning.Source: §3.1 · pp. 5–6
• 97% of the 2,170 packages visible through the Dataverse API were successfully retrieved; restricted access and erroneous encoding excluded the remainder.Source: §3.1 · p. 6

3.2 Code cleaning

• Code cleaning removes absolute paths, standardizes encoding, identifies and imports used libraries, and installs missing libraries while aiming to avoid introducing new errors.Source: §3.2 · p. 6

4 Results and discussion

• Replication datasets averaged 92 MB and 17 files, although most were under 10 MB and contained fewer than 15 files; 40% also included non-R code.Stata appeared in 620 datasets, Python in 46, and SAS, C++, or MATLAB in 9, 7, and 6 datasets, respectively.Source: §4 · pp. 6–7

RQ 2. Does the research code comply with literate programming and software best practices?

• Research code showed mixed adherence to software practices: 57% of datasets had identifiable documentation, comments comprised about 20% of code, and functions averaged 82 lines.About 32% of filenames contained spaces, average variable names were 10 characters, and only 2,934 of 8,875 R files contained functions or classes.Source: RQ 2. Does the research code comply… · pp. 8–10

RQ 3. What are the most used R libraries?

• Individual R files required an average of 4.3 external dependencies, most datasets explicitly depended on up to 10 external libraries, and ggplot2 was the most frequently used library.Testing, provenance, runtime-environment, and workflow libraries were absent. No install.R file was found, although similarly named installation files existed.Source: RQ 3. What are the most used R… · pp. 10–12

RQ 4. What is the code re-execution rate?

• 56% of R files succeeded in the best-of-both re-execution results, compared with 25% without code cleaning and 40% with cleaning alone.The study ran code in R 3.2, R 3.6, and R 4.0, excluding files or packages that exceeded time limits; results with code cleaning were treated as primary.Source: RQ 4. What is the code re-execution… · pp. 12–13

RQ 5. Can automatic code cleaning with small changes to the code aid in its re-execution?

• Automatic code cleaning increased combined success by about 10%, fixed all setwd-related errors, and installed missing libraries when needed without breaking previously successful code.Remaining failures included incompatible library versions, inaccurate paths or directories, missing outputs, and unavailable R objects.Source: RQ 5. Can automatic code cleaning with… · p. 13
• Dataset-level success reached 45% (648 of 1,447), closely matching the 40% file-level rate, and excluding non-R code produced comparable 38% file-level and 45% dataset-level rates.The similarity suggests most R files were designed to run independently, although unknown workflow ordering could reduce measured success.Source: RQ 5. Can automatic code cleaning with… · pp. 14–15

RQ 7. What is the success rate in the datasets belonging to journal Dataverse collections?

• Journal-linked datasets had 42% file-level and 47% dataset-level success, slightly above overall rates, while journals with the strictest policies had the highest re-execution rates.Policies were categorized as encouraged, required, reviewed, verified, or absent; the reported pattern suggests policy strictness is positively correlated with re-execution.Source: RQ 7. What is the success rate in the… · pp. 15, 17
• R 3.6 had the highest success rate per year, R 3.2 performed best for 2016–2017 packages, and R 4.0 generally performed poorly, but release year and R version were not significantly correlated.The findings indicate some backward compatibility while underscoring that changing software and library environments complicate reproducibility.Source: RQ 7. What is the success rate in the… · pp. 17–18

RQ 9. What is the success rate per research field?

• Health and life sciences had the highest re-execution rates, whereas physics had the lowest, but small samples and repository-selection effects prevented broad field-level conclusions.The authors suggest stronger proof-oriented research cultures may explain the health and life sciences result.Source: RQ 9. What is the success rate per… · p. 19

RQ 10. How does the re-execution rate relate to research reproducibility?

• In the Journal of Experimental Political Science package, recreated logs and tables were nearly identical, except for occasional decimal discrepancies possibly linked to unavailable SDMTools.Two files matched the originals, while the third produced a warning under R 3.6.Source: RQ 10. How does the re-execution rate… · p. 20
• The Research & Politics package printed correlation coefficients but could not save its plot because of a broken Linux dependency.The script was expected to run with R 3.2.Source: RQ 10. How does the re-execution rate… · p. 20
• The Review of Economics and Statistics package reproduced numerical estimates and probabilities without errors or warnings, but Docker suppressed its pop-up plotting functions.One file defined functions and the other invoked them.Source: RQ 10. How does the re-execution rate… · pp. 20–21
• Successful re-execution may be a good indicator of computational reproducibility, but it is not sufficient to establish reproducibility.The study identified 127 datasets in which all R files re-executed successfully.Source: RQ 10. How does the re-execution rate… · pp. 20–21

• Automated, large-scale analysis cannot detect all nuances in research code, limiting the sophistication of its reproducibility assessment.Source: §5 · p. 21
• Successful code re-execution can still produce results that differ from those reported, creating possible false positives for reproducibility.Source: §5 · p. 21
• R-version testing was limited, and incompatibilities between R versions and library versions can cause failures or unexpected behavior.Source: §5 · p. 21
• Cloud-resource constraints produced many time-limit outcomes, partly because the study used EC2 instances larger than necessary.Source: §5 · p. 21
• Using datasets from one research-focused repository limits the study’s source scope, although Harvard Dataverse is curated for research data.Source: §5 · pp. 21–22

Recommendations for researchers

• Researchers should capture library versions, use relative paths, automate execution and workflows, test in clean environments, and prefer open-source software.Source: §6 · p. 22

Recommendations for repositories

• Repositories should capture re-execution commands so users can determine file order and input arguments.Source: §6 · p. 22
• Repository integration with reproducibility platforms and containerized tools could capture dependencies and test code before publication.Source: §6 · p. 22
• A working group can coordinate expertise, identify community-wide problems, prioritize them, and implement improvements in repository software.Source: §6 · p. 23

Recommendations for journal editors

• Journals should review deposited material, use reproducibility checklists, and recommend tools that facilitate code automation and re-execution.Source: §6 · p. 23

• Manual reproduction succeeded for 33% of 67 economics papers without author contact and 43% with assistance, while another study reproduced 26% of findings from 204 Science papers.Source: §7 · p. 23
• Journal data-sharing policy improved access and reproducibility but was less effective than code review or verification in enabling re-execution.Source: §7 · pp. 23–24
• R-focused work found two studies ran without issues, 33 had resolvable issues, and two had unresolved issues, with recurring problems including wrong directories, deprecated functions, missing libraries, missing data, and faulty calls.Source: §7 · p. 24
• A large Jupyter-notebook study found 24.11% completed execution successfully and about 4% re-executed with the same result, with library problems recurring across languages.Source: §7 · p. 24

• The study contributes collected data, open-source code, analysis findings, and recommendations for researchers, repositories, and journals.Source: §8 · p. 24
• Most datasets were small and documented, but Rnw and RMD reporting was rare, and libraries for testing, provenance, and automated workflows were not used.Source: §8 · pp. 24–25
• About 40% of R-code executions succeeded overall, with 1,472 of 3,695 cases re-executing successfully across three R versions and code cleaning.Source: §8 · p. 25
• Stricter journal data policies, especially code review and verification, were associated with the highest re-execution rates, while same-year dataset and R-version releases showed no significant correlation.Source: §8 · p. 25
• Basic code cleaning improved code re-execution by about 10%, motivating recommendations for tools integrated into authoring or repository workflows.Source: §8 · p. 25

A Technical implementation of code cleaning

• The cleaner scans R files and applies simple automatic modifications to address common errors while reducing the risk of breaking code.It first converts non-ASCII R files to ASCII to reduce syntax errors caused by symbols from other operating systems.Source: A Technical implementation of code… · p. 25
• Replacing setwd commands with the downloaded-files directory prevents failures caused by unavailable original directory paths.The implementation detects setwd and assigns the current directory containing the downloaded files as the working directory.Source: A Technical implementation of code… · p. 25
• The implementation avoids standardized static-analysis packages because tools such as goodpractice and lintr do not automatically modify code.The US Cran repository was set as the default to avoid Cran errors.Source: A Technical implementation of code… · pp. 25–26
• Long file paths are replaced with basename-based references because replication-package files are downloaded into one directory.This approach removes path separators and retains only the target filename.Source: A Technical implementation of code… · p. 26
• Missing R libraries trigger installation only after require fails, reducing duplicated installation code and related errors.For example, library(dplyr) becomes a conditional require followed by install.packages when necessary.Source: A Technical implementation of code… · p. 26

Code Availability and Data Records

• The data and code are deposited in Harvard Dataverse under CC0 and mirrored on GitHub under the MIT license.The Dataverse record is identified as 10.7910/DVN/UZLXSZ, and the GitHub repository is dataverse-r-study.Source: Code Availability and Data Records · p. 26
• Harvard Dataverse provides the study’s primary public repository for its data and code records.The cited Dataverse deposit is available under a CC0 license.Source: Code Availability and Data Records · p. 26
• GitHub provides an additional public location for the study materials under the MIT license.The cited repository is dataverse-r-study.Source: Code Availability and Data Records · p. 26