Source-linked AI summary
6.5% of the Neuro-Symbolic Literature Can Be Reproduced from Its Published Artifacts, a Six-Stage Audit Framework and First Instantiation
Brandon Colelough, Vladimir Martirosyan, Ishan Tamrakar, William Regli, Aditya Kumar, Anh N. Nhu, Dhruv Dubey, Raj Ambavane, Haowei Deng
TL;DR
Published NSAI results are often difficult to recover even when papers advertise public code. The paper introduces and instantiates a six-stage same-artifact audit, finding 85 successful reruns and advocating complete, versioned, permanently archived artifact bundles.
Problem
The paper addresses limited evidence about whether independent teams can recover published NSAI results from authors’ released computational artifacts.
Method
The authors develop a six-stage framework and instantiate it as a same-artifact rerun audit of eligible NSAI studies.
Results
85 studies were fully or partially reproduced, representing 6.52% of the eligible corpus and 18.68% of attempted reruns.
Takeaways & Limitations
The findings support requiring complete, versioned, permanently archived artifact bundles for empirical NSAI publications.
Takeaways & Limitations
Missing checkpoints are treated as non-rerunnable under this protocol, although some papers might remain reproducible through retraining.
Abstract
from arXiv · showhide
We present a six-stage framework for auditing the reproducibility of scientific claims across a research literature within the computer science domain, and instantiate our framework for the neuro-symbolic AI (NSAI) subdomain. Instantiating the framework on the NSAI subdomain produced a multi-year audit. Stage one retrieved 5,497 records and removed 3,018 duplicates. Stage two screened the 2,479 unique records at title and abstract, identifying 1,365 self-identified NSAI records, then removed a further 61 at full text for off-topic, non-research, no-quantitative-evaluation, or inaccessible-full-text reasons. Stage three sought a verifiable public code artifact for each of the 1,304 eligible records and found none for 849, leaving 455 to enter the artifact inventory and bounded rerun of stages four and five. We fully or partially reproduced 85 studies, 6.52% of the eligible corpus and 18.68% of attempted reruns. We found that 321 attempted reruns were blocked by missing non- code artifacts and 42 by missing or unusable code repositories. These figures quantify a persistent reproducibility deficit that survives even nominal "code available" declarations, and signal the need for enforced, versioned, and permanently archived artifact bundles in future NSAI publications. We argue that empirical NSAI papers should be required at submission time to provide complete, versioned, and permanently archived artifact bundles.
Introduction
The paper examines whether published NSAI results can be recovered by independently rerunning their released computational pipelines. It frames the rapidly expanding field as needing stronger standards for executable, verifiable artifacts.
- The audit asks whether an independent team can rerun a published NSAI pipeline and recover its main reported result within a prespecified tolerance.
- NSAI integrates connectionist methods with logic and formal techniques for reasoning, spanning loosely connected pipelines to jointly trained systems.
- The paper reports that results in the majority of NSAI papers cannot be recovered by rerunning released artifacts.
- The authors argue that NSAI publications need greater emphasis on documenting reproducibility so others can verify empirical claims.
Background
The background situates computational reproducibility within a broader reproducibility crisis while emphasizing that computer science artifacts can, in principle, be shared exactly. The paper therefore audits whether NSAI’s shared artifacts support independent verification.
- Reproducibility failures across empirical disciplines involve unavailable data, undisclosed analytical choices, and unspecified experimental assumptions.
- The paper defines reproducibility as an independent team obtaining a consistent result using the original team’s artifacts.
- Computer science should be highly verifiable because algorithms, code, and data can in principle be shared exactly.
- Prior work shows that public datasets can degrade, move, or silently change versions, motivating immutable storage, version identifiers, and provenance metadata.
Methodological Audit Framework
The six-stage framework audits same-artifact rerun reproducibility through sequential corpus, eligibility, artifact, execution, rerun, and outcome-audit checkpoints. Its NSAI instantiation began with a broad search and left 455 code-bearing studies for rerun auditing.
- The framework constructs and screens the corpus, confirms eligibility, inventories six artifact categories, checks execution, reruns experiments, and audits outcome labels.
- The initial bibliographic sweep queried nine major digital libraries and returned 5,497 records.
- Figure 1 separates late full-text eligibility exclusions from rerun outcomes and marks successful reruns within the attempted set as an audit outcome.
- 849 records lacking a verifiable public code artifact were eliminated, leaving 455 code-bearing studies for the rerun audit.
Audit Protocol Stage 4-6
Stages 4–6 operationalize the audit by defining entry criteria, executing repositories under a bounded protocol, and assigning reproduction outcomes after rerun attempts. The protocol distinguishes full and partial recovery using explicit fidelity requirements.
- Entry criteria: Pipeline entry required NSAI self-identification, quantitative empirical evaluation, computer-science relevance, an auditable public code artifact, and accessible full text.
- Entry criteria: Studies were excluded when indispensable rerun artifacts could not be identified, accessed, or reconstructed under the protocol.
- Stages 4–6: Repositories were evaluated through auditable checkpoints covering environment builds, dependency management, metric verification, and licensing constraints.
- Stages 4–6: Outcome labels distinguished fully reproduced studies from partially reproduced studies whose core pipelines executed and main qualitative claims were preserved.
Evaluation Design
The audit distinguishes full reproduction from partial recovery and applies a prespecified fidelity criterion to attempted reruns.
- Full reproduction requires the primary metric to fall within ±5 % absolute error or the authors’ 95 % confidence interval.The run must also satisfy the headline claims and avoid architecture-level bug corrections.
- Partial recovery preserves the paper’s main qualitative claim despite quantitative deviations or incomplete reruns of headline experiments.
- Only studies entering the attempted rerun pool count toward full or partial reproduction outcomes.
Protocol Stages
The framework uses six auditable stages, moving from corpus screening and artifact verification through environment checks, reruns, and outcome auditing.
- The six stages construct and screen the corpus, confirm eligibility, identify artifacts, build environments, rerun experiments, and audit outcomes.
- Stage 1 – Full-text eligibility confirmation: Stage 1 re-checks language, topical relevance, quantitative evaluation, and full-text accessibility before eligibility is confirmed.
- Stage 2 – Repository identification and full-artifact verification: Stage 2 records repository details and verifies public access to code, data, weights, environment files, and documentation.Gated essential artifacts could trigger up to two email requests over 14 days.
- Stage 3 – Environment Build & Executable Integrity: Stage 3 rebuilds repositories with author-provided environment files, then records irreconcilable dependency or fatal code failures as O5 outcomes.
- Stage 4 – Result Re-execution: Stage 4 reruns the primary experiment and records metric fidelity, percentage gaps, and categorical reproduction status.
- Stage 5 – Data Extraction & Audit: Stage 5 extracts design, dataset, compute, metric, and evaluation details, followed by review of O1/O2 papers and a stratified sample of other outcomes.
Technical Infrastructure, Limitations, and Data Management
The audit ran on a university high-performance cluster, defined reproducibility as same-artifact rerunning, and preserved audit materials through institutional and online archiving.
- Technical Infrastructure: GPU jobs used one full node with either 4 × NVIDIA A100 or 4 × NVIDIA H100 GPUs, while CPU jobs used up to one 128-core node with a 7-day cap.Both configurations provided 512 GiB RAM under the cluster’s default quality-of-service limits.
- Limitations: The primary construct measures whether an independent team can obtain the reported result using authors’ released artifacts.
- Limitations: Missing checkpoints are treated as non-rerunnable under this protocol when evaluation depends on a fixed trained model state, not as universally irreproducible methods.Retraining was excluded from the primary analysis because it changes the experimental object and adds variance.
- Data Management: The audit trail and scripts for regenerating aggregates and figures were archived online, while examined artifacts also remained on institutional project storage.
Results
Across 455 attempted NSAI reruns, 85 studies were fully or partially reproduced, while incomplete artifacts remained the dominant barrier. Reproduction rates showed no sustained improvement across publication years or venue groups, and citation impact did not clearly track reproducibility.
- Overall outcomes: 85 of 455 attempted reruns succeeded (18.68%), comprising 48 full reproductions and 37 partial reproductions.Relative to the eligible corpus, 85 successful reruns represented 6.52%.
- Artifact failures: 321 attempted reruns were blocked by missing non-code artifacts, compared with 42 blocked by unavailable or unusable code repositories.When code, data, and weights were all available, reruns succeeded in 85 of 92 cases (92.4%).
- Artifact failures: Missing model weights, environment specifications, and incomplete codebases accounted for 235 of 321 missing-artifact failures (73.2%).The largest reported failure cells involved unexplained missing model weights and unpinned dependencies causing environment failures.
- Publication-year patterns: From 2019 to 2024, annual full-or-partial reproduction rates ranged from 10.3% to 22.1% despite sharply increasing publication volume.The study excludes pre-2019 and partial-year 2025 cohorts when describing the trend and reports no sustained upward trajectory.
- Venue patterns: Venue class was not associated with reproducibility (χ2 = 2.26, df = 2, p = .32), with conferences at 21.1%, journals at 16.5%, and preprints at 15.0%.The three confidence intervals overlapped substantially, and publication venue alone was not a reliable predictor.
- Citation impact: Citation distributions substantially overlapped across reproduction and exclusion buckets, with the fully reproducible bucket having mean citations of 14.1 and median 2.Comparable means occurred in non-reproducible buckets involving missing checkpoints or weights and environment issues.
Discussion
The audit shows that nominal code availability rarely guarantees executable NSAI pipelines: missing artifacts, repository failures, and configuration drift remain central barriers. The discussion therefore emphasizes complete, versioned artifact bundles and repository auditing across venues.
- Fewer than one in five studies with a verifiable public code artifact could be rerun.
- 70.55% of attempted reruns were blocked by missing non-code artifacts, while 9.23% were blocked by absent code.
- Only 7 of 455 attempts failed when code, data, and weights were complete, supporting automated artifact checks before peer review.
- The proposed artifact bundle includes code, datasets, model weights, evaluation scripts, environment files, and documentation archived in a DOI-minting repository.
- Case studies: Scallop’s reproducibility depended on frozen commits, dataset splits, pretrained weights, environment manifests, lockfiles, hashes, and automated validation.
- Case studies: Across Scallop, LogiCity, and MARS, all reported metrics were reproduced with absolute error no greater than three percentage points across publication strata.
Conclusion
This study presents a large-scale longitudinal audit of same-artifact rerun reproducibility in NSAI. Among 1,304 eligible records, 455 entered attempted reruns and 85 were fully or partially reproduced, motivating mandatory artifact audits and complete archived bundles.
- The study is presented as the first longitudinal, large-scale assessment of same-artifact rerun reproducibility in neuro-symbolic AI.
- Among 1,304 eligible NSAI records, 455 entered the attempted rerun audit and 85 were fully or partially reproduced.
- The recommended six-item bundle contains source code, immutable datasets, pretrained checkpoints, an executable evaluation script, an environment descriptor, and documentation.
Appendix
The supplementary archive deposits the study’s reproduction records, annotations, protocols, documentation, scripts, aggregate results, and an offline results browser.
- The supplementary materials are deposited in a Zenodo archive identified by DOI 10.5281/zenodo.21779234.
- The archive contains paper-level reproduction and data-extraction records, missing-artifact annotations, and audit protocols.
- It also includes annotator documentation, analysis scripts, generated aggregate results, and an offline copy of the results browser.
Broader impacts
The work may encourage more transparent and cumulative NSAI research, but stronger artifact requirements can burden authors and create tensions around sensitive or restricted materials. The passage identifies mitigations for these risks.
- The work can encourage more transparent, verifiable, and cumulative Neuro-Symbolic AI research.
- Stronger artifact-release requirements may increase burdens on authors, especially under-resourced groups.
- Requirements may pressure authors to release sensitive, proprietary, or legally restricted artifacts, with mitigations including gated access and documented exemptions.
Final List of Reproduced Works
The reproduced works span neuro-symbolic reasoning, perception, planning, verification, structured prediction, and symbolic representation. Together, they illustrate diverse combinations of neural models with formal or interpretable components.
- Reasoning and representation: Neural-symbolic reasoning frameworks combine neural perception or learning with symbolic processing, probabilistic logic, logical constraints, or explicit reasoning spaces.Examples include NeuPSL, Neural Logic Machines, VAEL, A-NESI, neurosymbolic vectors, and neurosymbolic diffusion models.
- Perception and uncertainty: Neuro-symbolic architectures also support visual reasoning and uncertainty-aware prediction by combining image or concept processing with logical structure.Examples include graph-structured visual question answering, constitutional filtering, and concept-level confidence calibration.
- Language and planning: Several works integrate structured or logical representations into language, story, and planning systems to support controllable inference or generation.The systems cover natural-language inference, narrative planning, story reasoning, and embodied task planning.
- Specialized applications: The list includes specialized systems for efficient arithmetic-circuit computation, physics-informed symbolic regression, and association-rule extraction from tabular data.These works emphasize scalability, benchmark accuracy, compact rule discovery, or computational efficiency in their respective settings.
- Interpretability and formalization: Other studies connect neural models with interpretable rules or formal specifications for classification, feature selection, verification, and domain-specific decision-making.Applications include hierarchical multi-label classification, neural-network verification, feature-subset generation, and drug discovery.