Source-linked AI summary
Papers Without Code: Availability of GitHub Repositories Linked in *CL Publications
Selina Meyer, Michael Roth
TL;DR
Long-term availability of research repositories linked from *CL papers has not been well evaluated. The paper checks linked repositories across venues and platforms, finding persistent unavailability, especially for GitHub; it recommends steps to support reliable access.
Problem
The long-term availability of repositories linked from *CL papers has not been well evaluated, although such artifacts matter for reproducibility and research reuse.
Source: §1 · p. 1Method
Results
Unavailable repositories and placeholders persist, and rising publication counts translate these rates into an accelerating growth of unavailable repositories and artifacts year after year.
Source: §6 · p. 8Takeaways & Limitations
The authors recommend supporting reliable access over time and extend aclpubcheck with a lightweight method for checking papers for invalid or broken GitHub links.
Source: §6 · p. 8The study assesses repository availability, not whether papers that should share code or data actually do so, and it does not evaluate the quality or documentation of provided code.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Source code and data published at computational linguistics (*CL) venues are increasingly being shared via GitHub. While this generally is a favourable development for the accessibility and potential reusability of research artifacts in natural language processing (NLP), the long-term availability of such repositories has not been evaluated. In this squib, we discuss the availability of repositories linked in papers published in the Computational Linguistics (CL) journal as well as at ACL and its co-located events over the past ten years. Contrary to our expectations, we find that GitHub repositories linked in more recent ACL publications are unavailable at similar rates as in older publications, in parts due to an increase in empty and placeholder repositories. Similar trends hold for other *CL venues, but not for platforms other than GitHub.
1. Introduction
Publicly sharing research artifacts is presented as important for reproducibility and good scientific practice, yet their long-term availability in NLP is little known. The paper evaluates linked repositories across ACL papers and other *CL venues, finding persistent unavailability and recommending ways to improve access.
1. Introduction
- The long-term availability of GitHub repositories linked in NLP papers is little known, despite public artifact sharing being treated as important for reproducibility and good scientific practice.Some national science funding organizations mandate long-term storage of at least ten years.Source: §1 · p. 1
- The study checks GitHub links in CL journal, ACL, and co-located-event papers over ten years, finding substantial unavailability even among ACL papers published in 2025, often for the papers’ own artifacts.The issue is not exclusive to ACL and occurs most commonly with GitHub.Source: §1 · pp. 1–2
- The authors recommend improving long-term artifact availability and provide a low-resource Python script integrated into aclpubcheck to check GitHub repository availability in camera-ready papers.The recommendations address authors, reviewers, organizers, and editors.Source: §1 · p. 2
2. Related Work
Prior *CL work has examined artifact availability, reproducibility, and the benefits of sharing, while efforts to support reproducibility remain fragmented. This study complements that work with a large-scale analysis of linked GitHub repositories over time.
2. Related Work
- Reproducibility efforts in *CL include tutorials, checklists, and a pilot track, but the field’s approach remains fragmented compared with some other disciplines.Other fields have dedicated tracks, code-submission policies, and workflows for evaluating artifacts.Source: §2 · p. 2
- Mieskes (2017) found over 15% of links to collected data from *CL papers were unavailable just one year after publication.Wieling, Rawee, and van Noord (2018) reported increased data and code availability and a higher share of working links in ACL 2011 and 2016 papers.Source: §2 · pp. 2–3
- Replication surveys identify unavailable resources as a common issue, while Arvan, Pina, and Parde (2022) reproduced results for only two of eight sampled papers.They also found that code availability alone does not ensure reproducibility.Source: §2 · p. 3
- Prior studies associate artifact availability with higher acceptance rates, improved reviewer scores, perceived reproducibility, and citation counts; this analysis focuses on linked GitHub repository availability over time.The reported associations are attributed to Magnusson, Smith, and Dodge (2023), Chaudhuri and Salakhutdinov (2019), Pineau et al. (2021), Kang, Kang, and Jang (2023), and Colavizza et al. (2024).Source: §2 · p. 3
3. Methods
The study extracts repository links from CL and ACL papers and comparison venues, checks GitHub availability, manually verifies links flagged as unavailable or empty, and classifies repository status and artifact ownership.
3. Methods
- The corpus covers CL and ACL papers from 2015–2025, plus LREC and EACL 2024 and EMNLP, NAACL, AACL, and COLING 2025 papers for comparison.The conference comparisons include co-located events.Source: §3 · p. 3
- GitHub links are extracted from embedded hyperlinks and paper text, filtered to repository links, normalized, and checked through the GitHub API for availability and file counts.Searching paper text captures URLs that are not embedded as hyperlinks, which is common in older papers.Source: §3 · p. 3
- Links flagged as unavailable or empty are manually checked in a browser and against the author's GitHub repositories; single-file repositories are inspected for placeholder README files.Formatting issues are corrected when necessary, and the authors search for the corresponding GitHub user if a link remains unresolved.Source: §3 · p. 3
- Repositories are classified as available or unavailable, with unavailable cases split into 404, empty, and placeholder categories.Placeholders contain only a license or README without links to code or data or contact information; unavailable, empty, and placeholder links are also checked for whether they refer to the paper’s own artifacts or related work.Source: §3 · p. 4
4. Results
ACL papers show substantial GitHub repository unavailability, including in recent years, while the CL journal has a lower rate. Similar unavailability patterns occur across computational linguistics conferences, whereas Zenodo links are less often unavailable than GitHub links.
4. Results
- 14,727 unique GitHub links were extracted from 12,087 ACL papers; 354 returned 404 errors, 326 were empty or placeholders, and 88% of unavailable links pointed to papers’ own artifacts.The paper also reports noticeable rates of unavailable repositories in recent years.Source: §4 · p. 4
- In 2025, 41% of unavailable links were placeholders, versus a 28% mean in previous years; 404 errors fell from 63% on average to 47%.In 2025, 80% of 404 errors concerned papers’ own code or data, which might indicate that those repositories were never created.Source: §4 · pp. 4–5, Figure 4 · p. 5
- Unavailable repositories appeared most commonly in Findings of the ACL papers, but also in the main track and co-located events.Figure 5 presents annual shares by paper type or event for 2021–2025.Source: §4 · p. 5, Figure 5 · p. 5
- Among 317 unique GitHub repositories extracted from 369 CL papers published between 2015 and 2025, 7 (2%) were unavailable, and six pointed to the paper’s own code or data.Source: §4 · p. 5
4.1 Comparison with Other Conferences
- Across six other computational linguistics conferences, 4–5% of GitHub repositories were unavailable in most cases, with similar overall rates and unavailability-type distributions.AACL had a slightly higher share of unavailable links; EACL, AACL, and NAACL had more 404s and fewer placeholders than other conferences.Source: §4.1 · p. 5, Table 1 · p. 6
4.2 Comparison with Other Online Repositories
- HuggingFace links became more frequent, while GitLab, Bitbucket, and Zenodo were rarely linked; 3% of unique HuggingFace links were unavailable, mostly links to others’ models or data.Manual verification found that most unavailable HuggingFace links were incorrect paths, often associated with name changes; HuggingFace appears to be used more for existing models and data than for publishing new research artifacts.Source: §4.2 · pp. 6–7, Figure 6 · p. 6, Figure 7 · p. 6
- Only two unique Zenodo URLs were unavailable (0.4%), compared with a 3% unavailability rate for GitHub repositories linked in articles since 2015.The paper says this reinforces the notion that persistent and safe repositories facilitate long-term use of research outcomes and artifacts.Source: §4.2 · p. 7
5. Discussion and Recommendations
The authors argue that insufficient verification of linked artifacts risks trust and uptake, and recommend checks by authors, reviewers, and publication chairs. They also provide an aclpubcheck tool whose evaluation found useful detection performance, though with false positives.
5. Discussion and Recommendations
- The authors warn that papers claiming to open-source code may link to missing or unfilled repositories, which they view as a breach of trust.They suggest review-process pressure may contribute and say stronger validation could increase potential uptake of results and methods.Source: §5 · p. 7
- Authors are encouraged to share code and data at submission, using OpenReview uploads or anonymized repositories when needed.The likelihood of meeting this aim may be higher when artifact deadlines follow the main review submission deadline.Source: §5 · p. 7
- Reviewers should check artifact accessibility and contents, while publication chairs and editors should check camera-ready papers for broken links and invalid GitHub references.Empty and placeholder repositories should be mentioned in reviews.Source: §5 · p. 7
- The authors integrated their extraction script into aclpubcheck to help check GitHub repository availability in the publication workflow.In a sample of 50 flagged and 50 unflagged papers, no unavailable repositories were found in the control sample, which the authors interpret as suggesting high recall.Source: §5 · p. 8
- The tool identified papers with problematic links with 89% accuracy and 78% precision; paper-level precision rose from 25% (2015) to 72% (2025).The paper-level increase was largely due to better PDF hyperlinking.Source: §5 · p. 8
6. Conclusion
The study finds persistent unavailability and placeholder repositories among linked artifacts at CL and ACL venues, with rising publication counts translating into more unavailable artifacts. It recommends workflow checks while limiting its assessment to long-term availability, not whether papers should share code or data.
6. Conclusion
- The study finds persistent unavailability rates and placeholder repositories among repositories linked in CL and ACL papers over the past ten years.With publication counts rising, these rates translate into accelerating growth in unavailable repositories and artifacts year after year.Source: §6 · p. 8
- The authors state that unavailable code and data significantly mitigate the reusability of methods and trustworthiness of results.They outline recommendations to support reliable resource access over time.Source: §6 · p. 8
- The authors expand aclpubcheck with a lightweight method for checking invalid or broken GitHub references and encourage its use in publication workflows.They specifically encourage authors and publication chairs to introduce the method into current workflows.Source: §6 · p. 8
- The study does not assess whether papers that should share code or data actually do so; it focuses on long-term availability of repositories linked by papers claiming to share artifacts.The authors propose future work on automatically evaluating code quality and documentation.Source: §6 · p. 8