Source-linked AI summary
Model Retirement Creates Reproducibility Risk in Biomedical AI Publications
Nathan Wolfrath, Meghan Conroy, Thomas Kosten, Dave Bell, Bhabishya Neupane, Jonah Kindel, Anjishnu Banerjee, Priya Deshpande, Bradley Taylor, Anai N. Kothari
TL;DR
Biomedical LLM studies depend heavily on commercial services whose updates and deprecations can complicate reproducibility. The study searched PubMed, extracted and normalized specific model mentions, and compiled lifecycle data for 50 frequently used models. It found that many publications are on a trajectory toward computational non-reproducibility after publication.
Problem
Commercial LLM services can undergo updates and deprecations that complicate reproducibility in biomedical research.
Method
The study systematically characterized this problem by extracting specific LLM mentions and compiling lifecycle data for 50 frequently used models.
Results
77.7% of LLM citations were commercial closed-weight models, while the study found a trajectory toward computational non-reproducibility after publication.
Takeaways & Limitations
Model deprecation should be treated as a core reporting and preservation issue for biomedical research.
Takeaways & Limitations
The analysis was restricted to abstract-level model mentions, and incomplete lifecycle data may understate the problem’s scope.
Abstract
from arXiv · showhide
Background. Large language models (LLMs) are being adopted in biomedical research at a rapid and accelerating pace, yet commercial services that host many widely used models operate under deprecation schedules that can complicate scientific reproducibility. Methods. We searched PubMed for original research articles from 2022 through March 2026 that applied a specific LLM to a biomedical task. An extraction agent identified model names from 61,077 article abstracts with human reviewers validating a subset for extraction accuracy. Extracted model names were normalized to canonical model identifiers. Lifecycle data (release date, retirement date, status) were compiled for the 50 most frequently used models. Results. We identified 8,931 paper-model mentions spanning 5,242 unique publications after restricting the analysis to the 50 most frequently used models. Among these mentions, 77.7% cited a commercial closed-weight model. Overall, 42% involved a model that was already retired by the time of official publication or is scheduled to retire within two years of publication. The median interval from publication to model retirement was 538 days. Conclusion. Many biomedical publications using LLMs are on a trajectory toward computational non-reproducibility after publication. Model deprecation should be treated as a core reporting and preservation issue for biomedical research.
INTRODUCTION
Biomedical LLM research increasingly depends on commercial services whose updates and deprecations researchers cannot fully control. Because reproducibility depends on re-executing published workflows, model retirement creates a model-dependent reproducibility gap that this study systematically characterizes.
- Commercial LLM workflows accessed through interfaces or APIs provide limited control over updates, behavioral changes, and deprecations.
- Reproducibility is foundational to evidence-based medicine, while inability to re-execute published workflows reduces reliability, external validation, and future extension.
- Exact model names, versions, access dates, and prompts are important for reproducing LLM-related work, but rapid model release and deprecation can still create a model-dependent reproducibility gap.
- The study systematically characterizes this problem by quantifying biomedical papers citing models already retired or expected to retire soon after publication.
METHODS
The study searched PubMed for biomedical research articles published between January 2022 and July 2026 using LLM-related terms in titles or abstracts. Publication-type filters excluded reviews and other non-original formats, and records without publication dates beyond the year were excluded.
- The PubMed search covered biomedical research articles published between January 2022 and July 2026.
- Search terms included large language model, LLM, generative AI, artificial intelligence, foundation model, and language model.
- Reviews, systematic reviews, meta-analyses, editorials, letters, and comments were excluded using PubMed publication-type filters.
- Records lacking publication dates more specific than the publication year were excluded.
Model Mention Extraction
An automated agent extracted specific existing LLM uses from article abstracts, human reviewers checked a subset, and extracted names were normalized while nonspecific or out-of-scope model families were excluded.
- A GPT-4o-based agent identified whether abstracts described use of an existing named LLM in a biomedical task.
- Papers whose primary contribution was a novel model or architecture were excluded.
- The agent returned the most specific available version strings, treating each model used in a multi-model paper as a separate paper-model mention.
- Human reviewers assessed extraction accuracy in a subset of 50 processed papers.
- Raw model strings were normalized to harmonize spelling variants and references to the same model.
- Encoder-only, protein-language, generic family-level, and version-insufficient model names were excluded from analysis.
Lifecycle Data
Lifecycle analysis compiled release dates, retirement dates, and status for the 50 most-mentioned models, then measured publication-to-retirement intervals. Open-weight models were treated as having no retirement date, and retirement within two years defined rapid post-publication obsolescence.
- Release dates, retirement dates, and current status were compiled for the 50 most-mentioned models from primary vendor sources where available.
- Models were classified as active, scheduled, retired, or open weight, and by commercial, open-weight, or untracked-commercial license type.
- For each paper-model mention, the analysis computed days between publication and the underlying model retirement date.
- Open-weight models were treated as having no retirement date, and publication dates were collapsed to the month for time-series analysis.
- Retirement within two years of publication defined papers that become rapidly outdated after publication.
RESULTS
The study identified 8,931 normalized paper-model mentions across 5,242 publications, with commercial APIs dominating usage. Model mentions increased over time, and OpenAI models accounted for most in-scope mentions.
- 96.2% average extraction accuracy was found in human review of a subset of 50 extractions.Most extraction errors were missed model mentions rather than hallucinated models.
- 77.7% of in-scope mentions involved commercial API models, while open-weight models accounted for the remainder.
- 65.7% of in-scope mentions involved OpenAI models, led by GPT-4, GPT-4o, and GPT-3.5-turbo.Their mention shares were 27.8%, 14.8%, and 14.2%, respectively.
Reproducibility Risk Due to Model Retirement
Model retirement affected a substantial share of biomedical LLM mentions, including papers published after withdrawal and papers nearing scheduled retirement. The average publication-to-retirement interval was 546 days.
- 41.6% of 8,931 in-scope mentions involved a model with a confirmed retirement date within 730 days of publication.
- 2.9% of mentions cited models that had already been retired at publication.
- 28.5% of 898 mentions with confirmed retirement dates were published after the model was withdrawn.
- 546 days was the mean interval between publication and retirement across retired and scheduled mentions.This corresponds to approximately 17.9 months.
- Claude Sonnet 3.5 had 33.1% of its 172 citing papers appear after its October 2025 retirement.
- All 182 papers citing base GPT-5 will meet the study’s reproducibility-risk definition when its December 2026 retirement occurs.
DISCUSSION
The discussion links commercial model dominance and provider retirement practices to reproducibility risk in biomedical AI. It recommends precise version reporting, persistent preservation, and greater attention to changing model behavior and execution context.
- Reproducibility implications: Open-weight models offer an advantage in persistent and accurate versioning, while remaining competitive with proprietary systems on some biomedical tasks.
- Reproducibility implications: Commercial APIs provided no equivalent preservation guarantee yet accounted for 77.7% of LLM citations in the corpus.
- Reproducibility implications: OpenAI models comprised 65.7% of in-scope mentions, so one vendor’s retirement schedule contributes substantially to corpus-wide reproducibility risk.
- Recommended interventions: Journals should require exact model version strings, such as gpt-4o-2024-08-06 rather than GPT-4o.
- Recommended interventions: For long-term reproducibility tasks, the paper recommends open-weight models and archiving their weights alongside the manuscript.
- Scope and limitations: The analysis was limited by abstract-level extraction, possible extraction errors, changing retirement schedules, and omitted execution-context parameters such as temperature and prompts.
- Scope and limitations: Model names alone may not ensure reproducibility because commercial systems can silently change weights, endpoints, instructions, or alignment behavior.
DATA AVAILABILITY
The authors state that code can be obtained upon reasonable request and that the PubMed abstracts used are publicly available.
- Code is available upon reasonable request to the corresponding author.
- The PubMed abstracts used in the study are publicly available.
AI DISCLOSURE
The authors disclose using ChatGPT and Claude for language refinement, organization, and editing, with all AI-assisted content reviewed by the authors.
- ChatGPT and Claude assisted with language refinement, organization, and editing of the manuscript.
- The authors reviewed and edited all AI-assisted content and accepted responsibility for the published content.
TABLES
The paper uses tables and figures to summarize model usage, study selection, license categories, and temporal trends in biomedical AI research.
- TABLES: Table 1 ranks models by the number of publications in which they were used and reports retirement status for commercial models.
- FIGURE LEGENDS: Figure 1 depicts record identification, screening, model-use assessment, name normalization, and final inclusion.
- FIGURE LEGENDS: Figure 2 shows monthly model mentions by commercial, open-weight, and untracked-commercial license category using 3-month rolling averages.
- FIGURE LEGENDS: Figure 3 shows 3-month rolling-average trends for the 10 most frequently mentioned trackable models, ranked by total mentions.