Source-linked AI summary

Model Retirement Creates Reproducibility Risk in Biomedical AI Publications

Nathan Wolfrath, Meghan Conroy, Thomas Kosten, Dave Bell, Bhabishya Neupane, Jonah Kindel, Anjishnu Banerjee, Priya Deshpande, Bradley Taylor, Anai N. Kothari

arXiv:2609.04699v1cs.AI

TL;DR

Biomedical LLM studies depend heavily on commercial services whose updates and deprecations can complicate reproducibility. The study searched PubMed, extracted and normalized specific model mentions, and compiled lifecycle data for 50 frequently used models. It found that many publications are on a trajectory toward computational non-reproducibility after publication.

  • Problem

    Commercial LLM services can undergo updates and deprecations that complicate reproducibility in biomedical research.

    Source: INTRODUCTION · pp. 4–5
  • Method

    The study systematically characterized this problem by extracting specific LLM mentions and compiling lifecycle data for 50 frequently used models.

    Source: INTRODUCTION · p. 5, Model Mention Extraction · p. 6, DISCUSSION · p. 13
  • Results

    77.7% of LLM citations were commercial closed-weight models, while the study found a trajectory toward computational non-reproducibility after publication.

    Source: DISCUSSION · pp. 12–13
  • Takeaways & Limitations

    Model deprecation should be treated as a core reporting and preservation issue for biomedical research.

    Source: DISCUSSION · p. 13

    The analysis was restricted to abstract-level model mentions, and incomplete lifecycle data may understate the problem’s scope.

    Source: DISCUSSION · pp. 13–14

Abstract

from arXiv · show

Background. Large language models (LLMs) are being adopted in biomedical research at a rapid and accelerating pace, yet commercial services that host many widely used models operate under deprecation schedules that can complicate scientific reproducibility. Methods. We searched PubMed for original research articles from 2022 through March 2026 that applied a specific LLM to a biomedical task. An extraction agent identified model names from 61,077 article abstracts with human reviewers validating a subset for extraction accuracy. Extracted model names were normalized to canonical model identifiers. Lifecycle data (release date, retirement date, status) were compiled for the 50 most frequently used models. Results. We identified 8,931 paper-model mentions spanning 5,242 unique publications after restricting the analysis to the 50 most frequently used models. Among these mentions, 77.7% cited a commercial closed-weight model. Overall, 42% involved a model that was already retired by the time of official publication or is scheduled to retire within two years of publication. The median interval from publication to model retirement was 538 days. Conclusion. Many biomedical publications using LLMs are on a trajectory toward computational non-reproducibility after publication. Model deprecation should be treated as a core reporting and preservation issue for biomedical research.

INTRODUCTION

Biomedical LLM research increasingly depends on commercial services whose updates and deprecations researchers cannot fully control. Because reproducibility depends on re-executing published workflows, model retirement creates a model-dependent reproducibility gap that this study systematically characterizes.

  • Commercial LLM workflows accessed through interfaces or APIs provide limited control over updates, behavioral changes, and deprecations.Source: INTRODUCTION · p. 4
  • Reproducibility is foundational to evidence-based medicine, while inability to re-execute published workflows reduces reliability, external validation, and future extension.Source: INTRODUCTION · p. 4
  • Exact model names, versions, access dates, and prompts are important for reproducing LLM-related work, but rapid model release and deprecation can still create a model-dependent reproducibility gap.Source: INTRODUCTION · pp. 4–5
  • The study systematically characterizes this problem by quantifying biomedical papers citing models already retired or expected to retire soon after publication.Source: INTRODUCTION · p. 5

METHODS

The study searched PubMed for biomedical research articles published between January 2022 and July 2026 using LLM-related terms in titles or abstracts. Publication-type filters excluded reviews and other non-original formats, and records without publication dates beyond the year were excluded.

  • The PubMed search covered biomedical research articles published between January 2022 and July 2026.Source: Literature Search · p. 5
  • Search terms included large language model, LLM, generative AI, artificial intelligence, foundation model, and language model.Source: Literature Search · p. 5
  • Reviews, systematic reviews, meta-analyses, editorials, letters, and comments were excluded using PubMed publication-type filters.Source: Literature Search · p. 5
  • Records lacking publication dates more specific than the publication year were excluded.Source: Literature Search · p. 5

Model Mention Extraction

An automated agent extracted specific existing LLM uses from article abstracts, human reviewers checked a subset, and extracted names were normalized while nonspecific or out-of-scope model families were excluded.

Lifecycle Data

Lifecycle analysis compiled release dates, retirement dates, and status for the 50 most-mentioned models, then measured publication-to-retirement intervals. Open-weight models were treated as having no retirement date, and retirement within two years defined rapid post-publication obsolescence.

  • Release dates, retirement dates, and current status were compiled for the 50 most-mentioned models from primary vendor sources where available.Source: Lifecycle Data · p. 7
  • Models were classified as active, scheduled, retired, or open weight, and by commercial, open-weight, or untracked-commercial license type.Source: Lifecycle Data · p. 7
  • For each paper-model mention, the analysis computed days between publication and the underlying model retirement date.Source: Lifecycle Data · p. 7
  • Open-weight models were treated as having no retirement date, and publication dates were collapsed to the month for time-series analysis.Source: Lifecycle Data · p. 7
  • Retirement within two years of publication defined papers that become rapidly outdated after publication.Source: Lifecycle Data · p. 7

RESULTS

The study identified 8,931 normalized paper-model mentions across 5,242 publications, with commercial APIs dominating usage. Model mentions increased over time, and OpenAI models accounted for most in-scope mentions.

  • 96.2% average extraction accuracy was found in human review of a subset of 50 extractions.Most extraction errors were missed model mentions rather than hallucinated models.Source: Corpus Characteristics · p. 8
  • 77.7% of in-scope mentions involved commercial API models, while open-weight models accounted for the remainder.Source: License and Vendor Distribution · p. 10
  • 65.7% of in-scope mentions involved OpenAI models, led by GPT-4, GPT-4o, and GPT-3.5-turbo.Their mention shares were 27.8%, 14.8%, and 14.2%, respectively.Source: License and Vendor Distribution · p. 10

Reproducibility Risk Due to Model Retirement

Model retirement affected a substantial share of biomedical LLM mentions, including papers published after withdrawal and papers nearing scheduled retirement. The average publication-to-retirement interval was 546 days.

DISCUSSION

The discussion links commercial model dominance and provider retirement practices to reproducibility risk in biomedical AI. It recommends precise version reporting, persistent preservation, and greater attention to changing model behavior and execution context.

Reproducibility implications

  • Open-weight models offer an advantage in persistent and accurate versioning, while remaining competitive with proprietary systems on some biomedical tasks.Source: DISCUSSION · p. 12
  • Commercial APIs provided no equivalent preservation guarantee yet accounted for 77.7% of LLM citations in the corpus.Source: DISCUSSION · p. 12
  • OpenAI models comprised 65.7% of in-scope mentions, so one vendor’s retirement schedule contributes substantially to corpus-wide reproducibility risk.Source: DISCUSSION · pp. 12–13

Recommended interventions

  • Journals should require exact model version strings, such as gpt-4o-2024-08-06 rather than GPT-4o.Source: DISCUSSION · p. 13
  • For long-term reproducibility tasks, the paper recommends open-weight models and archiving their weights alongside the manuscript.Source: DISCUSSION · p. 13

Scope and limitations

  • The analysis was limited by abstract-level extraction, possible extraction errors, changing retirement schedules, and omitted execution-context parameters such as temperature and prompts.Source: DISCUSSION · p. 13
  • Model names alone may not ensure reproducibility because commercial systems can silently change weights, endpoints, instructions, or alignment behavior.Source: DISCUSSION · p. 14

DATA AVAILABILITY

The authors state that code can be obtained upon reasonable request and that the PubMed abstracts used are publicly available.

AI DISCLOSURE

The authors disclose using ChatGPT and Claude for language refinement, organization, and editing, with all AI-assisted content reviewed by the authors.

  • ChatGPT and Claude assisted with language refinement, organization, and editing of the manuscript.Source: AI DISCLOSURE · p. 15
  • The authors reviewed and edited all AI-assisted content and accepted responsibility for the published content.Source: AI DISCLOSURE · p. 15

TABLES

The paper uses tables and figures to summarize model usage, study selection, license categories, and temporal trends in biomedical AI research.

TABLES

  • Table 1 ranks models by the number of publications in which they were used and reports retirement status for commercial models.Source: Table 1 · p. 16

FIGURE LEGENDS

  • Figure 1 depicts record identification, screening, model-use assessment, name normalization, and final inclusion.Source: Figure 1
  • Figure 2 shows monthly model mentions by commercial, open-weight, and untracked-commercial license category using 3-month rolling averages.Source: Figure 2
  • Figure 3 shows 3-month rolling-average trends for the 10 most frequently mentioned trackable models, ranked by total mentions.Source: Figure 3
Loading 2609.04699v1…