Source-linked AI summary
When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications
Hyungjin Lukas Kim
TL;DR
Commercial LLM applications rely on models providers can retire, but prior work had not measured ecosystem-wide migration timing, breakage, or code costs. The paper matches GitHub migration commits to official retirement events and validates a stratified sample. It finds that most genuine migrations occur after shutdown, with post-shutdown migration shares varying alongside provider notice policy.
Problem
Prior work documented model-migration challenges but had not measured ecosystem-wide migration timing, breakage, or code costs when retirements occur.
Source: §1 · p. 1Method
The study matches GitHub commits migrating away from officially retired models and endpoints to provider events, then validates a stratified sample and reweights estimates by coder labels.
Source: §3 · p. 2Results
82% (95% CI 79–84) of migrations away from already-retired models were committed after shutdown, and provider-level shares varied from 64% for Google to 89% for Anthropic.
Source: §4.1 · p. 2, §4.2 · p. 3Takeaways & Limitations
The authors conclude that retirement is often unplanned for applications and release the data and pipeline so providers, builders, and assessors can track this dependency.
Source: §8 · p. 6The study underrepresents projects that migrate silently or never migrate, and commit-message keyword matching admits unrelated commits.
Source: §6 · p. 5
Abstract
from arXiv · showhide
Applications built on commercial large language model (LLM) APIs depend on model versions that providers retire on their own schedule, with notice periods ranging from one year to two weeks. We ask what actually happens to applications when a model is retired. We mine GitHub for commits that migrate away from officially deprecated models and endpoints of OpenAI, Anthropic, and Google, matching each commit to the provider's published announcement and shutdown dates. From 22,555 commits in 17,703 non-fork repositories (2024-2026), 5,139 are matched to an official event; two independent coders validated a stratified sample of 300 (kappa = 0.89-0.95), and we reweight all estimates by their labels. We find that an estimated 82% (95% CI 79-84) of migrations away from retired models were committed after the shutdown date - after the application had started failing - regardless of repository popularity, prior retirement experience, or the presence of a provider-abstraction layer. The share tracks the provider's notice policy: 89% for Anthropic's 60-114-day notices versus 13% for OpenAI's one-year Assistants API notice, and each e-fold increase in notice length reduces the odds of post-shutdown migration by about three quarters. Model identifiers are hard-coded in 94% of migrating applications, migration effort scales from a median of 6 added lines for prompt-only applications to nearly 700 for fine-tuned ones, and only 8% of migrations switch provider. We release the dataset and pipeline and discuss implications for deprecation policy, dependency-risk assessment of LLM products, and tooling.
1 Introduction
Commercial LLM applications depend on models providers can shut down, making retirement a distinct dependency risk. Prior work studied migration challenges, but not ecosystem-wide timing, breakage, or code costs; this paper measures those outcomes across providers.
1 Introduction
- At shutdown, every request to a retired model fails, and providers control the service’s expiry date.The introduction also notes that reported model lifecycles have shortened from 18–24 months to 6–12 months.Source: §1 · p. 1
- Prior work examined migration difficulty and tooling, but not how many applications migrate before a retirement deadline, break, or incur migration costs.The cited studies cover prompt re-engineering, regression tooling, an end-of-life migration framework, and item-level regressions hidden by aggregate scores.Source: §1 · p. 1
- The paper contributes an event-matched migration dataset, ecosystem-scale measurement of post-shutdown migration, provider-policy comparisons, breakage taxonomy, and evidence on migration size and architecture.The architectural analysis includes abstraction layers, fallbacks, and evaluation suites.Source: §1 · p. 1
2 Background: Deprecation Policies
OpenAI, Anthropic, and Google publish different retirement-notice policies, and Google’s announcement dates are not available for measuring notice length. The study therefore compares Google migration timing only against shutdown dates.
2 Background: Deprecation Policies
- OpenAI promises at least six months’ notice for generally available models, three months for specialized variants, and as little as two weeks for previews.The provider distinguishes deprecation announcements from shutdown.Source: §2 · p. 1
- Anthropic promises at least 60 days’ notice for publicly released models, publishes “not sooner than” retirement floors, and has committed to preserving retired weights.Its published retirement floors apply to active models.Source: §2 · p. 1
- Google publishes earliest shutdown dates for GA Gemini models and at least two weeks for previews, but does not publish announcement dates.The study consequently measures Google timing relative to shutdown only; Table 1 summarizes the events used.Source: §2 · p. 1
3 Method
The study searches GitHub for migrations away from officially retired models and endpoints, matches commits to provider events, and validates a stratified sample to reweight estimates. Timing analyses classify migrations by whether they were committed before or after shutdown.
3 Method
- 5,139 commits across 4,544 repositories matched official retirement events, and 3,933 involved events already shut down for timing analyses.The wider search produced 22,555 unique commits in 17,703 non-fork repositories; commit queries combined retired identifiers or endpoints with migration vocabulary.Source: §3 · pp. 1–2
- Commit matching required a retired model or endpoint, a migration verb, and event timing, while excluding commits that migrated into the retired model and other specified false matches.The direction filter removed 33% of candidates, concentrated in Google events.Source: §3 · p. 2
- Only 62% of matched commits were judged genuine migrations by both coders; rates varied by stratum (24%–80% before shutdown, 48%–86% after, and lowest for Google), so population shares were reweighted by stratum-specific genuine-migration rates.Agreement was high on five codes (Cohen’s κ = 0.95, 0.89, 0.94, 0.94, 0.92) and moderate on pinned snapshots (κ = 0.62), reported only as a lower bound.Source: §3 · p. 2
- A migration was classified as reactive if committed after shutdown; dose–response analyses excluded recently retired events and commits outside the specified window.The window was [−365, +180] days around shutdown, and events had to be retired at least 28 days before collection.Source: §3 · p. 2
- The method also classified commit messages for breakage, silent failure, pinned snapshots, abstraction layers, evaluation, prompt or parameter changes, and cost.The event table lists selected official retirement events with announcement and shutdown dates.Source: §3 · p. 2, Table 1 · p. 2
4 Results
Across retired-model events, most migrations came after shutdown, and longer provider notice was associated with fewer post-shutdown migrations. Migration effort varied sharply by application architecture, while provider switching and prior-experience learning were limited.
4.1 RQ1: When do applications migrate?
- 82% of migrations tied to already-retired events were committed after shutdown (95% CI 79–84) after reweighting; only 9% of reactive fixes landed within a week.The median migration landed 39 days after shutdown; reactive shares stayed near 80% across repository popularity tiers after catalogue-like projects were excluded.Source: §4.1 · pp. 2–3, Figure 1 · p. 3
4.2 RQ2: Does the notice policy matter?
- 89% of Anthropic-related migrations were reactive, compared with 66% for OpenAI and 64% for Google; each e-fold increase in notice reduced post-shutdown migration odds by roughly 74–78%.The one-year Assistants API notice had 13% reactive migration, while several 60–76-day Anthropic retirements had 86–98%; Google’s movable date showed no deadline spike.Source: §4.2 · p. 3, Table 2 · p. 3, Table 3 · p. 3
4.3 RQ3: Breakage
- Human-validated migrations reported breakage more often after shutdown for Anthropic (63%) than Google (29%) or OpenAI (19%), with hard failures, silent failures, parameter incompatibility, and pinned snapshots among the recurring forms.Silent failure appeared in 8% of genuine migrations by human coding; parameter changes were mentioned in 7.5% of commits, and pinned snapshots were uncommon.Source: §4.3 · pp. 3–4
4.4 RQ4: Effort and where the model lives
- 94% of migrations edited source code, and median changed lines rose from 6 for prompt-only applications to 693 for fine-tuned applications.Only 3% of migrations used a pre-existing abstraction layer; 4.5% introduced one, while 93% renamed the identifier in place.Source: §4.4 · p. 4, Table 4 · p. 4
4.5 RQ5: Adaptation
- Only 4% of migrations actually added an abstraction layer, while 4.5% mentioned an evaluation or regression suite and 11% added an assertion or evaluation line.Although 20% of matched commit messages mentioned an abstraction layer, gateway, or fallback, file-level analysis found additions in only 4%.Source: §4.5 · p. 4
4.6 RQ6: Where do applications go, and do they learn?
- Only 8% of migrations naming a destination switched providers, and repositories hit by repeated retirements had the same 73% post-shutdown share at first and later events.Later migrations landed sooner (median 22 vs 45 days after shutdown), but the sample could rule out only large learning effects.Source: §4.6 · pp. 4–5, Figure 2 · p. 5, Figure 3 · p. 5
5 Discussion
Provider notice length is the policy lever most clearly associated with whether applications break, while builders and assessors can inspect observable practices such as identifier externalization and retirement watches. The discussion also proposes warning and CI mechanisms to surface retirements before shutdown.
For providers
- Longer notices are associated with more proactive migrations, and providers can give customers more uptime at little retention cost because only 8% of affected applications leave the provider.The discussion contrasts mostly reactive migrations after 60-day notices with mostly proactive migrations after one-year notices.Source: §5 · p. 5
For builders
- Most model swaps are hard-coded and unmeasured: 4% mention an evaluation suite and 6.5% are configuration-only.The discussion says environment-variable or registry indirection would have made the majority of these migrations a one-line change.Source: §5 · p. 5
For dependency-risk assessment
- A pre-existing abstraction layer can shrink a fix without making migration timely, and surviving a prior retirement does not predict proactive behavior at the next one.The discussion identifies notice policy, identifier externalization, and a retirement watch as observable risk indicators.Source: §5 · p. 5
For tooling
- A static check against provider deprecation tables, configured to fail CI before shutdown, could have prevented most post-shutdown commits in the corpus.The discussion also recommends surfacing deprecation in API responses and SDK logs rather than relying on email to an account that may belong to a contractor.Source: §5 · p. 5
6 Threats to Validity
The study’s estimates are constrained by incomplete migration capture, false-positive matches, query caps, right censoring, and a calendar-based definition of post-shutdown timing.
Recall
- Projects that migrate silently or never migrate are under-represented because the search observes migrations whose commit messages name the retired identifier.Abandonment without repository archiving is common, although fewer than 1% of matched repositories are marked archived.Source: §6 · p. 5
Precision
- Keyword matching admits false positives, so headline shares are reweighted by stratum-specific genuine rates.Two-coder validation found 38% of matched commits were not genuine migrations, concentrated before shutdown and in Google events.Source: §6 · p. 5
Search caps and reproducibility
- GitHub’s 1,000-result query cap and retained 200–300 results make counts lower bounds and truncate popular months.The Gemini 2.0 Flash event, which dominates the Google sample, is also the event most likely to be truncated.Source: §6 · p. 5
Right censoring
- Recent retirements have not accumulated all post-shutdown migrations, and the censoring rule mitigates but does not remove this right-censoring problem.Re-running 12 query windows reproduced results exactly below the cap; the three capped windows returned strict subsets.Source: §6 · p. 5
Construct
- Calendar-based post-shutdown timing can misclassify a project that stopped using a model before shutdown but cleaned up its code afterward.Source: §6 · p. 5
7 Related Work
Prior work studies API deprecation, LLM supply-chain risk, model migration, serving-chain opacity, and vendor dependence, but leaves gaps in ecosystem-scale evidence about commercial model retirement. This study measures client reaction to that dependency lifecycle.
API deprecation and breaking changes
- Unlike library deprecation, commercial model retirement ends service on shutdown, while notice windows are provider-controlled and can be short.Prior ecosystem studies document clients ignoring deprecation warnings [10]; this work measures reaction to the harsher retirement setting at ecosystem scale.Source: §7 · p. 6
The LLM application supply chain
- LLM supply-chain studies focus on open-weight artifacts and security risk, not what happens when a commercial API dependency is withdrawn.This study treats commercial model endpoints as lifecycle-bearing dependencies and measures downstream consequences of termination.Source: §7 · p. 6
Migrating between LLMs
Serving-chain opacity and model substitution
- Model substitution work detects silent changes in which model answers; retirement is the complementary failure mode in which the model disappears.The paper connects this distinction to its finding that some retirement-induced failures were silent, caught by exception handlers and surfaced as model output.Source: §7 · p. 6
Cost and vendor dependence
- Prior work discusses API price heterogeneity and vendor-controlled updates, while practitioner writing treats single-provider dependence as a valuation risk without measurement.This study estimates one component of that risk: forced-migration probability and lateness, and whether abstraction layers reduce migration effort.Source: §7 · p. 6
8 Conclusion
Model retirement is routine but unplanned for most applications: four in five genuine migrations happened after the model stopped answering. The paper releases its data and pipeline so providers, builders, and assessors can track this dependency.
8 Conclusion
- Four in five genuine migrations away from retired models happened after the model had stopped answering, and the share scales with how little notice the provider gave.Source: §8 · p. 6
- Retirement-induced failures were often quiet.Source: §8 · p. 6
- The authors release the data and pipeline for tracking this dependency as it evolves.Source: §8 · p. 6