PaperlayerView exact revision on arXiv

Source-linked AI summary

UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists

Ye Chen, Weining Zhang

arXiv:2608.20918v1cs.AI

TL;DR

Organizations must decide how to migrate fine-tuned specialists when better base models arrive, but prior studies do not measure these choices across real release sequences. UpgradeBench provides a longitudinal, cost-accounted evaluation of freezing, porting, refreshing, and retraining, finding that upgrade behavior varies by task and portability depends on pretraining continuity and distance. Experimental variance and benchmark scope constrain interpretation.

  • Problem

    Prior transfer studies evaluate isolated model pairs, leaving organizations without measured evidence for choosing among freezing, porting, refreshing, and retraining across release sequences.

  • Method

    UpgradeBench evaluates the freeze/port/refresh/retrain decision space across four Qwen generations, six tasks, two scales, and training-side costs, supplemented by lineage-controlled OLMo checkpoints.

  • Results

    Upgrade gains vary by task and release, while adapter portability depends on pretraining continuity and distance rather than architecture alone.

  • Takeaways & Limitations

    Upgrade policies should be chosen per task and release using lineage, distance, and task–base coupling rather than defaulting to retraining, freezing, or copying.

  • Takeaways & Limitations

    The benchmark’s conclusions remain bounded by task and release-series scope, and some comparisons are affected by measurable seed variance.

Abstract

from arXiv · show

Organizations maintain task-specific adapters for open-weight language models, and each new base-model release forces a migration decision: retain existing specialists, port adapters, refresh from preserved behavior, or retrain. Prior transfer work evaluates isolated model pairs, without studying these choices across real model release sequences. We present UpgradeBench, a decision-driven longitudinal benchmark covering four consecutive Qwen releases, one continuation checkpoint, six tasks, and two model sizes, augmented by OLMo checkpoints with known training lineage. The benchmark disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether specialization assets transfer across versions, and what recovery resources are usable. We observe upgrade gains differ across task-scale-release episodes: some retrained baselines improve while others stay within training noise, with durability ranging from under one release interval for text-to-SQL to over fourteen months for intent classification. Direct adapter copying depends neither on architecture nor model family: on OLMo, retention drops from 0.88-0.99 at 46B-token continued pretraining to zero at 2.9T tokens; annealing and model souping introduce no extra harm, with portability decaying with continued-pretraining distance. Given preserved input data, teacher relabeling recovers target-base specialists without fresh gold annotations, though compute savings are not guaranteed. Simulating a fixed decision policy over 33 upgrade episodes yields 0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining. A lightweight CKA probe over 256 prompts predicts cross-version adapter portability (Spearman 0.74 across eight model pairs). We release per-example predictions, cost logs, split manifests, and evaluation code.

1 Introduction

UpgradeBench measures the upgrade tax created when organizations must migrate fine-tuned specialists across rapidly improving base-model releases. Across longitudinal release sequences, durability and adapter portability vary sharply by task and pretraining continuity.

  • Benchmark scope: UpgradeBench evaluates freeze, port, refresh, and retrain across four Qwen generations, six tasks, three task classes, two scales, and training-side costs.The benchmark tracks a complete fourteen-month release series rather than isolated model pairs.
  • Durability: 99–101% retention across nine hops shows intent specialists can preserve nearly all attainable upgrade gains, while text-to-SQL forfeits up to 59% on one hop.Intent classification durability exceeds fourteen months, whereas text-to-SQL advantage reaches zero within three generations.
  • Portability: 92.8% falls to 42.9% when copying across identical architectures from independent pretraining, below 60.7% zero-shot, but continued-pretraining descendants retain reference-level accuracy.The measurements identify weight continuity, not architectural shape, as the relevant condition for copying.
  • Portability: Portability decays with continued-pretraining distance: OLMo retention is near-parity at 46B tokens and floor-level at 2.9T tokens, with annealing and souping adding no further damage.The benchmark’s strategy map uses lineage and distance to distinguish viable copying from harmful transfer.

2 Related Work

Prior work proposes many cross-base transfer methods, but evaluations use self-selected model pairs and tasks. UpgradeBench instead treats release sequences, costs, and upgrade decisions as the primary evaluation frame.

  • Cross-base transfer: Synthetic distillation, parameter mappings, subspace projection, diff-vector porting, and training-time immunization each target cross-base transfer but are evaluated on isolated model pairs.The resulting comparisons across papers are not directly comparable.
  • Reuse and transfer: Task-vector arithmetic, merging, and specialist reuse assume shape-compatible or shared-lineage bases, leaving portability across changing release architectures unresolved.UpgradeBench tests these assumptions along a real model lineage.
  • Model evolution: Capability density doubles on the order of months, making upgrade decisions recurrent while per-task uplift can still be negative.The paper uses this average capability-growth trend to motivate specialization half-life measurements.
  • Costs: Cost-aware evaluations report monetary costs, whereas UpgradeBench adds training-side accounting and defines the upgrade tax as the missing depreciation term for specialist fleets.The benchmark reports GPU-minutes and energy alongside quality outcomes.
  • Benchmark gap: Existing benchmarks evaluate systems at fixed times; UpgradeBench makes release sequence the primary axis of evaluation.It is presented as the first benchmark whose central dimension is model release time.

3 Formalizing the Upgrade Tax

The paper formalizes upgrade decisions by separating specialization gain, retention, quality shortfall, durability horizons, recovery budgets, and strategy costs. This keeps upgrade evaluation multi-dimensional rather than collapsing it into a single scalar.

  • Core quantities: Specialization gain is G(b,t) = S_t(s_b,t) − S_t(b), comparing a fixed-recipe specialist with the base’s cheap-adoption score.The setup defines a lineage of release-ordered checkpoints and a common PEFT procedure.
  • Core quantities: Retention R=1 matches new-base retraining, R=0 adds nothing over adoption, R<0 is worse than no transfer, and R>1 exceeds retraining.FREEZE is represented as transferring the old specialist without retraining on the target base.
  • Upgrade tax: The upgrade tax is T^m = (1−R^m)·(S_t(s_bj,t)−S_t(b_j)), paired with the strategy’s measured cost.The paper reports the signed quality shortfall and associated migration resources rather than only normalized retention.
  • Durability: Durability uses the advantage A_t(τ) and reports both half-value and crossover horizons in interval-censored real release time.The two horizons are kept separate because they can differ by an order of magnitude on some tasks.
  • Recovery: The 95% recovery budget is the smallest labeled-example and GPU-minute budget reaching 95% of attainable reference gain above the target floor.The gain-based criterion is invariant to the floor, unlike an absolute-score threshold.
  • Decision model: The decision model maximizes v·u(Q^m)−c^m/Δ over strategies, while the full upgrade tax remains a Pareto vector of quality, compute, energy, labels, teacher queries, and validation load.Organization-specific utility weights determine how these components are combined.

4 The UpgradeBench Benchmark

UpgradeBench follows real Qwen release transitions while varying task suites, model lineage, and upgrade strategies. Its design also validates transfer across documented OLMo checkpoints and records paired evaluation and resource measurements.

  • UpgradeBench covers every major Qwen generation from February 2024 to April 2025, using instruct checkpoints and consistent training/evaluation substrates.
  • Architecture changes at two of three Qwen major-version boundaries make weight-space transfer undefined for most real upgrades, while the 2→2.5 hop is shape-compatible.
  • OLMo supplies architecturally identical checkpoints with documented data order and training stages, enabling transfer tests grounded in known continuation lineage.
  • The benchmark spans six tasks across discriminative, structured-generation, and agentic tool-use classes, with released splits and task-specific evaluation protocols.
  • For LoRA specialists, copying an adapter and adding its merged ∆W = BA are equivalent operations, so COPY also measures diff-vector porting.
  • All condition comparisons use identical test sets and paired McNemar tests, while runs log wall-clock time, VRAM, and integrated board energy.

5 Experimental Setup

The experimental setup uses a fixed QLoRA recipe, a single RTX 4090, and deliberately sparse action–edge coverage. It combines extensive training and evaluation logging with explicit limits on which baselines are measured.

  • One fixed QLoRA recipe uses NF4 double quantization, bf16 computation, LoRA r=16, α=32, and task-specific sequence lengths and batching.
  • All experiments run on a single RTX 4090 with 24 GB memory, while the software stack and peak VRAM range are specified.
  • The study inventory contains 98 QLoRA trainings and 193 evaluation cells, with 148.7 training GPU-hours and 47.8 kWh of measured board energy.
  • The coverage matrix is sparse rather than factorial: label ladders are limited to the highest-coupling task, REFRESH-D requires retained inputs, and mapping or distillation baselines are unmeasured extensions.

6 Prospectively Specified Analyses

The preregistered-style analyses test durability, transfer, recovery, and task-class hypotheses across release episodes. Results show sharply task- and lineage-dependent specialization persistence, with portability declining as continued-pretraining distance grows.

  • The analyses were fixed in runner scripts before each corresponding wave, but without an external immutable registry; the paper therefore calls them prospectively specified.
  • H0 exceeded 14 months for the discriminative task, while text-to-SQL reached H0 at 14 months and had H50 within 0–4 months.
  • OLMo copying retained 0.88 and 0.99 across approximately 46B-token continuation steps, but fresh-pretraining copies fell to R=−0.32 and −0.02.
  • Template swaps did not change Qwen copy scores, constrained decoding reduced the CLINC decline by roughly one-third, and the wave-4 collapse was not attributable to annealing or souping alone.
  • The OLMo durability prediction was refuted: a frozen Spider specialist maintained a 39–58pp advantage across nine months of releases.

7 Results

UpgradeBench shows that specialist retention, adapter portability, and durability vary substantially with task competence provenance and the target base’s training lineage. Frozen specialists often remain effective on data-bound tasks, while base-bound tasks benefit from retraining; copying works mainly across short, documented continuation paths.

  • Retention across upgrade hops: 99–101% retention on Banking77 contrasts with 41–93% retention on Spider, showing that freezing preserves data-bound skills better than base-bound SQL competence.Banking77’s retrained reference remains flat across four generations, whereas Spider’s ceiling rises with each generation.
  • Retention across upgrade hops: Measured FREEZE retention ranks D ≈A > S, refuting the pre-registered discriminative-over-structured hypothesis.The paper attributes the difference to competence provenance rather than surface task form.
  • Task–base coupling: Reference scores rise on Spider and mildly on xLAM-FC but stay flat on Banking77, glaive-FC, and FinQA after multi-seed evaluation.CLINC150 is reported as flat within training noise after its single-seed decline is overturned by three seeds.
  • Copying and continuity: Fresh-hop copying is usually worse than freezing: across nine task/scale cells, seven have R ≤0.37, with Banking77 reaching R = −0.551 at 7–8B.The two higher-retention exceptions are format-heavy tasks with shallow adapters and still trail FREEZE.
  • Copying and continuity: Short continuation preserves copying, with R=0.985 on Banking77, 1.445 on Spider, and 0.815 on xLAM, while fresh independent runs collapse toward the floor.The continuation comparisons are not uniformly non-inferior at the strictest margins, especially for xLAM.
  • Copying and continuity: Portability decays with continued-pretraining distance: OLMo retention is 0.88–0.99 after 46B tokens but reaches the floor after 2.9T tokens, with annealing and souping adding little change.The distance-matched ablation overturns the earlier attribution of collapse to the anneal-plus-soup boundary.
  • Measured durability horizons: Spider loses 77% of its initial specialization advantage within four months but reaches zero advantage only after fourteen months, whereas Banking77 remains +28.6pp ahead at fourteen months.The two horizons distinguish rapid value decay from eventual crossover.
  • Measured durability horizons: The OLMo Spider specialist shows no half-value crossing through nine months, illustrating that durability depends on the release series rather than task identity alone.In that series, zero-shot SQL floors remain low while retrained references rise only modestly.

8 The Small-Evidence Regime

At enterprise-sized evidence budgets, UpgradeBench replaces point-estimate upgrade gates with disagreement-aware equivalence testing and targeted labeling. This reduces false upgrade openings while preserving decision accuracy.

  • Operating regime: The three-zone gate uses paired-difference confidence intervals and equivalence testing to distinguish gains, losses, and unresolved cases.It also applies a rule-of-three bound when systems agree on every item.
  • Labeling strategy: Disagreement-first labeling reduces annotation complexity from O(n) to O(#disagreements) by labeling only items where systems differ.Both systems can first run over unlabeled traffic, making disagreement collection annotation-free.
  • Operating regime: At n ≤300, the point-estimate gate falsely opens 13–18% of null episodes, triggering unnecessary retrain-and-migrate cycles.The benchmark replays gate decisions on per-example records from 21 task–upgrade cells.
  • Labeling strategy: Labeling disagreement items concentrates a fixed budget on the examples carrying decision information, unlike random labeling.The comparison is summarized in the annotation-efficiency analysis.
  • Operating regime: The conservative gate cuts false opens by an order of magnitude at roughly a tenth of a percentage point of mean serving-accuracy cost.Its equivalence verdicts are reported as essentially always correct in the tested regime.

9 Discussion

UpgradeBench argues that upgrade decisions should depend on task coupling and release genealogy rather than defaulting to retraining, freezing, or adapter copying. The study’s scope remains bounded by its model, task, lineage, and cost assumptions.

  • Discussion: Always retraining wastes 17–422 GPU-minutes per specialist per generation when data-bound tasks show no measurable accuracy gain.For xLAM, the newest reference score (.8455) does not nominally exceed the one-generation-old specialist (.8480).
  • Discussion: Never revisiting specialists forfeits 42–59% of attainable gain per hop on base-bound tasks.Adapter copying is harmful across independent checkpoints, near-free after short continuation, and harmful again after long continuation.
  • Discussion: Release genealogy and continued-pretraining distance, rather than configuration compatibility, determine whether adapter copying is viable.On the measured OLMo trajectory, retention falls from 0.88–0.99 after 46B tokens to the no-adapter floor after 2.9T tokens.
  • Limitations: The benchmark is limited to QLoRA specialists capped at 8B, English tasks, a narrow lineage, and an amortized decision model excluding re-validation and compliance costs.Its continuity result is presented as strongly supported but not yet universal.
  • Limitations: REFRESH-D’s relabel variant assumes retained task inputs, which is common but not universal in enterprise settings.The benchmark measures the upgrade tax rather than proposing a new transfer method.

10 Conclusion

UpgradeBench measures upgrade costs across a fourteen-month, multi-task release lineage and concludes that specialist durability and portability depend on task coupling and pretraining continuity. It releases artifacts supporting reproducible cost- and behavior-aware upgrade decisions.

  • Conclusion: UpgradeBench measures the base-model upgrade tax across four generations, six enterprise tasks, and two model scales over fourteen months.The study runs on a single RTX 4090 and reports 98 trainings and 193 evaluation cells.
  • Conclusion: Specialist durability varies by task: the most base-bound task has a two-to-three-generation half-life at 7–8B, while a classification specialist retains a 28.6-point lead after fourteen months.Retrained reference scores rise where task–base coupling is high and remain within training noise where coupling is low.
  • Conclusion: Adapter portability follows pretraining continuity rather than architectural compatibility, failing across independent runs and retaining 0.82–1.45 between continued-pretraining checkpoints.On OLMo, retention declines from 0.88–0.99 after 46B continuation tokens to the no-adapter floor after 2.9T tokens.
  • Reproducibility: The release artifacts include adapter checkpoints, per-query predictions, bootstrap intervals, split manifests, relabeled Refresh-D corpora, and raw cost logs.The implementation uses public datasets and checkpoints, with total measured training energy of 47.8 kWh.
  • Lineage constraints: The Qwen lineage changes vocabulary size and architecture across hops, leaving weight-space transfer methods undefined on two of three major-version transitions.The vocabulary alternates between 151,936 and 152,064 entries.

E Memorization probe

The memorization probe checks whether generation comparisons are differentially affected by verbatim continuation from task inputs. Reported rates are low and approximately constant across generations.

  • Probe design: Verbatim continuation rates are 0–1% for Banking77, 0% for Spider, and 3–5% for xLAM across generations.The probe uses 200 test items per task, truncates each item to 60%, and greedily generates 24 tokens.

F Full run inventory and cost log

The study logs training, evaluation, and teacher-relabeling costs across 98 runs, while releasing per-run resource records and uncertainty estimates for evaluation results.

  • Run inventory and costs: 98 training runs consumed 148.7 training GPU-hours, equivalent to 47.8 kWh measured, plus 79.4 evaluation GPU-hours and 4.6 teacher-relabel GPU-hours.Evaluation energy was not separately metered; the power integral covers training only.
  • Run inventory and costs: The longest single run was glaive-FC/Qwen3-8B at 422.5 minutes, while FinQA/Qwen3-8B used the most energy at 1,255 Wh and peaked at 21.1 GB VRAM.The smallest run was Spider ladder-256 at 8.0 minutes and 49.5 Wh.
  • Implementation safeguards: Completion-only masking required separate prompt and completion tokenization because BPE boundary merges can silently remove the completion’s first token from the loss.When this occurred, reference self-evaluation collapsed to 5–17% despite apparently normal training convergence.
  • Uncertainty reporting: Bootstrap 95% confidence intervals for all 193 evaluation cells use 10,000 resamples and are released in ci_table.json.Representative half-widths are Banking77 ±0.9pp, Spider ±2.8pp, and xLAM ±1.6pp.
  • Uncertainty reporting: Three-seed variance for Banking77/Qwen2.5 was mean .9301 with σ = 0.12pp.The reported seed statistic provides a compact estimate of run-to-run variability for that condition.

H Dataset licenses

The benchmark uses datasets with distinct licensing conditions, including gated access for xLAM-function-calling-60k.

  • Dataset licenses: Banking77 is licensed CC-BY-4.0, Spider is CC BY-SA 4.0, and xLAM-function-calling-60k is CC-BY-4.0 with gated Hugging Face access.The license texts were checked in 2026-07 and should be re-verified before redistribution.
Loading 2608.20918v1…