PaperlayerView exact revision on arXiv

Source-linked AI summary

What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations

Xiaonan Xu, Wenjing Wu

arXiv:2608.17719v1cs.SEcs.AIcs.CL

TL;DR

Aggregate benchmark scores can obscure item-level changes during commercial LLM API migrations. This paper measures reliable item-level improvements and regressions across migrations and benchmarks, finding that both coexist even when aggregate scores improve.

  • Problem

    Aggregate benchmark reporting compresses heterogeneous item-level behavior into a net score, leaving item-level migration changes insufficiently measured.

  • Method

    The study estimates item-level pass-probability changes across migrations and calibrates them against a permutation null.

  • Results

    Across every migration–benchmark cell, reliable improvements and regressions coexist, with aggregate gains up to 7.3 percentage points accompanying up to 8.3% reliably regressed items.

  • Takeaways & Limitations

    Aggregate scores alone do not fully characterize bidirectional item-level change during model migrations.

  • Takeaways & Limitations

    Because benchmarks are screened for headroom, the reported reliable-change shares are estimates for that selected benchmark scope.

Abstract

from arXiv · show

Context: Software systems that depend on commercial large language model APIs must migrate to successor versions when vendors deprecate older models. Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure. Objective: We measure what that compression conceals. Method: On three pairwise upgrades in the GPT-5.4 to GPT-5.6 Sol product sequence, we query 900 public benchmark items (graduate-level knowledge, olympiad mathematics, instruction following) 50 times per item per model, classify each item as reliably improved, reliably regressed, practically equivalent, or inconclusive under false-discovery-rate control and a practical-significance threshold, and calibrate the results against a label-permutation null. Results: Across all nine migration-benchmark cells, reliable improvements and reliable regressions coexist. Edges with aggregate gains of up to 7.3 percentage points contain up to 8.3% reliably regressed items; edges with aggregate losses contain up to 10.7% reliably improved items. On the instruction-following benchmark, the gap between strict and loose scoring widens by 3.9 percentage points on the latest migration: a 3.9-point regression under strict scoring shrinks to 0.04 points under loose scoring. Conclusion: Migration decisions based on aggregate scores alone miss substantial bidirectional item-level change. The complete response-level archive and per-item scoring outputs are released.

migration, repeated sampling · 1. Introduction

Commercial LLM API migrations can preserve aggregate benchmark performance while changing many individual items, and stochastic outputs make single-draw comparisons unreliable. This study addresses both problems through repeated sampling and permutation-calibrated item-level analysis of three frontier API upgrades.

  • Commercial large language model (LLM) APIs have become external soft-: Vendor deprecation schedules force downstream consumers to migrate to successor models, making item-level backward compatibility relevant to migration safety.Interface-preserving updates can still break client behaviour because each migration replaces the model behind downstream calls.
  • Commercial large language model (LLM) APIs have become external soft-: Aggregate benchmark scores report only the net balance of heterogeneous item changes, so gains can conceal simultaneous improvements and regressions.The introduction illustrates this compression with a two-point net gain comprising 100 more reliably answered items and 80 fewer.
  • Commercial large language model (LLM) APIs have become external soft-: A single correct-to-incorrect flip may reflect sampling noise rather than a model change, and prior single-draw evaluation missed 42% of reliably changed items.Item-level compatibility therefore requires estimating pass-probability changes rather than relying on one response per model.
  • A longitudinal study of 18 GPT models fits ability trajectories to single-: K=50 independent trials per model–item pair extend repeated-sampling evaluation to frontier commercial API migrations, with permutation-null calibration of the classification procedure.The study holds request parameters constant except the model identifier and calibrates at matched sample size.
  • A longitudinal study of 18 GPT models fits ability trajectories to single-: 3.9 percentage points widen the strict–loose scoring gap on the latest instruction-following migration: strict scoring shows a 3.9-point regression versus 0.04 points under loose scoring.The comparison tests whether measured regressions depend on exact output compliance.
  • A longitudinal study of 18 GPT models fits ability trajectories to single-: All nine migration–benchmark cells contain reliable improvements and reliable regressions, including positive aggregate edges with up to 8.3% reliably regressed items.The coexistence of both directions occurs despite the aggregate change being positive.
  • A longitudinal study of 18 GPT models fits ability trajectories to single-: Negative aggregate edges contain up to 10.7% reliably improved items, showing that aggregate losses also conceal item-level gains.The reported item-level classifications are calibrated against a permutation null.

1. We measure item-level backward compatibility across the GPT-5.4 to GPT-

The section measures item-level backward compatibility using a permutation-calibrated classification procedure. It shows bidirectional changes across all nine migration–benchmark cells, scoring-dependent conclusions, and releases data for verification and rescoring.

  • Measurement: Item-level judgements are calibrated against a permutation null that reruns the complete classification procedure at matched sample size.This calibration supports the item-level classification procedure.
  • Item-level results: Reliable improvements and reliable regressions coexist in all nine migration–benchmark cells, including edges where the aggregate score changes.The finding applies across the full set of migration–benchmark cells, including cases with aggregate-score movement.
  • Scoring sensitivity: The latest instruction-following migration widens the gap between strict and loose scoring, causing the strict-scoring regression to shrink under loose scoring.Migration conclusions therefore depend on the scoring definition used for the instruction-following benchmark.
  • Reproducibility: The complete response-level archive and per-item scoring outputs are released, enabling verification and alternative rescoring without re-querying mutable systems.The release supports independent checking of the reported item-level results.

2. Related Work

Prior work shows that aggregate scores obscure heterogeneous item-level changes and that stochastic, evolving LLM APIs challenge classical regression testing. This study extends repeated-sampling and item-level comparison to frontier commercial API migrations while releasing archived response traces.

  • Aggregate evaluation and its limits: Aggregate scores remain useful summaries of overall capability but collapse heterogeneous item-level behaviour, conflating uniform improvement with mixtures of gains and losses.The study measures what this compression hides.
  • Model-update regression in LLM APIs: Stochastic LLM outputs invalidate deterministic regression assumptions, while prior work documents behaviour shifts across time-stamped commercial-model snapshots.Negative flips and migration frameworks address the problem from model-development and production-decision perspectives.
  • Item-level measurement and repeated sampling: A dynamic item-response study estimated probability changes across 18 GPT models, but its single ability parameter forced all item-level shifts within a comparison to share one sign.The authors note that heterogeneous directions require observed response flips, which single-draw evaluation does not support.
  • Output variability: Repeated queries can vary even at temperature zero, with documented sources including sampling, silent updates, numerical rounding, and expert routing.Community guidelines identify output non-determinism and model evolution as reproducibility threats and call for archived interaction traces.

3. Method

The study compares three GPT migration edges using 900 fixed benchmark items, repeated 50 times per model under controlled API conditions. It combines benchmark-specific scoring with aggregate accuracy changes to examine item-level migration behaviour.

  • Migration design: Three migration edges compare GPT-5.4, GPT-5.5, and GPT-5.6 Sol, varying only the API model identifier while holding request parameters constant.Reasoning effort is fixed at medium for all models.
  • Data collection: 50 independent trials per item and model produce repeated-response data for each migration comparison.The final response matrix is complete after transient API errors were retried until success.
  • Benchmark selection: Widely used candidates GPQA Diamond, MMLU-Pro, and IFEval fell below the selection threshold, with qualifying proportions between 7% and 27%.Benchmark screening used 30 items, three models, and 10 repetitions per item before the main collection.
  • Benchmark selection: 900 items span SuperGPQA knowledge, Omni-MATH hard mathematics, and IFBench instruction following, with sampled lists fixed before collection.The sample includes 500 SuperGPQA items, 100 Omni-MATH hard items, and all 300 IFBench prompts.
  • Scoring: Official IFBench verifiers provide strict and loose readings that differ only in format tolerance, with strict scoring designated primary.Unparseable replies are counted as failures because they are operationally indistinguishable from wrong answers for an API consumer.
  • Analysis: Aggregate change is defined per benchmark as Δ = mean_i(p̂_t,i) − mean_i(p̂_s,i), using each item’s observed accuracy across K trials.For knowledge and mathematics, correctness is determined by extracting and symbolically normalising the final answer.

meani(ˆps,i); the reliable-improvement share P +; the reliable-regression share

The analysis defines item-level reliable change using statistical significance and a minimum 20-percentage-point effect threshold, then calibrates aggregate change shares against a label-permutation null. This identifies improvements and regressions only when they exceed changes attributable to label noise.

  • Change-share definitions: P_chg = P_+ + P_- combines the reliable-improvement share P_+ and reliable-regression share P_- into the reliable-change share.These shares summarize item-level classifications.
  • Statistical control: Benjamini–Hochberg controls the false discovery rate at 5% within each migration–benchmark cell.The statistical procedure uses a Fisher exact test on 2 × K correct/incorrect counts.
  • Classification criteria: Items meeting both criteria are classified as reliably improved or reliably regressed according to the shift’s sign, while the remainder are inconclusive.Classification therefore requires both statistical evidence and the minimum effect size.
  • Null calibration: 1,000 label-permutation replications generate null distributions for P_+, P_-, and P_chg under zero true change at matched sample size.The reported shares are compared with the null’s 95th percentile so reliable change is claimed only beyond label noise.

For the secondary analysis on IFBench, let Rm denote the mean difference

This section defines the aggregate and item-level quantities used to analyze migration edges, including mean strict-accuracy change, classification shares, and a permutation-null threshold.

  • ∆ denotes the change in mean strict accuracy in percentage points, computed before rounding.
  • P +, P −, and P chg denote the shares of items classified as reliably improved, reliably regressed, and reliably changed.
  • Null95 is the 95th percentile of P chg under a permutation null based on 1,000 replications.
  • All shares are reported in percent.

4. Results

Results show that aggregate migration scores conceal substantial item-level bidirectional change: reliable regressions occur alongside gains, and reliable improvements occur alongside losses. The calibrated null places these changes above the noise floor, while IFBench’s strict-versus-loose scoring reveals that the latest decline is concentrated in exact constraint compliance.

  • Aggregate and item-level change: 8.3% of IFBench items regressed reliably despite a positive aggregate change.Reliable regressions also affected 6.0% of Omni-MATH items and SuperGPQA items, demonstrating bidirectional change within aggregate gains.
  • Aggregate and item-level change: 13.3% of IFBench items reliably regressed while 10.7% reliably improved on the direct 5.4→Sol edge.This edge had a −2.0 aggregate loss; SuperGPQA instead had a +3.8 gain and a 10.8% reliable-improvement share.
  • Null calibration: 0.0% was the 95th-percentile reliable-change share in every migration–benchmark cell under 1,000 label permutations with zero true change.Every reliably changed item therefore lay strictly above the calibrated noise floor; observed total churn shares of 12.0–24.0% were comparable in magnitude to prior work but used this zero baseline.
  • IFBench scoring sensitivity: +3.9 points was the increase in IFBench’s strict-versus-loose scoring gap on the 5.5→Sol edge.Sol’s strict score fell 3.9 points below GPT-5.5’s, while its loose score fell 0.04 points; the regression was concentrated in exact constraint compliance.
  • IFBench scoring sensitivity: 6.3 percentage points was the strict-versus-loose gap for GPT-5.4, compared with 5.1 for GPT-5.5 and 9.0 for Sol.The corresponding gap changes were −1.2 points on 5.4→5.5, +3.9 on 5.5→Sol, and +2.7 on 5.4→Sol.

5. Discussion · The appendix compares single-draw evaluation with the repeated-sampling · 6. Threats to Validity

Aggregate scores and single-draw evaluations both omit important item-level migration information: reliable regressions can coexist with gains, while one draw detects none of the 457 reliable item-level changes. IFBench results further show that migration risk depends on whether output-format compliance is treated as functional.

  • 5. Discussion: Reliable improvements and regressions coexist across all nine migration–benchmark cells, exceeding sampling noise under the K=50 permutation null.The null calibrates the observed shares against sampling noise alone.
  • 5. Discussion: 4.4–8.3% of items regressed on aggregate-improving cells that an aggregate-based gate would have accepted.This shows why aggregate scores are necessary but insufficient for migration decisions.
  • 5. Discussion: A 3.9-point strict-scoring regression on the 5.5-to-Sol edge shrinks to 0.04 points under loose scoring.The two readings differ only in format tolerance, locating part of the strict-scoring regression in exact format compliance.
  • 5. Discussion: Migration risk depends on output acceptance criteria: format non-compliance is functional for programmatic parsers but less costly for tolerant consumers.Both perspectives are needed when reporting compatibility costs.
  • The repeated-sampling study on open-weight models [6] asked whether its: The study extends repeated-sampling evidence to frontier commercial models at K=50 across three GPT versions.Its higher per-item trial count and permutation null separate reliable item-level changes from sampling artefacts.
  • The appendix compares single-draw evaluation with the repeated-sampling: None of the 457 reliable item-level changes across nine cells is detectable from one single-draw observation per model.A single binary observation cannot reach significance under the item-level test.
  • The appendix compares single-draw evaluation with the repeated-sampling: The single-draw versus K=50 gap provides a lower bound on information lost when migration decisions rely on one draw.Raw single-draw flips are additionally distributed over the K=50 classes.
  • The appendix compares single-draw evaluation with the repeated-sampling: Item-level regression shares can serve as reference rates for designing acceptance tests when treating model upgrades as dependency updates.The study presents these shares as an implementation reference for organisations.

Three limitations bound the scope of these measurements. The benchmark … Xiaonan Xu: Conceptualization, Methodology, Investigation, Data cura-

The measurements are bounded by benchmark selection, limited vendor and task coverage, and an inconclusive category for smaller changes. Across the studied migrations, item-level improvements and regressions coexist, with released archives enabling verification and reuse.

  • Three limitations bound the scope of these measurements. The benchmark: The benchmark screening selects informative benchmarks, so reliable-change shares are estimates rather than population rates over arbitrary workloads.The study covers one vendor’s GPT-5.4 through GPT-5.6 Sol product line and three public benchmarks spanning knowledge, mathematics, and instruction following.
  • Three limitations bound the scope of these measurements. The benchmark: Smaller genuine changes remain inconclusive, limiting which item-level effects the procedures can detect.The inconclusive category is reported as such.
  • 7. Conclusion: Every migration–benchmark cell contains both reliably improved and reliably regressed items across three upgrades, 900 items, 50 trials per item, and permutation-null calibration.The upgrades span the GPT-5.4 to GPT-5.6 Sol product sequence.
  • 7. Conclusion: 7.3 percentage points of aggregate gain can accompany 8.3% reliably regressed items.These figures summarize the reported maximum coexistence of aggregate gains and item-level regressions.
  • Data Availability: The complete response-level archive and per-item scoring outputs are released to support verification and alternative analyses.The archive includes request parameters, raw responses, rescoring outputs, model identifiers, usage metadata, and timestamps; reported statistics can be recomputed without re-querying commercial APIs.
  • Funding: The research received no specific grant from public, commercial, or not-for-profit funding agencies.
  • Declaration of Competing Interest: The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.

Appendix A. Benchmark screening … Table F.9 reports mean strict accuracy per model and the per-edge change

The appendices document benchmark screening, single-draw limitations, threshold sensitivity, token usage, response categories, difficulty stratification, and strict-accuracy results. Together, these materials describe the robustness checks and supplementary measurements accompanying the migration analysis.

  • of the eleven candidate benchmarks, 30 randomly drawn items were queried: 30 randomly drawn items from each candidate benchmark were queried K=10 times by each of the three models.
  • Appendix A. Benchmark screening: ComplexBench screening did not complete.
  • single binary observations never reaches significance under the Fisher/Benjamini–: Every item on every edge was classified as inconclusive under the single-draw Fisher/Benjamini–Hochberg procedure.
  • single binary observations never reaches significance under the Fisher/Benjamini–: 457 reliable item-level changes from the full K=50 procedure were not recovered by single-draw comparisons.
  • Appendix D. Token consumption: Table D.7 reports mean total tokens per request on each migration edge.
  • Appendix E. Response category breakdown: Refusals occur only on IFBench, while truncation and API errors do not occur in the final response matrix.
  • Appendix F. Difficulty stratification: SuperGPQA and Omni-MATH hard provide benchmark-native difficulty metadata, whereas IFBench provides no comparable difficulty metadata.
  • Table C.5 reports the reliable-improvement, reliable-regression, and reliable-: Table C.5 reports P +, P −, and P chg for four practical-significance thresholds.
Loading 2608.17719v1…