What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations

Commercial large language model (LLM) APIs have become external soft-

• Vendor deprecation schedules force downstream consumers to migrate to successor models, making item-level backward compatibility relevant to migration safety.Interface-preserving updates can still break client behaviour because each migration replaces the model behind downstream calls.Source: Commercial large language model (LLM)… · p. 2
• Aggregate benchmark scores report only the net balance of heterogeneous item changes, so gains can conceal simultaneous improvements and regressions.The introduction illustrates this compression with a two-point net gain comprising 100 more reliably answered items and 80 fewer.Source: Commercial large language model (LLM)… · p. 2
• A single correct-to-incorrect flip may reflect sampling noise rather than a model change, and prior single-draw evaluation missed 42% of reliably changed items.Item-level compatibility therefore requires estimating pass-probability changes rather than relying on one response per model.Source: Commercial large language model (LLM)… · p. 2

A longitudinal study of 18 GPT models fits ability trajectories to single-

• K=50 independent trials per model–item pair extend repeated-sampling evaluation to frontier commercial API migrations, with permutation-null calibration of the classification procedure.The study holds request parameters constant except the model identifier and calibrates at matched sample size.Source: A longitudinal study of 18 GPT models… · p. 3
• 3.9 percentage points widen the strict–loose scoring gap on the latest instruction-following migration: strict scoring shows a 3.9-point regression versus 0.04 points under loose scoring.The comparison tests whether measured regressions depend on exact output compliance.Source: A longitudinal study of 18 GPT models… · p. 3
• All nine migration–benchmark cells contain reliable improvements and reliable regressions, including positive aggregate edges with up to 8.3% reliably regressed items.The coexistence of both directions occurs despite the aggregate change being positive.Source: A longitudinal study of 18 GPT models… · p. 3
• Negative aggregate edges contain up to 10.7% reliably improved items, showing that aggregate losses also conceal item-level gains.The reported item-level classifications are calibrated against a permutation null.Source: A longitudinal study of 18 GPT models… · p. 3

Measurement

• Item-level judgements are calibrated against a permutation null that reruns the complete classification procedure at matched sample size.This calibration supports the item-level classification procedure.Source: §5.6 · p. 3

Item-level results

• Reliable improvements and reliable regressions coexist in all nine migration–benchmark cells, including edges where the aggregate score changes.The finding applies across the full set of migration–benchmark cells, including cases with aggregate-score movement.Source: §5.6 · p. 3

Scoring sensitivity

• The latest instruction-following migration widens the gap between strict and loose scoring, causing the strict-scoring regression to shrink under loose scoring.Migration conclusions therefore depend on the scoring definition used for the instruction-following benchmark.Source: §5.6 · p. 4

Reproducibility

• The complete response-level archive and per-item scoring outputs are released, enabling verification and alternative rescoring without re-querying mutable systems.The release supports independent checking of the reported item-level results.Source: §5.6 · p. 4

Aggregate evaluation and its limits

• Aggregate scores remain useful summaries of overall capability but collapse heterogeneous item-level behaviour, conflating uniform improvement with mixtures of gains and losses.The study measures what this compression hides.Source: §2 · p. 4

Model-update regression in LLM APIs

• Stochastic LLM outputs invalidate deterministic regression assumptions, while prior work documents behaviour shifts across time-stamped commercial-model snapshots.Negative flips and migration frameworks address the problem from model-development and production-decision perspectives.Source: §2 · p. 4, testing assumptions: vendor-controlled… · p. 5

Item-level measurement and repeated sampling

• A dynamic item-response study estimated probability changes across 18 GPT models, but its single ability parameter forced all item-level shifts within a comparison to share one sign.The authors note that heterogeneous directions require observed response flips, which single-draw evaluation does not support.Source: testing assumptions: vendor-controlled… · p. 5

Output variability

• Repeated queries can vary even at temperature zero, with documented sources including sampling, silent updates, numerical rounding, and expert routing.Community guidelines identify output non-determinism and model evolution as reproducibility threats and call for archived interaction traces.Source: a permutation null separates true… · p. 6

Migration design

• Three migration edges compare GPT-5.4, GPT-5.5, and GPT-5.6 Sol, varying only the API model identifier while holding request parameters constant.Reasoning effort is fixed at medium for all models.Source: §3.2 · p. 7

Data collection

• 50 independent trials per item and model produce repeated-response data for each migration comparison.The final response matrix is complete after transient API errors were retried until success.Source: §3.2 · p. 7, Collection proceeded in per-model… · p. 8

Benchmark selection

• Widely used candidates GPQA Diamond, MMLU-Pro, and IFEval fell below the selection threshold, with qualifying proportions between 7% and 27%.Benchmark screening used 30 items, three models, and 10 repetitions per item before the main collection.Source: Measuring item-level change requires… · p. 7
• 900 items span SuperGPQA knowledge, Omni-MATH hard mathematics, and IFBench instruction following, with sampled lists fixed before collection.The sample includes 500 SuperGPQA items, 100 Omni-MATH hard items, and all 300 IFBench prompts.Source: Measuring item-level change requires… · p. 8

Scoring

• Official IFBench verifiers provide strict and loose readings that differ only in format tolerance, with strict scoring designated primary.Unparseable replies are counted as failures because they are operationally indistinguishable from wrong answers for an API consumer.Source: matches the reference; mathematical… · p. 9

Analysis

• Aggregate change is defined per benchmark as Δ = mean_i(p̂_t,i) − mean_i(p̂_s,i), using each item’s observed accuracy across K trials.For knowledge and mathematics, correctness is determined by extracting and symbolically normalising the final answer.Source: Scoring is decoupled from collection… · p. 8, §3.6 · p. 9

Change-share definitions

• P_chg = P_+ + P_- combines the reliable-improvement share P_+ and reliable-regression share P_- into the reliable-change share.These shares summarize item-level classifications.Source: meani(ˆps,i); the reliable-improvement… · p. 9

Statistical control

• Benjamini–Hochberg controls the false discovery rate at 5% within each migration–benchmark cell.The statistical procedure uses a Fisher exact test on 2 × K correct/incorrect counts.Source: Item-level judgements require both… · p. 9

Classification criteria

• Items meeting both criteria are classified as reliably improved or reliably regressed according to the shift’s sign, while the remainder are inconclusive.Classification therefore requires both statistical evidence and the minimum effect size.Source: benchmark cell, establishes… · p. 9

Null calibration

• 1,000 label-permutation replications generate null distributions for P_+, P_-, and P_chg under zero true change at matched sample size.The reported shares are compared with the null’s 95th percentile so reliable change is claimed only beyond label noise.Source: Observed shares are calibrated against… · p. 9

• ∆ denotes the change in mean strict accuracy in percentage points, computed before rounding.Source: Table 1 · p. 10, For the secondary analysis on IFBench,… · p. 10
• P +, P −, and P chg denote the shares of items classified as reliably improved, reliably regressed, and reliably changed.Source: Table 1 · p. 10, For the secondary analysis on IFBench,… · p. 10
• Null95 is the 95th percentile of P chg under a permutation null based on 1,000 replications.Source: For the secondary analysis on IFBench,… · p. 10
• All shares are reported in percent.Source: For the secondary analysis on IFBench,… · p. 10

Aggregate and item-level change

• 8.3% of IFBench items regressed reliably despite a positive aggregate change.Reliable regressions also affected 6.0% of Omni-MATH items and SuperGPQA items, demonstrating bidirectional change within aggregate gains.Source: Bench). All three benchmarks also show… · p. 11
• 13.3% of IFBench items reliably regressed while 10.7% reliably improved on the direct 5.4→Sol edge.This edge had a −2.0 aggregate loss; SuperGPQA instead had a +3.8 gain and a 10.8% reliable-improvement share.Source: Both declining benchmarks still… · p. 11

Null calibration

• 0.0% was the 95th-percentile reliable-change share in every migration–benchmark cell under 1,000 label permutations with zero true change.Every reliably changed item therefore lay strictly above the calibrated noise floor; observed total churn shares of 12.0–24.0% were comparable in magnitude to prior work but used this zero baseline.Source: §4.2 · p. 11, study’s sample sizes, the Fisher/BH/ε… · p. 12

IFBench scoring sensitivity

• +3.9 points was the increase in IFBench’s strict-versus-loose scoring gap on the 5.5→Sol edge.Sol’s strict score fell 3.9 points below GPT-5.5’s, while its loose score fell 0.04 points; the regression was concentrated in exact constraint compliance.Source: §4.3 · p. 12
• 6.3 percentage points was the strict-versus-loose gap for GPT-5.4, compared with 5.1 for GPT-5.5 and 9.0 for Sol.The corresponding gap changes were −1.2 points on 5.4→5.5, +3.9 on 5.5→Sol, and +2.7 on 5.4→Sol.Source: §4.3 · p. 12

5. Discussion

• Reliable improvements and regressions coexist across all nine migration–benchmark cells, exceeding sampling noise under the K=50 permutation null.The null calibrates the observed shares against sampling noise alone.Source: Across all nine migration–benchmark… · p. 12
• 4.4–8.3% of items regressed on aggregate-improving cells that an aggregate-based gate would have accepted.This shows why aggregate scores are necessary but insufficient for migration decisions.Source: Across all nine migration–benchmark… · pp. 12–13
• A 3.9-point strict-scoring regression on the 5.5-to-Sol edge shrinks to 0.04 points under loose scoring.The two readings differ only in format tolerance, locating part of the strict-scoring regression in exact format compliance.Source: Across all nine migration–benchmark… · p. 13
• Migration risk depends on output acceptance criteria: format non-compliance is functional for programmatic parsers but less costly for tolerant consumers.Both perspectives are needed when reporting compatibility costs.Source: Across all nine migration–benchmark… · p. 13

The repeated-sampling study on open-weight models [6] asked whether its

• The study extends repeated-sampling evidence to frontier commercial models at K=50 across three GPT versions.Its higher per-item trial count and permutation null separate reliable item-level changes from sampling artefacts.Source: The repeated-sampling study on… · p. 13

The appendix compares single-draw evaluation with the repeated-sampling

• None of the 457 reliable item-level changes across nine cells is detectable from one single-draw observation per model.A single binary observation cannot reach significance under the item-level test.Source: The appendix compares single-draw… · p. 13
• The single-draw versus K=50 gap provides a lower bound on information lost when migration decisions rely on one draw.Raw single-draw flips are additionally distributed over the K=50 classes.Source: The appendix compares single-draw… · p. 13
• Item-level regression shares can serve as reference rates for designing acceptance tests when treating model upgrades as dependency updates.The study presents these shares as an implementation reference for organisations.Source: The appendix compares single-draw… · p. 13

Three limitations bound the scope of these measurements. The benchmark

• The benchmark screening selects informative benchmarks, so reliable-change shares are estimates rather than population rates over arbitrary workloads.The study covers one vendor’s GPT-5.4 through GPT-5.6 Sol product line and three public benchmarks spanning knowledge, mathematics, and instruction following.Source: Three limitations bound the scope of… · p. 14
• Smaller genuine changes remain inconclusive, limiting which item-level effects the procedures can detect.The inconclusive category is reported as such.Source: Three limitations bound the scope of… · p. 14

7. Conclusion

• Every migration–benchmark cell contains both reliably improved and reliably regressed items across three upgrades, 900 items, 50 trials per item, and permutation-null calibration.The upgrades span the GPT-5.4 to GPT-5.6 Sol product sequence.Source: §7 · p. 14
• 7.3 percentage points of aggregate gain can accompany 8.3% reliably regressed items.These figures summarize the reported maximum coexistence of aggregate gains and item-level regressions.Source: §7 · p. 14

Data Availability

• The complete response-level archive and per-item scoring outputs are released to support verification and alternative analyses.The archive includes request parameters, raw responses, rescoring outputs, model identifiers, usage metadata, and timestamps; reported statistics can be recomputed without re-querying commercial APIs.Source: §7 · p. 14, The complete response-level archive… · p. 14

Funding

• The research received no specific grant from public, commercial, or not-for-profit funding agencies.Source: Funding · p. 15

Declaration of Competing Interest

• The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.Source: Declaration of Competing Interest · p. 15

of the eleven candidate benchmarks, 30 randomly drawn items were queried

• 30 randomly drawn items from each candidate benchmark were queried K=10 times by each of the three models.Source: of the eleven candidate benchmarks, 30… · p. 19

Appendix A. Benchmark screening

• ComplexBench screening did not complete.Source: of the eleven candidate benchmarks, 30… · p. 19

single binary observations never reaches significance under the Fisher/Benjamini–

• Every item on every edge was classified as inconclusive under the single-draw Fisher/Benjamini–Hochberg procedure.Source: single binary observations never… · p. 19
• 457 reliable item-level changes from the full K=50 procedure were not recovered by single-draw comparisons.Source: single binary observations never… · p. 19

Appendix D. Token consumption

• Table D.7 reports mean total tokens per request on each migration edge.Source: Appendix D · p. 21, Table D.7 · p. 23

Appendix E. Response category breakdown

• Refusals occur only on IFBench, while truncation and API errors do not occur in the final response matrix.Source: Appendix E · p. 21

Appendix F. Difficulty stratification

• SuperGPQA and Omni-MATH hard provide benchmark-native difficulty metadata, whereas IFBench provides no comparable difficulty metadata.Source: Table F.9 reports mean strict accuracy… · p. 21

Table C.5 reports the reliable-improvement, reliable-regression, and reliable-

• Table C.5 reports P +, P −, and P chg for four practical-significance thresholds.Source: Table C.5 · p. 22