PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-Horizon Scaling: How Model Capabilities Shape the Returns to Computation

Haoyu Zheng, Zhengyu Chen, Huaisheng Zhu, Ruishan Fang, Teng Xiao, Yiwei Li, Jingang Wang, Wenqiao Zhang

arXiv:2609.35236v1cs.LG

TL;DR

The paper studies how existing model capabilities relate to long-horizon growth and the distribution of gains from further computation. It models category-specific capability–time responses and derives a continuation policy based on a run’s observed progress. In replay, the policy saves roughly one-third of full-run time with reported relative score losses of 2.4% on AutoLab and 3.3% on EdgeBench.

  • Problem

    How shared model capabilities shape long-horizon growth and the distribution of gains from further computation remains less understood.

    Source: §1 · p. 1

  • Method

    The paper models category-specific logistic power laws with shared capability readouts, then conditions predicted growth on a run’s observed history to guide continuation decisions.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    Average scores rise as participation falls, and replay of the continuation policy saves roughly one-third of full-run time with relative score losses of 2.4% on AutoLab and 3.3% on EdgeBench.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    Capability-based growth can describe long-horizon progress and guide run-level decisions about the value of further computation.

    Source: §7 · p. 10

    The paper’s trajectory-based results cannot yet be reproduced from the planned public artifact because AutoLab individual-run trajectories and replay inputs remain withheld pending redaction.

    Source: Reproducibility Statement · p. 11

Abstract

from arXiv · show

Long-horizon agents improve solutions through sustained interaction, execution, and task feedback. Scaling studies relate performance to resources and capabilities, yet how existing capabilities shape returns to extended interaction remains less understood. To address this gap, we analyze AutoLab and EdgeBench, two long-horizon benchmarks. We find that starting performance and subsequent growth are associated with different capabilities: within a task category, similar early scores can precede different later gains. To formalize this finding, we model capability-time scaling with category-specific logistic power laws shared across models. Fitted to early trajectories, these curves extrapolate the observed models' category-average scores to later computation. However, rising average scores mask narrowing improvement opportunities: later gains concentrate among fewer improving models. High final scores and continued improvement also have distinct capability profiles. Predicted mean gains estimate each model's fraction of improving tasks; averaging these estimates forecasts the average share of improving models. These uneven returns motivate deciding whether a specific run should continue. We therefore derive a continuation policy to save time and compute with limited score loss. The policy conditions growth predictions on the run's observed progress and weighs immediate and delayed gains against computation costs. In replay with training and price calibration based on other models' histories, the policy saves roughly one-third of full-run time, with relative score losses of 2.4% on AutoLab individual runs and 3.3% on EdgeBench published mean curves. Our repository is available at https://github.com/Chihaya-Anon-chan/long-horizon-scaling.

1 Introduction

The paper asks how shared model capabilities shape long-horizon growth and returns to computation. It models category-specific growth, analyzes how improvement is distributed, and derives a continuation policy that saves time with limited score loss in replay.

1 Introduction

  • Shared capability combinations describe starting performance and growth separately, so similar early scores can precede different later gains.The category-specific logistic power laws are fitted to early trajectories and predict observed models’ later category-average scores.Source: §1 · pp. 1–2
  • Average scores rise as fewer models improve; declining participation concentrates gains, partly offset by more balanced gains among those still improving.Predicted mean gains estimate each model’s fraction of improving tasks, and averaging these estimates forecasts task-averaged participation.Source: §1 · p. 2
  • The continuation policy conditions growth predictions on a run’s observed history and purchases an interval when forecast immediate or delayed gains justify its cost.It reassesses after each interval.Source: §1 · p. 2
  • The analysis uses AutoLab individual-run trajectories and official EdgeBench task–model mean curves across ten categories, ten model versions, and 71 tasks.Continuation training and price calibration exclude target models’ long-horizon histories.Source: §1 · p. 2
  • In replay, the continuation policy saves roughly one-third of full-run time, with relative score losses of 2.4% on AutoLab and 3.3% on EdgeBench.Training and price calibration use other models’ histories.Source: §1 · p. 3

2 Measurement Framework and Experimental Setting

The framework measures verified-score trajectories over computation, with interval gains and positive increments defining improvement. Category comparisons use fixed task and model membership, while the study combines AutoLab and EdgeBench trajectories with distinct data sources.

2 Measurement Framework and Experimental Setting

  • Growth is measured from verified scores over normalized computation time; gains are interval increments, and improvements are positive increments.Natural termination is absorbing, and checkpoints end at the predeclared budget endpoint.Source: §2 · p. 3
  • The category panels cover 71 tasks, four AutoLab and six EdgeBench categories, and four or five models per category.AutoLab curves average available replicates; EdgeBench uses published task–model means.Source: §2 · p. 3
  • Fixed category panels keep temporal comparisons on the same tasks and models, with membership selected by trajectory availability rather than scores.Continuation replay uses all eligible histories under a held-out-model protocol and does not require every task to be shared by the category panel.Source: §2 · p. 3

3 Capability–Time Scaling of Long-Horizon Performance

Category-specific readouts of shared capability coordinates model both initial performance and growth over time. Fits to early trajectories forecast later category-average scores across the observed models, with five-PC fitting improving on the reported alternatives.

3.1 Shared Capabilities, Category-Specific Growth

  • Shared five-PC capability coordinates feed category-specific readouts whose weights change over time to represent initial performance and growth.The same model coordinates serve all categories; PC1 is a broad positive direction, while PC2 contrasts agent-execution with knowledge and reasoning measurements.Source: §3.1 · pp. 3–4

3.2 Forecasting Later Category Performance

  • Mean category R2 is 0.844 across 90 later model–checkpoint means, with category RMSE ranging from 0.57 to 4.08 points.Five categories exceed R2 = 0.95.Source: Figure 2 · p. 4, §3.2 · p. 5

3.1 Shared Capabilities, Category-Specific Growth

  • The response predicts category means with a logistic power law, separating the first-checkpoint score logit from the modeled logit change to the endpoint.Separate shared readouts allow similar starting performance to coexist with different subsequent growth.Source: §3.1 · p. 4
  • A matched comparison with all 29 external inputs favors the logistic over the affine log-time response in eight of ten categories.The growth response uses normalized logarithmic time, which measures proportional compute increases from the first checkpoint to the endpoint.Source: §3.1 · pp. 3–4

3.2 Forecasting Later Category Performance

  • Figure 3 shows substantial early gains followed by flatter observed trajectories in System Optimization, and diverging later scores for similarly starting models in Formal.Opus 4.8 and GPT 5.5 start at similar scores in Formal.Source: Figure 3 · p. 5, §3.2 · p. 5
  • Five PCs attain RMSE 2.31 on later points, improving on independent logistic curves in seven categories.The reported alternatives have mean RMSE 3.10 for retaining the last early score, 2.52 for independent logistic curves, and 2.66 for regularized model indicators.Source: §3.2 · p. 5

4 Shrinking Improvement Opportunities and Concentrated Gains

Average scores rise even as later improvement reaches fewer models and tasks. Capability-dependent growth helps estimate improvement frequency and reveals that high attainment and continued progress have distinct capability profiles.

4.1 Later Gains Reach Fewer Models

  • Participation fell in every category, by 35.2 percentage points on average; analysis of complete task–model curves found a 33.5-point average decline.Category-mean scores nevertheless rose at every recorded interval, and the participation decline persisted when gains below 0.1 points were excluded.Source: §4.1 · pp. 5–6
  • Fewer improving models drove declining gain coverage, while more balanced positive gains partly offset the decline in seven categories.Participation measures how widely improvement occurs; U measures how evenly positive gains are distributed among improving models, and their product is effective gain coverage.Source: §4.1 · p. 5

4.2 From Mean Gains to Improvement Frequency

  • Predicted mean gains estimate the fraction of tasks each model improves on, and grouped forecasts track observed improvement frequencies.In the fitted low-gain regime, halving predicted mean increments leaves about 30% of predicted improvement frequency.Source: §4.2 · p. 6

4.3 Capability Profiles of Long-Horizon Progress

  • Across 29 benchmarks, average signed rank correlations are more positive for final scores than for terminal improvement frequency.In Systems & SE, repository engineering and execution measurements track final scores more closely, while scientific reasoning tracks continued improvement more closely.Source: §4.3 · p. 7
  • Optimization’s eight-benchmark profile spans repository engineering, execution and tool use, and information seeking, aligning with continued improvement in executable solvers.Seven benchmarks retain positive associations across nine time-window and improvement-threshold choices; six also track continued progress in Knowledge.Source: §4.3 · p. 7
  • In AutoLab System Optimization, several benchmarks associate with both fewer terminal improvements and earlier gain realization.By 60% budget, Opus 4.7, Gemini 3.1 Pro, and DeepSeek V4 Pro had realized 91–100% of their observed 20–100% gains, versus 71.5% for GLM 5.2.Source: §4.3 · p. 7

5 From Scaling to Adaptive Continuation

The continuation policy conditions category-level growth on a run’s observed progress, then weighs immediate and remaining-budget gains against computation costs. In replay, it improves cost–loss tradeoffs and saves roughly one-third of full-run time at limited score loss.

5.1 Conditioning Growth on the Current Run

  • At each checkpoint, a bounded curve predicts next-interval and remaining-budget gains using reference progress and the run’s attained score, recent improvement, and stagnation.The response is conditioned on the observed prefix; its intensity and shape adjust gain magnitude and dependence on reference progress.Source: §5.1 · p. 7

5.2 Purchase One Interval and Reassess

  • The policy buys another interval when either forecast’s net value is positive, then reassesses; the remaining-budget forecast can justify continuing when the next interval alone cannot.Source histories select a nonnegative price or run-to-end option, fixed for the target model.Source: §5.2 · p. 8

5.3 Cost–Performance Tradeoffs

  • The replay evaluates 492 AutoLab individual seed runs across seven target models and 252 published EdgeBench mean curves across five.Target models’ long-horizon histories were excluded from training.Source: §5.3 · p. 8, §5.1 · p. 8
  • Conditioned continuation had the lowest mean loss over the evaluated cost ranges in both suites.Relative to fixed-budget stopping, mean loss fell from 14.38 to 8.83 points on AutoLab and from 2.44 to 1.77 on EdgeBench.Source: §5.3 · p. 8
  • 32.8% of full-run time was saved on AutoLab at 2.4% relative score loss, and 30.9% on EdgeBench at 3.3% relative score loss.Prices were calibrated on other models’ runs to meet a target mean fraction of declared budget, then fixed for each target model.Source: §5.3 · p. 8

6 Related Work

Prior work studies capability-based scaling, long-horizon performance across computation budgets, and adaptive stopping. This paper combines capability-dependent growth with gain concentration and a continuation policy across tasks and models.

Capability-based scaling

  • Capability-based scaling uses latent capabilities to organize benchmark performance, while shared performance structure can also support efficient score estimation.This paper uses external capability coordinates for both starting performance and within-task temporal growth, with separate readouts for each task category.Source: §6 · p. 9

Long-horizon performance and compute returns

  • Long-horizon evaluations compare agent performance across computation budgets, while test-time scaling studies repeated sampling, longer reasoning, and difficulty-dependent allocation.The paper studies capability-dependent growth, gain concentration, and continuation across tasks and models.Source: §6 · pp. 9–10

Adaptive stopping and computation value

  • Adaptive-stopping work includes rational metareasoning and systems that predict unsuccessful episodes or use hidden-state probes.This policy prices next-interval and remaining-budget gains from a bounded, prefix-conditioned response to capability-dependent progress.Source: §6 · p. 10

7 Discussion and Conclusion

Capabilities organize both the level and timing of long-horizon progress, while average scores can rise as fewer models continue improving. Conditioning growth on observed run progress improves time–score tradeoffs in replay, with broader model coverage and individual EdgeBench trajectories identified as extensions.

7 Discussion and Conclusion

  • Average scores rise across ten categories while participation falls; shrinking participation concentrates gains, and capability-dependent growth helps explain later improvement frequency.The paper concludes that evaluations should report performance, gain timing, and breadth of improvement.Source: §7 · p. 10
  • Conditioned category-level growth forecasts immediate and delayed gains for a specific run, improving replay tradeoffs on AutoLab individual runs and EdgeBench published mean curves.Broader model coverage and individual EdgeBench trajectories would extend the evaluation.Source: §7 · p. 10

Reproducibility Statement

The logistic response better predicts later category means than its affine counterpart, while the shared external capability representation uses five PCs. The planned public artifact cannot reproduce trajectory-based results because AutoLab trajectories and continuation-replay inputs remain withheld.

Reproducibility Statement

  • Trajectory-based results cannot yet be reproduced from the planned public artifact because AutoLab individual-run trajectories and continuation-replay inputs remain withheld pending redaction.The artifact is planned to include external inputs, code, and an index of already-public trajectories.Source: Reproducibility Statement · p. 11

A.1 External Measurements and the Shared PCs

  • The shared capability representation uses 29 external measurements and five unwhitened PCs, with missing target entries completed before long-horizon response fitting.Family weighting limits the total squared weight of related benchmark variants; covariance shrinkage is selected using reference-model validation splits.Source: §A.1 · p. 14
  • The five-PC choice is informed by compactness and later-checkpoint errors, while PC1 is a broad positive direction and later PCs express relative benchmark patterns.Dimension and response fitting use early checkpoints; PC signs are conventional.Source: §A.1 · p. 14, Table 2 · p. 14

A.2 Response on the Score Scale

  • Equation 16 translates the temporal coefficient from category trajectories into score gains, whose magnitude diminishes near saturation.Figure 6 links external loadings, category readouts, and observed growth in actual time; for Formal, PC2 has opposite signs in early-performance and temporal readouts.Source: §A.2 · pp. 14–15
  • Mean category RMSE falls from 2.83 to 2.50 points (11.4%) with the logistic response, which improves eight categories.The comparison holds the 29-dimensional external input space fixed and predicts the same 90 later means.Source: §A.2 · p. 15, Table 3 · p. 15

A.3 Category Response Estimation and Metrics

Category responses are fitted on fixed task–model panels using early checkpoints, then evaluated on later scores. Participation in improvement declines across categories, even as coverage can be partly sustained by more balanced gains among improving models.

A.3 Category Response Estimation and Metrics

  • Category responses use fixed task–model panels and early checkpoints, with later scores used for evaluation; the same tasks and models recur at every checkpoint within a category.The response is selected using early-point fits, with ties favoring the restricted response and then stronger regularization.Source: §A.3 · pp. 15–16, Table 4 · p. 15

A.4 Temporal Controls without External Scores

  • Temporal controls compare persistence, independent time curves, and model-identity indicators against the five-PC response under matched panels and evaluation points.The coefficients are regularized associations, and later-score prediction is evaluated with the controls in Table 5.Source: §A.4 · p. 16, Table 5 · p. 17

B.1 The Coverage Identity and Its Change over Time

  • The paired decomposition attributes −23.95 percentage points of coverage change to participation and +2.92 to balance.It uses the 62 tasks with positive gains in both intervals; the macro participation decline has a 95% bootstrap interval of 29.5–40.8 percentage points.Source: §B.1 · p. 17
  • 35.2 percentage points is the average first-to-last participation decline across primary panels; expanded panels show a 33.5-point decline.Participation declines in every category under both panel definitions; expanded panels include 80 tasks and 402 task–model pairs, versus 71 tasks and 329 pairs in the primary panels.Source: §B.1 · p. 17

B.2 Opportunity Response and Later Calibration

The opportunity response uses predicted mean gain to estimate the fraction of improving tasks. Its later calibration compares those predictions with observed improvement fractions across bins and model–category–interval observations.

B.2 Opportunity Response and Later Calibration

  • The opportunity model uses logistic regression on log predicted mean gain, with a category intercept and shared slope fitted to observed fractions of improving tasks.The standardized slope is selected using the last early interval and is positive in the selected fit; its bootstrap 95% interval is 1.54–2.05.Source: §B.2 · pp. 17–18
  • Figure 7b plots later model–category–interval fractions and calibration bins, with bootstrap intervals retaining sampled tasks’ models and intervals.The five populated bins contain 51, 22, 9, 5, and 3 fractions, respectively.Source: §B.2 · p. 18
  • The category-macro Brier score is 0.2071 for predicted gain and 0.2187 for time on 90 later observations.Both matched responses use category intercepts and a shared slope; lower Brier scores indicate lower squared probability error.Source: §B.2 · p. 18

B.3 Task-Grounded Interpretation of Progress

External-benchmark profiles differ between final-score attainment and terminal improvement, while participation in improvement declines across categories. Gain timing also varies across task profiles, distinguishing retrospective trajectory interpretation from continuation decisions based on a visible prefix.

B.3 Task-Grounded Interpretation of Progress

  • The task-signature analysis relates 29 external measurements to trajectories on a fixed 71-task panel, with correlations computed across four or five models per category.The task counts are not additional independent model samples.Source: §B.3 · p. 18
  • Final scores have a mean association of 0.454 across external benchmarks, versus 0.186 for terminal improvement frequency, a difference of 0.268.The difference is positive in seven categories, although Games, Scientific & ML, and Systems & SE have more positive terminal-frequency associations.Source: §B.3 · p. 20, Table 8 · p. 20
  • The aggregate attainment-versus-improvement contrast remains positive across model-membership and measurement sensitivity checks.Reported differences range from 0.089–0.355 when removing models and 0.268–0.432 across window–threshold combinations.Source: §B.3 · p. 20
  • Systems & SE final-score correlations are 0.90 for six benchmarks, while the first three have terminal-frequency correlations of 0.30.DeepSWE remains at 0.80 for terminal frequency, illustrating distinct associations for attainment and continued improvement.Source: §B.3 · pp. 20–21
  • In Formal, AA-LCR correlations are 0.80, 0.87, and 0.67 at improvement thresholds of 0, 0.1, and 0.5 points.At the >0.5-point threshold, AIME, SciCode, and τ 3-Banking each correlate at 0.97; the thresholds distinguish increment sizes, not time stages or proof difficulty.Source: §B.3 · p. 21
  • Across the remaining category profiles, associations vary by task and condition, including Puzzle & Challenge’s positive initial-performance and negative terminal-frequency correlations.SciCode, GPQA, and CritPt each correlate 0.80 with initial performance and −0.95 with terminal frequency; Scientific & ML profiles also depend on increment size and window.Source: §B.3 · p. 22
  • 35.2 percentage points: macro participation declines from the first to last interval, while effective gain coverage declines 23.6 points.For the last two and three intervals, participation declines 31.7 and 27.9 points, and coverage declines 20.5 and 18.2 points, respectively.Source: §B.3 · p. 22

B.4 Gain Realization over the Observed Trajectory

  • In System Optimization, six benchmarks associate negatively with terminal improvement frequency and gain timing, while first-score associations vary by benchmark.The fraction of observed 20–100% gain realized by 60% budget ranges from 71.5% to 100%; IMO-AnswerBench has the opposite timing association (1.00).Source: §B.4 · p. 23, Table 12 · p. 23

B.5 Joint Readouts and Observed Benchmark Profiles

The joint readouts use shared capability coordinates to predict growth, while empirical benchmark profiles describe associations with observed task demands and gain timing. The continuation model uses a monotone decision reference and source-only training to estimate next-checkpoint and endpoint gains from run progress.

B.5 Joint Readouts and Observed Benchmark Profiles

  • Joint readouts predict growth from the five principal components, while empirical benchmark profiles describe task demands and gain timing through associations with observed outcomes.The benchmark profiles are descriptive associations, not independent benchmark effects.Source: §B.5 · pp. 23–24

C.1 The Monotone Decision Reference

  • The monotone decision reference shares the external coordinates and log-time form of the category response, but uses a positive slope and an analysis-specific first checkpoint.It is refitted on source-model histories; AutoLab and EdgeBench use different stated decision checkpoints.Source: §C.1 · p. 24, §C.2 · p. 24

C.2 Source-Only Estimation

  • The continuation response uses seven observed-prefix features and jointly fits normalized next-checkpoint and endpoint gains, including histories with zero gain.The features include budget progress, current and initial scores, recent gains, observed-prefix improvement rate, and trailing stagnation; both targets are zero when no score space remains.Source: §C.2 · p. 24
  • Training, reference construction, model selection, and price calibration exclude the target model’s long-horizon histories.The design combines standardized state features and task indicators; source-only selection compares predicted and realized decision signs, and active full-score states have zero normalized targets.Source: §C.2 · pp. 24–25

C.3 Replay Metrics and Operating Points

This section defines replay populations and cost–loss comparisons, qualifies sequential continuation under prediction error, and documents the external capability representation and AI assistance.

C.3 Replay Metrics and Operating Points

  • Replay includes 492 eligible AutoLab runs and 252 EdgeBench task–model mean curves, with costs measured against full-run hours.AutoLab decisions use declared-budget checkpoints; EdgeBench uses two-hour checkpoints, and missing earlier scores are not filled from later observations.Source: §C.3 · p. 25
  • Table 14 summarizes matched frontier tradeoffs, while Table 15 evaluates source-calibrated operating points frozen for each target.The operating-point selection procedure uses source predictions and budget constraints; run-to-end is selected for AutoLab at the 62.5% and 75% source constraints.Source: §C.3 · pp. 25–26, Table 14 · p. 26, Table 15 · p. 26

C.4 Sequential Reassessment with Imperfect Gain Estimates

  • The sequential-reassessment bound charges prediction error only when it changes a decision, along visited checkpoints; fitted curves need not equal true conditional means.The error term includes finite-data fitting error, reference mismatch, and information lost by compressing the prefix into seven features.Source: §C.4 · p. 27

D External Capability Representation

AI Use Statement

  • AI tools assisted with research and mathematical development, experimental implementation and analysis, and language polishing; the authors checked numerical results against recorded outputs.The authors state that they take responsibility for the work.Source: AI Use Statement · p. 30

Related papers

Loading 2609.35236v1…