PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Scaling Reinforcement Learning Really Require More Training?

Bangji Yang, Jiajun Fan, Hongba Ma, Ruihan Guo, Ge Liu

arXiv:2610.01133v1cs.LGcs.AI

TL;DR

The paper asks whether a completed RL history can yield stronger policies than its measured checkpoints. SURGE recombines updates from two checkpoints using an anchor-defined spectral block and weight-only calibration. Across mathematical reasoning and coding, it reports policies that exceed measured native checkpoints, identifying stored RL history as a reusable scaling resource.

  • Problem

    The paper asks whether stored updates from a completed RL run can support a higher accuracy ceiling than native checkpoint selection alone.

    Source: §1 · pp. 1–2

  • Method

    SURGE combines two checkpoints’ updates relative to their shared initialization, preserving an anchor-defined component and admitting the donor’s complementary component.

    Source: §2.3 · p. 4

  • Results

    54.17% vs. 50.83% on DeepSeek AIME24 and 83.7% vs. 82.8% on OLMo HumanEval+ exceed the best measured native accuracies.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Stored RL history can serve as a reusable policy-space scaling resource, with deployment remaining a single policy.

    Source: §5 · p. 11

    On HumanEval+, the neighboring FC = 0.8 reaches 83.2% versus 83.7% at FC = 0.7, and the small difference does not establish a statistically resolved optimum.

    Source: §3.4 · p. 8

Abstract

from arXiv · show

Scaling reasoning typically spends more compute on reinforcement learning (RL) or on inference. We show that a completed RL training history can yield policies stronger than the checkpoints visited by its optimizer. We call this policy-space scaling: expanding the deployable policy set accessible from a fixed RL history, without extending training or increasing per-query inference computation. We instantiate it with SURGE (Scaling Up RL Gradient-free via Eigenspace fusion). SURGE combines two checkpoints from the same RL run: a high-accuracy anchor and a competitive donor that generates shorter responses. It expresses both checkpoints as changes from their shared initialization, then spectrally decomposes the anchor's update to retain its dominant component and incorporate the donor's complementary component. With a fixed target for how much of the anchor update to retain, SURGE determines the block size from the weights without testing candidate policies. We evaluate two 1.5B mathematical-reasoning histories, DeepSeek and Nemotron, and one 7B coding history, OLMo. SURGE improves benchmark-average accuracy over both input checkpoints while using fewer reasoning tokens than the anchor. It reaches 54.17% on DeepSeek AIME24 against a measured native maximum of 50.83%, and 83.7% on OLMo HumanEval+ against 82.8%. These gains exceed the observed training curves. Geometric controls support the importance of RL-update structure beyond weight displacement or token reduction alone. Each constructed model runs as a single policy. Our findings identify stored RL history as a reusable scaling resource: the capability available from a training run need not end at its best checkpoint.

1 Introduction

The paper asks whether a completed RL history can support stronger policies than the optimizer’s measured checkpoints. It introduces policy-space scaling and SURGE, then reports gains across mathematical reasoning and coding.

Results

  • 54.17% vs. 50.83% on DeepSeek AIME24 and 83.7% vs. 82.8% on OLMo HumanEval+ exceed the best measured native accuracies.Across nine math benchmarks, average accuracy improves over the DeepSeek and Nemotron anchors while using fewer tokens on every benchmark.Source: Figure 1 · p. 2, §1 · p. 2

Motivation

  • The optimizer’s measured checkpoints need not exhaust a run’s capability because recombined update components can yield policies the optimizer never visited.The paper frames the question around whether stored history can raise the accuracy ceiling beyond native checkpoint selection.Source: §1 · pp. 1–2

Contribution

  • Policy-space scaling expands the policies deployable from a completed RL history without further training or a larger inference budget.SURGE expresses two checkpoints as updates from shared initialization and combines the anchor’s protected component with the donor’s complementary component.Source: §1 · p. 2

2 Methodology

SURGE constructs candidate policies by recombining updates from a fixed RL history. Its anchor-defined spectral block protects part of one update, while a weight-only retained-energy rule sets the block size.

2.1 An RL trajectory as a resource for scaling

  • The fixed-history comparison tests whether a constructed policy exceeds the best measured native checkpoint, without claiming gains for every pair or a fitted scaling law.The anchor is accuracy-strong and the donor is competitive with shorter responses; their roles need not follow temporal order or per-benchmark ranking.Source: §2.1 · p. 3

2.2 Delta-first singular decomposition

  • SURGE subtracts shared initialization before decomposing the anchor update, so its spectral coordinates describe accumulated RL changes rather than inherited absolute-weight structure.The two checkpoints share an architecture and RL initialization, and exchanging anchor and donor generally changes the output.Source: §2.2 · pp. 3–4

2.3 Protected and complementary update components

  • The anchor defines the protected update block and SURGE admits the donor’s complementary component, with exact parameter-component identities that do not guarantee behavior preservation.The protected block is a two-sided matrix block, including interactions among selected left and right directions.Source: §2.3 · p. 4

2.4 Calibrating fusion control by retained update energy

  • Fusion control sets protected-block size, and a retained-energy target selects it from cumulative singular values without an accuracy sweep.The retained energy is the fraction of anchor-update squared Frobenius norm preserved; the target is empirical, not a reward-optimality guarantee.Source: §2.4 · pp. 4–5

2.5 Why an optimization path need not exhaust capability

  • The method explores recombinations beyond the optimizer’s visited checkpoints, and implementation produces one policy while accounting separately for offline construction and evaluation.A shorter-response donor may help when its useful updates complement the anchor; the ablations test whether structured recombination matters beyond token reduction.Source: §2.5 · p. 5

3 Experiments

Across two 1.5B math histories and a 7B coding history, SURGE constructs policies that improve accuracy over their anchors with fewer reasoning tokens. The experiments test whether this gain extends beyond native checkpoints and which update structures matter.

3 Experiments

  • The evaluation spans three fixed RL histories, with full-trajectory comparisons covering 19 native checkpoints each for DeepSeek AIME24 and OLMo HumanEval+.The main tables evaluate one pair from each history.Source: §3 · p. 5

3.1 Experimental setup

  • Math evaluation averages correctness over four responses per problem across nine benchmarks, using fixed decoding; the tested anchor–donor pairs keep history, architecture, and protocol fixed.The math runs follow JustRL’s single-stage GRPO recipe.Source: §3.1 · pp. 5–6

3.2 Scaling a fixed mathematical-reasoning history

  • DeepSeek Avg. accuracy rises from 53.26% to 55.35% (+2.08 points), and Nemotron’s from 63.29% to 65.08% (+1.78 points).Both gains use fewer reasoning tokens than their anchors.Source: Table 1 · p. 6, §3.2 · pp. 6–7
  • Across all three histories in Figure 3, spectrally calibrated policies exceed both sources on the displayed benchmark, using fewer tokens than the anchor under the same response budget.For Nemotron, AIME24 reaches 70.00%, above the anchor’s 68.33% and donor’s 60.00%, with 12.5% fewer tokens than the anchor.Source: §3.2 · p. 7

3.3 Cross-scale and cross-domain validation: 7B code

  • 83.7% HumanEval+ accuracy is 1.7 points above the OLMo anchor and 0.9 above the best native checkpoint, while reasoning tokens fall 10.3%.A selected HumanEval+ example has direct set equality in three of SURGE’s four responses, versus one for the donor and none for the anchor; it does not estimate prevalence.Source: §3.3 · p. 7, Table 2 · p. 8, Table 3 · p. 8

3.4 A shared spectral coordinate across trajectories

  • A fixed retained-energy target selects strong policies across three trajectories despite differing rank fractions, without an accuracy sweep for new pairs.On HumanEval+, the neighboring FC = 0.8 scores 83.2%, versus 83.7% at the calibrated FC = 0.7; the difference does not establish a statistically resolved optimum.Source: §3.4 · p. 8

3.5 Ablations: which update geometry matters?

  • Geometric controls favor structured RL updates: delta-first SVD and donor addition outperform absolute-weight, scalar, random-basis, and last-direction alternatives.The controls include lower-token alternatives that lose accuracy; across repeated math evaluations, accuracy and reasoning tokens have a weak negative association (r = −0.181), which does not establish causality.Source: §3.5 · pp. 9–10
  • SURGE spends offline construction compute to reuse a completed RL history, then deploys one policy; evaluating the broader family is optional and separately costly.Once the checkpoint pair and retention target are fixed, the anchor spectrum determines the deployed policy.Source: §3.5 · p. 10

4 Related Work

Prior work improves reasoning through training or inference changes and develops methods for composing or forecasting model updates. SURGE instead asks what policies can be constructed from one completed RL history.

Scaling reasoning with verifiable rewards

  • Training-time approaches include prolonged optimization and fixed RL recipes, while inference-time budget forcing allocates reasoning compute; SURGE complements them by constructing a policy from stored checkpoints.The related-work discussion also describes objective changes aimed at token efficiency.Source: §4 · p. 10

Model merging and update composition

  • Model soups, task arithmetic, and spectral merging compose model updates, while ResMerge combines RL experts; SURGE focuses on capability beyond one measured RL trajectory.Its experiments examine policy-space scaling across domains and geometric controls.Source: §4 · p. 10

Reusing RL training trajectories

  • Trajectory-reuse methods forecast policies from observed RL history, whereas SURGE recombines observed update components and calibrates them from weights without fitting a predictor or specifying a future step.It is presented as complementary to extrapolative weight averaging for code RL.Source: §4 · pp. 10–11

5 Conclusion

The conclusion argues that a fixed RL history can yield deployable policies stronger than its best measured checkpoint. Policy-space scaling reuses already-computed updates while keeping deployment to a single policy.

5 Conclusion

  • On the evaluated DeepSeek and OLMo trajectories, constructed policies exceed every measured native checkpoint.The conclusion places these results across 1.5B math and 7B code histories.Source: §5 · p. 11
  • One weight-only calibration procedure constructs policies that improve on their sources across the evaluated 1.5B math and 7B code histories.The conclusion states construction and evaluation costs are explicit, while deployment remains a single policy.Source: §5 · p. 11
  • Policy-space scaling treats stored RL history as a reusable policy space alongside new optimization and additional inference.The conclusion says capability may reside in policies the optimizer never visited.Source: §5 · p. 11

A Training, evaluation, and fusion details

The appendix specifies the evaluation settings and SURGE’s weight-only calibration and fusion procedure. It also states a conditional smooth-objective criterion for improvement and clarifies that it does not guarantee sampled benchmark gains.

A Training, evaluation, and fusion details

  • Main-table results use four responses per question, with math settings recorded in Table 5 and code settings in Appendix H.The qualitative examples are per-problem diagnostics.Source: A Training, evaluation, and fusion… · p. 15, Table 5 · p. 15
  • SURGE selects a fusion setting by matching anchor-update retained energy to a target, then forms the fused policy from an anchor-protected block and donor complement.The calibration uses cached singular values and chooses the setting without policy evaluation; non-target parameters are copied from the anchor.Source: A Training, evaluation, and fusion… · p. 15
  • A smooth-objective bound gives a sufficient condition for fusion to improve the anchor, but not a guarantee for sampled benchmark accuracy or discontinuous decoding.SURGE does not measure the gradient or assume its projector enforces the condition; dense economy SVD costs on the order of mnr operations for an m × n matrix.Source: A Training, evaluation, and fusion… · p. 16
  • For a fixed projector, fusion is the closest donor update whose protected component matches the anchor.The proof characterizes the unique update through the anchor and complement projectors.Source: A Training, evaluation, and fusion… · p. 16

B Additional ablation results

The ablations compare delta-first spectral fusion with alternative constructions and controls. They find higher accuracy for delta-first SVD across nine benchmarks, while the controls distinguish update geometry from displacement and token reduction alone.

B Additional ablation results

  • The ablations compare decomposition reference, donor complement, displacement direction, protected block, and checkpoint averaging using DeepSeek sources and the same target matrices.Projector-based variants use FC = 0.8; checkpoint averages uniformly combine all parameters over specified windows.Source: B Additional ablation results · p. 17
  • The expanded AIME24 controls report normalized Frobenius distance over body linear matrices and response-limit truncation rate, while distances remain nonnegative even for negative α.The table caption marks a dash as an unreported distance.Source: Table 6 · p. 17, B Additional ablation results · p. 17
  • Delta-first SVD gives higher accuracy on all nine benchmarks than absolute-weight SVD, though it does not uniformly use fewer reasoning tokens.The comparison changes only the SVD input, from Wa − W0 to Wa, while holding the fusion rule fixed.Source: Table 7 · p. 17, B Additional ablation results · p. 18
  • Random-subspace controls independently sample left and right orthonormal bases for each target matrix, while a trailing-subspace control uses the last k singular vectors.Both controls use the same block size as SURGE; the displacement ratio α = 0.71 approximately matches the measured ratio 0.713.Source: B Additional ablation results · p. 18, Table 8 · p. 18

C Evaluation variability

Evaluation variability is measured across four response-sampling rounds, not across independent RL runs. The within-cell analysis reports a negative pooled accuracy–token association, with benchmark-level variation and explicit limits on interpretation.

C Evaluation variability

  • Four round-level scores estimate four-response accuracy, and their variability describes evaluation sampling rather than independent RL training runs.The paper reports descriptive differences and does not claim per-benchmark statistical significance.Source: C Evaluation variability · p. 19, Table 9 · p. 19

C.1 Within-cell accuracy–token correlation

  • The observed pooled within-cell accuracy–token correlation is −0.18067, with a two-sided permutation estimate p = 0.02063.The analysis uses four rounds per cell, standardizes within cells, and reports a Monte Carlo standard error of approximately 0.00045.Source: §C.1 · p. 19, Table 10 · p. 20
  • Eight of nine pooled benchmark correlations are negative, but the association is not a universal within-benchmark rule and does not establish that shorter generations cause higher accuracy.Olympiad is the exception; local signs vary by family, and the table reports correlations by benchmark.Source: §C.1 · p. 20, Table 11 · p. 20

D Controller measurements

  • Controller measurements include AIME24 sweeps with native references at FC = 0 and 1, while Figure 5 shows raw FC coordinates with energy-calibrated settings marked by stars.The figure omits native reference positions and uses four evaluation rounds for all nine interior settings per family; HumanEval+ sensitivity measurements are also reported.Source: Table 12 · p. 21, Figure 5 · p. 21, Table 13 · p. 21

E Qualitative reasoning examples

  • For the selected rounding example, the decisive step is completing the final k = 9 case rather than stopping at a smaller feasible candidate, with feasibility requiring m ≤ 18.The cases were selected to show specific success and failure patterns, not to estimate prevalence.Source: E Qualitative reasoning examples · pp. 22–23

E.1 Coding examples: specification fidelity

The coding examples illustrate specification fidelity, while the spectral diagnostics describe how a shared energy target selects retention settings across models. The procedure uses weight-only calibration rather than accuracy sweeps.

E.1 Coding examples: specification fidelity

  • Table 18’s counting-lists example distinguishes the requested type predicate from a shortcut suggested by homogeneous examples.The coding examples were selected to expose failure modes, not to estimate how often fusion fixes or introduces errors across the benchmark.Source: Table 18 · p. 24, §E.1 · p. 24

F Spectral calibration of RL updates

  • The spectral diagnostics use anchor deltas and target matrices, with Equation 6 aggregating squared singular values before taking an update-energy-weighted ratio.All quantities are computed from weights without generated responses.Source: F Spectral calibration of RL updates · p. 25

F.1 One target, different rank fractions

  • At the selected target, FC is 0.8/0.8/0.7 for DeepSeek/Nemotron/OLMo, with distances from τ = 0.95 of 0.009/0.003/0.004.The rule selects the nearest candidate, so Nemotron’s selected retention is slightly below 0.95.Source: §F.1 · p. 25
  • The target was fixed before applying the weight-only selection procedure to OLMo, but agreement across math and code does not establish a universal optimal energy level.Finer-grid OLMo diagnostics bracket the selected value, but behavioral measurements are unavailable at those settings.Source: §F.1 · p. 25

F.2 Why equal rank fractions preserve different energy

Equal rank fractions can preserve different amounts of update energy across architectures, and retained anchor energy is distinct from fusion displacement. The section also contrasts update-defined and absolute-weight blocks and reports the cost of fusion diagnostics.

F.2 Why equal rank fractions preserve different energy

  • At FC = 0.8, OLMo’s shape-matched Gaussian reference retains 95.8%, versus 91.3% for the math architectures, while its actual update retains 97.6%.The reference is weighted by each anchor’s module energies; matching ξa adjusts for observed energy distributions rather than treating raw FC as equivalent across models.Source: Table 20 · p. 25, §F.2 · p. 26

F.3 Protected energy is distinct from fusion displacement

  • Retaining about 95% of anchor energy does not mean moving 5% toward the donor or predict the donor’s behavioral contribution.At the selected settings, normalized displacement η depends on the donor as well as the anchor.Source: §F.3 · p. 26
  • The unprotected anchor tails contain 4.1%, 5.3%, and 4.6% of update energy for DeepSeek, Nemotron, and OLMo; most of each tail comes from MLP projections.The tails are descriptive: low energy is not necessarily behaviorally irrelevant, and 95% is not an intrinsic spectral boundary.Source: §F.3 · p. 26

F.4 Absolute-weight blocks versus update-defined blocks

  • Equal block dimensions preserve different amounts of RL-update energy for absolute-weight versus update-defined blocks in the DeepSeek comparison.For delta-defined blocks, captured energy equals ξa(FC).Source: Table 21 · p. 26, §F.4 · p. 26

G.1 Measured fusion time

  • The ξ curve reuses singular values and takes 0.07 seconds for either family; evaluating the reported η curve adds 0.34 seconds.These diagnostics do not require repeated SVDs or an inference sweep, though constructing and evaluating additional policies still incurs costs.Source: §G.1 · p. 27, Table 22 · p. 27

G.2 Reconstructing one RL step

The section reconstructs RL-step resource costs from single-GPU measurements rather than an end-to-end training run. It compares fusion costs with those estimates and identifies omitted costs and assumptions.

G.2 Reconstructing one RL step

  • Fusion uses 0.45%/0.28% of synchronous reconstructed RL-step resources and 0.70%/0.34% of throughput-based resources for DeepSeek/Nemotron.These are aggregate GPU-resource comparisons, not one-GPU versus 32-GPU wall-time comparisons.Source: Table 23 · p. 27, §G.2 · p. 28
  • The synchronous reconstruction estimates 180/277 seconds per step for DeepSeek/Nemotron, while the separate 256-response measurements report 3,327/3,094 generated tokens per second.Mean response lengths were 4,472/9,579 tokens, respectively; the throughput-based estimate uses the larger-batch rates.Source: §G.2 · p. 28
  • Both reconstructions omit synchronization, communication, reward scoring, data loading, validation, and checkpoint I/O, so they are lower-bound accounting estimates under stated assumptions.The synchronous estimate is not a hardware-independent lower bound for asynchronous rollout systems.Source: §G.2 · p. 28
  • Accuracy evaluation is a separate cost from fusion and weight-space diagnostics, and end-to-end search cost depends on candidate count and decoding protocol.The reported timings do not measure search under the paper’s four-response evaluation protocol.Source: §G.2 · p. 28

H 7B coding setup and checkpoint measurements

The coding setup uses execution-based verification and evaluates source checkpoints and fusions under a fixed decoding protocol. The measured trajectory determines anchor and donor roles, while reported precision constrains interpretation of small changes.

H 7B coding setup and checkpoint measurements

  • Coding RL starts from Olmo-3-1025-7B and follows the OlmoRL recipe on Dolci-RL-Zero-Code-7B with execution-based verification and a GRPO-derived objective.The recipe uses 32 prompts per batch, eight responses per prompt, and a constant learning rate of 10^-6.Source: H 7B coding setup and checkpoint… · p. 29
  • HumanEval+ has 164 problems and MBPP+ has 378; reported accuracy is mean single-response pass rate across four rounds, requiring both base and additional EvalPlus tests.The coding decoding settings are fixed across sources and fusions, but differ from the math decoding configuration.Source: H 7B coding setup and checkpoint… · p. 29
  • Among six later candidates, step 1950 has the highest two-benchmark mean accuracy and serves as anchor; step 1500 is the donor, with highest native HumanEval+ accuracy and shorter responses on both tasks.Source-policy measurements determine these roles; the fixed spectral target determines FC.Source: H 7B coding setup and checkpoint… · p. 29
  • The reported 0.0-point MBPP+ change denotes equality at displayed precision; the four HumanEval+ round accuracies are reported separately for SURGE, anchor, and donor.The respective round scores are 84.1/83.5/83.5/83.5, 82.3/81.7/82.3/81.7, and 84.1/81.1/82.3/83.5.Source: H 7B coding setup and checkpoint… · p. 29

Related papers

Loading 2610.01133v1…