PaperlayerView exact revision on arXiv

Source-linked AI summary

Expanding LLM Reasoning

Rian Atri, Evan Luo

arXiv:2610.05584v1cs.LGcs.AIcs.CL

TL;DR

The paper asks where to spend an additional continuation within an existing reasoning chain. It measures correctness gains from restarting at stored steps and evaluates placement rules across models and benchmarks. Held-out audits find selected positions beat uniform placement, while always-last is a strong baseline and the learned router shows no detected gain over it.

  • Problem

    Most test-time methods sample or rank complete trajectories, leaving open where an additional continuation helps most within an existing chain.

    Source: §1 · p. 1

  • Method

    The paper measures correctness gains from restarting at every eligible stored step and tests opportunity, held-out selection, oracle headroom, and feasible selectors.

    Source: §1 · p. 1, §2 · p. 3

  • Results

    Selected restart positions beat uniform placement in held-out audits, while the learned router exceeds uniform but shows no detected gain over always-last.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Within-chain placement is a separate test-time allocation decision, and always-last is the reference baseline for learned placement methods.

    Source: §10 · p. 8

    Results are conditional on a designed model-benchmark grid, automatic graders, and stored step boundaries; the resolved self-consistency advantage is measured in aggregate generated output on one panel.

    Source: §11 · p. 8

Abstract

from arXiv · show

Extra inference compute is usually spent on sampling more reasoning chains. We study where inside an existing chain an additional continuation should begin. We define expansion utility, the change in correctness from restarting a chain at a stored step, and measure it at every eligible step for nine models on six benchmarks (41 model and benchmark cells). Restart position matters: steps selected on one set of continuations beat uniform placement when scored on disjoint ones, in held-out audits on 5, 16, and 38 cells (+4.25 points [+2.51, +6.63] in a fresh five-cell audit). A fixed rule that restarts from the last eligible steps, always-last, is a strong baseline: our learned router beats uniform placement but shows no detected gain over it, and on DeepSeek-R1-Distill-Qwen-14B/MATH-500 always-last exceeds the exact self-consistency frontier at matched aggregate generated output by +0.052 [+0.008, +0.098], using 0.774x the aggregate generated output of four-sample self-consistency. Cross-fitted oracle selection still finds held-out headroom beyond declared positional classes, a target for future selectors. Finally, breaking step-label ties by earliest index flips the sign of a pointwise selector's gain over uniform placement in every seed of a five-seed diagnostic with four rollouts per step; randomized ties remove the bias.

1 Introduction

The paper asks where an additional continuation should begin within an existing reasoning chain, defining expansion utility as the correctness gain from restarting at a stored step. Across models and benchmarks, selected restart positions retain an advantage over uniform placement on held-out continuations, while always-last is a strong fixed baseline.

2 Expansion Utility

  • Expansion utility is the correctness gain from restarting a stored chain at a step, measured by intervening at every eligible step and observing outcomes.A restart keeps the prefix fixed and samples a new suffix; the same-draw statistic compares top-ranked eligible steps with the eligible-step mean.Source: §1 · p. 1, §2 · p. 3

1 Introduction

  • +14.80 pp same-draw opportunity across 41 primary cells; selected restart positions also beat uniform placement in held-out audits on 5, 16, and 38 cells.The same-draw statistic is optimistic by construction, so the held-out audits test whether the selection advantage persists on disjoint outcomes.Source: §1 · p. 1, §2 · p. 3
  • Across 25 of 25 split-and-refit runs, the learned router exceeds uniform placement but shows no detected gain over always-last.On DeepSeek-R1-Distill-Qwen-14B/MATH-500, always-last clears the exact self-consistency frontier at 0.774× the aggregate generated output of SC(4).Source: §1 · p. 2
  • Cross-fitted oracle selection finds held-out headroom beyond declared positional classes, while randomized tie-breaking removes early-position labeling bias.The evaluated ListNet loss is invariant to tie resolution.Source: §1 · p. 2

2 Expansion Utility

  • Held-out scoring selects positions on one set of continuation outcomes and scores them on disjoint outcomes; its protocols differ in draw counts and estimate different selectors.The same-draw statistic measures opportunity, not the gain of a feasible selector.Source: §2 · p. 3

3 Experimental Setup

The study evaluates restart opportunity across a designed grid of models and benchmarks, then tests whether selected positions retain an advantage under held-out scoring. The measured opportunity is substantial on average, but varies across cells and protocols.

3 Experimental Setup

  • The primary set has 41 nondegenerate model-benchmark cells spanning nine models and six benchmarks, with 30 to 500 problems per cell.Putnam is exploratory; its two cells are reported separately and excluded from aggregates.Source: §3 · p. 3
  • Held-out scoring intervals use a crossed percentile bootstrap over shared problems and cells; router-comparison intervals describe split stability, not problem-population uncertainty.The self-consistency comparison instead uses a paired problem bootstrap that keeps every seed of a resampled problem together.Source: §3 · p. 3

4 Restart Opportunity Is Concentrated and Survives Held-Out Scoring

  • The 41 primary cells average +14.80 pp same-draw opportunity, with substantial variation across models and benchmarks.The two exploratory Putnam cells are excluded from aggregates.Source: §4 · p. 3, Figure 3 · p. 4
  • Positive restart utility is concentrated within cells and uneven within chains, while greater cell-level concentration is associated with smaller advantages (r = −0.563).The pooled Gini averages 0.896, with at least 0.80 in 37 of 41 cells; mean within-chain Gini is 0.585 among chains with nonzero clipped utility.Source: §4 · pp. 3–4
  • +4.25 pp [+2.51, +6.63] held-out restart-success advantage over uniform in a five-cell audit of 718 examples.Positions selected on four fresh continuations were scored on the complementary four; other audits also found aggregate advantages over uniform placement.Source: §4 · pp. 4–5, Figure 4 · p. 4
  • Across 38 primary cells, selecting on two of four stored continuations and scoring on the other two yields +11.03 pp [+7.74, +14.54], positive in 36 cells.The held-out audits estimate different selectors, but all place selected positions above uniform in aggregate.Source: §4 · pp. 4–5

5 Where to Restart Versus Whether to Spend

The router’s held-out placement advantage over uniform does not distinguish learned placement from the late-position rule always-last. The section also separates placement effects from gating and reports no detected improvement from added feature components.

5 Where to Restart Versus Whether to Spend

  • +0.127 GR@10 for the router over uniform placement across 25 of 25 split seeds, but no detected gain over always-last in either ungated or gated comparisons.The router-minus-always-last differences were −0.006 [−0.016, +0.005] ungated and +0.008 [−0.023, +0.037] gated.Source: §5 · p. 5, Figure 5 · p. 5
  • Gate-on minus gate-off GR@10 was +0.056 [+0.020, +0.092] for the router and +0.042 [+0.005, +0.083] for always-last.The paired difference between these gate effects was +0.013 [−0.020, +0.044], so comparisons must apply the gate consistently to isolate placement.Source: §5 · pp. 5–6, Figure 5 · p. 5
  • Removing the learned depth prior improved GR@10 by +0.0215, while dropping late support or adding PRM, log-probability, or hidden-state features showed no detected improvement.The results do not separate the router’s gain over uniform from the late-position rule.Source: §5 · pp. 5–6

6 Oracle Headroom Beyond Declared Positional Classes

Always-last exceeds the tested self-consistency frontier at matched aggregate generated output in one evaluated cell. Cross-fitted oracle analysis also finds held-out restart-success headroom beyond declared positional classes, but that residual is not a feasible selector’s gain.

6 Oracle Headroom Beyond Declared Positional Classes

  • Cross-fitted oracle residuals exceeded the 21-policy depth class by +0.0275 [+0.0183, +0.0368] and +0.0248 [+0.0157, +0.0340] across the two sample directions.Residuals were positive in 14 and 15 of 16 cells; they remained positive against a richer positional learner that did not outperform the original class out of sample.Source: §6 · p. 6
  • The oracle residual is a target, not feasible-policy gain; tested PRM, log-probability, and hidden-state summaries showed no detected selector improvement.The richer positional family does not span every positional function.Source: §6 · p. 6

7 Always-Last Versus the Self-Consistency Frontier

  • +0.052 [+0.008, +0.098] for always-last over the exact SC(1) to SC(4) frontier at matched aggregate output, using 0.774× SC(4) output.Always-last exceeded the frontier in 3 of 3 generation seeds; the router also cleared it, with no detected router-minus-always-last difference.Source: Figure 6 · p. 6, §7 · pp. 6–7
  • The frontier result comes from one DeepSeek-R1-Distill-Qwen-14B/MATH-500 cell; the other available cell’s margin is unresolved, and the experiment establishes no latency or total-token benefit.Always-last runs after the base chain, so latency and total tokens require separate measurement.Source: §7 · p. 7

8 Label Ties Can Manufacture Position Effects

With four rollouts per step, earliest-index tie breaking injects an early-position label signal, while randomized ties remove it; pointwise BCE’s GR@10 changes sign across the five-seed diagnostic.

(b) GR@10

  • −0.175 to +0.095: pointwise BCE GR@10 changes sign under randomized rather than earliest-index ties in all five seeds.The five-seed ranges are −0.218 to −0.137 and +0.050 to +0.134, respectively.Source: §8 · p. 7, Figure 7 · p. 7
  • +0.123: ListNet GR@10 is unchanged under either tie-breaking rule.The evaluated ListNet formulation is exactly invariant to tie resolution.Source: §8 · p. 7, Figure 7 · p. 7

8 Label Ties Can Manufacture Position Effects

  • At the selection cutoff, earliest-index tie breaking labels the first tied step and injects an artificial early-position signal; randomized ties choose one tied step at random.The diagnostic uses four rollouts per step, so cutoff ties are common.Source: §8 · p. 7, Figure 7 · p. 7

(b) GR@10

  • Both 24-seed ListNet-minus-BCE intervals cross zero, so the comparisons establish neither a ranking nor equivalence; a 64-seed post-lock extension is not independent confirmation.The extension met its frozen criteria for a positive contrast without positional support and equivalence within ±0.02 GR@10 inside the late support.Source: (b) GR@10 · p. 7
  • +14.25 pp with same-eight selection and scoring versus +4.25 pp with fresh-four selection and complementary-four scoring on five audited cells; the selectors differ.The attenuation is consistent with selection on noise, so the passage recommends validating rollout-derived step importance on held-out rollouts.Source: (b) GR@10 · pp. 7–8

9 Related Work

The paper measures where to spend a continuation within an existing chain, complementing methods that allocate compute across full trajectories. Its discussion and conclusion emphasize held-out evaluation, always-last as a baseline, and limits on the scope of the evidence.

9 Related Work

  • Prior methods allocate compute across complete trajectories, score intermediate steps, branch from partial trajectories, or adapt computation; this work measures interventional restart effects at stored prefixes.The paper tests how much fixed placement, declared positional classes, and a feasible selector capture.Source: §9 · p. 8

10 Discussion

  • Restart value is uneven within chains and its advantage survives held-out scoring at reduced magnitude; the discussion identifies same-draw optimism and tie-induced position bias as failure modes.The discussion names always-last as the reference baseline for this separate allocation decision.Source: §10 · p. 8
  • The authors recommend reporting whether gains are same-draw, held-out, or from a feasible selector, paired contrasts against uniform and always-last, and generated output, total tokens, and wall-clock time separately.The learned router showed no detected gain over the rule without a learned component.Source: §10 · p. 8
  • Restart value is an interventional quantity rather than a judgment of step correctness, and adding Math-Shepherd PRM summaries showed no detected improvement in the tested selector.The discussion identifies held-out residual beyond declared positional classes as a reference target for future selectors trained on restart value.Source: §10 · p. 8

11 Limitations and Broader Impact

  • Results are conditional on a designed model-and-benchmark grid, automatic graders, and stored step boundaries; the resolved self-consistency advantage is from one panel and uses aggregate generated output.A miscalibrated selector could damage correct answers or waste compute.Source: §11 · p. 8

12 Conclusion

  • Across 41 cells, held-out audits retain a positive advantage over uniform placement; the learned router shows no detected gain over always-last, which exceeds the exact self-consistency frontier in one tested cell.The conclusion reports +0.052 at matched aggregate generated output for always-last on DeepSeek-R1-Distill-Qwen-14B/MATH-500.Source: §12 · p. 9

A Measurement Definitions and Cell Inclusion

The appendix distinguishes the primary measurement estimand from other utility summaries and specifies which model-benchmark cells enter the analysis. It also describes held-out audits as separate protocols that estimate different selectors.

A Measurement Definitions and Cell Inclusion

  • +14.80 pp: Eq. (2) is the budgeted eligible-position advantage; the maximum-observed-utility statistic is a distinct quantity at +21.34 pp.The maximum-observed statistic averages the largest observed utility per example.Source: A Measurement Definitions and Cell… · p. 11
  • The primary roster has 41 cells: 38 nondegenerate main-matrix cells, one GPT-OSS HMMT cell, and two MBPP cells; the two Putnam rows are exploratory.The complete display has 43 rows, while 11 of the 54 model-benchmark pairs were not run and two were degenerate.Source: A Measurement Definitions and Cell… · p. 11, Table 1 · p. 11
  • The max-utility column is not Eq. (2): it can fall below it because utility subtracts the base outcome; the two Putnam rows are excluded from primary aggregates.The primary aggregate weights cells equally, while problem-weighting yields +18.27 pp instead of +14.80 pp.Source: Table 2 · p. 11, A Measurement Definitions and Cell… · p. 11

B Held-Out Scoring Audits

  • Held-out protocols select on one set of continuation outcomes and score on disjoint outcomes; fresh-eight, stored-winner, fresh 4/4, and 2/2 audits estimate different selectors and are reported separately.The 2/2 matrix audit exhausts complementary partitions and both directions, with analytic tie-neutral inclusion at the budget cutoff.Source: B Held-Out Scoring Audits · p. 12, Table 3 · p. 12
  • The separately generated 4/4 audit gives +5.48 pp [+3.67, +7.59] across 3421 examples in 16 cells, positive in every cell.Its cross-sample cell-Gini reliability is r = 0.994; boundary-density controls change the candidate set and do not test semantic segmentation.Source: B Held-Out Scoring Audits · p. 13, Table 4 · p. 13

C Policy-Class Protocol

The policy-class protocol cross-fits selection and scoring across continuation samples to estimate residual opportunity beyond declared positional rules. It also describes the learned router and limits the transfer claim to a shared late-only policy within the benchmark suite.

C Policy-Class Protocol

  • The 21-policy depth class selects the relative-depth target by five-fold cross-validation, then scores on a separate continuation sample with the direction reversed; the resulting residual is oracle opportunity, not feature-based predictability.The enriched sensitivity nests the original grid and regularized length-by-rank tables, but continuation outcomes remain in its oracle selection rule.Source: C Policy-Class Protocol · p. 13, Table 5 · p. 13
  • The enriched positional candidate family does not cover the full arbitrary positional class, and its selected learner trails the original grid out of sample.The residual estimates remain oracle targets rather than gains of a feasible selector.Source: Table 6 · p. 13, Table 5 · p. 13
  • The evaluated router combines within-example text and structure features, PRM and token-log-probability summaries, a listwise objective, learned depth prior, late positional support, and a confidence gate.The gate determines whether to spend a restart on a chain; attribution rests on paired always-last controls.Source: C Policy-Class Protocol · p. 13
  • Router intervals are paired split-seed stability intervals on one frozen store, not problem-population intervals; crossing-zero contrasts are no detected difference, not equivalence.The fitted gate threshold is applied unchanged to both placement rules.Source: Table 7 · p. 14
  • The leave-domain-out control removes target domains from fitting and validation, disables benchmark identity, and applies shared late-only support; it establishes within-suite transfer, not general content-based restart localization.Paired differences from always-last remain unresolved, and the control does not establish transfer outside the benchmark suite.Source: C Policy-Class Protocol · p. 14, Table 9 · p. 14

D Exact Self-Consistency Frontier Protocol

The exact self-consistency frontier is constructed on a fixed DeepSeek-R1-Distill-Qwen-14B/MATH-500 panel by enumerating sample subsets and mixing adjacent frontier points. Its scope is a partial fixed-panel sensitivity, not the incomplete planned model-by-benchmark panel.

D Exact Self-Consistency Frontier Protocol

  • The frontier uses the first 100 MATH-500 problems and three generation seeds, with all subsets for SC(1) through SC(4) enumerated per problem and seed.SC uses the plurality parsed answer, breaks ties toward the earliest sample, and counts tokens from unparsable samples even though they do not vote.Source: D Exact Self-Consistency Frontier… · p. 15
  • For a target aggregate generated-output budget, a linear program selects a globally randomized mixture over adjacent frontier points.Policy accuracy is paired with the same problem’s frontier construction, and percentile-bootstrap resampling keeps each problem’s seeds together.Source: D Exact Self-Consistency Frontier… · p. 15
  • Table 10 reports an exact aggregate generated-output frontier, whose ratios measure generated output only—not total tokens or latency.The comparison is conditional on the frozen DeepSeek-R1-Distill-Qwen-14B/MATH-500 panel.Source: Table 10 · p. 15, D Exact Self-Consistency Frontier… · p. 15
  • The two-model MATH result is a fixed-panel sensitivity using two available cells; missing GPQA cells leave the planned 2-by-2 panel unidentified.PRM, tree-search, and H200 rows use separate cohorts and ledgers rather than being pooled with the exact DeepSeek frontier.Source: D Exact Self-Consistency Frontier… · p. 15, Table 11 · p. 15

E Loss and Feature Diagnostics

The locked seed extension finds a positive mask-free contrast and equivalence within the late-support margin, but it is post-lock stability evidence rather than independent confirmation. The primary problem-cluster inference crosses zero, and evaluated feature additions show no detected improvement.

E Loss and Feature Diagnostics

  • The primary unmasked problem-population interval crosses zero despite a positive seed-stability interval, while both late-support contrasts are unresolved.No equivalence claim is made for the primary comparison because its protocol did not freeze and power an equivalence margin.Source: E Loss and Feature Diagnostics · p. 15
  • +0.0105 to +0.0263 is the paired-seed interval for the mask-free contrast in the post-lock extension; the late-support contrast is equivalent within ±0.02.The primary inference remains the 24-seed problem-cluster interval, so the extension is stability evidence, not independent confirmation.Source: Table 12 · p. 15, E Loss and Feature Diagnostics · p. 16
  • PRM summaries, token log-probabilities, and hidden states show no detected improvement on matched coverage; PRM best-of-N and a shallow tree also do not beat self-consistency in scoped controls.These diagnostics do not exclude other representations, objectives, or search algorithms.Source: E Loss and Feature Diagnostics · p. 16

F Additional Analyses and Scope

Additional analyses are reported separately when cohorts, cost ledgers, or uncertainty units differ from the primary results. Reproducibility materials and asset disclosures describe what is available and where historical provenance remains incomplete.

F Additional Analyses and Scope

  • The predeclared feasible-spend analysis did not pass its randomization gate and therefore establishes no learned-spend effect.Domain-transfer, H200 timing, tree-search, and fixed-panel controls are reported separately because their cohorts, cost ledgers, or uncertainty units differ.Source: F Additional Analyses and Scope · p. 16

G Reproducibility and Release

  • Measurement records, utility maps, inclusion records, split seeds, JSON certificates, and a verification script are available from the authors on request.Figures are compiled directly from TikZ/PGFPlots.Source: G Reproducibility and Release · p. 16
  • Historical settings were reconstructed from commands and logs, but original runs lack per-cell manifests, leaving their settings not cryptographically bound and row-level provenance incomplete.The paper records settings, seeds, graders, and protocol details for recollected and later collections; exact-frontier assays use a separate 8192-token-cap protocol.Source: G Reproducibility and Release · pp. 16–17
  • The paper identifies evaluated model repositories and benchmark datasets but does not provide a complete per-asset license inventory; no human-subject data were collected.Language models assisted with prose editing and code review, and authors verified tables, figures, and numerical statements against frozen sources and the hash manifest.Source: G Reproducibility and Release · p. 17
Loading 2610.05584v1…