PaperlayerView exact revision on arXiv

Source-linked AI summary

From Base Rollouts to RL Reasoning: A Budgeted Search Perspective

Wenhe Sun, Cunxiang Wang, Zijun Yao, Yixin Cao

arXiv:2609.01274v1cs.CL

TL;DR

The paper asks whether RLVR creates reasoning absent from base models or shifts sampling toward trajectories they can already reach, and studies this behavior with UDF and structured Base operating-point paths. Across benchmarks and model settings, pass@k recovery follows BOPTR, supporting a qualified internalized-search interpretation while remaining behavioral rather than parameter-level.

  • Problem

    It is unclear whether RLVR creates new reasoning behavior or mainly reweights trajectories already present in the base model, a distinction complicated by heterogeneous decoding and search budgets.

  • Method

    UDF represents decoding and search as executable policies over a shared budgeted operating space, while SearchPath/BOPTR tests low-complexity recovery of RL curves by Base operating points.

  • Results

    Across paired Base/RL checkpoints, RL default-policy curves are often recoverable as Base operating paths, with pass@k recovery following benchmark-regime-conditioned BOPTR scaling.

  • Takeaways & Limitations

    Under the tested recipe, much measured RL gain corresponds behaviorally to improved sampling efficiency toward operating points the base model can reach under search.

  • Takeaways & Limitations

    The results are behavioral diagnostics rather than parameter-level mechanistic identification, and the rule’s behavior beyond the tested cohort is not established.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but how these gains relate to inference-time decoding and search remains unclear. Does RL create reasoning the base model lacks, or shift the rollout distribution toward trajectories it can already reach but rarely samples? We study this behaviorally with a Unified Decoding Framework (UDF), which expresses token-level sampling, beam-like search, tree search, and sequence-level resampling as executable policies over a shared budgeted operating space, scored post hoc with pass@$k$, self-consistency, best-of-$N$, and first-finish success. Using paired Base/RL checkpoints from SimpleRL-Zoo, we ask whether an RL default-policy curve can be approximated by a structured path of Base operating points. On Math500, AIME, GPQA, and IFEval, the pass@$k$ recovery path follows a Budgeted Operating-Point Transition Rule (BOPTR), $N_{\mathrm{Base}} \approx αN_{\mathrm{RL}}^β$, with benchmark-conditioned exponents. On Qwen2.5-7B, BOPTR gives the lowest transfer error among the non-oracle rules we test, 3.41 pp (95% CI [2.32, 5.53]); a three-seed replication gives 3.07 $\pm$ 0.39 pp. The rule extends to ten models across four families (3.28 to 4.87 pp on checkpoints added after fitting), to four benchmarks it was never fitted on (5.03 pp vs. 4.44 pp in fit), and holds without an RL checkpoint for the target model (4.19 pp) or without RL supervision of any kind (5.08 pp). These results support a qualified internalized-search reading: under the recipe we test, much of the measured RL gain corresponds to a change in sampling efficiency toward operating points the base model can already reach under search. We treat the scaling patterns as descriptive of this recipe and cohort, report where they break down, and use UDF and BOPTR as behavioral diagnostics rather than evidence of parameter-level equivalence.

1 Introduction

The paper asks whether RLVR creates new reasoning or shifts sampling toward trajectories already reachable by the base model. It introduces UDF and BOPTR to test behavioral recovery across budgets, benchmarks, and models.

  • Motivation: RL may improve sampling efficiency by shifting default rollout allocation toward reward-aligned reasoning trajectories.Base models can contain correct trajectories that are difficult to sample by default, while RL increases their probability under small budgets.
  • Approach: UDF represents sampling and search methods as executable policies in a shared budgeted operating space, with evaluation metrics applied after generation.The framework covers token-level sampling, beam-like expansion, tree-style exploration, and sequence-level resampling.
  • Findings: RL default-policy curves are often recoverable as Base operating paths across paired Base/RL checkpoints from SimpleRL-Zoo.In pass@k, the recovered Base budget follows a regime-conditioned transition rule, N_Base ≈ αN_RL^β.
  • Findings: Transfer is stronger within the same SimpleRL-Zoo difficulty split and weaker for OOD benchmarks, cross-family models, and different training distributions.The formula family remains informative, but multipliers and selected policies are model-specific.
  • Approach: BOPTR tests whether RL default-policy curves can be recovered by a low-complexity path of Base operating points rather than unrelated per-budget matches.The pass@k rule uses a benchmark-regime-conditioned exponent for the effective Base budget.
  • Resources: The released code, configurations, and analysis scripts support reproducibility of the SearchLens and BOPTR analyses.Materials are available at the repository identified in the paper.

2 Related Work

Related work frames RLVR gains as potentially reflecting amplified base-model behaviors and improved sampling efficiency rather than wholly new reasoning. It also motivates explicit treatment of inference-time budget and search.

  • RL and reasoning: Recent studies ask whether RLVR creates new reasoning abilities or reweights behaviors already present in the base distribution.RL-trained models often lead at small sampling budgets, while base models can recover much of the gap under larger pass@k.
  • Inference-time computation: Training-free sampling from base models can approach or outperform RL-posttrained models on several reasoning benchmarks.This comparison motivates treating inference budget as an explicit axis rather than comparing Base and RL at one budget.
  • Inference-time computation: How a fixed test-time budget is allocated can matter as much as model scale, so Base–RL comparisons may vary across budgets.This motivates analyzing decoding and search policies jointly with their rollout budgets.
  • Inference-time computation: Inference-time compute improves reasoning through prompting, repeated sampling, search, and verifier- or reward-model-based selection.The paper places these heterogeneous mechanisms in a common budgeted policy space through UDF.

3 SearchLens: A Unified Decoding/Search Landscape

SearchLens defines a shared operating landscape for decoding and search by combining a policy configuration with an inference budget. UDF separates policy execution from post-generation behavioral evaluation.

  • Operating space: A decoding or search method is represented as an operating point consisting of a UDF policy configuration and an inference budget.The experimental budget set is {1, 2, 4, 8, 16}.
  • UDF components: The scheduler allocates inference budget across samples, branches, or resampling steps.This makes different decoding and search procedures comparable as budget-allocation policies.
  • Evaluation: Evaluation metrics summarize generated candidate sets after generation and are not part of the UDF policy definition.The metrics include pass@k, self-consistency, best-of-N, and first-finish success.
  • UDF components: UDF uses five policy components: local policy, controller, evaluator, transition, and scheduler.Together they specify how candidates are proposed, searched, scored, updated, and allocated across the finite budget.
  • Policy geometry: SearchLens associates operating points with behavioral and cost coordinates, including pass@k, self-consistency, first-finish or validity, diversity or pass–SC gap, and cost.The combined distance is used for path diagnostics rather than mechanistic identification.
  • SearchPath: SearchPath/BOPTR tests whether the RL curve forms a low-dimensional trajectory through the SearchLens operating space.The five-component parameterization instantiates eleven mainstream decoding algorithms against reference implementations.

4 Experimental Setup

The experiments use paired Base/RL checkpoints, four benchmark families, and budgets from 1 to 16 to test horizontal, vertical, and cross-family transfer. Results are reported as errors relative to RL default-policy targets.

  • Evaluation design: Qwen2.5-7B/Math500 is the anchor cell for constructing the SearchLens landscape, recovery analysis, and BOPTR rule.Other model–benchmark cells test horizontal transfer, vertical transfer, and cross-family stress settings.
  • Models: The model cohort includes Qwen2.5, Qwen2.5-Math, Llama3.1, and Mistral Base/RL checkpoint pairs when compatible generations are available.SimpleRL-Zoo Hard-split models form the upper cohort, while cross-family or weak-capacity models form the lower cohort.
  • Benchmarks: The benchmarks are Math500, AIME 2024/2025, GPQA-Diamond, and IFEval.IFEval stresses instruction following and termination, with pass rate, validity, and first-finish success as primary metrics.
  • Policies and metrics: Budgets are 1, 2, 4, 8, and 16 across token-level and, when available, expanded search policies.Main analyses compare same-policy curves, Base+UDF envelopes, near-match policies, and low-complexity rule families.
  • Policies and metrics: Errors are reported as mean absolute error in percentage points relative to the RL default-policy target.This metric is used to compare transfer rules across model–benchmark settings.

5 From Same-Policy Gains to Base Search Recovery

The paper separates same-policy RL gains from behavioral recovery through Base decoding and search. On the Qwen2.5-7B/Math500 anchor, the RL target lies within the Base+UDF support region, motivating structured recovery beyond same-budget comparisons.

  • Same-Policy Gains: Same-policy comparisons isolate post-training effects from gains caused by richer inference-time policies.Accuracy is defined by model, metric, policy, and budget, with π0 denoting the RL default policy.
  • Same-Policy Gains: RL improves low-budget pass@k and self-consistency on math-style tasks, while Base can recover or exceed RL pass@k at larger budgets.GPQA and IFEval exhibit different metric behavior, with GPQA resembling concentration and IFEval emphasizing validity and first-finish success.
  • Base Search Recovery: The recovery question asks whether external decoding and search can approximate the RL default-policy curve, not whether RL implements an external policy.This frames the analysis as behavioral recoverability rather than a parameter-level equivalence claim.
  • Base Search Recovery: The RL default target lies inside the Base+UDF support region at every budget on Qwen2.5-7B/Math500.The support region combines the Base policy cloud with the Base+UDF upper envelope.
  • Base Search Recovery: Near-match candidates are selected into a recovery path so structured recovery is distinguished from unrelated post-hoc per-budget lookup.A Base model may underperform RL at the same policy and budget yet match RL when policy and budget vary.

6 SearchPath: Low-Dimensional Recovery Paths

SearchPath formalizes Base recovery as low-dimensional budget transitions whose exponents depend on benchmark regime and whose offsets and policies depend on the model. BOPTR transfers best within related training regimes, while transfer degrades across families and distributions but remains informative without target-model RL calibration.

  • SearchPath: The RL target remains inside the Base+UDF support region at every budget, providing the feasible region from which SearchPath selects recovery paths.Figure 4 shows Base policy trajectories, the Base policy cloud, and the Base+UDF upper envelope relative to the RL target.
  • SearchPath: BOPTR-P1 models recovery with a regime-specific budget exponent and a model-specific scalar offset, while selecting a structured Base operating path.The rule is designed to predict RL pass rates through Base operating points rather than perform independent per-budget matching.
  • Cross-Benchmark Transfer: The fitted regimes are sublinear for Math500 (βmath = 0.60), approximately linear for GPQA and IFEval (βnonmath = 1.00), and floor-bound for AIME (βfloor = 0.00).The AIME label is descriptive rather than a capability claim; refitting AIME into the math regime changes its cell error by +1.67 pp with a 95% CI of [−5.00, +4.00].
  • Cross-Benchmark Transfer: 3.41 pp is the mean error for BOPTR-P1, down from 6.01 pp for anchor per-budget copy and identified as the strongest non-oracle horizontal rule.The 3.41 pp anchor mean has a 95% CI of [2.32, 5.53], and three decoding seeds yield 3.07 ± 0.39 pp.
  • Limitations: Budget-grid extension raises mean error to 5.02 pp, concentrated on AIME at 12.50 pp, showing a boundary for the frozen floor map.The reported large-budget misfit is characterized as a budget-map allocation error rather than a ceiling of Base search.
  • Vertical Transfer: Direct anchor-path transfer works best within the same SimpleRL-Zoo difficulty split and reaches 10.16 pp error in the cross-family Llama setting.Errors for the 7B, 14B, and Math-7B models are 1.40, 2.55, and 3.73 pp, respectively.
  • Vertical Transfer: A one-dimensional model offset reduces overall error to 4.44 pp, while a base-only predicted offset reaches 4.19 pp without target-model RL calibration.The offset absorbs much of the model-specific variation, though richer rule families can worsen transfer when they add free axes without matching training signal.

7 Discussion and Conclusion

The paper interprets RL as often moving models through the base operating landscape via structured Base+UDF paths, while emphasizing benchmark- and model-specific recovery and diagnostic limits.

  • 7 Discussion and Conclusion: Recovery can reveal access to behavior already reachable by the base model, while failures localize behavior outside the tested policy pool or behavioral channel.UDF therefore serves both as a matching framework and as a diagnostic for where matching breaks.
  • 7 Discussion and Conclusion: The shared BOPTR formula family has benchmark-conditioned exponents and model-specific offsets and selected policies.The paper advises re-estimating these quantities for other post-training recipes rather than assuming unchanged transfer.
  • 7 Discussion and Conclusion: BOPTR traces RL behavior through the UDF map using a low-dimensional, regime-conditioned Base operating path.The recovered budget follows N_Base ≈ αN_RL^β, with policy selection and offsets completing the operating point.
  • 7 Discussion and Conclusion: BOPTR is positioned as a behavioral diagnostic under the tested SimpleRL-Zoo recipe, not a universal post-RL performance predictor.The conclusion notes that stronger prediction or checkpoint-selection claims require broader cohorts or additional calibration mechanisms.

Limitations

The paper bounds its claims to behavioral recoverability under evaluated UDF policy pools, benchmark-specific metric regimes, and a limited SimpleRL-Zoo model cohort.

  • Limitations: Recoverability is behavioral rather than parameter-level: Base+UDF operating points can match RL targets without identifying an RL model’s mechanism.The paper explicitly does not claim that RL literally executes an external decoding policy.
  • Limitations: Recoverability statements are valid only relative to the UDF policies evaluated in each cell.AR-only and expanded-pool cells are separated, and envelopes from mismatched pools are not compared.
  • Limitations: Cross-benchmark metrics have different semantics, so Math500, AIME, GPQA, and IFEval results should not be read as one uniform metric.IFEval emphasizes first-finish and validity, while GPQA often behaves as a sharpening or concentration task.
  • Limitations: BOPTR parameters and prediction behavior remain model-specific and unestablished beyond the tested cohort.The rule depends on a cohort of RL models, and stronger absolute-prediction claims require larger cohorts or additional calibration.
  • Limitations: The study covers a limited set of model families, scales, benchmarks, and one SimpleRL-Zoo recipe, so transfer requires re-anchoring.Training-duration effects and broader regime behavior remain untested.

A.1 Notation

The notation appendix provides a reference for recurring symbols used throughout the paper.

  • A.1 Notation: Table 5 lists every recurring symbol, one line each.
  • A.1 Notation: Table 5 is the notation table used throughout the paper.

A.2 UDF expressiveness: equivalence to mainstream decoding algorithms

The appendix validates UDF’s expressiveness by instantiating mainstream decoding algorithms and comparing outputs with reference implementations.

  • A.2 UDF expressiveness: equivalence to mainstream decoding algorithms: Eleven mainstream decoding algorithms are instantiated as UDF operating points and checked against reference implementations on identical prompts and seeds.The evaluation uses a Math500 subset with n=50 and Qwen2.5-7B.
  • A.2 UDF expressiveness: equivalence to mainstream decoding algorithms: All seven Tier 1–2 strategies pass the pre-registered equivalence threshold against vLLM SamplingParams references.The comparison reports per-prompt string-match rate and extracted-answer agreement.
  • A.2 UDF expressiveness: equivalence to mainstream decoding algorithms: All four Tier 3–4 strategies match external references within |ΔEM| ≤0.08.The table also reports a 0.657 MCMC accept ratio for power_sampling and token-identical output for entropy_tree.

A.3 Same-policy and Recovery supplements

The supplements quantify transfer, ablations, recoverability, calibration, and scope boundaries for BOPTR-P1. Results show a mostly shared formula with model-specific parameters, substantial compute costs, and failures under some model or distribution shifts.

  • Compute and accounting: ≈461 H100 GPU-hours are required for the anchor cell, dominated by eight expanded-controller runs; RL training cost is excluded.The canonical post-paper sweeps consume approximately 230 wall-hours, 461 H100 GPU-hours, and 0.31B tokens.
  • Transfer costs: 16/23 cells have zero anchor-vs-cell-fit gap, while the largest positive gaps are +6.10 pp and +2.11 pp.The two outliers involve the smallest models on OOD-nonmath benchmarks, where the cell fit may itself be unstable.
  • Ablations: +2.76 pp is the largest ablation drop, identifying the behavior-coordinate selector as the load-bearing component.Removing the budget exponent adds +1.17 pp overall and +5.5 pp in the math regime, while removing the model offset helps cross-family transfer.
  • Self-consistency recoverability: 19/23 cells recover RL self-consistency within 0.05 of maximum Q, but four Math500 cells remain unrecoverable.Those four cases span multiple capacities and families, so unrecoverability is not monotone in model size.
  • Base-only prediction: −0.38 LOO pseudo-R^2 shows that the top base-only feature does not reliably predict model offsets with six models.The authors report that a usable estimate requires a richer, gated predictor rather than a larger linear ridge.
  • Calibration and interpretation: 4.08 pp with one RL anchor and 5.13 pp with no RL data indicate that post-RL behavior can be predicted with very limited calibration information.Across seven models, base@rltrain saturation correlates with log αmath at Pearson r = −0.96, but the broader paper treats the result as behavioral rather than parameter-level equivalence.
Loading 2609.01274v1…