PaperlayerView exact revision on arXiv

Source-linked AI summary

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

Qiancheng Zhou, Ruizhe Li

arXiv:2608.29188v1cs.LGcs.AIcs.CL

TL;DR

RLVR improves single-sample accuracy but can narrow the solution space, raising whether diversity is lost through inaccessible solution families or failed downstream execution. The paper separates these mechanisms with exhaustive Countdown analysis, finding that breadth loss concentrates at trajectory entrances and can be mitigated by entrance-targeted interventions while preserving accuracy. Broader evaluations show the pattern across math benchmarks and model scales, while alternative training regimes retain more early-step diversity.

  • Problem

    Existing diversity measures conflate failure to access a valid solution family with failure to execute it after initiation, leaving the mechanism of RLVR-induced narrowing unidentified.

  • Method

    The paper exhaustively enumerates Countdown solutions into entrance families and compares access, entrance-clamped execution, and RLVR behavior across PPO and GRPO models.

  • Results

    Early entrance narrowing recurs across six math benchmarks and 7B and 14B models, while entrance-targeted interpolation increases solution coverage by 37% without loss in pass@1.

  • Takeaways & Limitations

    Reasoning breadth is primarily lost when the policy selects an opening trajectory, while alternative downstream solutions remain executable and can be recovered by targeting early decisions.

  • Takeaways & Limitations

    Direct access–execution analysis depends on exhaustive enumeration in Countdown; the PPO run uses G=1, no actor KL penalty, and a 1,024-token rollout cap.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: https://github.com/ershiyidian/early-branch-locking.

1 INTRODUCTION

The paper identifies where RLVR-induced reasoning diversity is lost and evaluates interventions that recover breadth without sacrificing single-sample accuracy. Using exhaustively enumerable Countdown solution families, it separates failure to access a solution family from failure to execute it.

  • Motivation and findings: RLVR increases pass@1 while reducing solution coverage across PPO and GRPO training checkpoints.Under PPO, pass@1 increases more than fiftyfold while coverage falls from 0.337 to 0.111; GRPO shows tripled accuracy and a 43% coverage drop.
  • Mechanistic localization: Likelihood shifts before the first arithmetic operation are 16× larger under PPO and 11× larger under GRPO than during later reasoning.Clamping an entrance prefix restores low-access-family completion from 0.018 to 0.212 under PPO, while matched-prefix execution capability improves during training.
  • Recovery interventions: Late-layer parameter interpolation increases solution coverage by 37% without reducing pass@1, whereas surface prompting and forced operator shifts fail to recover coverage.Checkpoint sampling and reasoning-phase logit mixing also recover breadth by unlocking alternative opening trajectories.
  • Generalization and training regimes: Early entrance narrowing recurs across six math benchmarks and 7B and 14B models, while SFT and staged SFT–DPO–RLVR preserve more breadth.On longer reasoning horizons, downstream execution becomes a compounding bottleneck, but entrance selection remains primary.
  • Access versus execution: The framework partitions Countdown solutions into entrance families and separately measures family access and downstream execution.Entrances are defined by the first operand and operator, while entrance-clamped rollouts leave subsequent arithmetic open.

2 RELATED WORK

Related work studies RLVR-induced distribution collapse, sparse branching in autoregressive reasoning, and inference-time or weight-space methods for recovering diversity. These strands motivate locating policy narrowing mechanistically and restoring unreachable solution branches.

  • Distributional sharpening and exploration collapse: Prior studies examine RLVR distribution collapse through reward shaping, exploration bonuses, and altered rollout objectives.The paper positions its analysis within work on winner-take-all sharpening and degraded solution coverage.
  • Bifurcation points and prefix-guided exploration: Research on bifurcation points treats sparse high-entropy tokens as decisions that disproportionately shape downstream trajectory semantics.Other approaches cluster semantic branches from rollouts or steer generation with reasoning prefixes.
  • Test-time scaling and weight-space recovery: Test-time search and verifier guidance are bounded by the policy’s reachability support, motivating checkpoint ensembling and weight interpolation.The paper contributes a mechanistic explanation for why these recovery strategies can restore diversity.

3 SETUP: ENTRANCES, ACCESS, AND EXECUTION

The setup exhaustively enumerates Countdown solution spaces into entrance families, then separates family access from downstream execution using unconstrained sampling, likelihoods, and solver-constructed prefixes. It evaluates these diagnostics across independent PPO and GRPO training pipelines with controlled multi-rollout protocols.

  • Training regimes: The study compares self-trained PPO on Qwen2.5-3B with public GRPO checkpoints for Qwen2.5-3B-Instruct under disjoint evaluation protocols.The PPO run evaluates 150 held-out problems, while the filtered GRPO evaluation uses 135 solver-feasible instances.
  • Solution-space construction: Countdown solutions are exhaustively enumerated as canonical arithmetic expressions, enabling exact coverage measurement over the valid solution space.An independent enumerator verified feasibility and leaf-set completeness on 500 held-out instances with zero discrepancy.
  • Entrance families: Entrance families partition valid solutions by the initial operand–operator tuple while leaving subsequent arithmetic unconstrained.Across 150 held-out problems, the solver finds 379 feasible families, averaging 2.53 families per instance.
  • Access and execution: Access is estimated from unconstrained generation and prefix likelihoods, whereas execution is measured by clamping a solver-constructed entrance prefix and leaving downstream reasoning open.The prefix ends immediately after the opening operator, isolating downstream completion without supplying further arithmetic cues.
  • Evaluation protocol: Evaluations use repeated stochastic rollouts and structural proxies to quantify solution coverage, opening entropy, trace diversity, and early-versus-downstream likelihood shifts.Countdown uses 320 rollouts per problem; broader math benchmarks use first-calculation entropy and canonicalized trace diversity.

4 THE ACCURACY–BREADTH TRADEOFF ACROSS RLVR TRAINING

Across PPO and GRPO, RLVR improves single-sample accuracy while contracting solution breadth, with the strongest losses concentrated at opening computational branches. The contraction persists on invariant solvable problems rather than reflecting only reduced problem reachability.

  • Accuracy–breadth tradeoff: 67%: PPO solution coverage falls from 0.337 to 0.111 while pass@1 increases more than fiftyfold; GRPO coverage drops 43% as accuracy more than triples.The inverse accuracy–breadth relationship persists across decoding temperatures, top-p thresholds, and generation budgets.
  • Invariant solvable subsets: 64% of early-solved problems remain solvable late, but only 31% of early-discovered valid leaves are generated by the late PPO policy.Mean coverage on the invariant 58-problem subset falls from 0.564 to 0.286, while formatting compliance approaches 100%.
  • Initial computational branches: Two-thirds: coverage across fine-grained first-evaluated-pair families drops to 0.117, while coarse operator-class coverage decreases by half.The same opening-diversity decline recurs across five independent solver-enumerated structural partitions.

5 MECHANISTIC LOCALIZATION: BREADTH IS LOST AT THE ENTRANCE

The paper separates access from execution and finds that RLVR’s breadth loss is concentrated in initiating arithmetic branches, while supplied entrances remain executable.

  • Localizing breadth loss: Likelihood shifts are 16× larger under PPO and 11× larger under GRPO before the first arithmetic operation than during subsequent reasoning.Teacher-forced NLL profiles localize probability-mass migration to the entrance.
  • Experimental decomposition: Table 3 separates designated-family completion across all problems, failed-trace retries, and previously unobserved families using entrance interventions.Panel A uses 150 problems; Panel B concerns 33 failed problems; Panel C uses 64 continuations per cell.
  • Testing execution capability: Minimal entrance prefixes raise PPO designated-family completion from 0.018 to 0.212, matching the downstream efficiency of a fully evaluated first calculation.The intervention specifies only the first operand and operator, leaving later arithmetic open.
  • Access versus execution: Conditional execution capability rises by +0.107 while entrance entropy falls between PPO steps 50 and 275 on matched problem-family-scaffold tuples.The paired evaluation covers 139 problems and 359 feasible family instances.

6 TARGETING OPENING DECISIONS RECOVERS SOLUTION BREADTH

The recovery experiments target opening decisions rather than surface formatting or final answers, showing that internal interventions and entrance allocation restore breadth more effectively.

  • Surface interventions: Surface prompting changes solution coverage by only +0.014, while temperature increases coverage by +0.016 but lowers pass@1 from 0.278 to 0.264.Blending earlier logits only inside final answer tags leaves coverage at 0.105.
  • Representation interventions: Late-layer interpolation with early checkpoints increases solution coverage by 37% without sacrificing pass@1.Blending step-50 logits into non-formatting reasoning tokens raises coverage from 0.111 to 0.124 while maintaining late-checkpoint pass@1.
  • Entrance-aware allocation: Uniformly distributing rollouts across feasible entrance families improves solution coverage by +0.077 over free resampling under a matched per-problem token cap.This provides a test-time recovery strategy when earlier checkpoints or weight averages are unavailable.

7 GENERALIZATION ACROSS MODEL SCALES, HORIZONS, AND TRAINING PIPELINES

Early entrance narrowing extends across larger models and six math benchmarks, but longer horizons add execution bottlenecks and alternative training curricula can preserve breadth.

  • Model scales and benchmarks: RLVR consistently triggers early distributional collapse across six benchmarks on Qwen2.5 7B and 14B models.The study measures first-calculation entropy, same-trace likelihoods, and cross-checkpoint trace diversity.
  • Model scales and benchmarks: SimpleRL raises macro pass@1 by 13% at 7B and 17% at 14B while pass@64 shifts only from 0.754 to 0.760 at 7B and remains 0.774 at 14B.First-calculation entropy falls sharply in the same comparisons.
  • Reasoning horizons: Longer open-domain reasoning introduces a compounding downstream execution bottleneck, although entrance selection remains the primary gatekeeper.Countdown leaves only 6–10 downstream tokens, whereas GSM8K and MATH500 can require hundreds after the first calculation.
  • Training pipelines: Across OLMo-3’s SFT → DPO → RLVR stages, pass@1 increases while first-calculation entropy and distinct-trace rates remain near their initial SFT baselines.The comparison indicates that reasoning optimization need not inherently contract search breadth.
  • Trajectory-level evidence: Direct RLVR extinguishes a backward-arithmetic entrance from 10 of 59 to 0 of 64 correct rollouts on GSM8K Item 108, while staged alignment preserves it.All evaluated models reach pass@64 =1 on this instance.
  • Training pipelines: SFT reaches pass@1 = 0.333 with solution coverage 0.709, more than 2.6× the coverage of the late GRPO checkpoint.The SFT models use solver-enumerated multi-solution demonstrations with semantic-key holdout.

8 CONCLUSION

The paper concludes that RLVR narrows reasoning primarily by restricting entrance access rather than eliminating downstream execution. It argues that preserving alternative opening decisions is central to maintaining scalable reasoning breadth.

  • 8 CONCLUSION: RLVR-induced solution narrowing is fundamentally an access failure rather than an execution collapse.The conclusion attributes the finding to Countdown analyses that separate opening branch selection from downstream computation.
  • 8 CONCLUSION: 11×–16× larger likelihood shifts occur at the entrance than during downstream reasoning.This localization is supported by likelihood analysis and entrance-clamped rollouts.
  • 8 CONCLUSION: Up to 38% of solution coverage is restored at invariant pass@1 through entrance-targeted interventions.The conclusion specifically names late-layer parameter interpolation and structured entrance allocation.
  • 8 CONCLUSION: Exhaustive enumeration currently ties the direct access–execution analysis to Countdown.On standard math benchmarks, the first complete calculation is only an observable proxy for an entrance.
  • 8 CONCLUSION: The supplied-entrance measure is interventional and differs from execution conditioned on policy-selected entrances.It measures completion after an entrance is externally fixed, rather than after the policy selects it.

B.3 OPERATOR-CLASS COVERAGE OVER TRAINING

The appendix reports that breadth contracts progressively across checkpoints, including among problems that remain solvable, while decoding changes alone trade coverage against pass@1. Controlled entrance interventions instead reveal preserved downstream capability and improved completion after access is supplied.

  • B.3 OPERATOR-CLASS COVERAGE OVER TRAINING: Opening breadth declines gradually across checkpoints, so endpoint comparisons are not selection artifacts.The appendix explicitly states that the decline is gradual across checkpoints.
  • B.3 OPERATOR-CLASS COVERAGE OVER TRAINING: 31% of step-50 leaves reappear by step 275 while 64% of step-50-solved problems remain solvable.Retention falls steadily even though most of the same problems remain solvable.
  • B.3 OPERATOR-CLASS COVERAGE OVER TRAINING: On Sboth, solution coverage falls from 0.564 to 0.286 and operator-class coverage from 0.940 to 0.654.Sboth contains problems solved at both steps 50 and 275, showing contraction within persistently reachable problems.
  • B.3 OPERATOR-CLASS COVERAGE OVER TRAINING: Every Sloss problem remains unrecovered at every tested budget, while all 33 matched controls are recovered within 64 samples.The budget stress test uses up to 2,048 samples per problem and distinguishes persistent loss from ordinary sampling scarcity.
  • B.3 OPERATOR-CLASS COVERAGE OVER TRAINING: No decoding setting reaches step-50 coverage of 0.337; the best late-policy coverage is 0.194, with pass@1 at 0.238.The best setting is T=2.0, top-p=1.0, min-p=0.05.
  • B.3 OPERATOR-CLASS COVERAGE OVER TRAINING: Designated-family completion improves by +0.107 on supplied entrances, linking contracted access with preserved conditional execution.For low-access families, mean completion is 0.216 versus a 0.054 path-enumeration reference.

C.6 ACCESS-TAIL RECOVERABILITY

Access-tail recoverability is graded: some families absent from late free sampling reopen strongly when their entrance is supplied, while deeper zero-access families recover less readily. The public GRPO replication likewise shows that early-state interventions can restore breadth without requiring downstream arithmetic to be relearned.

  • Recoverability: 0.529 designated-family completion for unobserved families exceeded the 0.115 mean path-enumeration reference when minimal entrances were supplied.The corresponding completion was 0.475 at the earlier checkpoint, while retained families reached 0.658.
  • Recoverability: 0.087 mean supplied-entrance completion across 276 zero-access family cells shows that recoverability declines deeper in the access tail.Branches that previously carried appreciable policy mass reopen more readily than branches with less historical access.
  • Access-tail stability: 336 of 337 families absent from one 512-sample half remained absent from the other, with zero hits implying a 95% upper access bound of 0.0058.The zero-access designation therefore remains stable under a larger 1,024-sample budget.
  • Prefix controls: 0.617 completion before the first operator and 0.999 after it show that successful-trace prefixes retain substantial downstream execution information.Because successful natural-language and successful-trace conditions fail the predictability gate, the identification analysis uses solver-constructed entrances.
  • Entrance allocation: +0.148 any-valid success, +0.077 coverage, and +0.430 distinct entrance families show that allocating attempts across feasible entrances helps at step 275.At step 50, the corresponding contrasts are flat, indicating that the benefit depends on the later training state.

F.3 SAMPLING DEPTH AT 256 SAMPLES

At 256 samples, larger-budget accuracy approaches saturation while early-calculation and trace diversity remain substantially lower under RLVR. The pattern generalizes across model scales and benchmarks, although longer reasoning horizons add downstream execution constraints.

  • Sampling depth: Large-budget accuracy nears saturation for both policies while first-calculation and trace diversity remain several-fold lower under SimpleRL.This comparison uses Qwen2.5-7B evaluations on GSM8K and MATH500 with 256 samples over 200 problems per benchmark.
  • Execution limits: The depth-by-late interaction is γ = −0.0152, showing that the late-policy completion advantage decreases as more downstream computation remains.This result comes from 534 four-number Countdown problems evaluated at four successive supplied depths.
  • Execution limits: Entrance selection remains the primary gatekeeper even though downstream execution becomes a compounding second bottleneck on extended reasoning horizons.The multi-benchmark analysis uses first-calculation entropy, same-trace likelihood profiles, and cross-checkpoint trace diversity across six benchmarks.
  • Scale comparison: RLVR raises single-sample accuracy while first-calculation entropy falls sharply for both 7B and 14B Qwen2.5 base–SimpleRL pairs.The matched GSM8K correct-only comparison agrees after conditioning on correctness.
  • Scale comparison: RL after distillation at 1.5B raises single-sample accuracy while large-budget accuracy and first-calculation structure stay level.Entropy changes point in opposite directions on GSM8K and MATH500, indicating a different breadth profile.

G.3 SUPERVISED FINE-TUNING CONTROLS ON COUNTDOWN

Supervised fine-tuning with entrance-diverse solutions preserves substantially more Countdown coverage than late RL, showing that learning correct reasoning paths alone does not require solution-space contraction. Continuing GRPO from a diverse SFT initialization reintroduces concentration.

  • SFT controls: 0.471 coverage with one supervised path per problem exceeds the late RL policy’s 0.268 coverage.Coverage increases monotonically with supervision breadth k: 0.471 → 0.551 → 0.709 → 0.766.
  • SFT controls: Coverage rises monotonically from 0.471 to 0.766 as the number of supervised solutions per problem increases.The training set uses solver-enumerated solutions selected with an entrance-diverse round-robin strategy.
  • Interpretation: SFT preserves more Countdown solution coverage than late RL, indicating that contraction is not an unavoidable byproduct of learning correct reasoning paths.The comparison uses the exact base model of the GRPO series and held-out evaluation tasks.
  • GRPO continuation: 0.085 nats mean rollout response entropy after continued GRPO shows that concentration re-emerges from a high-diversity k=4 SFT initialization.Continuation used 500 GRPO steps, with the entropy statistic averaged over the final 100 updates.

H.3 COMPANION ANALYSIS ON MATH500 (ITEM 451)

On MATH500 Item 451, direct RLVR improves pass@1 but sharply narrows the correct first-step support, whereas staged SFT–DPO–RLVR preserves or expands opening diversity. All models nevertheless reach pass@64 = 1.

  • MATH500 Item 451: pass@64 = 1 for all models, while Qwen SimpleRL raises pass@1 from the base model’s 0.797 to 1.000.The item’s gold answer is 639/40.
  • MATH500 Item 451: Qwen SimpleRL collapses the correct first-step support from 10 forms to 2, with H = 0.525 nats and 50/64 rollouts using 15 + 0.975.The Qwen Base reference has H = 1.590 nats and pass@1 = 0.797.
  • MATH500 Item 451: The staged OLMo-3 progression expands correct opening forms from 7 to 13 and raises entropy from 1.678 to 1.804 nats across SFT, DPO, and RLVR.This provides a contrasting breadth profile to direct RLVR on the same item.
Loading 2608.29188v1…