PaperlayerView exact revision on arXiv

Source-linked AI summary

The Model Knows Another Way: Strategy Switching for Effective RLVR Exploration

Jin Cui, Xinyue Long, Boran Zhao, Pengju Ren, Hao Dong

arXiv:2609.33085v1cs.AI

TL;DR

RLVR exploration can fail when finite rollouts miss viable strategies, leaving hard problems with no learning signal. PSRA allocates a fixed rollout budget across unguided and strategy-conditioned prompts, preserving useful routes while transferring successful guided behaviors. Across training settings, it improves reasoning performance and reduces saturation while broadening exploration.

  • Problem

    On hard problems, uniformly unsuccessful rollout groups provide little learning signal, and finite sampling may miss viable strategies already supported by the model.

    Source: §1 · pp. 1–2

  • Method

    PSRA allocates a fixed rollout budget across unguided and strategy-conditioned arms, preserving useful strategy routes while transferring successful guided behaviors to the unguided policy.

    Source: §1 · p. 2

  • Results

    PSRA recovers 26% of initially unsolved problems versus 17% for GRPO, while improving reasoning across settings, reducing saturation, and strengthening transfer.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Pretrained strategy repertoires provide an effective interface for RLVR exploration, with PSRA sustaining broader exploration as training proceeds.

    Source: §7 · p. 10

    Evaluation is limited to the Qwen2.5 family up to 7B; broader model families and larger models remain to be validated.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) is often limited by insufficient exploration: difficult problems can yield uniformly incorrect rollout groups and therefore little learning signal. We show that such failures need not reflect missing capability. Instead, finite sampling often concentrates on a problem-specific dominant reasoning strategy while leaving alternative strategies already supported by the model unexplored. Moreover, the accessibility of these strategies evolves during RL: some are internalized into autonomous behavior, while others become difficult to elicit before being absorbed. Motivated by these observations, we introduce Problem--Strategy Rollout Allocation (PSRA), which treats unguided and strategy-conditioned prompts as competing exploration arms and uses Bayesian sequential allocation to direct a fixed rollout budget toward arms most likely to yield informative, non-saturated groups. A preservation objective keeps useful strategy-conditioned routes accessible while successful guided behaviors are transferred to the unguided policy. Across Qwen2.5 models from 1.5B to 7B and two RL training corpora, PSRA consistently improves reasoning performance, reduces dead saturation, strengthens out-of-distribution transfer, and maintains larger gains under increased inference budgets.

1 INTRODUCTION

RLVR can produce uninformative all-failure groups when finite sampling misses viable reasoning strategies. PSRA allocates rollouts across unguided and strategy-conditioned prompts, preserving useful guided routes while transferring successful behaviors to the unguided policy.

1 INTRODUCTION

  • Finite autonomous sampling often concentrates on a problem-specific dominant strategy, so an all-failure group can reflect strategy-selection failure rather than missing capability.On initially all-incorrect groups, switching to another strategy rescues problems more often than continuing the dominant strategy or default sampling.Source: §1 · pp. 1–2
  • Strategy accessibility changes during RL: some routes become autonomous, while others lose steerability before their behaviors are absorbed.Guided exploration therefore requires keeping useful conditional routes available long enough for their behaviors to transfer.Source: §1 · p. 2
  • PSRA treats unguided and strategy-conditioned prompts as competing arms, allocating a fixed rollout budget toward informative groups and preserving routes while transferring successful guided trajectories.Its allocation uses Bayesian estimates of success and expected non-saturated groups.Source: §1 · p. 2
  • PSRA recovers 26% of initially unsolved problems versus 17% for GRPO, while reducing dead saturation and increasing advantages at larger inference budgets.The experiments span model scales and training distributions.Source: §1 · p. 2

2 RELATED WORK

Prior RLVR approaches address saturated learning signals or add guidance, but PSRA treats a model's own strategy repertoire as an online exploration resource.

Learning signals in RLVR

  • GRPO receives no reward-driven update from all-correct or all-incorrect groups; DAPO filters saturated groups and resamples prompts to maintain informative batches (Shao et al., 2024; Yu et al., 2025).PSRA instead treats an all-failure prompt as potentially recoverable through another reasoning strategy already available to the model.Source: §2 · p. 2

Guided RLVR exploration

  • BREAD branches from partial expert anchors (Zhang et al., 2025b), HINT uses abstract Meta-Hints (Wang et al., 2025), NudgeRL conditions rollouts on strategy-level contexts (Lee et al., 2026b), and Uniqueness-Aware RL upweights correct trajectories instantiating rare observed strategies (Hu et al., 2026); these methods largely treat guidance as given or act on sampled patterns.Source: §2 · p. 3
  • PSRA builds a reusable strategy repertoire from the model, estimates problem-specific exploration value online, and preserves useful strategy conditions as the policy changes.This contrasts with approaches that act on patterns already sampled.Source: §2 · p. 3

3 FROM STRATEGY TO PROBLEM-CONDITIONED EXPLORATION

The paper identifies steerable pretrained strategies, shows that problem-level sampling can miss useful alternatives, and develops measures of uncovered opportunities and changing strategy accessibility.

Reusable pretrained strategy repertoire

  • Five of seven strategy candidates pass behavioral certification; natural-language instructions retain 88% of full-exemplar displacement, and the same strategy families recur across Qwen2.5 scales.The repertoire is clustered from rollout representations and candidates are tested for reliable redirection toward their associated regions.Source: §3 · pp. 3–4

Problem-conditioned strategy selection

  • Switching to the best alternative strategy rescues more initially all-incorrect problems than continued dominant-strategy or unguided sampling, showing that zero-reward groups can reflect finite-budget selection failure.Autonomous generation concentrates most strategy-associated mass on one dominant strategy for a problem.Source: §3 · p. 4

Predicting missed strategy opportunities

  • A planning-state predictor estimates a distribution over frozen DPMM components and performs substantially better than an early-layer content representation, including on the most novel trajectories.Its novelty-weighted log score emphasizes trajectories far from the observed rollout group.Source: §3 · p. 4
  • The five certified strategies cover most predicted gap mass, substantially exceeding matched random component sets, with coverage saturating rapidly as strategies are added.This supports a small strategy basis for the structured exploration opportunities exposed by finite sampling.Source: §3 · p. 4

Strategy accessibility evolves during RL

  • During RL, some strategies become internalized into autonomous behavior, while others lose conditional accessibility before their behaviors are absorbed.Internalization and residual loss of conditional accessibility are distinct contributors to changing strategy displacement.Source: §3 · p. 4
  • The problem–strategy pair is the unit of structured exploration: it combines remaining coverage opportunity, current utility, and whether the strategy remains accessible.This links the uncovered-mass estimate and strategy accessibility to the allocation decision.Source: §3 · p. 4

4 PROBLEM–STRATEGY ROLLOUT ALLOCATION

PSRA treats unguided and strategy-conditioned prompts as competing arms for allocating a fixed rollout budget across problem–strategy pairs. It uses Bayesian estimates of arm success and expected non-saturated groups, while learning from guided trajectories and preserving strategy accessibility.

4 PROBLEM–STRATEGY ROLLOUT ALLOCATION

  • PSRA allocates a fixed rollout budget across unguided and strategy-conditioned arms for each problem, whose success probabilities evolve during training.The unguided condition is ∅, and each strategy condition supplies a context for the problem.Source: §4 · p. 5

4.1 BAYESIAN ALLOCATION OVER PROBLEM–STRATEGY ARMS

  • Bayesian priors combine problem difficulty, global arm effectiveness, and uncovered strategy opportunity; allocation greedily favors arms with the largest expected gain in non-saturated groups.The uncovered-opportunity estimate removes components reached by probed arms and discounts strategies whose steerability has degraded.Source: §4.1 · pp. 5–6

4.2 LEARNING FROM STRATEGY-CONDITIONED ROLLOUTS

  • Within-arm reward groups with zero variance are discarded, while guided trajectories use strategy conditioning to produce informative updates for the unguided policy used at inference.The update uses within-arm advantages rather than a common baseline across strategy conditions.Source: §4.2 · p. 6

4.3 MAINTAINING STRATEGY ACCESSIBILITY

  • A small on-context PPO preservation term keeps guided strategy conditions usable as successful guided trajectories are transferred into autonomous behavior.Its coefficient can increase as measured strategy displacement declines.Source: §4.3 · p. 6

5 EXPERIMENT

Across Qwen2.5 model scales and training corpora, PSRA improves reasoning results and broadens strategy coverage. The experiments compare its performance with GRPO and other baselines, examine training dynamics, and ablate guidance, allocation, and preservation choices.

5.1 EXPERIMENTAL SETUP

  • Experiments use Qwen2.5-Instruct models at 1.5B, 3B, and 7B, train mainly on DAPO-Math-17K-Processed, and also test DeepScaleR-40K at 7B.Evaluation covers ID mathematics, OOD reasoning and knowledge benchmarks, and pass@k up to 128; baselines include GRPO and guided or off-policy methods, with matched generation budgets.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • PSRA beats vanilla GRPO by 2.3 points on DAPO-Math-17K and 1.3 points on DeepScaleR-40K at 7B, and exceeds the strongest competing baseline by 1.2 points in both settings.At 3B and 1.5B, it improves over vanilla GRPO by 0.9 and 1.7 points, respectively.Source: Table 1 · p. 7, §5.2 · p. 7
  • PSRA improves the OOD average over GRPO by 3.3, 4.7, and 0.4 points for DAPO-trained 7B, 3B, and 1.5B models, and by 1.7 points for DeepScaleR-trained 7B.Its advantage generally widens with larger inference budgets, especially on AIME.Source: Table 1 · p. 7, §5.2 · pp. 7–8, Figure 3 · p. 8
  • At matched rollout budgets, PSRA learns faster and more stably, maintains higher entropy, and reduces dead-saturated groups; these changes appear earlier at 7B than 1.5B.The best-strategy arm remains strong as the unguided policy approaches it, consistent with preservation keeping routes callable while transfer internalizes them.Source: Figure 4 · p. 8, §5.2 · p. 9
  • On initially failed problems, PSRA recovers 26%, versus 17% for GRPO and 19% for Fixed Strategy; 73% of its recovered problems are first solved within an alternative strategy footprint.Its unguided policy shifts more probability mass toward alternative strategies rather than simply increasing diversity uniformly.Source: §5.2 · p. 8, Figure 3 · p. 8

5.3 ABLATION STUDIES

  • Random fixed guidance underperforms PSRA, recovering 19% of initially unsolved problems versus 26%, with higher saturation and weaker training progress.The authors conclude that effective exploration requires steering toward strategies both accessible to the model and useful for the problem.Source: §5.3 · p. 9
  • Performance remains close with four or three strategies but drops sharply with only two, supporting a compact, sufficiently diverse strategy set.The two-strategy variant can perform worse than the single-strategy variant.Source: §5.3 · p. 9
  • Randomly assigning extra rollouts performs worst among allocation variants and can underperform B=0, while Bayesian allocation improves with added budget; removing preservation weakens final performance.Beyond the default B=10, further budget yields only marginal gains at proportionally higher rollout cost; without preservation, entropy decays faster and saturation increases.Source: §5.3 · p. 9, Figure 5 · p. 9

6 DISCUSSION AND LIMITATIONS

The experiments are limited to the Qwen2.5 family up to 7B, while PSRA reaches comparable reward levels in fewer training steps with little online overhead and stronger exploration.

6 DISCUSSION AND LIMITATIONS

  • Experiments cover multiple model scales and training corpora but only the Qwen2.5 family up to 7B; broader families and larger models remain unvalidated.PSRA reaches comparable reward levels in markedly fewer training steps while maintaining stronger exploration and introducing little online overhead.Source: §6 · pp. 9–10

7 CONCLUSION

The paper frames RLVR exploration failures as finite-budget strategy-selection failures rather than missing capability, and presents PSRA as a way to allocate rollouts across strategies while preserving and transferring useful guidance.

7 CONCLUSION

  • RLVR exploration failures can arise from finite-budget strategy selection rather than missing capability.PSRA allocates rollouts over problem–strategy arms, preserves useful strategy conditions, and transfers successful guided behaviors into the unguided policy; experiments report improved reasoning, reduced saturation, and broader exploration.Source: §7 · p. 10

A.1 DETAILED STRATEGY DISCOVERY AND PROBLEM-CONDITIONED EXPLORATION

This appendix adds details on five analyses introduced in Section 3 and specifies the representation space used for most analyses.

A.1 DETAILED STRATEGY DISCOVERY AND PROBLEM-CONDITIONED EXPLORATION

  • The appendix details strategy discovery, behavioral certification, problem-conditioned rescue analysis, the coverage predictor, and longitudinal accessibility analysis.Unless otherwise stated, representation-space analyses use the frozen base-model coordinate system constructed before RL training.Source: §A.1 · p. 12

A.1.1 STRATEGY REPERTOIRE DISCOVERY

The strategy repertoire is built by clustering frozen base-model rollout representations and merging components with the same high-level solution procedure; behavioral certification yields the final strategy set.

A.1.1 STRATEGY REPERTOIRE DISCOVERY

  • Rollout representations pool hidden states from early and intermediate response regions after re-encoding each complete problem–response sequence in its original context.The extractor is frozen at the base checkpoint so representation-space changes are not confounded with policy-behavior changes.Source: §A.1.1 · p. 12
  • A Dirichlet process mixture model partitions rollout representations into fine-grained components, which remain the coordinate system for analysis.Semantic strategies are unions of components, preserving a finer-grained basis for coverage estimation.Source: §A.1.1 · p. 12
  • Five of seven semantic strategy candidates survive behavioral certification to form the final strategy set.Candidates are formed by merging stable components that exhibit the same high-level solution procedure.Source: §A.1.1 · p. 12

A.1.2 BEHAVIORAL CERTIFICATION AND STRATEGY FOOTPRINTS

The section behaviorally certifies candidate strategies through exemplar and natural-language steering tests against unguided and placebo controls, then defines strategy footprints from the components most associated with invoking each strategy.

A.1.2 BEHAVIORAL CERTIFICATION AND STRATEGY FOOTPRINTS

  • Candidates are tested on a disjoint calibration set, and retained only when exemplar and natural-language conditioning consistently increase occupancy of the associated region over unguided and placebo controls.This distinguishes steerable reasoning directions from clusters that are only descriptively coherent.Source: §A.1.2 · pp. 12–13
  • A strategy footprint is the smallest prefix of components ranked by intervention lift whose cumulative probability accounts for at least 50% of strategy-conditioned mass within that strategy’s component set.The footprint targets components most causally associated with invoking the strategy, rather than every component sharing its semantic label.Source: §A.1.2 · p. 13
  • Natural-language instructions retain most of exemplar conditioning’s representation displacement, while matched placebo instructions produce negligible displacement.The authors consequently use natural-language instructions as lightweight strategy handles during RL.Source: §A.1.2 · p. 13

A.1.3 PROBLEM-CONDITIONED CONCENTRATION AND RESCUE

This section examines how finite-budget concentration on unsuccessful reasoning strategies can produce all-failure rollouts despite reachable successful behavior, and how changing strategies can expose successful trajectories. It also tracks strategy coverage and steerability during RL and describes compute-matched baselines and a consistent evaluation setup.

A.1.3 PROBLEM-CONDITIONED CONCENTRATION AND RESCUE

  • The oracle alternative strategy rescues substantially more all-failure groups, indicating that many reflect finite-budget strategy concentration rather than absent successful behavior.The oracle is computed from evaluation rollouts and is not available to PSRA during training; Figure 1d reports Rescue@B over increasing budgets.Source: §A.1.3 · p. 13

A.1.4 TRAINING THE PRE-GENERATION COVERAGE PREDICTOR

  • The pre-generation predictor estimates future reasoning-component distributions from a problem’s planning representation, while novelty-weighted evaluation tests whether it assigns probability to under-sampled regions.The early-layer baseline shares the predictor architecture and objective, isolating information in the planning representation; novelty weighting emphasizes trajectories poorly represented by finite autonomous samples.Source: §A.1.4 · p. 14

A.1.5 FROM PREDICTED COVERAGE TO STRATEGY OPPORTUNITIES

  • Strategy opportunities are high when predicted probability mass lies in unobserved regions reachable through a strategy’s footprint.The coverage tail comprises high-probability predicted components not already observed, which strategy footprints translate into executable exploration directions.Source: §A.1.5 · p. 14
  • Certified strategies cover the predicted coverage tail rapidly, saturating with few strategies and outperforming matched random subsets in structured coverage opportunities.The analysis greedily adds strategies by marginal gap mass covered and compares the resulting curve with matched random subsets of DPMM components.Source: §A.1.5 · p. 15

A.1.6 LONGITUDINAL STRATEGY ACCESSIBILITY

  • A shrinking strategy-conditioned displacement indicates internalization only when autonomous occupancy rises; otherwise, residual accessibility loss signals that the strategy has become harder to elicit.Measurements use frozen base-model DPMM footprints across RL checkpoints, separating autonomous movement into a strategy region from loss of conditional accessibility.Source: §A.1.6 · p. 15

A.1.7 ONLINE ESTIMATION OF THE PSRA PRIOR

  • The online prior uses all-arm probe outcomes, leave-one-arm-out problem difficulty, centered global arm effects, and a non-negative coverage coefficient; adaptive samples do not update its prior statistics.When coverage opportunity carries little information about success, shrinkage drives the coefficient toward zero, leaving problem and global-arm effects.Source: §A.1.7 · pp. 15–16

A.3 FIXED STRATEGY BASELINE

  • Discovered strategy prompts specify reusable high-level procedures without problem-specific solution information, while the fixed-strategy baseline uses five externally specified, model-agnostic instructions across settings.The fixed baseline excludes demonstrations, few-shot examples, solution prefixes, hints, and privileged solution information.Source: §A.3 · p. 17, §A.2 · p. 17

A.4 TRAINING AND IMPLEMENTATION DETAILS

  • PSRA allocates rollouts across six arms per problem, first probing each arm twice and then distributing an additional per-prompt-equivalent budget of 10 across problem–arm pairs.The additional allocation is capped at eight rollouts per arm; shared training settings include Qwen2.5-Instruct models at 1.5B, 3B, and 7B parameters.Source: §A.4 · pp. 18–19
  • Vanilla GRPO uses the same backbone, data, training configuration, optimization steps, and total generation budget as its corresponding PSRA run.The evaluation uses a shared decoding and verification pipeline, with 16 responses per problem for the main evaluation and up to 128 for the inference-budget analysis.Source: §A.4 · p. 20

A.5.1 EXAMPLE 1: FORMULA APPLICATION TO STEPWISE DECOMPOSITION

  • In the problem, the failed route prematurely reduces the parity condition to the even-even case, whereas the alternative decomposes the target event into exhaustive subcases before combining their contributions.The first scenario has one favorable outcome because the only even number is 2; in the second, each can choose from five odd numbers. The cited calculation reports 26/36 = 13/18.Source: §A.5.1 · p. 21

A.5.2 EXAMPLE 2: EQUATION SETUP TO FORMULA APPLICATION

  • In the triangle problem, formula application avoids the failed coordinate-based route by combining standard geometric formulas instead of solving for the circumcenter.The alternative uses the circumradius and the distance from the circumcenter to the chord.Source: §A.5.2 · pp. 21, 23

A.5 QUALITATIVE EXAMPLES OF STRATEGY SWITCHING

  • Two examples show the same model reaching a solution after switching from its failed default reasoning route to an alternative strategy.The examples contrast direct formula use with case decomposition, and coordinate-based equation setup with formula application.Source: §A.5 · p. 20, §A.5.2 · p. 23

Related papers

Loading 2609.33085v1…