Source-linked AI summary
Guided-RLVR: Guided Exploration in Reinforcement Learning with Verifiable Rewards
Bo Yang, Zishuai Zhang, Weihua Li, Qinnan Zhang, Sijia Wen, Hainan Zhang
TL;DR
RLVR can concentrate on high-probability reasoning paths, leaving alternative solutions underexplored. Guided-RLVR steers rollout sampling with a state-conditioned controller, and it achieves the highest average performance across five benchmarks for every evaluated backbone–algorithm pairing.
Problem
RLVR rollouts can concentrate on high-probability reasoning patterns and leave large regions of the reasoning space unexplored (Tomihari and Sato 2026; Yue et al. 2025; Wu et al. 2025).
Source: §1 · p. 1Method
Guided-RLVR jointly optimizes a state-conditioned controller and policy, selecting semantic steering tokens at high-entropy sentence boundaries to perturb rollout sampling.
Source: §1 · p. 2Results
Takeaways & Limitations
Higher trajectory diversity and broader problem coverage without longer generations support targeted perturbation as an effective approach to exploration in RLVR.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) has shown strong empirical gains in improving the reasoning ability of large language models (LLMs) on tasks such as mathematics and programming. However, recent studies have shown that RLVR rollouts often collapse onto a narrow set of reasoning trajectories, limiting the explored reasoning space. Motivated by findings that thinking tokens at critical steps serve as critical structural elements in LLM reasoning pathways, we propose Guided-RLVR, a novel approach that augments RLVR with a lightweight controller. The controller strategically injects semantically meaningful steering tokens into the sampling distribution, encouraging the model to approach problems from multiple reasoning perspectives and thereby expanding the exploration space. Experiments on mathematical reasoning and code generation benchmarks show that Guided-RLVR consistently outperforms existing RLVR methods while producing more diverse reasoning trajectories.
1 Introduction
RLVR can concentrate on high-probability reasoning patterns, leaving alternatives unexplored (Tomihari and Sato 2026; Yue et al. 2025; Wu et al. 2025). Guided-RLVR steers sampling at uncertain reasoning points and improves benchmark performance while broadening sampled trajectories.
1 Introduction
- In 200 MATH instances, guided steering spreads rollout representations and raises total variance by nearly fourfold versus vanilla RLVR.The study uses 32 responses per problem; t-SNE is for visualization, while variance is computed before projection.Source: §1 · p. 1, Figure 1 · p. 1
- Thinking tokens such as discourse connectives and reflective markers occur at reasoning turning points and can be promoted to guide exploration.These tokens are described as carrying disproportionately high mutual information with respect to the final answer.Source: §1 · p. 2
- Guided-RLVR uses a state-conditioned controller to select semantic steering tokens at high-entropy sentence boundaries and jointly optimizes the controller and policy.The controller boosts the selected token’s logit before sampling; joint optimization uses group-relative advantages.Source: §1 · p. 2
- Across five benchmarks, Guided-RLVR achieves the highest average performance for all four backbone–RLVR algorithm combinations, outperforming vanilla RLVR, ENT, and REP.The evaluated benchmarks are MATH, AIME2024, AIME2025, AMC2023, and Code-R1.Source: §1 · p. 2
- Under a finite rollout budget, low-prior trajectories are rarely sampled, and a union-bound analysis links their vanishing learning signals to sampling probabilities.The paper argues that standard RLVR mainly reweights trajectories already favored by the current policy.Source: §1 · pp. 1–2
- Guided-RLVR delivers stronger pass@k scaling and more diverse trajectories without increasing generation length.These results suggest broader exploration of the reasoning space.Source: §1 · p. 2
2 Related Work
Prior RLVR exploration methods shape optimization signals or target selected positions, while other guided-rollout approaches use reference reasoning, demonstrations, or inference-time steering. Guided-RLVR learns semantic token interventions during RL rollout generation.
2.1 Exploration in RLVR
- Exploration methods shape optimization signals through entropy, diversity, perplexity, uncertainty, or likelihood-based rewards; Guided-RLVR instead intervenes in rollout sampling.ENT uses a clipped, gradient-detached entropy term; REP and DIVER use representation-based and sequence-level diversity incentives, respectively.Source: §2.1 · p. 2
- Targeted exploration methods focus on high-entropy tokens or selected positions, motivating Guided-RLVR’s learned promotion of semantic steering tokens at high-entropy sentence boundaries.The cited approaches include Wang et al. (2025), SIREN (Jiang et al. 2025), and Lp-Reg (Huang et al. 2026a).Source: §2.1 · p. 2
2.2 Guided Rollouts and Reasoning-Time Steering
- Guided-RLVR differs from reference-step, expert-demonstration, and inference-time steering approaches by jointly optimizing semantic-token steering with the reasoning policy during RL rollouts.Related approaches include G2RPO-A, ICPO (Huang et al. 2026b), Soft Reasoning (Zhu et al. 2025), ESamp (Zeng et al. 2026), and Qian et al. (2026).Source: §2.2 · p. 2
3 Methodology
Standard RLVR reweights sampled high-reward trajectories but remains tied to its rollout distribution, limiting access to low-prior reasoning paths. Guided-RLVR perturbs sampling with a learned Logic Controller and adds a log-perplexity exploration reward, jointly optimizing the controller and policy.
3.1 Problem Formulation
- When valid low-prior trajectories have sampling probability far below 1/G, or zero under top-p truncation, finite-budget rollouts rarely or never sample them, and their learning signal vanishes.The union-bound analysis connects the probability of observing such trajectories to their sampling probability.Source: §3.1 · p. 3
3.2 Guided-RLVR
- Guided-RLVR uses a lightweight controller to perturb sampling toward semantically meaningful steering tokens, exposing the policy to broader candidate reasoning trajectories.The controller conditions on policy hidden states and is queried at sentence boundaries when next-token entropy exceeds a dynamic threshold.Source: §3.2 · p. 3, Logic Controller · p. 3
- A log-perplexity bonus measures the reduction in answer log-perplexity when conditioned on the guided reasoning chain, supplementing sparse correctness feedback.The reward combines task correctness with a scaled, normalized and clipped bonus; the bonus applies only to correct math rollouts where a promoted token was sampled.Source: Exploration via a Log-Perplexity Reward · p. 3, Figure 2 · p. 4
- Guided and unperturbed completions share group-relative advantages for separate policy and controller updates, while policy optimization evaluates sampled completions with unboosted logits.The rollout group mixes guided completions with unperturbed baselines.Source: Joint Optimization of Policy and… · p. 4
4 Experiments
Experiments compare Guided-RLVR with vanilla and exploration-enhanced baselines across two backbones, two RLVR algorithms, and five benchmarks. The reported results cover benchmark performance, exploration behavior, component ablations, and compute cost.
4.1 Experimental Setup
- Experiments use Qwen2.5-7B-Instruct and Qwen3-4B, with 2,000 training examples per dataset and pass@128 or pass@16 evaluation budgets.Baselines include vanilla GRPO and DAPO, plus ENT and REP; the training data, schedule, decoding, and evaluation are held fixed across variants.Source: Backbone Models · pp. 4–5, Baselines · p. 5
4.2 Main Results
- Guided-RLVR achieves the highest average score in every backbone–algorithm pairing, with margins of 1.1–2.8 points over the strongest baseline.The comparison covers two backbones, five benchmarks, and GRPO and DAPO.Source: §4.2 · p. 5, Table 1 · p. 6
4.3 Exploration Analysis
- At k=128, Guided-RLVR reaches TD=0.70 and ED=0.16 versus vanilla GRPO’s TD=0.60 and ED=0.13, while its pass@k advantage persists across budgets.On AIME2024, AIME2025, and AMC2023, it is best or tied-best at pass@128 for every backbone–algorithm pairing.Source: §4.3 · p. 5, §4.2 · p. 5, Figure 3 · p. 5
- Guided-RLVR stays close to vanilla GRPO in generation length on MATH, is shorter on the competition benchmarks, and uses fewer tokens than ENT and REP on every benchmark.The reported comparison uses four mathematical benchmarks with Qwen2.5-7B-Instruct.Source: §4.3 · p. 6, Table 2 · p. 6
4.4 Ablation and Sensitivity
- Random steering lowers pass@1/pass@16 from 65.6/80.6 to 63.4/77.2, while always choosing “Wait” lowers them to 63.9/78.2.Removing the entropy threshold also reduces both metrics; the authors report decreases of 3.5 points in pass@1 and 1.4 points in pass@16.Source: §4.4 · pp. 6–7
- Removing the log-perplexity reward lowers pass@1/pass@16 from 65.6/80.6 to 62.0/79.6, and granting the bonus unconditionally lowers them to 64.6/79.4.The α=0.01 setting has the highest peak accuracy and outperforms larger coefficients at most checkpoints; excluding steering-attempt positions from policy loss lowers scores from 65.6/80.6 to 63.8/80.0.Source: §4.4 · pp. 7–8, Hyperparameter Sensitivity · p. 8, Figure 5 · p. 8
- Without supervised controller initialization, Guided-RLVR reaches pass@1=64.0 and pass@16=79.60; adding cold start improves these by 1.56 and 1.00 points, respectively.The no-cold-start variant exceeds ENT and REP on pass@16 but remains below both on pass@1.Source: Baselines · p. 5, §4.4 · p. 7
4.5 Computational Cost
- Guided-RLVR requires 17.92 GPU-hours versus 15.92 for vanilla GRPO, an added 2.00 GPU-hours or 12.6% overhead.The timing run uses the controller without supervised cold start.Source: §4.5 · p. 8, Table 4 · p. 8
5 Conclusion
Guided-RLVR uses a state-conditioned controller to steer generation toward underexplored reasoning trajectories. The appendix details its token-bias calculation, vocabulary, intervention triggers, and hidden-state input.
5 Conclusion
- The state-conditioned controller steers generation toward underexplored reasoning trajectories that standard RLVR rarely reaches under finite rollout budgets.The controller and policy are jointly optimized to learn where and how to redirect generation.Source: §5 · p. 8
A Closed-Form Derivation of Perturbation Injection
- A nonnegative logit bias raises the selected token toward a target probability, while clipping prevents intervention when its original probability already exceeds the target.The perturbed logits then pass through the standard decoding pipeline.Source: A Closed-Form Derivation of… · pp. 10–11
B.1 Steering Token Vocabulary
- The controller selects among six semantic steering categories, with surface tokens sampled according to their frequency in cold-start DeepSeek-R1 traces.More common connectives are injected more often, while rarer variants remain reachable.Source: §B.1 · p. 11, Table 5 · p. 11
B.2 Trigger Conditions and Entropy-Threshold Schedule
- The controller is queried at sentence boundaries after a minimum interval and only when next-token entropy exceeds a dynamic threshold that grows stricter during training.A warm-up and sliding window of recent punctuation-position entropy values help calibrate the threshold.Source: §B.2 · p. 11
B.3 Hidden-State Aggregation
- For its input, the controller aggregates the hidden states of the last L model layers at the last generated token position.L is a hyperparameter.Source: §B.3 · p. 11
C.1 Cold-Start Training
The appendix describes controller cold-start and online updates, policy-loss evaluation, intervention settings, exploration rewards, and verification-based data processing.
C.1 Cold-Start Training
- Cold-start training uses sentence-boundary hidden states to predict six steering categories from rule-labeled DeepSeek-R1 traces.The class-balanced set contains 100,000 boundary samples per category; the controller continues to update online with the policy during RL training.Source: §C.1 · pp. 11–12
C.2 Policy Loss Evaluation
- Policy loss uses the sampled completion unchanged and evaluates current and old unboosted policies, without correcting for the guided sampling distribution.GRPO averages clipped terms across valid tokens and completions, whereas DAPO normalizes by the generation batch’s completion-token count before validity masking.Source: §C.2 · p. 12
C.3 Controller Learning Rates and Intervention Details
- Interventions boost the selected token’s probability to 0.5, and controller learning rates vary by task; updates are skipped without a valid successful intervention.The exploration baselines use the entropy coefficient of ENT and representation-bonus scale of REP.Source: §C.3 · p. 12
C.4 Log-Perplexity Reward Parameters
- For math tasks, a clipped, scaled log-perplexity bonus is added to task reward; code generation uses correctness reward alone.The bonus compares the same answer under the guided reasoning context and a counterfactual context without that reasoning chain.Source: §C.4 · p. 12
C.5 Data Processing and Reward Design
- Prompts require separate think and answer tags, and automatic verification combines format and answer-correctness rewards.A soft overlength penalty may also be included for DAPO.Source: §C.5 · p. 12
C.6 Training Hyperparameters
Training configurations are specified for each backbone and dataset combination, with shared hyperparameters reported separately.
C.6 Training Hyperparameters
- Model- and dataset-specific training configurations are provided alongside hyperparameters held fixed across configurations.The configurations were tuned to balance training stability, memory efficiency, and generation quality.Source: §C.6 · p. 12, Table 6 · p. 12, Table 7 · p. 13
D Evaluation Protocol
Evaluation uses an unbiased pass@k estimator with benchmark-specific sampling budgets. The controller’s action distribution remains active across tokens after an initial adjustment period.
D.1 Pass@k Estimator
- Pass@k is computed with the unbiased estimator of Chen et al. (2021), using 128 samples for AIME2024/2025 and AMC2023 and 16 for MATH and Code-R1.Source: §D.1 · p. 13
D.2 Sampling Budget K
- Sampling budgets use larger K for the few, extremely hard AIME and AMC benchmarks, while MATH and Code-R1 are evaluated at K = 16.The stated rationale is to reveal reachable solution space for the harder, smaller benchmarks and estimate the others adequately.Source: §D.2 · p. 13, Table 8 · p. 13
E.1 Controller Behavior
- After an initial adjustment period, the controller’s action distribution stabilizes without collapsing to one token; all nine tokens remain active, with “Because” selected about one third of the time.Post-intervention accuracy varies across tokens, and the random-selection ablation indicates gains do not arise from token injection alone.Source: §E.1 · p. 13
E.2 Entropy-Threshold Training Dynamics
The entropy threshold keeps later training entropy lower and is associated with better pass@k at the selected checkpoints. Without the threshold, entropy rises after its initial decrease but does not improve solution coverage at the evaluated sampling budgets.
E.2 Entropy-Threshold Training Dynamics
- Removing the threshold lowers pass@1 from 65.6% to 62.1% and pass@16 from 80.6% to 79.2% at the respective selected checkpoints.The higher training entropy of the ablation does not translate into better solution coverage under the evaluated sampling budgets.Source: §E.2 · p. 13
- Without the threshold, entropy rises overall after its early decrease and ends above its initial level; with it, later entropy remains lower and fluctuates within a narrower range.The unthresholded run has a temporary reversal in the middle of training.Source: §E.2 · p. 13
- The threshold restricts steering to sentence boundaries with unusually high next-token uncertainty, making interventions more selective.The entropy dynamics and pass@k results are consistent with this selectivity helping control perturbations while preserving useful exploration; entropy curves alone do not establish changes in trajectory diversity or correctness.Source: §E.2 · p. 13
E.3 Intervention-Position Loss Masking
The intervention-position ablation excludes steering-attempt positions from the policy loss, while retaining guided generation and the controller update. On MATH with Qwen2.5-7B-Instruct and GRPO, this masking has lower pass@1 and pass@16 than Full loss, though the comparison does not isolate the effect of successful steering-token emissions.
E.3 Intervention-Position Loss Masking
- The trigger mask counts every steering attempt on a non-baseline completion, whereas the controller update uses a successful-hit filter.Full loss retains policy loss at all valid completion positions; the ablation excludes steering-attempt positions.Source: §E.3 · p. 13
- Masked loss excludes intervention positions; GRPO averages over remaining valid tokens, changing term inclusion and weighting, while DAPO retains its generation-token normalizer.Both variants use the unboosted policy ratio in Equation (2), not guided sampling probabilities in the denominator.Source: §E.3 · p. 13
- Masked loss achieves pass@1 of 63.8 and pass@16 of 80.0, compared with 65.6 and 80.6 for Full loss on MATH with Qwen2.5-7B-Instruct and GRPO.The decreases are 1.8 and 0.6 percentage points, respectively.Source: §E.3 · pp. 13–14
- The comparison supports Full loss inclusion and weighting together, but does not isolate successfully emitted steering tokens or correct the guided behavior-distribution mismatch.Guided generation and the controller update are retained, and later token losses still condition on generated prefixes.Source: §E.3 · p. 14
F Theoretical Analysis
The analysis splits the gradient expectation over the unperturbed rollout distribution into contributions from trajectories outside and inside A_x. Under bounded-gradient assumptions, the contribution from A_x is bounded by its probability and vanishes when that probability is sufficiently small or zero.
F.1 Gradient Bound for Low-Prior Trajectories
- The gradient of Equation (1) is split into contributions from trajectories outside and inside A_x under the unperturbed rollout distribution.The inside-A_x contribution is the second term in the split.Source: §F.1 · p. 14
- With bounded rewards, clipped importance ratio, and policy-score gradient, the per-trajectory gradient norm is bounded by a constant C independent of y.The bound follows under standard regularity assumptions.Source: §F.1 · p. 14
- The gradient contribution from A_x is bounded by C · P(A_x | x), so its learning signal vanishes when P(A_x | x) ≪ 1/G or is zero under top-p truncation.This mirrors the sampling bound discussed earlier.Source: §F.1 · p. 14