PaperlayerView exact revision on arXiv

Source-linked AI summary

Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR

Chanuk Lee, Sangwoo Park, Minki Kang, Sung Ju Hwang

arXiv:2605.15726v1cs.AIcs.CL

TL;DR

RLVR exploration is limited by the cost of finding rare reasoning trajectories. NudgeRL uses lightweight strategy contexts and context-aware learning to induce diverse exploration, matching or surpassing larger-budget GRPO and oracle-guided baselines.

  • Problem

    RLVR exploration remains limited because discovering rare reasoning trajectories through brute-force rollouts is computationally prohibitive.

  • Method

    NudgeRL uses lightweight strategy-level prompts to induce diverse reasoning trajectories and context-aware advantages with distillation to transfer discoveries to the base policy.

  • Results

    NUDGERL outperforms GRPO with up to 8× larger rollout budgets and oracle-guided baselines across models and challenging math benchmarks.

  • Takeaways & Limitations

    Structured, diversity-oriented exploration is an effective alternative to brute-force rollout scaling and privileged-information baselines in RLVR.

  • Takeaways & Limitations

    A fixed context pool may become less informative as the trained policy adapts, motivating model-adaptive context generation.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable paradigm for improving the reasoning capabilities of large language models. However, its effectiveness is fundamentally limited by exploration: the policy can only improve on trajectories it has already sampled. While increasing the number of rollouts alleviates this issue, such brute-force scaling is computationally expensive, and existing approaches that modify the optimization objective provide limited control over what is explored. In this work, we propose NudgeRL, a framework for structured and diversity-driven exploration in RLVR. Our approach introduces Strategy Nudging, which conditions each rollout on lightweight, strategy-level contexts to induce diverse reasoning trajectories without relying on expensive oracle supervision. To effectively learn from such structured exploration, we further propose a unified objective, which decomposes the reward signal into inter- and intra-context components and incorporates a distillation objective to transfer discovered behaviors back to the base policy. Empirically, NudgeRL outperforms standard GRPO with up to 8 times larger rollout budgets, while outperforming oracle-guided RL baseline on average across five challenging math benchmarks. These results demonstrate that structured, context-driven exploration can serve as an efficient and scalable alternative to both brute-force rollout scaling and feasibility-oriented methods based on privileged information. Our code is available at https://github.com/tally0818/NudgeRL.

1 Introduction

RLVR’s exploration is constrained by the computational cost of brute-force rollout scaling and the limited control or supervision of existing alternatives. NUDGERL addresses this bottleneck with lightweight Strategy Nudging and a unified learning approach, outperforming larger-budget GRPO and oracle-guided baselines.

  • Exploration bottleneck: Brute-force rollout scaling improves rare-trajectory discovery but quickly becomes computationally prohibitive.This limitation motivates methods that improve exploration efficiency.
  • Prior approaches: Objective-level exploration methods broaden distributions but provide limited control over semantically meaningful reasoning strategies.Examples include entropy regularization and decoupled clipping.
  • Prior approaches: Oracle-guided methods improve correct-trajectory discovery but require expensive supervision and may restrict exploration diversity.These approaches rely on oracle solutions or intermediate reasoning steps and guide policies toward predefined successful trajectories.
  • NUDGERL: NUDGERL appends lightweight heuristic strategy prompts to inputs, forcing exploration across distinct reasoning modes without expensive oracle data.The prompts can specify mathematical strategies or reasoning keywords.
  • NUDGERL: Context-conditioned rollouts create grouped samples whose rewards mix trajectory quality with context-specific biases, making standard group-wise advantage estimation unreliable.The method must also address a mismatch between context-forced sampling and final policy learning.
  • Empirical results: 8× larger rollout budgets still leave GRPO behind NUDGERL, which also outperforms oracle-guided baselines.The result supports diversity-oriented exploration as an alternative to brute-force scaling and privileged information.

2 Preliminaries

RLVR samples groups of rollouts per prompt and uses verifiable rewards, while GRPO derives relative advantages from group-wise rewards within a PPO-style clipped objective. Its exploration bottleneck arises because correct but unsampled trajectories can hinder improvement, and long-tail coverage requires prohibitively large rollout budgets.

  • RLVR and GRPO: For each prompt x0, policy πθ samples N rollouts yi ∼ πθ(· | x0), each scored by verifiable reward R(x0, yi) ∈ {0, 1}.The rollouts form a group generated from the same prompt.
  • RLVR and GRPO: GRPO computes group-wise relative advantages from rollout rewards, using the group mean and standard deviation plus δ > 0 for numerical stability, without a learned value function.This contrasts with standard PPO, which typically uses a learned value function.
  • RLVR and GRPO: GRPO retains PPO’s clipped objective while optimizing with group-relative advantages.The policy is optimized with a PPO-style clipped objective.
  • Exploration Bottleneck: The core RLVR bottleneck is unexplored correct regions: when Qneg ≫ Qpos, unsampled correct-token mass Upos,2 can create a dominant negative force that hinders performance gain.The expected one-step improvement depends on sampled and unsampled correct and incorrect token moments, with SR ∈ [0, 1].
  • Exploration Bottleneck: Increasing rollout size N monotonically decreases expected unsampled second moment, but small-probability tokens decay slowly, making full long-tail coverage prohibitively expensive.Blindly scaling N therefore leaves long-tail correct trajectories unlikely to be sampled and motivates structured exploration.

3 NUDGERL

NudgeRL structures RLVR exploration with strategy-conditioned rollouts, context-aware credit assignment, and advantage-weighted distillation. The framework diversifies sampled reasoning trajectories, balances intra- and inter-context rewards, and transfers useful behaviors to the base policy for context-free inference.

  • Framework overview: NudgeRL combines Strategy Nudging, Inter-Intra Group Advantage, and a distillation-augmented RL objective for structured exploration and learning.Strategy Nudging induces diverse trajectories; Inter-Intra Group Advantage controls credit assignment; distillation transfers effective strategies.
  • Exploration motivation: Contexts act as lightweight controls that shift sampling toward rare trajectories without providing solutions, reducing the rollout cost of discovering them.If π(y|x, c) ≫π(y|x), contexts increase the probability of trajectory y and reduce the required discovery budget.
  • Strategy Nudging: Strategy Nudging assigns one sampled strategy context to each rollout, using uniform context selection and context dropout to enforce coverage while retaining compatibility with the original prompt.Varying conditioning across rollout indices creates input-level diversity rather than relying only on sampling from a single prompt.
  • Strategy Nudging: Under the same rollout budget, Strategy Nudging more often increases distinct reasoning structures than naive sampling, whose base-policy rollouts frequently collapse to similar patterns.The comparison uses 8 rollouts per prompt: 4 from each of 2 contexts for Strategy Nudging versus 8 from the original prompt for the baseline.
  • Inter-Intra Group Advantage: Inter-Intra Group Advantage combines intra-context trajectory quality with inter-context context reliability to prevent context-induced reward variation from distorting credit assignment.For λ ∈[0, 2], higher reward always receives higher advantage; λ < 1 favors lower-reward contexts, while λ > 1 favors higher-reward contexts.
  • Distillation-augmented objective: Advantage-weighted distillation updates the context-free base policy using context-conditioned trajectories while emphasizing only behaviors with high normalized advantage.The RL term improves exploration under contexts, whereas distillation projects successful improvements onto the base-prompt policy for inference without external context.

4 Experiments

Across five challenging math benchmarks, NudgeRL improves exploration efficiency by using strategy-level contexts, achieving stronger results than larger-budget GRPO and competing with oracle-guided methods. Ablations further show that context dropout, random context sampling, exploitation weighting, and moderate distillation are important for performance.

  • Main Results: NUDGERL achieves the best average performance on both models using only 8 rollouts per prompt.On Qwen3-4B-Instruct-2507, it reaches 0.489 average pass@1 versus 0.487 for GRPO at 32 rollouts and 0.451 at 64 rollouts; on Olmo3-7B-Instruct-SFT, it reaches 0.285 versus 0.281 at 32 rollouts.
  • Training Dynamics: NUDGERL improves pass@1 faster than GRPO variants and exceeds 0.42 pass@1 on AIME24/25 by 200 steps.GRPO variants remain around or below 0.41 and show slower or less stable gains as rollout budgets increase.
  • Exploration Analysis: Strategy Nudging increases the frequency of rare effective strategies, while the Inter-Intra Group Advantage reinforces reliable strategies after discovery.In the geometry example, NUDGERL exploited the shoelace-formula strategy, whereas GRPO sampled it only once and explored ineffective alternatives that caused all 32 trajectories to fail.
  • Ablations: A moderate dropout rate of pdrop = 0.5 consistently yields the best performance across benchmarks.Context dropout enables exploration beyond fixed contexts and stabilizes group-wise statistics, while pdrop = 0 restricts exploration and large values weaken context forcing.
  • Ablations: Random sampling consistently outperforms top-ranked context selection in pass@1 because it induces a broader distribution of reasoning trajectories.Top-ranked contexts provide more correctness but concentrate exploration on fewer strategies, whereas random sampling is preferable under limited rollout budgets.
  • Ablations: λ = 1.1 and λdistill = 0.1 achieve the best results, while removing distillation causes a clear performance drop.The results indicate that strategy nudging supplies diversity, exploitation prioritizes reliable contexts, and moderate distillation transfers context-discovered trajectories without over-constraining the policy.

5 Conclusion … A.2 Exploration in RLVR

NUDGERL structures RLVR exploration through lightweight strategy-level contexts and distillation-augmented reinforcement learning, achieving stronger performance than GRPO with up to 8× larger rollout budgets and oracle prefix-based baselines. The related work frames exploration as a central RLVR bottleneck, noting that rollout scaling is costly while objective-level methods lack reliable, structurally guided coverage.

  • 5 Conclusion: NUDGERL induces diverse reasoning trajectories with lightweight, strategy-level context-conditioned distributions and learns from them through a distillation-augmented RL objective.The framework is presented as structured exploration for RLVR.
  • 5 Conclusion: Up to 8× larger rollout budgets: NUDGERL achieves superior performance compared to GRPO and outperforms oracle prefix-based baselines across models.The conclusion reports both comparisons without specifying a separate metric value.
  • 5 Conclusion: Generating strategy-level contexts is a practical cost, but the offline process runs once with a lightweight LLM such as gpt-4o-mini and contexts can be reused across training runs.A fundamental limitation is that contexts are generated independently of the model being trained.
  • A.1 Reinforcement Learning with Verifiable Rewards: RLVR uses automatically verifiable signals, including exact mathematical answers and code test-case correctness, to optimize policies without dense human supervision.The passage presents RLVR as a scalable paradigm for improving LLM reasoning abilities.
  • A.1 Reinforcement Learning with Verifiable Rewards: GRPO replaces value-function estimation with group-wise comparisons among sampled rollouts, while later work adds decoupled clipping and alternative normalization for training stability.These methods build on the group-relative policy optimization formulation.
  • A.1 Reinforcement Learning with Verifiable Rewards: Exploration is a key RLVR bottleneck because the policy can improve only on trajectories it has already sampled, making insufficient exploration directly limit learning.The passage identifies exploration as fundamental across reasoning tasks and model scales.
  • A.2 Exploration in RLVR: Scaling sampled rollouts can improve performance by reducing unsampled-region probability mass, but it is computationally expensive and often impractical at scale.This is described as the straightforward approach to improving exploration.
  • A.2 Exploration in RLVR: Entropy regularization and decoupled clipping can steer updates toward exploration, yet they do not guarantee meaningful trajectory coverage and may produce incoherent or unproductive paths without structural guidance.Distribution-level exploration is characterized as stochastic, unconstrained, and lacking control over how the policy explores.

A.3 Usage of Privileged Information · B Details on Strategy Nudging

The paper identifies limitations of privileged-information methods and positions NudgeRL as targeting exploration diversity rather than only feasibility. Strategy Nudging generates lightweight mathematical hints, conditions rollouts on them, and evaluates resulting strategy diversity with an LLM judge.

  • A.3 Usage of Privileged Information: GRPO-like group-based advantage methods fail to provide informative learning signals when every rollout in a group is either correct or incorrect.
  • A.3 Usage of Privileged Information: Privileged information, including oracle prefixes or intermediate solutions, can enable successful trajectories that would otherwise be unreachable on hard problems.
  • A.3 Usage of Privileged Information: Privileged-information approaches are difficult to scale, unclear to internalize without assistance, and often assume multi-turn settings unlike standard single-turn RLVR.
  • A.3 Usage of Privileged Information: Existing work mainly improves the feasibility of correct trajectories, whereas this work targets diverse exploration even when successful trajectories are already attainable.
  • B Details on Strategy Nudging: For each problem, Strategy Generating Prompt uses gpt-4o-mini to produce keyword-level mathematical hints; main experiments use two hints, while top-ranked settings generate five candidates and select a subset by oracle evaluation.
  • B Details on Strategy Nudging: Strategy Nudging constructs prompts with optional hints that direct the model toward a primary reasoning approach while separating working and final answers with predefined delimiters.
  • B Details on Strategy Nudging: The diversity evaluation samples 8 rollouts from Qwen3-4B-Instruct-2507 on 200 DAPO-17k-Processed problems, comparing conditions with and without nudging via an LLM judge.The judge clusters solutions by conceptual reasoning strategy, ignores phrasing and minor computational differences, and counts distinct solution modes.

C Details on Baselines

This section details controlled rollout-scaling experiments, oracle-prefix baseline implementation, and lightweight strategy-level contexts designed to diversify reasoning without directly solving problems.

  • Rollout Scaling in GRPO: Rollout counts are varied while gradient accumulation steps and generation batch size are adjusted to keep optimization dynamics comparable.These controlled settings are summarized in Table 2.
  • Implementing POPE [16]: POPE is reimplemented with the original prompt format and dataset mixture, covering settings with and without privileged information.Oracle prefixes are standardized by truncating each oracle solution to 15% of its full length.
  • Example of Generated Contexts: Strategy-level contexts are lightweight keyword hints that steer models toward distinct reasoning modes rather than providing intermediate steps or solutions.Examples include "Generating functions" and "Parity of sums."

D Training Detail

Training used TRL to implement both baselines and NudgeRL, with training hyperparameters reported in Table 2.

  • Framework: TRL was used to implement the baselines and NudgeRL.The framework supported implementation of both the comparison methods and the proposed algorithm.
  • Hyperparameters: Training hyperparameters are reported in Table 2.The paper directs readers to Table 2 for the hyperparameter settings used during training.

E Details on Evaluation

Evaluation uses the same hyperparameters as Tab. 2, with temperature set to 0.7.

  • Evaluation keeps all hyperparameters identical to Tab. 2 except temperature, which is set to 0.7.

F Details on Case study

The case study contrasts GRPO’s lengthy, frequently truncated geometric reasoning with NUDGERL’s shorter, more reliable use of the shoelace formula. The example concerns finding n in a right-triangle area problem, whose shown final answer is 104.

  • Qualitative comparison: GRPO predominantly used coordinate geometry, heuristic symmetry assumptions, and case-by-case area decomposition, often producing long derivations that were truncated before the final answer.The case study describes these approaches as only occasionally reaching partial solutions.
  • Qualitative comparison: NUDGERL exploited the shoelace-formula strategy to compute polygon areas directly from vertex coordinates, yielding shorter and more reliable trajectories that completed within the generation budget.This contrasts directly with the GRPO-trained model’s frequent truncation.

G Full Evaluation Results

The section reports ablations of pdrop and hint sampling, with pass@1 estimated from 128 rollouts. Best results are bolded in both tables.

  • Table 3 evaluates pdrop ablations using pass@1 estimated with 128 rollouts, with the best results shown in bold.
  • Table 4 evaluates hint sampling ablations using pass@1 estimated with 128 rollouts, with the best results shown in bold.

H Broader Impacts

NudgeRL improves RLVR exploration efficiency without extremely large rollout budgets or expensive oracle supervision, potentially reducing training costs and improving accessibility. However, more capable reasoning systems may be misused, motivating continued safety, oversight, and responsible-deployment research.

  • Positive impacts: NudgeRL improves structured-exploration efficiency without relying on extremely large rollout budgets or expensive oracle supervision.The framework is proposed for reinforcement learning with verifiable rewards (RLVR).
  • Positive impacts: Avoiding large rollout budgets and expensive oracle supervision may reduce reasoning-model training costs and improve accessibility for smaller research groups.
  • Risks and responsibilities: More efficient exploration may contribute to increasingly capable reasoning systems that could be misused in harmful or unintended ways.
  • Risks and responsibilities: The authors emphasize continued research on safety, oversight, and responsible deployment.
Loading 2605.15726v1…