PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR

Daijiao Liu, Huiyi Wang, Lina Yao, Dong Gong

arXiv:2610.16105v1cs.LG

TL;DR

Small RLVR rollout groups can yield similar trajectories and weak group-relative signals, while existing remedies leave rollout conditioning unchanged. EDEN adds anchored input perturbations to a fraction of rollouts and selectively incorporates them into GRPO updates. Across model scales and mathematical reasoning benchmarks, it outperforms GRPO and same-pipeline exploration controls.

  • Problem

    Small RLVR rollout groups can produce similar trajectories and reduce within-group reward variation, weakening group-relative advantage signals.

    Source: §1 · p. 1

  • Method

    EDEN replaces a small fraction of standard rollouts with same-policy rollouts conditioned on an appended, noise-perturbed embedding around a shared anchor, then selectively incorporates them with downweighted advantages.

    Source: §1 · p. 2

  • Results

    Across model scales and benchmarks, EDEN outperforms GRPO and same-pipeline controls; on Qwen2.5-Math-1.5B it exceeds GRPO by 2.44 Pass@1 points and 3.06 Pass@8 points.

    Source: §4.1 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    Input-side perturbations provide an exploration lever that preserves the model architecture, rollout budget, and test-time inference path.

    Source: §1 · p. 2, §5 · p. 9

    The current method uses a fixed bank of anchor embeddings and Gaussian perturbations centered at those anchors.

    Source: §5 · p. 9

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals. Existing remedies such as higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets broaden sampling or modify policy updates but leave rollout conditioning unchanged. In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations. EDEN explores beyond the policy's own rollouts. Within the same rollout budget, it replaces a small fraction of standard rollouts in each group with rollouts that the same policy generates from the prompt plus a single appended, perturbed embedding position, rather than from the prompt alone. Gaussian noise drives variation, while a shared anchor provides a common conditioning direction within each prompt group. Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and test-time inference remain unchanged. Across model scales and mathematical reasoning benchmarks, EDEN outperforms GRPO and the exploration controls evaluated in the same pipeline, with the largest gains in sampled solution coverage. Analyses show that more prompt groups provide non-zero group-relative learning signals and that noise and anchoring play complementary roles. Intervening at the rollout source is thus an effective lever for exploration in RLVR.

1 Introduction

Small rollout groups can produce homogeneous trajectories and weak group-relative learning signals, while existing exploration methods leave rollout conditioning unchanged. EDEN uses anchored input perturbations to broaden exploration and improves benchmark performance over GRPO and same-pipeline controls.

1 Introduction

  • Small rollout groups can yield highly similar trajectories, narrowing the solution space represented in updates and weakening group-relative advantages when rewards are uniform.The paper identifies useful variation within the rollout pool, not rollout count alone, as a key constraint.Source: §1 · p. 1
  • Existing temperature, objective, and trajectory-selection approaches broaden sampling or alter updates, but leave rollout conditioning unchanged.The paper frames a minimal input-embedding intervention as a way to explore at the rollout source while retaining discrete decoding and unchanged test-time inference.Source: §1 · pp. 1–2
  • EDEN replaces a small fraction of standard group rollouts with same-policy rollouts conditioned on anchored, perturbed inputs.Gaussian noise supplies variation around an anchor shared within each prompt group; informative perturbed trajectories enter GRPO updates through advantage gating and reweighting.Source: §1 · p. 2
  • Across three model scales and eight mathematical reasoning benchmarks, EDEN improves average Pass@1 and Pass@8 over GRPO, with gains persisting at Pass@256 on AIME.Analyses also find more prompt groups with nonzero training signals and complementary roles for noise and anchoring.Source: §1 · p. 2

2 Related Work

Related RLVR work addresses exploration through sampling, optimization, trajectory selection, or generation-time changes. EDEN is distinguished by using shared-anchor input perturbations at the rollout source.

RLVR and GRPO foundations

  • RLVR methods including PPO, RLOO, GRPO, and DAPO share a group-rollout backbone, while DAPO adds asymmetric clipping, dynamic sampling, token-level loss, and overlong-reward shaping.EDEN builds on GRPO and is compatible with the described refinements.Source: §2 · pp. 2–3

Exploration approaches in RLVR

  • Temperature changes, entropy methods, KL resets, and diversity-aware objectives target sampling or optimization while leaving input conditioning unchanged.EDEN instead intervenes at the rollout source through anchored input perturbations.Source: §2 · p. 3

Rollout methods for LLM RLVR

  • Rollout methods select, prune, or reuse trajectories, while generation interventions include tree search, soft-token or logit perturbations, policy-weight noise, and quantisation noise.EDEN uses a shared anchor to direct continuous perturbations at one appended input embedding for a subset of rollouts.Source: §2 · p. 3

3 Method

EDEN extends group-relative policy optimization with a small pool of noise-perturbed rollouts, selecting informative trajectories and updating the unperturbed policy under clean prompts.

3.1 Preliminaries

  • GRPO computes group-relative advantages by standardizing each rollout's reward against the prompt group's mean and standard deviation, then shares that scalar across its tokens.The group-rollout setup supplies the baseline that EDEN builds on.Source: §3.1 · p. 3

3.2 Embedding-space Directional Exploration via anchored Noise

  • EDEN perturbs a fraction of each group's rollouts, regulates exploration with shared anchors and a temperature schedule, and absorbs selected trajectories through advantage-based selection and scaling.The overview describes broadening trajectory-level exploration without increasing the total rollout budget.Source: §3.2 · p. 3, Figure 2 · p. 4

3.2.1 Noise-Perturbed Rollout Generation

  • For each prompt, EDEN appends one perturbed embedding position; independent Gaussian noise varies around an anchor shared across that prompt's perturbed rollouts.The anchor bank is fixed during training, and its screened anchors aim to induce non-trivial variation without substantially disrupting generation. The temperature schedule limits early randomness and enables stronger exploration later.Source: §3.2.1 · pp. 4–5

3.2.2 Noise-Aware Trajectory Selection and Advantage Estimation

  • EDEN selects the top fraction of batch-wide noisy rollouts by absolute group-relative advantage, favoring rare within-group outcomes that change the relative signal.Selected perturbed advantages are scaled by γ in (0, 1] to reduce their gradient contribution relative to standard rollouts.Source: §3.2.2 · p. 5

3.2.3 Noise-Aware Policy Update

  • Selected noisy trajectories are generated under perturbed conditioning but evaluated for updates under the clean prompt, with advantage filtering and scaling instead of an importance-sampling correction.The policy update uses a GRPO-style objective over standard and selected perturbed rollouts.Source: §3.2.3 · p. 5

4 Experiments

Experiments compare three Qwen2.5 scales across mathematical reasoning benchmarks and same-pipeline exploration controls, using a default budget of 16 rollouts per prompt. Results cover benchmark performance, solution coverage, training signals, and design ablations.

4.1 Main Results

  • On Qwen2.5-Math-1.5B, EDEN exceeds GRPO by 2.44 Pass@1 points and 3.06 Pass@8 points, and outperforms it on average at both model scales.The comparison includes English and Chinese mathematics benchmarks spanning grade-school to olympiad-level tasks.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6

4.2 Analysis

  • EDEN improves average Pass@256 over GRPO by 6.67 points across model scales, including 13.33 points on AIME’25 with Qwen2.5-Math-7B.The reported results indicate that solution-coverage gains persist at a large evaluation budget.Source: §4.2 · p. 6, Table 3 · p. 7
  • EDEN yields more correct AIME rollouts from problems solved by at least four distinct approaches and fewer from problems solved by one approach.Training also produces fewer zero-advantage groups under EDEN from about step 750 onward, reaching 0.53 versus 0.58 for GRPO at step 2000.Source: §4.2 · pp. 7–8, Figure 3 · p. 7
  • Increasing GRPO from 8 to 32 rollouts per prompt improves Pass@1 by only 0.5 points and slightly reduces Pass@8, while EDEN maintains higher entropy than GRPO configurations.The GRPO configurations reach similarly low entropy by step 200, whereas EDEN remains higher after warm-up.Source: §4.2 · p. 7, Table 5 · p. 8
  • Removing noise or using norm-matched random noise without an anchor fails to reproduce EDEN’s gains, while a fixed random anchor recovers part of the improvement.Sharing a curated-bank anchor within each prompt group combines a common conditioning center within groups with anchor variation across groups.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 Ablation Studies

  • Removing advantage filtering causes frequent gradient-norm spikes, while retaining perturbed trajectories with large absolute advantages yields smoother updates and better performance.The comparison is between EDEN and a variant that uses all noisy rollouts.Source: §4.3 · p. 8, Figure 8 · p. 9
  • On Qwen2.5-0.5B, a 25% noisy-rollout fraction performs best, while 50% matches GRPO; a single bank anchor recovers nearly all Pass@8 gains.Larger anchor banks provide modest additional gains, especially on Pass@1.Source: §4.3 · p. 9, Figure 10 · p. 9, Table 6 · p. 9
  • With temperature warmup but no embedding perturbations, baseline entropy quickly collapses, whereas EDEN maintains higher entropy; warmup lengths of 200 and 400 steps yield similar accuracy.Using the target temperature from the start leads to high entropy and near-zero rewards.Source: §4.3 · p. 9, Figure 9 · p. 9

5 Conclusion

The conclusion presents EDEN as rollout-source exploration for RLVR and reports stronger performance than GRPO and same-pipeline exploration controls. It identifies the anchor-bank construction and Gaussian perturbation scheme as directions for further development.

5 Conclusion

  • EDEN explores beyond the policy’s own rollouts through directional input-embedding perturbations, incorporating informative trajectories into the unperturbed policy.The method uses selective incorporation and advantage reweighting without changing architecture, rollout budget, or test-time inference.Source: §5 · p. 9
  • Across model scales and benchmarks, EDEN outperforms GRPO and the exploration controls evaluated in the same pipeline.The conclusion frames the results as gains from input-side perturbations beyond output-side controls alone.Source: §5 · p. 9
  • The current noise bank uses anchor embeddings and Gaussian perturbations centered at anchors; adaptive construction and more structured directions remain future-work options.The paper suggests dynamic selection or joint optimization and anisotropic or task-dependent noise as possible extensions.Source: §5 · p. 9

A Detailed Experimental Setup

The setup evaluates general-purpose and math-specialized Qwen2.5 models across mathematical reasoning datasets, using binary outcome rewards and a default group of 16 rollouts per prompt.

Datasets

  • Evaluation spans GSM8K, MATH, AMC, AIME 2024 and 2025, OlympiadBench, Gaokao, and MinervaMath for the math-specialized models.The datasets cover standard reasoning, competition-level, olympiad-style, and advanced mathematical reasoning tasks.Source: A Detailed Experimental Setup · p. 13

Models and datasets

  • Experiments use Qwen2.5-0.5B, Qwen2.5-Math-1.5B, and Qwen2.5-Math-7B, covering general-purpose and math-specialized models.The smaller model trains on GSM8K; the math-specialized models train on MATH and DAPO-Math.Source: A Detailed Experimental Setup · p. 13

Training details

  • Each prompt uses 16 rollouts by default; EDEN assigns 12 to standard inputs and 4 to noise-perturbed inputs, sharing one sampled anchor across the perturbed rollouts.Anchors are sampled per prompt from a bank of eight, and each perturbed rollout receives an independent Gaussian perturbation.Source: A Detailed Experimental Setup · p. 13
  • Perturbed-rollout temperature follows a cosine schedule to 1.0, while rewards and group-relative advantages are computed jointly across all 16 rollouts.All standard rollouts are retained; perturbed rollouts are ranked by absolute advantage, and retained advantages are scaled by γ = 0.5.Source: A Detailed Experimental Setup · p. 13

B Full Training Procedure of EDEN

The full training procedure combines standard and noise-anchored rollouts, computes their rewards and group-relative advantages, then filters samples before policy updates.

B Full Training Procedure of EDEN

  • For each prompt, EDEN samples standard rollouts and then constructs noise-anchored rollouts using embedding-space perturbations.The full algorithm explicitly lays out construction of mixed rollout batches.Source: B Full Training Procedure of EDEN · p. 14
  • EDEN combines the rollout types and computes advantages before filtering samples by advantage scores.The procedure computes rewards and group-relative advantages over the combined samples.Source: B Full Training Procedure of EDEN · p. 14
  • The policy update uses the selected samples, integrating structured noise injection with standard policy optimization.The procedure describes this combination as balancing exploration and stability during training.Source: B Full Training Procedure of EDEN · p. 14

C Additional Details on Noise Bank Construction

EDEN builds a fixed noise bank from vocabulary-token anchors, screening anchor-relative directions for complementary, viable perturbations. The bank uses a one-time heuristic rather than an objective optimized during RL training.

C Additional Details on Noise Bank Construction

  • EDEN clusters normalized anchor-relative embedding directions and retains 32 representative candidate anchors for offline screening.Normalizing displacement removes scale influence while retaining the direction from the reference token to each candidate anchor.Source: C Additional Details on Noise Bank… · p. 14
  • Screening uses 200 prompts per model and generates eight perturbed rollouts for each candidate-anchor and prompt pair.Qwen2.5-0.5B uses GSM8K training prompts; the 1.5B and 7B models use DAPO-Math training prompts.Source: C Additional Details on Noise Bank… · p. 14
  • The final bank contains eight anchors, selected to prioritize prompt coverage and response-length feasibility, then rank remaining candidates by rollout diversity and correctness.Coverage is the fraction of prompts with at least one correct perturbed rollout; length ratio enforces response-length feasibility.Source: C Additional Details on Noise Bank… · pp. 14–15
  • The bank is screened to exclude anchors whose perturbations have little effect on generation or substantially disrupt coherent responses, not to identify a unique optimum.Its construction is a one-time screening heuristic, not an objective optimized during RL training.Source: C Additional Details on Noise Bank… · p. 15

D Additional Experimental Results

EDEN consistently improves Avg@4 across small-scale and math-specialized models, and its macro-average gains on Qwen2.5-0.5B hold across three paired training seeds. The section also describes pre- and post-RL analyses of rollout diversity.

D Additional Experimental Results

  • The pre-RL diversity analysis embeds correct standard and anchor-perturbed trajectories for 28 prompts using distances based on token-level 4-gram Jaccard similarity.The prompt-specific MDS embeddings are centered, aligned, and normalized using the standard-rollout trajectories before pooling.Source: D Additional Experimental Results · p. 15
  • The post-RL AIME analysis samples 16 responses per problem from final GRPO and EDEN checkpoints and groups correct responses by distinct reasoning-approach signatures.It covers 60 AIME 2024 and AIME 2025 problems.Source: D Additional Experimental Results · p. 15
  • EDEN improves the macro average for all three paired Qwen2.5-0.5B training seeds under both greedy and Pass@8 evaluation.Table 7 reports the mean and sample standard deviation over three independent training runs.Source: D Additional Experimental Results · p. 15, Table 7 · p. 15
  • EDEN consistently improves Avg@4 across small-scale and math-specialized models under T=1 and top-p=0.7, with comparisons including Qwen2.5-0.5B and Qwen2.5-Math-1.5B.Source: D Additional Experimental Results · p. 15, Figure 11 · p. 16, Table 8 · p. 16

E Extended Ablation Studies

Temperature scheduling stabilizes EDEN training, while performance is relatively insensitive to schedule duration. Extreme temperature alone does not consistently improve results, whereas EDEN has the best overall performance at T=1 in the controlled comparison.

E Extended Ablation Studies

  • Temperature scheduling is necessary for stable optimization, but performance is relatively insensitive to schedule duration.The results suggest EDEN’s primary performance gains stem from its noise-based exploration mechanism rather than fine-tuned temperature scheduling.Source: E Extended Ablation Studies · p. 16
  • Without temperature scheduling, gradient norms diverge to numerical instability (NaNs), and validation accuracy collapses to zero after initially improving.The results identify gradual temperature increases as essential for stabilizing noise-perturbed rollouts during early training.Source: E Extended Ablation Studies · p. 16, Figure 12 · p. 17
  • Cosine schedules of 0–200 and 0–400 steps yield comparable performance and consistently outperform GRPO across most benchmarks under Pass@1 and Pass@8.Extending the schedule to 0–400 steps yields slight improvements on more challenging tasks, but overall differences remain modest.Source: E Extended Ablation Studies · p. 16, Figure 13 · p. 17
  • Extreme temperature does not consistently improve performance and often degrades it, particularly for Pass@8; EDEN achieves the best overall performance at T=1.For a controlled T=1.4 comparison, the entropy coefficient is set to zero in both GRPO and EDEN because high-temperature exploration is already sufficient.Source: E Extended Ablation Studies · pp. 16, 18

F Case Studies.

The case studies illustrate rollout reasoning through selected correct AIME traces and a GRPO solution to a divisibility problem.

Case study setup

  • The comparison uses correct rollouts from 16 samples per prompt at T=0.6 and top-p=0.95; correctness refers to final answers, not every reasoning step.The selected AIME problem illustrates the qualitative pattern alongside aggregate rollout diversity in Fig. 3.Source: F Case Studies · p. 18

Analysis of Reasoning Traces

  • EDEN’s correct rollouts use several derivations of the same core reduction, n + 2 | 39, while GRPO’s two correct solutions follow essentially the same derivation.EDEN’s approaches include polynomial division, parameterization, the remainder theorem, and substitution followed by algebraic reduction; GRPO’s differences are mostly in expansion order and intermediate steps.Source: F Case Studies · p. 18

F Case Studies.

  • For GRPO Baseline—Rollout 2, the positive solutions are 1, 11, and 37, summing to 49.The rollout is labeled correct.Source: F Case Studies · pp. 22–23
  • Setting k = n + 2 reduces the divisibility condition to k dividing 39; since k > 2, the candidates are 3, 13, and 39.Subtracting 2 gives n = 1, 11, and 37.Source: F Case Studies · p. 23
Loading 2610.16105v1…