Source-linked AI summary
Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification
Yiju Guo, Tianyi Hu, Zexu Sun, Yankai Lin
TL;DR
RLVR reasoning is limited by inefficient exploration and sparse successful rollouts, especially in complex tasks. LENS purifies prompts by removing interfering tokens, transfers successful rollouts to optimization on the original prompts, and outperforms GRPO in performance and convergence speed across math and broader reasoning tasks.
Problem
RLVR depends on sampling correct rollouts, but sparse delayed binary rewards and high-dimensional decisions make successful exploration inefficient in complex reasoning tasks.
Method
LENS identifies and removes interference tokens from low-success prompts, then uses successful denoised-prompt rollouts to supervise policy optimization on the original noisy prompts.
Results
3.88% average performance gain and over 1.6× faster convergence versus GRPO are reported across seven math reasoning benchmarks, with a 1.83% average gain on scientific and general reasoning tasks.
Takeaways & Limitations
Pruning interference tokens improves rollout efficiency, and LENS outperforms both rollout-scaling and prompt-filtering baselines while using fewer computational resources.
Takeaways & Limitations
Experiments used models up to 8B parameters and primarily binary-reward tasks, leaving larger models and multidimensional scoring environments for further investigation.
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework (LENS), which first prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that LENS significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6$\times$ speedup on math reasoning, and a 1.83% gain on scientific and general reasoning. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research.
1 Introduction
RLVR exploration can fail because correct rollouts are rare in complex tasks, but LENS finds that a few interfering prompt tokens often drive these failures and purifies them before transferring successful rollouts back to noisy prompts. Across benchmarks, LENS improves rollout efficiency and reasoning performance over GRPO and related baselines.
- Correct rollouts become exceedingly rare when delayed binary rewards meet long-horizon decisions and high-dimensional action spaces.
- Increasing rollout counts raises computational cost without improving efficiency, while filtering zero-variance prompts sacrifices exploration on challenging samples.
- Fewer than 5% of prompt tokens can introduce excessive interference, causing failures that are not attributable to problem difficulty.
- Over 20% rollout-accuracy improvement follows interference-token pruning on previously failed DeepMath samples across all model families.
- LENS identifies and removes interference tokens from low-success prompts, then transfers successful denoised-prompt rollouts to optimize policy behavior on original noisy prompts.
- 3.88% average performance gain and over 1.6× faster convergence distinguish LENS from GRPO across seven math reasoning benchmarks, with further gains out of domain.
2 LENS: Less Noise Sampling Framework
LENS identifies and prunes interference tokens from low-success prompts, then transfers successful denoised-prompt rollouts to calibrate optimization on the original prompts. Its two-stage design uses token-level policy deviations to improve exploration while preserving prompt semantics with minimal pruning.
- Framework overview: LENS combines interference-token purification with Calibrated Rollout Policy Optimization to transfer supervision from denoised prompts to original noisy prompts.The framework is described as plug-and-play and is intended to support effective policy exploration under noisy inputs.
- Interference Token Identification: Interference Tokens are identified through token-level deviations between the learned policy πθ and reference policy πref, with large deviations treated as interference.The Interference Score measures this deviation for a token prefix and next generated token.
- Interference Token Identification and Purification: Purification ranks prompt tokens by Interference Score and removes the top k = ⌈γ · |xi|⌉ tokens, using a small deletion ratio γ of 1%–5%.The denoised prompt is defined as x′i = xi \ Ii, where Ii is the selected interference-token set.
- Calibrated Rollout Policy Optimization: Pruning is constrained because removal is not always beneficial; about 20% of prompts improve in rollout accuracy after removal.The method therefore uses selective activation rather than universally replacing rollouts.
- Calibrated Rollout Policy Optimization: CRPO calibrates training with sample reweighting, importance correction, group-relative advantages, and a PPO-style clipped objective with KL regularization.The objective accounts for whether each rollout was sampled from the original or denoised prompt.
- Calibrated Rollout Policy Optimization: CRPO activates rollout replacement when denoised prompts have higher empirical accuracy, substituting selected failed original-prompt rollouts with successful denoised-prompt rollouts.This selective replacement is designed to transfer high-quality learning signals while retaining the original noisy setting for optimization.
3 Experiments
LENS is evaluated across multiple model families, rollout baselines, mathematical benchmarks, and out-of-domain reasoning tasks. Across these settings, it improves accuracy or learning efficiency over GRPO and related strategies, including on challenging benchmarks.
- Experimental Setup: Experiments cover five models from the Llama-3.2, Qwen2.5, and Qwen3 families across several parameter scales.The models include Llama-3.2-3B-Instruct, Qwen2.5-3B, Qwen2.5-7B, Qwen3-4B-Base, and Qwen3-8B-Base.
- Baselines: LENS is compared with vanilla GRPO, doubled-budget GRPOextended, and zero-variance filtering methods DAPO and GRESO.The comparisons test rollout scaling and prompt filtering as alternative approaches to handling exploration failures.
- Experimental Setup: Training uses Openr1-Math-46k, while evaluation spans seven mathematical reasoning benchmarks with varied difficulty levels.The benchmarks are MATH500, AMC23, AIME24, AIME25, GaokaoEN-2023, Minerva, and OlympiadBench.
- Main Results: LENS consistently outperforms GRPO and GRPOextended under the same training corpus, and also surpasses DAPO and GRESO, including with fewer rollouts.The reported advantage is larger on challenging benchmarks such as AMC23 and AIME24.
- Main Results: LENS achieves higher progressive and final accuracy than GRPO across Qwen3-4B-Base and Qwen3-8B-Base on MATH-500 and OlympiadBench.On OlympiadBench, LENS shows more stable and continuous improvement than the fluctuations observed with GRPO.
- Analysis: LENS attributes its gains to removing interference tokens and contrasting successful purified rollouts with erroneous original samples to improve exploration and focus.These mechanisms are described as improving sample quality and helping the model focus on key information.
4 Further Analysis
Further analysis shows that LENS improves rollout quality, training efficiency, and stability across training stages and model scales. It also matches or exceeds GRPO while the best pruning threshold varies with model capacity.
- Training Dynamics: LENS reduces zero-reward prompts across early, middle, and late training, shifting prompts into more informative Mid and High categories.The stages are Steps 1–100, 101–200, and 201–300.
- Training Dynamics: LENS gains accuracy and response length more rapidly while maintaining moderate entropy, with earlier emergence of long-form reasoning.This trajectory is associated with more decisive reasoning in the analysis.
- Training Efficiency: 1.67× fewer gradient steps on MATH-500 and 1.64× acceleration on OlympiadBench are required for LENS to reach GRPO’s peak performance.Efficiency is measured by gradient steps needed to reach the GRPO baseline’s peak average accuracy.
- Threshold Sensitivity: LENS consistently matches or outperforms GRPO across pruning thresholds ranging from 1% to 5%.The threshold sensitivity analysis uses Qwen2.5-3B and Qwen2.5-7B on MATH-500.
- Training Dynamics: LENS produces steadier accuracy improvements than GRPO and GRPOextended across model scales.The comparison covers Qwen3-4B-Base and Qwen3-8B-Base.
- Threshold Sensitivity: The Qwen2.5-3B model performs best with a higher threshold, whereas Qwen2.5-7B performs best with a lower threshold.The analysis suggests that limited-capacity models are more strongly affected by interference tokens.
5 Related Work
Related work addresses RLVR exploration and credit assignment through prompt filtering, scaling exploration, and reward-function design. These approaches optimize rollout strategies or reward functions but do not focus on interference tokens.
- Related Work: Prompt filtering removes extremely difficult prompts to maintain a productive training set.This direction can sacrifice exploration on challenging samples.
- Related Work: Scaling exploration allocates more rollouts to harder samples to increase reward variance within groups.The approach increases rollout computation for difficult examples.
- Related Work: Reward function design revises advantage computation to prevent vanishing gradients.These approaches optimize rollout strategies or reward functions rather than prompt-token interference.
6 Conclusion
The paper identifies interference tokens as a source of failed RLVR exploration and proposes LENS to transfer successful denoised-prompt rollouts to optimization on noisy prompts. LENS consistently outperforms GRPO in performance and efficiency and generalizes beyond mathematics.
- Conclusion: LENS identifies interference tokens and transfers successful rollouts from denoised prompts to calibrate optimization on original noisy prompts.The framework is presented as a new approach to improving RLVR sampling efficiency.
- Conclusion: 3.88% average gain and over 1.6× speedup are reported against GRPO, with a 1.83% average gain on scientific and general reasoning tasks.The results also outperform scaling-exploration and prompt-filtering baselines.
Limitations
The paper’s evaluation is limited to models with up to 8B parameters, primarily binary-reward tasks, and the GRPO framework. Larger models, richer rewards, and other GRPO-based variants remain untested.
- Scope: Experiments use models with up to 8B parameters, leaving larger-scale performance and scalability for future research.The stated examples of larger scales are 32B and 70B models.
- Scope: The approach has been validated primarily on binary-reward tasks, so applicability to multi-dimensional scoring requires further investigation.The limitation concerns the reward structure of evaluated environments.
- Scope: The method is demonstrated within GRPO but has not yet been integrated with other GRPO-based variants.The paper suggests combining it with methods that optimize rollout frequency or reward functions.
A Detailed Training Settings
LENS and GRPO training hyper-parameters are documented in Table 3, while LENS is evaluated across additional model backbones and consistently achieves the strongest math-reasoning performance.
- A Detailed Training Settings: Table 3 contains the complete training hyper-parameters for GRPO and LENS.The table is identified as the basic training-parameter comparison.
- A Detailed Training Settings: LENS is evaluated on Qwen3-4B-Base, Qwen3-8B-Base, Llama3.2-3B-Instruct, Qwen2.5-3B, and Qwen2.5-7B.The additional Qwen2.5 models extend evaluation beyond the initially listed backbones.
- A Detailed Training Settings: LENS consistently achieves the strongest performance across all reported math reasoning benchmarks.The results are summarized in Table 4.
C Success Rate Sensitivity Analysis
The analysis tests four success-rate thresholds to examine LENS stability, revealing a trade-off between concentrating on difficult samples and covering medium- and high-difficulty samples.
- C Success Rate Sensitivity Analysis: LENS stability is evaluated with success-rate thresholds τ ∈ {0.125, 0.25, 0.375, 0.5}.The experiments investigate how threshold choice affects training stability.
- C Success Rate Sensitivity Analysis: τ = 0.125 concentrates training on difficult samples near the model’s capability frontier.This setting particularly benefits the high-difficulty Minerva and AMC23 benchmarks.
- C Success Rate Sensitivity Analysis: τ = 0.5 provides broader coverage of medium- and high-difficulty samples.
D Computational Overhead
LENS incurs additional computation per update to prioritize signal quality, but remains more efficient than doubling the rollout group size while improving reasoning accuracy.
- D Computational Overhead: LENS is compared with standard GRPO under identical hardware using the verl framework on 8× NVIDIA A800 GPUs.
- D Computational Overhead: LENS incurs 1.27×–1.62× the computational overhead of standard GRPO with G = 8.Specific average step times are reported in Table 6, whose group-size column is defined by the table caption.
- D Computational Overhead: LENS remains more efficient than brute-force doubling of the sample size to G = 16.The additional computation is associated with 2–3% absolute accuracy gains reported in Table 1.
- D Computational Overhead: The reported overhead translates into 2–3% absolute accuracy gains.
E Pruning Strategies Analysis
LENS outperforms alternative pruning strategies across seven benchmarks, while functional and quantitative evidence indicates that pruning preserves prompt semantics.
- E Pruning Strategies Analysis: LENS is compared with Resampling, Random Pruning, and Gradient-based Pruning under the same training setup and pruning ratio.The alternatives respectively replace unsuccessful rollouts, prune tokens uniformly at random, or prune tokens with the smallest gradient norm.
- E Pruning Strategies Analysis: LENS achieves the best accuracy on all seven evaluated benchmarks.The comparison results are presented in Table 7.
- E Pruning Strategies Analysis: Postpruning accuracy gains provide functional evidence that pruned prompts remain semantically equivalent to the originals.The reasoning would otherwise diverge from the correct solution and receive zero reward under RLVR’s unique ground-truth answer evaluation.
- E Pruning Strategies Analysis: BERTScore F1 exceeds 0.98 between original and pruned prompts across three backbones.Table 8 reports this as evidence of strong semantic preservation.
- E Pruning Strategies Analysis: Removed tokens are predominantly formatting noise, template redundancy, and irrelevant entities.