PaperlayerView exact revision on arXiv

Source-linked AI summary

DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes

Caijun Xu, Changyi Xiao, Zhongyuan Peng, Yixin Cao

arXiv:2605.28421v2cs.AI

TL;DR

Existing reasoning RL methods often depend on stronger teachers or carefully curated difficult data, leaving a gap in scalable capability improvement without stronger supervision. DenoiseRL trains recovery from noisy prefixes generated by weak models and adaptively controls their intensity, achieving consistent gains over on-policy baselines across mathematical reasoning and agentic tasks. The authors conclude that this provides a scalable route to stronger self-corrective reasoning, while noting limitations tied to weak-model perturbations and prefix-based corruption.

  • Problem

    Reasoning RL often relies on stronger-model supervision or carefully engineered difficult data, motivating improvement without stronger supervisors or manually curated hard datasets.

  • Method

    DenoiseRL converts weak-model failures into structured noisy prefixes and adapts each problem’s noise intensity using a sample-level curriculum based on recovery accuracy.

  • Results

    DenoiseRL consistently gains over on-policy baselines across mathematical reasoning and agentic tasks, with fine-grained control and external noise identified as key design choices.

  • Takeaways & Limitations

    The framework supports scalable reasoning improvement by making recovery from weak-model failures and self-correction direct training targets.

  • Takeaways & Limitations

    Perturbation quality depends on weak-model behavior, and the experiments instantiate recovery only through prefixes of failed weak-model trajectories.

Abstract

from arXiv · show

Reinforcement learning has become a central paradigm for advancing reasoning in large language models, yet most existing methods still depend on stronger teacher models or heavily curated difficult datasets, limiting scalable capability improvement. In this paper, we introduce DenoiseRL, a reinforcement learning framework that substitutes external supervision with recovery-oriented optimization over failures from weak models. Instead of relying on stronger supervision or carefully engineered data, DenoiseRL learns directly from noisy reasoning prefixes by converting them into opportunities for improvement, while exercising fine-grained control over the noise intensity, making training more scalable and effective. This yields a richer and more diverse learning signal, improving exploration efficiency by leveraging imperfect model behavior. Empirically, DenoiseRL consistently outperforms strong on-policy RL baselines across competitive mathematical reasoning tasks and interactive decision-making tasks, while promoting stronger self-corrective behavior as training difficulty increases, highlighting an effective and scalable pathway for improving agentic reasoning capabilities of large language models.

1 Introduction

DenoiseRL addresses the difficulty of improving reasoning without stronger teachers or heavily curated hard data by training models to recover from weak-model failures. It injects controllable noisy prefixes and adapts their difficulty per sample, producing consistent gains over on-policy baselines.

  • Motivation: Existing reasoning RL methods often rely on stronger-model supervision, while alternative difficulty-increasing approaches require engineered pipelines and substantial curation.These limitations motivate learning from weak-model failures without stronger teachers or manually designed hard datasets.
  • Method: DenoiseRL repurposes weak-model failures as structured reasoning-state perturbations rather than teacher trajectories or synthetic hard data.The policy learns to reconstruct valid solution paths from corrupted intermediate states.
  • Adaptive curriculum: DenoiseRL adapts each problem’s prefix ratio using recovery accuracy, increasing difficulty for easy samples and decreasing it for hard samples.This sample-level curriculum keeps training near the policy’s evolving capability boundary.
  • Method: Erroneous prefixes are injected into rollouts to force recovery from corrupted intermediate states and make self-correction a direct training target.The prefix controls the starting reasoning state, while the policy must revise incorrect directions rather than continue them.
  • Results: DenoiseRL consistently improves over on-policy baselines across mathematical reasoning and agentic tasks, with external noise and fine-grained control identified as key design choices.The reported contribution emphasizes broad gains rather than dependence on stronger supervision.

2 Related Work

Prior work improves reasoning through on-policy exploration, weak-to-strong supervision, or injected successful trajectories, but each leaves important constraints on exploration or supervision quality. DenoiseRL instead uses weak-model prefixes as misleading reasoning states that expand failure-based exploration.

  • Bootstrapping Reasoning via On-Policy RL: On-policy RL is bounded by the model’s self-generated state distribution, causing an exploration bottleneck when informative failures become scarce.Policy saturation can leave mostly correct rollouts or narrowly confined failure modes for gradient updates.
  • Weak-to-Strong Generalization (W2SG): Weak-to-strong methods use weaker models as supervisors, but student performance remains vulnerable to noisy pseudo-labels and the supervisor’s limited capacity.DenoiseRL reverses this role by using the weak model to generate out-of-distribution mistakes.
  • Prefix-Conditioned and Off-Policy Exploration: Prefix-conditioned and off-policy methods typically inject successful traces, expert solutions, or oracle hints to make sparse-reward problems more reachable.DenoiseRL differs by treating weak-model prefixes as misleading reasoning states rather than privileged guidance.

3 Method

DenoiseRL trains policies to recover from weak-model reasoning failures by injecting noisy prefixes and adapting their intensity to each problem’s recovery accuracy. Its sample-level curriculum keeps training near the policy’s capability boundary while preserving compatibility with other RL backbones.

  • 3.1 Denoising Reasoning with Noisy Prefixes: DenoiseRL treats prefixes of incorrect weak-model trajectories as structured reasoning-space noise and trains the policy to escape those states.The problem and answer remain unchanged; only the intermediate reasoning state is perturbed.
  • 3.1 Denoising Reasoning with Noisy Prefixes: The training objective starts rollouts from noisy prefixes, retains terminal verifier rewards, and updates only the on-policy continuation.Prefix tokens receive a zero loss mask because they were not sampled by the current policy.
  • 3.2 Adaptive Denoise Curriculum Learning: Each problem carries a prefix ratio that controls how much of its failed trajectory is revealed, allowing noise intensity to become an adjustable difficulty variable.Revealing more of a wrong solution makes the same problem harder.
  • 3.2 Adaptive Denoise Curriculum Learning: The controller increases a problem’s prefix ratio when recovery exceeds the target and decreases it when recovery is too difficult, tracking an informative noise level as the policy improves.The ratio is updated online from per-problem recovery accuracy.
  • 3.2 Adaptive Denoise Curriculum Learning: DenoiseRL rounds requested prefix lengths to complete line boundaries, producing coherent potentially flawed reasoning states rather than arbitrary syntactic truncations.If no internal line boundary exists, it uses the nearest token position.
  • 3.2 Adaptive Denoise Curriculum Learning: Problems with stabilized prefix ratios are retired and replaced by unseen problems initialized near the active set’s current mean difficulty.A least-squares slope over recent ratio updates identifies stability before replacement.
  • 3 Method: Because DenoiseRL changes only rollout construction and loss masks, it can be combined with other RL backbones.The method’s intervention is confined to how training trajectories are constructed and optimized.

4 Experiments

DenoiseRL is evaluated against on-policy baselines across mathematical reasoning and agentic decision-making, with experiments testing noisy-prefix sources and adaptive noise control. Results favor external structured noise, fine-grained per-problem adaptation, and recovery-oriented training.

  • Main mathematical reasoning results: DenoiseRL improves over corresponding RL baselines by 2.6 and 2.5 average points on Qwen3-4B-Base, and by 1.9 and 2.4 points on Qwen3-8B-Base.DenoiseRL-DAPO reaches 42.3 and 45.2, respectively, and DenoiseRL variants achieve the best results across most reported benchmark columns.
  • Main mathematical reasoning results: Both DenoiseRL variants outperform Critique-GRPO, with the best DenoiseRL result exceeding it by 1.8 points at each model scale.The comparison evaluates recovery training over structured errors against a critique-based self-correction baseline.
  • Noisy-prefix source: External small-model prefixes achieve the best mathematical and overall averages under dynamic noise control.Structured external noise outperforms random-token prefixes and errors sampled from the policy’s own pre-RL checkpoint.
  • Noisy-prefix source: Small-model prefixes retain the highest late-stage noise intensity, averaging 3.9 points during steps 1,000–1,200 versus 2.2 for pre-RL self prefixes and 0.6 for random tokens.Pre-RL self prefixes have the highest initial and overall intensity at 14.9 points, while small-model prefixes average 6.9 and random tokens 1.9.
  • Adaptive noise control: Fine-grained control keeps recovery accuracy within the target interval for 96.9% of moving-average windows, versus 12.4% for coarse control and 14.6% for fixed ratios.The alternatives drift above the upper boundary to approximately 0.82–0.86, whereas per-problem adaptation maintains the intended operating point.
  • Adaptive noise control: Fine-grained control yields a 15.6-point average gain over Qwen3-4B-Base, compared with 13.9 points for coarse control and 13.5 points for fixed control.It gives the largest improvement on four of five benchmarks, including AIME24 at +16.5 points.
  • Agentic decision-making: DenoiseRL-GRPO reaches 96.3% success on seen ALFWorld environments and 88.1% on unseen environments, improving over standard GRPO by 15.6 and 8.2 points.It is best on every seen task category and best or tied for best on every unseen category.
  • Recovery behavior: A representative rollout shows the model preserving useful partial reasoning while rechecking constraints, recomputing the feasible range, and repairing an incorrect answer.The prefix contains a partially correct derivation that fails because of faulty enumeration.

5 Conclusion

DenoiseRL trains models to recover from incorrect weak-model trajectories using adaptive, recovery-oriented reinforcement learning. The framework improves reasoning and interactive decision-making while strengthening revision of flawed reasoning.

  • DenoiseRL trains models to recover from incorrect intermediate trajectories generated by weak models.It converts weak-model failures into structured perturbations rather than relying on stronger teachers or manually curated hard datasets.
  • Per-problem prefix-length control adapts training difficulty according to online recovery accuracy.The curriculum tracks the policy’s evolving capability boundary and keeps training examples informative as the model improves.
  • Fine-grained per-problem control outperforms fixed and coarse-grained noise strategies.
  • External weak-model noise provides more persistent late-stage interference and stronger overall performance than self-generated or random noise.
  • Recovery-centric optimization consistently improves reasoning and interactive decision-making.
  • DenoiseRL strengthens the model’s ability to revise flawed reasoning.

Limitations

DenoiseRL’s recovery signal depends on the quality and realism of weak-model errors, and its tested corruption mechanism is limited to failed-trajectory prefixes. Whether the objective transfers to other perturbed reasoning representations remains unclear.

  • Generated perturbations may provide limited training value when weak-model errors are trivial, repetitive, or unrealistic.
  • The recovery signal’s effectiveness depends on the behavior of the weak models used to generate corruptions.
  • Experiments instantiate perturbed reasoning states only as prefixes of failed weak-model trajectories.
  • It remains unclear whether the recovery objective extends to erroneous conclusions, structured plans, tool-use histories, or latent reasoning states.

A Training Prompt and Hyperparameters for Mathematical Tasks

The mathematical-task training prompt presents each question to a helpful assistant and requires step-by-step reasoning with a boxed final answer.

  • A.1 Training Prompt: The system prompt frames the model as a helpful assistant.
  • A.1 Training Prompt: The user message supplies the mathematical question through a question placeholder.
  • A.1 Training Prompt: The training prompt combines a helpful-assistant system instruction with a user-provided question.
  • A.1 Training Prompt: The prompt requests reasoning step by step.
  • A.1 Training Prompt: The final answer is required inside a boxed-answer format.
  • A.1 Training Prompt: The response format separates intermediate reasoning from the final boxed answer.

A.2 Hyperparameters

Training uses GRPO with fixed rollout and batch sizes, while DenoiseRL adapts independent per-problem prefix ratios toward a target recovery accuracy.

  • Optimization and rollout: Qwen3-4B-Base is trained with GRPO using 16 active prompts and 16 recovery rollouts per prompt.The setup uses no additional clean rollouts, and both training and PPO mini-batch sizes are 16.
  • Optimization and rollout: The optimization uses a 10^-6 learning rate, no warmup or weight decay, gradient clipping at 1.0, and PPO clipping bounds of 0.2.
  • Optimization and rollout: KL regularization is disabled in both the reward and actor loss.
  • DenoiseRL curriculum: DenoiseRL uses line-level prefixes from incorrect weak-model trajectories with an independent prefix ratio for each problem.
  • DenoiseRL curriculum: Each prefix ratio is initialized and lower-bounded at 0 and upper-bounded at 0.5.
  • DenoiseRL curriculum: The ratio controller targets recovery accuracy of 0.75 with step size α=0.2.

B Training Efficiency

DenoiseRL adds preprocessing for noisy prefixes but avoids weak-model inference during RL training. Its longer responses increase training time, though wall-clock growth is smaller than the response-length increase.

  • Stored noisy prefixes are constructed once from Qwen2.5-1.5B-Instruct and reused across training runs.The preprocessing is lightweight and occurs outside the RL loop.
  • 1.89× longer responses accompany only a 1.37× increase in mean step time versus GRPO.DenoiseRL-GRPO averages 1,843 tokens and 67.0 seconds per step, compared with 974 tokens and 48.8 seconds for GRPO.
  • Response length and wall-clock step time are strongly associated during training.The per-step Pearson correlation is 0.96 across the shared training horizon.

C Supplementary Cases

The supplementary cases show DenoiseRL recovering by abandoning flawed local strategies while retaining useful structure. The examples cover probability and algebraic reasoning, with noisy prefixes and corrective continuations explicitly contrasted.

  • Training measurements: Figure 5 tracks response length and wall-clock training time over the shared first 1,400 updates.Faint traces represent per-step observations, while curves show 50-step moving averages under matched batch and rollout settings.
  • Probability reasoning: In probability reasoning, the continuation replaces an individual-quitter probability framing with counting unordered pairs.The prefix preserves the useful fact of two tribes of size 8, while the continuation rejects the unproductive sample space.
  • Algebraic reasoning: In algebraic reasoning, the continuation abandons repeated rational-root checks and uses Vieta’s formulas to recover the target aggregate.It obtains A from the missing x_2 coefficient and then computes B through the linear relation y = 1 − x.
  • Cross-case interpretation: The cases suggest denoise rollouts train reassessment, useful-structure retention, and replacement of flawed or inefficient local strategies.This extends beyond simply continuing from noisy contexts toward solution-relevant correction.

D Hyperparameters for Agentic Tasks

Agentic-task training uses Qwen2.5-7B-Instruct as policy with Qwen2.5-1.5B-Instruct generating recovery prefixes, combined with GRPO and dynamic prefix-ratio curriculum control. Episodes, prompts, responses, and optimizer settings are tightly bounded.

  • Model, environment, and rollout: Each optimizer step uses 16 ALFWorld tasks and 16 recovery rollouts per task, with episodes capped at 50 environment steps.Maximum prompt and response lengths are 4,096 and 512 tokens, respectively.
  • Model, environment, and rollout: The agentic setup pairs Qwen2.5-7B-Instruct as policy with Qwen2.5-1.5B-Instruct as the online weak model.The weak model generates one prefix candidate per task group using the full_then_ratio strategy.
  • Optimization and curriculum: GRPO training runs for up to 500 epochs with a 10^-6 learning rate and 256-example PPO mini-batches.The actor uses a low-variance KL loss with coefficient 0.01, while invalid actions receive a 0.1 penalty.
  • Optimization and curriculum: Dynamic prefix ratios are shared within ALFWorld task categories and refreshed as optimizer steps progress.Fresh game files are sampled at every optimizer step for the step-level DenoiseRL curriculum.
  • Supplementary recovery case: The algebraic recovery case contrasts explicit root search in the noisy prefix with Vieta-based recovery in the continuation.This supplementary case operationalizes the recovery behavior in an agentic-task training setup.
Loading 2605.28421v2…