PaperlayerView exact revision on arXiv

Source-linked AI summary

R$^3$L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification

Weijie Shi, Yanxi Chen, Zexi Li, Xuchen Pan, Yuchang Sun, Jiajie Xu, Xiaofang Zhou, Yaliang Li

arXiv:2601.03715v2cs.LGcs.AI

TL;DR

R3L targets inefficient exploration, coarse credit assignment, and unstable optimization in difficult RL tasks for LLMs. It synthesizes corrected trajectories through language-guided reflect-then-retry, masks shared prefixes, and amplifies successful advantages. Experiments report 5% to 52% relative improvements over baselines, while the method remains limited to tasks with verifiable ground truth and incurs an additional reflection pass.

  • Problem

    Difficult LLM reinforcement-learning tasks suffer from low-success stochastic exploration, costly retries, coarse credit assignment, and failure-dominated training signals.

  • Method

    R3L uses language-guided reflect-then-retry, Pivotal Credit Assignment on diverging suffixes, and Positive Amplification for successful trajectories.

  • Results

    5% to 52% relative improvements over baselines are reported across agentic tasks and mathematical reasoning benchmarks.

  • Takeaways & Limitations

    R3L provides a training approach that synthesizes corrective trajectories, preserves valid prefixes, and emphasizes constructive signals during optimization.

  • Takeaways & Limitations

    Experiments are limited to tasks with verifiable ground truth, and reflection requires an additional inference pass despite reported rollout savings.

Abstract

from arXiv · show

Reinforcement learning drives recent advances in LLM reasoning and agentic capabilities, yet current approaches struggle with both exploration and exploitation. Exploration suffers from low success rates on difficult tasks and high costs of repeated rollouts from scratch. Exploitation suffers from coarse credit assignment and training instability: Trajectory-level rewards penalize valid prefixes for later errors, and failure-dominated groups overwhelm the few positive signals, leaving optimization without constructive direction. To this end, we propose R$^3$L, Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification. To synthesize high-quality trajectories, R$^3$L shifts from stochastic sampling to active synthesis via reflect-then-retry, leveraging language feedback to diagnose errors, transform failed attempts into successful ones, and reduce rollout costs by restarting from identified failure points. With errors diagnosed and localized, Pivotal Credit Assignment updates only the diverging suffix where contrastive signals exist, excluding the shared prefix from gradient update. Since failures dominate on difficult tasks and reflect-then-retry produces off-policy data, risking training instability, Positive Amplification upweights successful trajectories to ensure positive signals guide the optimization process. Experiments on agentic and reasoning tasks demonstrate 5\% to 52\% relative improvements over baselines while maintaining training stability. Our code is released at https://github.com/shiweijiezero/R3L.

1 Introduction

R3L addresses exploration and exploitation failures in difficult, sparse-reward tasks by synthesizing corrected trajectories, localizing credit, and amplifying positive signals. Experiments report relative improvements over baselines across agentic and mathematical reasoning tasks.

  • Challenges: Difficult tasks produce mostly failed stochastic rollouts, costly retries, and no actionable guidance from scalar rewards.Natural-language feedback such as error messages and execution traces is available but underused by standard RL methods.
  • Challenges: Trajectory-level rewards penalize valid earlier reasoning when a later error causes failure, creating noisy gradient estimates.Process-level alternatives require costly annotation and can provide unstable supervision.
  • R3L: R3L uses language-guided reflect-then-retry to diagnose failure points and restart generation with corrective guidance, reducing rollout cost.Training inputs omit the guidance descriptions so corrections must be internalized by the model.
  • R3L: Pivotal Credit Assignment updates only the diverging suffix shared by contrastive base and retry trajectories, while Positive Amplification strengthens successful signals.These mechanisms target valid-prefix preservation and failure-dominated optimization.
  • Results: 5% to 52% relative improvements over baselines are reported on agentic tasks and mathematical reasoning benchmarks.The cited experiments include ALFWorld, WebShop, and ScienceWorld among the agentic tasks.

2 Related Work

Prior work addresses sparse-reward exploration and unstable learning through oversampling, feedback-guided correction, variance reduction, and adaptive weighting. R3L combines language feedback with failure-point retries and group-wide positive amplification.

  • Exploration: Sampling-based methods compensate for rare successful trajectories through oversampling and filtering, but incur substantial computational cost.Correction-based methods instead synthesize improved trajectories using heuristics, critics, scaffolds, or reflection.
  • Credit and stability: Existing methods reduce gradient variance with sequence-level ratios or adaptive clipping, while trajectory-level rewards still penalize valid prefixes after later errors.Critique-GRPO uses critiques and weighted advantages for selected refinements.
  • R3L distinction: R3L differs by identifying specific failure points, learning from all trajectories, and applying a simple amplification factor across the exploration group.It also uses model-driven judgments of suboptimality rather than only binary verification.

3 Preliminaries

The paper formulates multi-turn agent interaction and reviews GRPO’s group-relative policy optimization. GRPO normalizes trajectory rewards within sampled groups and uses clipped importance sampling and KL regularization for off-policy control.

  • Problem formulation: A trajectory consists of K turns, each pairing an environment observation x_k with an agent response y_k.The policy history at turn k contains preceding observations and responses plus the current observation.
  • GRPO: GRPO samples a group of N trajectories from the behavior policy and estimates each trajectory’s advantage from group reward statistics.The group-relative formulation avoids requiring a learned critic.
  • GRPO: The behavior policy synchronizes with the trainable policy every S steps, while importance sampling with clipping manages off-policy divergence.A frozen reference policy supplies KL regularization in the standard GRPO objective.

4 Methodology

R3L alternates language-guided trajectory synthesis with policy optimization. It retries from diagnosed pivots, masks shared prefixes for pivotal credit, and amplifies successful advantages to stabilize learning from off-policy data.

  • Framework: R3L alternates trajectory synthesis and policy optimization, using reflect-then-retry to transform failed attempts into corrected trajectories.Optimization uses both base and retry trajectories.
  • Data construction: The exploration data combines base trajectories, corrected distillation trajectories, and auxiliary reflection and retry tasks.The auxiliary tasks maintain the exploration skills during training.
  • Reflect-then-retry: Base trajectories use half the sampling budget, while unsuccessful samples are diagnosed for outcome, root cause, suggestions, and a pivot turn.Generation restarts at the pivot with corrective guidance and pairs the original prefix with the corrected suffix.
  • Reflect-then-retry: Training omits guidance descriptions from paired prefix-correction inputs so the model internalizes corrections for inference.This design separates synthesis-time feedback from the learned policy behavior.
  • Pivotal Credit Assignment: Pivotal Credit Assignment masks the shared prefix and updates only the diverging suffix where base and retry trajectories provide contrastive information.The shared prefix is treated as a control variate whose inclusion would add variance without useful discrimination.
  • Positive Amplification: Failure-dominated groups dilute successful advantages, so Positive Amplification scales positive signals while leaving negative advantages unchanged.The maximum-reward trajectories receive the full amplification factor, and α = 3.0 works well across tasks.
  • Objective: R3L removes importance sampling and KL constraints for retry data because guided behavior differs from the current policy and amplification supports stability.The paper reports reduced memory and computational cost with maintained training stability.

5 Experiments

R3L is evaluated across agentic and mathematical reasoning benchmarks against rejection-sampling, group-relative, and language-feedback baselines. It achieves broad performance gains, with active reflection, positive amplification, and pivotal credit contributing to exploration, stability, and sustained retry improvement.

  • Experimental Setup: The evaluation covers ALFWorld, WebShop, ScienceWorld, and six mathematical reasoning benchmarks, using Average Reward as the primary metric.Experiments use Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-4B; ablations use Qwen2.5-1.5B-Instruct.
  • Main Results: R3L achieves the best or second-best performance on all 27 settings and ranks first on all nine agentic settings.With the 1.5B model, R3L reaches 0.928 on ALFWorld, 0.663 on WebShop, and 0.385 on ScienceWorld.
  • Main Results: On mathematical reasoning, R3L reaches 0.897 on GSM8K, 0.658 on Math500, and 0.301 on OlympiadBench with the 7B model.The only first-place exception is GSM8K with the 1.5B model, where Critique-GRPO reaches 0.798 versus R3L's 0.721; R3L still reports 52% gains over GRPO.
  • Main Results: On Qwen3-4B, R3L improves GSM8K from GRPO's 0.934 to 0.948 and Math500 from GSPO's 0.722 to 0.753.R3L attains the best or tied-best score on all nine benchmarks in this evaluation.
  • Ablation Study: Removing Reflect-then-Retry causes the largest ablation degradation, reducing GSM8K from 0.721 to 0.562 and WebShop from 0.663 to 0.628.Removing Positive Amplification also lowers ALFWorld from 0.928 to 0.881 and GSM8K from 0.721 to 0.685, while omitting Pivotal Credit has smaller but consistent effects.
  • Efficacy of Language-Guided Reflection: After warm-up, retry trajectories show sustained improvement, with Reward Gain reaching 0.6 on ALFWorld at 1.5B, 0.8 at 7B, 0.3 on DAPO, and 0.15 on WebShop.Average Retry Improvement Rate reaches 73.9% on ALFWorld and 36.5% on WebShop for Qwen2.5-7B-Instruct.

6 Conclusion

R3L addresses inefficient exploration, coarse credit assignment, and instability in failure-dominated regimes through trajectory synthesis, prefix protection, and positive-signal amplification. The resulting framework supports robust learning where standard methods may collapse or converge prematurely.

  • Conclusion: R3L targets inefficient exploration, coarse credit assignment, and training instability in failure-dominated regimes.Its components are Reflect-then-Retry, Pivotal Credit Assignment, and Positive Amplification.
  • Conclusion: R3L transforms sparse-reward environments into richer learning opportunities by synthesizing trajectories, protecting valid prefixes, and emphasizing successful signals.The conclusion links these mechanisms to robust learning relative to standard methods.

Limitations

R3L has limitations involving reflection overhead, smaller-model cold starts, and an experimental scope restricted to tasks with verifiable ground truth.

  • Reflection overhead: Reflection requires an additional inference pass, although restarting from failure points reduces per-step rollout turns by 28–34% on multi-step agentic tasks.Across four model-task settings, per-step rollout time remains below GRPO’s, while multi-step training tokens are roughly halved.
  • Model scale: Smaller models can exhibit cold starts because they initially struggle to produce actionable self-corrections, requiring a longer warm-up period.The paper reports this behavior for the 1.5B model, unlike the 7B model.
  • Evaluation scope: Experiments are limited to tasks with verifiable ground truth, leaving open-ended domains with subjective criteria unvalidated.The paper specifically identifies creative writing as an untested setting where automated retry validation remains uncertain.

A.1 Training Dynamics and Convergence

R3L’s training dynamics combine a warm-up phase with later rapid improvement, while its pivotal credit and positive amplification mechanisms target gradient instability, entropy collapse, and off-policy retry learning.

  • Performance and Convergence: R3L initially warms up for reflection adaptation, then after step 120 shows a steeper learning curve than GRPO, which plateaus near reward 0.4.GRPO gains earlier around step 100 but later stagnates with visible variance.
  • Policy Drift and Gradient Anomalies: GRPO exhibits policy drift, clipping, and gradient spikes, whereas R3L maintains smooth gradients and consistently low KL divergence.GRPO’s Reference KL exceeds 10.0 after step 200, Clip Fraction reaches 30%, and gradient norm exceeds 400 around step 190.
  • Gradient Decomposition and Entropy Collapse: When negative samples dominate, gradients primarily suppress high-probability erroneous tokens, and without a positive attractor the released probability mass disperses, increasing entropy.The analysis connects this entropy collapse to reward stagnation and policy drift.
  • Positive Amplification: Positive Amplification makes constructive gradients dominate destructive gradients when α · p · ¯A+ exceeds (1 −p) · | ¯A−| under comparable gradient norms.The paper reports α = 3.0 as covering practical success fractions and advantage ratios.
  • Pivotal Credit Assignment: Pivotal Credit Assignment masks shared-prefix tokens and uses base-retry covariance to reduce gradient variance, with stronger reductions as pivot points move later.The method updates the diverging suffix while excluding the shared prefix from gradient weighting.
  • Off-Policy Stability: R3L’s verified successful retry trajectories support stable off-policy learning without importance sampling or KL constraints, while convergence requires positive retry success and sufficient amplification.The framework is characterized as filtered behavioral cloning from verified high-reward trajectories.

B Cross-Architecture Evaluation

On Llama-3.2-3B-Instruct, a more volatile backbone, R3L remains competitive across agentic and mathematical reasoning benchmarks, ranking first on five of nine and second on the rest.

  • Cross-architecture setting: Llama-3.2-3B-Instruct shows volatile RL dynamics, with sampling-based methods collapsing on agentic tasks.The evaluation tests whether R3L generalizes beyond the Qwen family under the same protocol.
  • Cross-architecture results: R3L ranks first on 5 of 9 benchmarks and second on the remaining 4, staying within 0.05 of the top method in every column.The benchmarks include WebShop, ScienceWorld, GSM8K, MinervaMath, and DAPO among the first-place results.

C.1 Amplification Factor

Ablations show that moderate positive amplification and one retry provide most of R3L’s gains, while N = 8 offers a practical group-size trade-off and full R3L outperforms Positive Amplification alone.

  • Amplification Factor: WebShop improves from 0.647 at α = 1.0 to 0.663 at α = 3.0, while pushing α beyond 5.0 degrades performance on most tasks.GSM8K performs best at α = 1.0, whereas Math500 peaks at α = 3.0.
  • Component decomposition: Positive Amplification alone reaches 0.807 on ALFWorld versus 0.720 for GRPO, but remains below R3L’s 0.928.The paper attributes roughly 40% of R3L’s total gain over GRPO to Positive Amplification and the remaining 60% to the other components.
  • Rollout structure: R3L uses N/2 base trajectories and N/2 conditionally triggered partial retries, with retry counts ranging between N and 2N depending on reflection triggers.Critique-GRPO instead issues 2N rollouts and discards all but the best refinement.
  • Retry Count: A single retry lifts ALFWorld from 0.807 to 0.928, while 2 retries reach 0.913 and 4 retries reach 0.896 under the fixed N = 8 budget.Additional retries reduce base-trajectory diversity and do not accumulate reflection experience across rounds.
  • Group Size: R3L leads every group-size and benchmark setting, with larger gains from N = 4 to N = 8 and only marginal improvement from N = 8 to 16.ALFWorld rises from 0.886 to 0.928, then to 0.932.

D.2 Per-step Measurements

R3L reduces per-step rollout and training costs while using language-guided correction, pivot localization, and contrastive trajectory structure. Its analyses connect evolving pivots and accurate diagnosis with improved correction of task-specific failures.

  • Per-step cost: R3L has the lowest rollout time across four representative settings, reaching 773.5 s/step versus 1148.0 s for GRPO and 2318.4 s for Critique-GRPO on Qwen3-4B DAPO.The reported reductions are −33% versus GRPO and −67% versus Critique-GRPO.
  • Per-step cost: R3L roughly halves training tokens on multi-step tasks, using 22,300 versus 44,141 tokens/sample on Qwen2.5-1.5B ALFWorld and 26,900 versus 70,176 on Qwen3-4B.The reductions are attributed to pivot-based partial retry and conditional retry activation.
  • Per-step cost: On single-step DAPO tasks, R3L uses more rollout turns than GRPO but still has lower rollout time because its KL-free objective removes the reference-model forward pass.For Qwen3-4B, rollout time is 773.5 versus 1148.0, while turns are 8.72 versus 11.24.
  • Per-step cost: R3L’s train time can be slightly higher than GRPO’s because auxiliary SFT runs on reflection and retry data, but rollout-time leadership favors its overall wall-clock cost.On Qwen3-4B DAPO, train time is 225.8 versus 121.3, while rollout time remains lower.
  • Design comparisons: R3L restarts only unsuccessful trajectories from identified pivots, unlike methods that retry every trajectory or restart from scratch.Its comparison is framed across exploration, credit assignment, and rollout-budget distinctions.
  • Design comparisons: R3L combines step-level credit without external supervision, the GRPO rollout budget, and one new hyperparameter.The method uses contrastive base-retry structure for credit and a single amplification factor.
  • Pivot analysis: Average pivot positions move later during training, from approximately 2 to over 6 steps on ALFWorld and from around 2 to over 12 on ScienceWorld.The shift suggests increasingly preserved prefixes and later-stage refinement as training progresses.
  • Pivot analysis: On ALFWorld with the 7B model, Oracle Agreement rises from 57% at step 100 to 76% at step 400, while correct-pivot retries achieve 85% success versus 43% for wrong pivots.ScienceWorld agreement also rises from 36% to 61% over the same steps.

J.3 In-domain Training Controls

Matched-distribution controls show that post-training improves over zero-shot under both output formats. The controls separate methodological effects from format and cross-distribution effects observed in broader evaluations.

  • Matched-distribution controls: Under matched distributions, all methods improve over zero-shot under both output formats.The controls train and evaluate on GSM8K-on-GSM8K and MATH-on-Math500.
  • Matched-distribution controls: The apparent degradation in broader evaluations is attributed to cross-distribution training rather than a methodological deficiency.This conclusion follows from the matched-distribution controls.
  • Gap decomposition: For Qwen2.5-1.5B GSM8K, switching formats accounts for roughly 28% of the apparent gap, while cross-domain shift accounts for the remaining 72%.Under the same <think>/<answer> format, DAPO training lowers GSM8K from 0.665 to 0.474.

K.1 Training Configuration

R3L is evaluated across multiple model families and task domains with full-history conditioning, off-policy stabilization, and method-specific components that omit explicit KL regularization. The configuration preserves temporal context while adding substantial long-context and compute considerations.

  • Models and infrastructure: Experiments use Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-4B, with Llama-3.2-3B-Instruct used for cross-architecture evaluation.The implementation uses Trinity-RFT, VLLM, and distributed training across 40 NVIDIA A100 and 56 NVIDIA H20 GPUs.
  • Shared configuration: Training uses Adam with learning rate 1e-6, global batch size 96, group size N = 8, and up to 20 epochs with manual early stopping.The behavior and learner policies synchronize at every update step with S = 1.
  • Sequence lengths: The maximum context length is 20,480 tokens, with response limits of 512 tokens for agentic tasks and 4,096 tokens for mathematical reasoning and reflection.These limits accommodate long interaction histories and longer reasoning outputs.
  • Trajectory modeling: R3L uses full-history concatenation rather than context compression, preserving the exact uncompressed cumulative history for reflection and causal diagnosis.This produces a more complex long-context distribution than compression-based baselines.
  • Trajectory modeling: Full-history modeling risks losing efficiency relative to compression approaches because it operates on a more complex long-context distribution.Compression-based methods are described as computationally efficient but dependent on the compression policy and vulnerable to losing historical details.
  • Off-policy stabilization: R3L’s experience replay buffer activates when S exceeds 1 and uses decay-limit randomized priority sampling to balance recency and diversity.The supplied shared configuration otherwise uses S = 1.
  • R3L components: R3L omits explicit KL regularization and relies on Positive Amplification to prevent entropy collapse, eliminating the frozen reference model.This reduces memory and computational overhead.
  • R3L components: Positive Amplification assigns maximum-reward trajectories with R(τ) ≥1.0 an amplified advantage of 1.0 and scales nonnegative advantages by α = 3.0.Negative advantages remain unchanged.
Loading 2601.03715v2…