ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning

Motivation

• Existing trajectory-guided methods rely on correct expert demonstrations, leaving failed expert trajectories unused because their structured reasoning cannot guide standard policy updates or be imitated as positive demonstrations.The introduction frames leveraging these discarded but informative negative trajectories as an open question.Source: §1 · pp. 1–2

Reflection Advantage

• GNTs yield a stronger Reflection Advantage than target-model or weaker-model failures, retaining higher process rewards and valid reasoning prefixes before localized errors.Interventions show that both the valid prefix and error region contribute to the gain.Source: §1 · p. 2, Figure 1 · p. 2

Framework

• ReflectRL keeps the base objective unchanged, combining Reflective and Direct Reasoning rollouts for RLVR and using GNT context on the OPD teacher side.Its Reflective-to-Direct Policy Transition gradually shifts training rollouts toward Direct Reasoning to reduce dependence on expert models.Source: §1 · pp. 2–3

Evaluation

• ReflectRL consistently improves reasoning performance with minimal overhead across 9 benchmarks, 4 LLM backbones, and 4 training methods spanning RLVR and OPD.The reported analysis also supports the Reflection Advantage and the role of Reflective-to-Direct Policy Transition in transferring GNT benefits to Direct Reasoning.Source: §1 · p. 3

Contributions

• ReflectRL introduces Golden Negative Trajectories as on-policy training signals and releases OpenR1-GNT-69k, a dataset of 69k expert failure trajectories.The framework is presented as a lightweight way to integrate these trajectories through Reflective-to-Direct Policy Transition.Source: §1 · p. 3

Notation

• Notation defines queries q, prompt-answer dataset D, ground-truth answers, autoregressive response rollouts, their history prefixes, and response lengths.A rollout is denoted o = (o1, …, o|o|), generated under policy πθ given q.Source: §2 · pp. 3–4

Group Relative Policy Optimization (GRPO)

• GRPO samples N outputs per prompt from a frozen rollout policy, scores them with verifiable rewards, and normalizes group rewards into token-level advantages without a separate critic.The normalized advantage for each rollout is broadcast to its token positions.Source: §2 · p. 4

On-Policy Distillation (OPD)

• OPD trains a student on its own sampled trajectories by minimizing reverse KL divergence from the student’s predictive distribution to a fixed teacher’s.Using student-generated states rather than offline expert demonstrations alleviates exposure bias and distribution shift in offline imitation learning.Source: §2 · p. 4

3.5 Computational Overhead

• ReflectRL requires no online expert queries and preserves rollout and verifier budgets and the baseline update rate; its only extra cost is GNT-token prefilling, which declines to zero with the reflective proportion.GNTs are pre-generated offline, and the prefilling overhead is described as a negligible fraction of training time.Source: §3 · p. 4, §3.5 · p. 7

3 ReflectRL

• ReflectRL adds no auxiliary loss, reward changes, or inference-time interface changes, preserving the original RLVR/OPD objectives.This makes the framework lightweight and plug-and-play within existing on-policy training.Source: §3 · p. 4

3.1 Direct & Reflective Reasoning Interfaces

• Chat_TempD prompts direct solutions from the query alone, while Chat_TempR presents a GNT for the policy to identify errors, repair reasoning, and derive a corrected solution.The Reflection Advantage is the empirical condition Δref > 0, where correcting a structured expert failure is easier than solving from scratch.Source: §3.1 · pp. 4–5

3.2 Reflective-to-Direct Policy Transition

• ReflectRL gradually reduces reflective rollouts with a cosine-decay transition, compiling GNT-elicited correction behavior into the direct pathway and ending with fully direct RLVR rollout groups.The schedule sets the reflective-rollout count from the target proportion; its terminal allocation rounds to zero in RLVR.Source: §3.2 · p. 5

3.3 ReflectRL in RLVR

• In RLVR, ReflectRL jointly optimizes mixed direct and reflective rollouts using group-relative advantages, so binary rewards do not systematically disadvantage direct reasoning.Correct rollouts retain positive advantages and incorrect rollouts negative advantages in every non-degenerate group, regardless of interface.Source: §3.3 · pp. 5–6
• ReflectRL consistently boosts GRPO, DAPO, and OPD on in-distribution and out-of-distribution Qwen2.5-Math-7B evaluations, with competitive performance against prior zero-RL paradigms.The supplied Table 1 caption reports these comparisons but provides no cell values.Source: Table 1 · p. 6

3.4 ReflectRL in OPD

• Across three Llama and Qwen model sizes, Figure 4 reports that ReflectRL outperforms standard GRPO on nine in-distribution and out-of-distribution benchmarks.The caption describes strong generalization across model sizes and families without giving benchmark-level values.Source: Figure 4 · p. 7
• In OPD, GNTs are teacher-side privileged context: the teacher reflects on errors to guide targets, while the student retains direct inputs and distills correction behavior without GNT dependence at inference.The transition schedule is applied across a batch because OPD generates one student rollout per query.Source: §3.4 · pp. 6–7

4.1 Experiment Setup

• ReflectRL trains on 69k verified-incorrect DeepSeek-R1 trajectories and is evaluated on nine in- and out-of-distribution reasoning benchmarks across four on-policy methods and models from 1.5B to 8B parameters.The benchmark suite includes six in-distribution mathematical tasks and three out-of-distribution tasks.Source: §4.1 · pp. 7–8

4.2 RQ1: Overall Effectiveness

• ReflectRL consistently improves GRPO, DAPO, and OPD across Qwen and LLaMA models from 1.5B to 8B parameters, with ARC-c gains indicating transfer beyond the mathematical training distribution.OpenR1-GNT-69k and ARC-c have no data overlap; the paper links the transfer to ARC-c’s causal and logical checks and distractor rejection.Source: §4.2 · p. 8

Takeaway 2: ReflectRL elicits more concise reasoning through Reflective-to-Direct Policy

• 800+ response tokens at step 500 for GRPO contrast with ReflectRL’s near 420; at step 250, GRPO entropy falls from 0.97 to below 0.03, while ReflectRL remains around 0.15.The paper interprets the shorter trajectories as more concise reasoning and the sustained entropy as supporting continued exploration and performance growth.Source: Takeaway 2: ReflectRL elicits more… · pp. 8–9

4.3 RQ2: Computational Efficiency

• Approximately 13 seconds per step for ReflectRL, versus around 20 for GRPO, reflects lower eventual training time alongside higher reasoning performance.Offline GNTs and unchanged rollout and verifier budgets limit added cost; there is no persistent memory overhead, and temporary reflective-context KV-cache use declines to zero during transition.Source: §4.3 · p. 9

4.4 RQ3: Mechanistic Validity

• GNTs provide more useful reflection than low-quality or mismatched failures, and combining a valid prefix with the localized error yields the largest gain.Shuffled trajectories or final-answer-only context underperform Direct Reasoning, while correct expert-trajectory guidance causes severe policy drift and collapse; GNT guidance remains stable.Source: Figure 5 · p. 8, §4.4 · pp. 9–10
• Smooth policy-transition kernels yield stronger, more stable performance than abrupt or overly rapid schedules, with cosine achieving the best final accuracy.Inverse-sigmoid follows a similar trend, suggesting gradual transition matters more than the precise functional form; sharper transitions remove reflective guidance too early.Source: Figure 6 · p. 9, §4.4 · p. 10

5 Conclusion

• ReflectRL learns from Golden Negative Trajectories through reflection rather than imitation.The framework is presented as lightweight.Source: §5 · p. 10
• Reflective-to-Direct Policy Transition transfers reflective behavior to direct reasoning while preserving the original on-policy pipeline, with consistent gains in accuracy, efficiency, and training stability across RLVR and OPD.Source: §5 · p. 10

A Prompt Templates

• A shared system prompt structures on-policy inputs and guides systematic step-by-step reasoning within a designated thought container.Source: A Prompt Templates · p. 15
• The direct reasoning interface uses Chat_TempD to present the original problem alongside the system prompt.Source: A Prompt Templates · p. 15, Figure 7 · p. 15
• The reflective reasoning interface uses Chat_TempR to present a pre-generated Golden Negative Trajectory as context for identifying errors and repairing the reasoning process.Source: A Prompt Templates · p. 15
• In OPD, the teacher’s reflective prompt treats the Golden Negative Trajectory as privileged guidance rather than ground truth, directing the teacher to avoid its errors and derive a correct solution for the student to match.Source: A Prompt Templates · p. 15

• The section reports exact evaluation-accuracy numbers corresponding to Figure 4 in the main paper.Source: B Detailed Evaluation Results · p. 15

C Policy Transition Schedulers

• Six transition kernels govern the guidance probability g(t) as training moves from high reflection to standard RL.The generalized expressions use initial and final probabilities, warm-up steps, decay horizon, and hard cutoff step.Source: C Policy Transition Schedulers · p. 15
• Experiments set ph = 0.5, pl = 0.05 (or 0.0 for cutoff variants), tw = 50, td = 250, and tc = 300.The probability drop is denoted ∆p = ph −pl.Source: C Policy Transition Schedulers · p. 15
• Figure 8 presents the format of the Reflective Reasoning Prompt (Chat_TempR).Source: Figure 8 · p. 16

4. Plateau + Linear

• Figure 10 presents a successful Reflective Reasoning case study.Source: Figure 10 · p. 18

6. Inverse Sigmoid

• In the Inverse Sigmoid section, k controls the curve’s steepness.Source: §6 · p. 18

Reinforcement Learning with Verifiable Rewards (RLVR)

• ReflectRL uses Golden Negative Trajectories to elicit reflective reasoning and address sparse rewards when RLVR fails to produce correct trajectories on complex problems.RLVR uses rule-based verifiers or deterministic outcomes, but relies on occasional correct samples; ReflectRL adds constructive signals through reflection.Source: D Related Work · p. 18

On-Policy Distillation (OPD)

• ReflectRL gives the teacher a Golden Negative Trajectory as privileged context, enabling higher-quality target distributions that distill error correction into the student’s direct reasoning pathway.Unlike standard OPD, the student does not rely on the external trajectory during inference.Source: D Related Work · p. 19

Learning from Expert Trajectories

• Unlike methods that imitate successful expert trajectories [10] or use correct trajectories to guide on-policy search [9], ReflectRL uses structured expert failures to elicit reflective reasoning.Prior approaches typically discard expert failures, while ReflectRL treats them as contextual scaffolds rather than negative contrastive targets.Source: D Related Work · p. 19

• The training pipeline is implemented based on the open-source verl framework.Source: E Training Details · p. 19

Data Configuration

• Training uses a subset of OpenR1-Math-220k, global batch size 128, validation batch size 512, and 2048-token prompt and response limits.Source: E Training Details · pp. 19–20

Rollout Generation

• Each query generates N = 8 responses with vLLM and tensor parallel size 2; sampling temperature is 1.0 for training and 0.6 for validation.Source: E Training Details · p. 20

PPO & Optimization

• Optimization uses the GRPO advantage estimator, actor learning rate 1 × 10−6, PPO micro-batch and mini-batch sizes 64, entropy coefficient 0.001, and KL penalty coefficient 0.0.The zero KL penalty is intended to let the policy shift freely from the initial reference model without interfering with the Reflective-to-Direct transition.Source: E Training Details · p. 20

Hardware and Distribution

• Training runs across NVIDIA GPUs with DeepSpeed integration, gradient checkpointing, and Ulysses sequence parallel size 1.Source: E Training Details · p. 20

F Case Study

• Direct Reasoning fails the arithmetic problem and violates the structural constraint by fragmenting its reasoning into disjoint, nested <think> blocks.Under standard RLVR, the trajectory receives zero reward and provides an uninformative gradient update.Source: F Case Study · p. 20
• In sparse-reward environments, repeated failures of this kind can lead to early convergence to sub-optimal policies.Source: F Case Study · p. 20
• Using a Golden Negative Trajectory as a contextual anchor, ReflectRL critiques the prior attempt, follows the <think> structure, and correctly recomputes the arithmetic.The paper characterizes this as converting a zero-reward exploration path into a dense, high-reward learning signal.Source: F Case Study · p. 20