PaperlayerView exact revision on arXiv

Source-linked AI summary

ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning

Jinhe Bi, Chennan Zhou, Zengjie Jin, Aniri, Shuo Lu, Wenke Huang, Hu Cao, Xun Xiao, Zhihong Zhu, Volker Tresp, Fei Shen, Yunpu Ma, Tat-Seng Chua

arXiv:2608.03972v1cs.AI

TL;DR

When expert models fail on hard problems, their reasoning trajectories are usually discarded, leaving on-policy training without useful structured supervision. ReflectRL instead treats these Golden Negative Trajectories as flawed reasoning to reflect upon and transfers the resulting behavior to direct reasoning, consistently improving performance across evaluated settings with minimal overhead.

  • Problem

    When expert models fail on harder problems, existing trajectory-guided methods typically discard their failed trajectories, leaving potentially informative reasoning signals unused.

  • Method

    ReflectRL learns from Golden Negative Trajectories through reflective rather than imitative reasoning, then transfers the acquired behavior to direct reasoning while preserving on-policy training.

  • Results

    ReflectRL consistently improves reasoning performance across representative RLVR and OPD training settings with minimal additional overhead.

  • Takeaways & Limitations

    Golden Negative Trajectories provide useful reasoning signals, and hard problems can be easier and more effective to solve through reflection than direct reasoning.

Abstract

from arXiv · show

On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.

1 Introduction

ReflectRL turns discarded expert failures into useful supervision by learning reflective reasoning from Golden Negative Trajectories and transferring it to direct reasoning. Across broad on-policy settings, it consistently improves reasoning performance with minimal overhead.

  • Motivation: Expert-generated negative trajectories are typically discarded, motivating their use as an additional reasoning signal for on-policy training.Correct golden trajectories guide generation or shape learning signals, but failed trajectories do not provide the same positive-reward advantage.
  • Reflection Advantage: The Reflection Advantage shows that hard problems can be easier and more effective to solve through Reflective Reasoning than Direct Reasoning.The model reflects on a flawed expert trajectory instead of solving the problem from scratch without external context.
  • ReflectRL: ReflectRL is a lightweight framework that incorporates Golden Negative Trajectories into on-policy training while keeping the base training objective unchanged.It constructs reflective and direct rollouts for RLVR and uses Golden Negative Trajectories as privileged context for OPD.
  • Reflective-to-Direct Policy Transition: Reflective-to-Direct Policy Transition gradually shifts training from Reflective Reasoning toward Direct Reasoning while reducing dependence on expert models.This transition transfers reasoning behavior learned from Golden Negative Trajectories to the desired inference-time behavior.
  • Evaluation: 9 benchmarks, 4 LLM backbones, and 4 training methods show consistent reasoning improvements with minimal additional overhead.The evaluation covers both RLVR and OPD, and further analysis supports the Reflection Advantage and the transfer mechanism.
  • Contribution: OpenR1-GNT-69k provides 69k expert failure trajectories as a released dataset for on-policy training research.The dataset operationalizes Golden Negative Trajectories as a reusable source of reasoning signal.

2 Preliminaries

This section defines the paper’s notation and reviews the GRPO and OPD objectives. It also introduces Reflection Advantage as motivation for the proposed method and analysis.

  • Reflection Advantage: Reflection Advantage is introduced as a concise concept that motivates ReflectRL and its analysis.The supplied passage does not provide its formal definition.
  • Notation: Notation defines q as an input query, D as prompt-answer pairs, o as an autoregressive response rollout, and |o| as response length.The rollout history prefix is denoted by (o1, . . . , os−1).
  • Group Relative Policy Optimization (GRPO): GRPO samples N outputs from a frozen rollout policy, evaluates verifiable rewards, and normalizes rewards within each group to compute token-level relative advantages.The normalized advantage is broadcast to each token position in the corresponding rollout.
  • On-Policy Distillation (OPD): OPD trains a student policy on its own sampled trajectories by minimizing reverse KL divergence from a fixed teacher policy at each token position.Using student-generated states rather than offline expert demonstrations addresses exposure bias and distribution shift associated with offline imitation learning.

3 ReflectRL

ReflectRL learns from Golden Negative Trajectories by eliciting reflective error correction and transferring it into direct reasoning through a scheduled policy transition. The framework preserves standard on-policy objectives while supporting joint direct-reflective optimization and teacher-side privileged information in OPD.

  • Reasoning Interfaces: ReflectRL uses Chat_TempD for Direct Reasoning from the query alone and Chat_TempR for Reflective Reasoning over the query plus a Golden Negative Trajectory.Reflective prompts ask the policy to identify errors, repair the reasoning process, and derive a corrected solution.
  • Reflection Advantage: Positive Reflection Gain (∆ref > 0) means correcting errors in a structured expert failure is easier than generating a correct path from scratch.The Reflection Advantage is defined as the expected correctness difference between reflective and direct reasoning.
  • Reflective-to-Direct Policy Transition: Reflective-to-Direct Policy Transition schedules the target proportion of reflective reasoning with a transition kernel g(t), preventing inference-time dependence on unavailable expert failures.Cosine decay is the default schedule, and in RLVR the terminal allocation rounds to zero, yielding fully direct rollout groups.
  • RLVR Instantiation: In RLVR, ReflectRL jointly optimizes mixed direct and reflective rollouts within one group using a joint group-relative objective.With binary rewards, correct rollouts retain positive advantages and incorrect rollouts retain negative advantages regardless of interface.
  • OPD and Efficiency: In OPD, the GNT remains teacher-side privileged context while the student always uses the direct interface, distilling error-avoidance and correction capabilities without GNT dependence during inference.ReflectRL pre-generates GNTs offline, requires no online expert queries, retains baseline rollout and verifier budgets, and adds only negligible prompt-prefilling overhead.

4 Experiments

ReflectRL is evaluated on diverse reasoning benchmarks by integrating it with multiple on-policy methods and model families. The experiments show consistent performance gains alongside more concise, exploratory, and computationally efficient training, with Golden Negative Trajectories and gradual policy transitions providing key benefits.

  • Datasets and Benchmarks: ReflectRL trains on 69k verified-incorrect Golden Negative Trajectories and evaluates in-domain learning and transfer across 9 reasoning benchmarks.The training set, OpenR1-GNT-69k, is derived from OpenR1-Math-220k using NuminaMath 1.5 prompts and DeepSeek-R1 candidate trajectories verified by Math-Verify.
  • Overall Effectiveness: ReflectRL consistently improves GRPO, DAPO, and OPD across Qwen and LLaMA models ranging from 1.5B to 8B parameters, including transfer beyond mathematical training data.The reported ARC-c gain reflects transfer beyond the mathematical training distribution because OpenR1-GNT-69k and ARC-c have no data overlap.
  • Training Dynamics: ReflectRL produces shorter successful reasoning while improving accuracy, remaining near 420 tokens when GRPO exceeds 800 tokens by step 500.The experiments interpret shorter successful trajectories as a behavioral indicator of more direct and selective problem solving.
  • Training Dynamics: ReflectRL sustains exploration: its entropy remains around 0.15 at step 250, roughly five times GRPO’s value after GRPO falls below 0.03.GRPO entropy falls from 0.97 to below 0.03 by step 250, while ReflectRL’s sustained entropy correlates with continued performance growth.
  • Computational Efficiency: ReflectRL reduces update time to approximately 13 seconds per step versus around 20 seconds for GRPO by offsetting reflective-prefill costs with shorter continuations.The shorter continuations reduce the more expensive part of rollout generation, lowering training time during the Reflective-to-Direct Policy Transition.
  • Mechanistic Validity: Golden Negative Trajectories yield increasing Reflection Gain, while gradual policy-transition kernels outperform sharper schedules; the cosine kernel achieves the best final accuracy.Golden Negative Trajectories are more informative than self-generated failures, and the exact transition functional form matters less than maintaining a gradual shift from Reflective Reasoning to Direct Reasoning.

5 Conclusion · Appendix · A Prompt Templates

ReflectRL learns from Golden Negative Trajectories through reflection rather than imitation, then transfers reflective behavior to direct reasoning while preserving on-policy training. Its prompt templates structure direct reasoning, reflective repair, and OPD teacher guidance around flawed trajectories.

  • 5 Conclusion: ReflectRL learns from Golden Negative Trajectories through reflection rather than imitation.The framework is designed as a lightweight approach for learning from failed expert trajectories.
  • 5 Conclusion: Reflective-to-Direct Policy Transition transfers reflective behavior to direct reasoning while preserving the original on-policy training pipeline.Experiments across RLVR and OPD show consistent gains in accuracy, efficiency, and training stability.
  • Appendix: ReflectRL structures all on-policy inputs with a shared system prompt for systematic step-by-step reasoning inside a designated thought container.This shared structure underlies the prompt templates used during training.
  • A Prompt Templates: The Direct Reasoning interface applies Chat_TempD to present the original problem alongside the system prompt.The structured direct-reasoning format is illustrated in Figure 7.
  • A Prompt Templates: The Reflective Reasoning interface applies Chat_TempR, presenting a pre-generated Golden Negative Trajectory to prompt error identification and reasoning repair.The template uses the trajectory as contextual information rather than as a direct solution to imitate.
  • A Prompt Templates: In OPD, a reflective teacher prompt treats the Golden Negative Trajectory as guidance rather than ground truth, avoids its errors, and derives the correct solution for the student.The OPD template is illustrated in Figure 9.

B Detailed Evaluation Results

This section reports the exact evaluation accuracy numbers corresponding to Figure 4 in the main paper.

  • The section provides exact evaluation accuracy numbers for Figure 4.

C Policy Transition Schedulers … 3. Cosine

The section defines policy-transition schedulers that move guidance probability from reflection toward standard RL across training steps. It specifies shared scheduler hyperparameters and includes cosine, cutoff, linear, prompt-format, and evaluation references.

  • C Policy Transition Schedulers: The scheduler family transitions guidance probability g(t) from high reflection to standard RL over training steps t.The framework evaluates six transition kernels governed by initial and final probabilities, warm-up, decay, and cutoff steps.
  • C Policy Transition Schedulers: The experiments use ph = 0.5, pl = 0.05, tw = 50, td = 250, and tc = 300.For cutoff variants, the final probability is instead pl = 0.0.
  • 1. UFT Cosine + Cutoff: The UFT Cosine + Cutoff subsection is represented among the evaluated transition-kernel variants.The supplied scheduler passage states that six transition kernels are evaluated, including cutoff variants.
  • 3. Cosine: The cosine subsection is included among the six evaluated transition kernels for guidance-probability scheduling.The supplied passage gives the shared scheduler parameters but does not provide the cosine equation itself.
  • C Policy Transition Schedulers: Figure 8 presents the format of the Reflective Reasoning Prompt used for Chat_TempR.This figure reference is listed within the policy-transition-scheduler material.
  • C Policy Transition Schedulers: Table 2 reports detailed Baseline, GRPO, and ReflectRL (+ReflectRL) results across three backbones and 9 ID/OOD benchmarks.The listed backbones are Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, and Llama-3.1-8B-Instruct.
  • C Policy Transition Schedulers: Figure 9 presents the Reflective Reasoning Prompt format for the OPD Teacher.The figure is listed alongside the scheduler and evaluation material.
  • 2. Linear: The linear scheduler passage specifies a decay-horizon condition for t < td.The supplied text fragment contains the condition “td , t < td,” without further formula context.

4. Plateau + Linear · 5. Plateau + Cutoff · 6. Inverse Sigmoid

The section covers three cases: Plateau + Linear, Plateau + Cutoff, and Inverse Sigmoid. The supplied passages identify a successful reflective-reasoning case, present fragments of a piecewise specification, and note that k controls curve steepness.

  • 4. Plateau + Linear: A case study in Plateau + Linear is labeled “Reflective Reasoning (Success).”The supplied figure passage identifies the case study but provides no further result or measurement.
  • 4. Plateau + Linear: The Plateau + Linear subsection includes a case-study figure on reflective-reasoning success.This restates the figure’s role without adding an unsupported outcome.
  • 5. Plateau + Cutoff: Plateau + Cutoff presents the opening fragment of a piecewise formulation.The supplied passage contains only brace notation and does not expose the formulation’s conditions or values.
  • 5. Plateau + Cutoff: Together, the Plateau + Cutoff passages show a multi-line mathematical specification without readable defining cases.The supplied excerpts expose notation fragments but no complete equation.
  • 5. Plateau + Cutoff: A second passage continues the Plateau + Cutoff piecewise-formulation fragment.The continuation is shown only as closing brace notation in the supplied text.
  • 6. Inverse Sigmoid: In Inverse Sigmoid, k controls the steepness of the curve.The supplied passage also references “2,” but does not provide enough context to identify its exact role.

D Related Work

Related work centers on verifiable-reward reinforcement learning, on-policy distillation from stronger experts, and learning from expert trajectories. These approaches enhance reasoning through verifiable rewards, distribution-aware teacher guidance, or imitation of successful demonstrations.

  • Reinforcement Learning with Verifiable Rewards: RLVR has emerged as a predominant post-training paradigm for enhancing reasoning capabilities on complex tasks.The passage contrasts RLVR with traditional RLHF, which relies on reward models trained on human preferences.
  • On-Policy Distillation: On-Policy Distillation transfers knowledge from a stronger teacher to a student using the student’s generated trajectories.It computes KL divergence between teacher and student distributions on on-policy states to mitigate distribution shift and exposure bias in offline SFT.
  • Learning from Expert Trajectories: Traditional expert-trajectory methods primarily treat successful expert trajectories as positive demonstrations for behavioral cloning.Recent on-policy approaches also use correct expert trajectories to initialize policy search or guide rollout generation with high-reward anchors.

E Training Details

ReflectRL training uses OpenR1-Math-220k subsets with bounded 2048-token prompts and responses, eight-response rollouts, and GRPO-based PPO optimization. The configuration emphasizes exploration while enabling reflective-to-direct policy shifts through a zero KL penalty.

  • Data Configuration: Training uses an OpenR1-Math-220k subset with global and validation batch sizes of 128 and 512, respectively.Both maximum prompt and response lengths are limited to 2048 tokens.
  • Rollout Generation: Each query generates N = 8 responses with vLLM, tensor parallel size 2, and temperatures of 1.0 for training and 0.6 for validation.The higher training temperature encourages exploration, while the lower validation temperature supports stable evaluation.
  • PPO & Optimization: Optimization uses the GRPO advantage estimator, actor learning rate 1 × 10^-6, and PPO micro-batch and mini-batch sizes of 64.The entropy coefficient is 0.001 to maintain continuous exploration.
  • PPO & Optimization: The KL penalty coefficient is 0.0, allowing the policy to shift away from the initial reference model during Reflective-to-Direct transition.This setting is intended to prevent interference with the transition mechanism.
  • Hardware and Distribution: Training runs across NVIDIA GPUs with DeepSpeed, enabled gradient checkpointing, and Ulysses sequence parallel size 1.Gradient checkpointing is used to save memory.

F Case Study

The case study contrasts standard direct exploration with ReflectRL’s reflective approach on an arithmetic problem. Direct reasoning fails and earns no useful supervision, whereas a Golden Negative Trajectory enables structured reflection and step-by-step reevaluation.

  • Behavioral Contrast: Direct Reasoning fails on a relatively straightforward arithmetic problem, fragmenting its reasoning into disjoint and nested <think> blocks.The structural violation causes a logical collapse.
  • Behavioral Contrast: Under standard RLVR, the failed trajectory receives a zero reward and an uninformative gradient update.The update does not guide the model toward correct formatting or logic.
  • Behavioral Contrast: Repeated failures in sparse-reward environments can lead to early convergence to sub-optimal policies.This describes the training consequence of receiving no informative supervision.
  • Reflection Advantage: A Golden Negative Trajectory as a contextual anchor elicits Reflective Reasoning that acknowledges prior errors, preserves <think> structure, and reevaluates arithmetic step-by-step.The policy explicitly recognizes shortcomings before recomputing.
Loading 2608.03972v1…