Source-linked AI summary
TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning
Changle Qu, Sunhao Dai, Hengyi Cai, Yuqi Zhou, Xinran Chen, Simon, Jun Xu
TL;DR
TurnSight addresses weak turn-level credit assignment in tool-integrated reasoning by deriving execution-conditioned hindsight supervision and using it to modulate reinforcement-learning advantages. Across three benchmarks, it outperforms all baselines and improves the previous best method by 7.7% in overall average performance.
Problem
Trajectory-level and token-level supervision provide limited, potentially incoherent credit assignment for distinct tool interactions in long-horizon TIR.
Method
TurnSight aggregates execution-conditioned, multi-horizon hindsight evidence at the turn level and uses reliable signals to modulate RL advantages without changing their direction.
Results
7.7% improvement over the previous best method in overall average performance, with TurnSight consistently outperforming all baselines across three benchmarks.
Takeaways & Limitations
TurnSight provides fine-grained, reliable credit assignment while preserving RL exploration across diverse multi-turn tool-use scenarios.
Abstract
from arXiv · showhide
Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at https://github.com/quchangle1/TurnSight.
Introduction
TurnSight addresses multi-turn TIR’s credit-assignment challenges by deriving supervision from on-policy tool-execution outcomes and aligning it with complete interaction turns. It combines multi-horizon hindsight assessments with reliable selection to modulate RL advantages without changing their direction.
- Motivation: TIR interleaves reasoning with external tool interactions, creating a temporal credit-assignment problem because decisions can have different causal effects on final outcomes.Agents repeatedly select tools, construct arguments, observe execution results, and adapt subsequent behavior.
- Motivation: Trajectory-level methods such as GRPO propagate a single advantage across all generated actions, assigning similar credit to decisions with different causal effects.This limitation is illustrated for outcome-based reinforcement learning with verifiable rewards.
- Motivation: Reference-trajectory supervision can become state-misaligned after the learned policy makes a different tool call, while OPSD evaluates on-policy behavior using privileged training-time views.Tool execution changes the environment, making the trajectory itself a source of state-aligned hindsight supervision.
- Motivation: Token-level self-distillation can assign conflicting credit within one interaction, whereas TIR’s fundamental decision unit is a complete tool interaction.A turn comprises reasoning, tool selection, and argument construction; formatting and argument variations can change token-level signal magnitude or sign.
- TurnSight: TurnSight aggregates token-level hindsight into turn-level signals, constructs execution-conditioned views across lookahead horizons, selects reliable supervision, and modulates RL advantages without changing their direction.The method is designed to satisfy on-policy state alignment and interaction-level coherence, with experiments spanning in-domain and out-of-domain benchmarks.
Related Work
Related work improves tool-integrated reasoning through task-level RLVR, turn-level supervision, and on-policy self-distillation. However, reference-trajectory methods constrain exploration, motivating TurnSight’s ground-truth-free combination of RL and self-distillation for fine-grained credit assignment.
- RLVR for TIR: RLVR methods, especially GRPO-style algorithms, optimize tool-integrated reasoning with task-level feedback, using sparse outcome rewards or trajectory-level supervision.These approaches improve TIR by directly optimizing models, but early methods provide relatively coarse supervision across reasoning turns.
- Turn-level supervision: Turn-level methods use LLM evaluation or compare predicted tool calls with ground-truth executions, but annotated reference trajectories make optimization closer to imitation learning and limit exploration.TurnSight instead combines RL with on-policy self-distillation without ground-truth trajectories.
- On-policy self-distillation: On-policy self-distillation evaluates student trajectories with a privileged teacher view from the same model, providing dense supervision on states actually visited by the student.Recent methods use teacher–student disagreement to reweight RL advantages, allocate fine-grained credit, or bound distillation interventions rather than directly matching teacher distributions.
Preliminaries
The paper formulates tool-integrated reasoning as policy-driven interaction with tools that produces turn-structured trajectories. It then presents GRPO optimization and OPSD, which uses privileged hindsight to modulate advantage magnitudes without changing their directions.
- TIR formulation: An agent with policy πθ interacts with M schema-defined tools to produce a trajectory of turns containing reasoning and JSON-argument tool calls until a final answer.Non-terminal turns are represented as sk = (nk, Ck, ok), where nk is reasoning and Ck is the set of issued tool calls.
- Agentic RL framework: GRPO samples G trajectories per prompt and computes policy-gradient advantages by comparing same-prompt trajectory rewards instead of training a critic.This group-wise comparison provides a prompt-dependent baseline and reduces value-function estimation cost.
- Agentic RL framework: The GRPO objective uses importance-sampling ratios, normalized group-relative advantages, clipping, and KL regularization, optimizing policy-generated tokens while excluding environment observations.The KL term regularizes toward the reference policy πref, and observation tokens receive no gradient computation.
- OPSD mechanism: OPSD compares student predictions from rollout information with detached teacher predictions that additionally access privileged training-time information.For each sampled token, the resulting log-probability gap measures how privileged information changes confidence in the realized token.
- OPSD mechanism: Rather than matching distributions directly, OPSD uses the hindsight log-probability gap to modulate RL advantage magnitudes while preserving their directions.This prevents the teacher from determining the update direction.
Our Approach: TurnSight
TurnSight derives privileged supervision from execution-conditioned hindsight on the student’s own trajectory, aggregates it at the interaction-turn level, and selects reliable signals across multiple lookahead horizons. It then normalizes the selected evidence across sibling rollouts and uses bounded, sign-aware weighting to modulate RL advantages while preserving the base objective.
- Execution-Conditioned Privileged Context: TurnSight constructs privileged context from the student trajectory’s execution outcomes, aligning hindsight supervision with the states actually visited.Ground-truth answers and reference rollouts may provide misaligned supervision because they are not conditioned on the student’s trajectory.
- Turn-Level Aggregation: TurnSight aggregates token-level hindsight gaps into one assessment per interaction turn, avoiding inconsistent credit assignment within tool-use decisions.The turn-level signal is shared by all policy tokens in the turn and serves as the evidence unit for teacher selection.
- Multi-Lookahead Hindsight Views: Using lookahead depths DH = {1, 2, 3}, TurnSight compares complementary hindsight views spanning immediate execution quality and downstream trajectory contributions.Different interaction states may require different horizons because some tool-use decisions are assessed immediately while others depend on later interactions.
- Direction-Consistent Teacher Selection: TurnSight determines the consensus direction by majority voting, then selects the largest hindsight signal among teachers consistent with that direction.This filters isolated conflicting assessments while retaining the strongest reliable supervision rather than indiscriminately combining all teachers.
- Group-Relative Evidence Normalization and Advantage Integration: The selected signal is normalized across valid interactions in sibling on-policy rollouts, then bounded sign-aware weighting amplifies aligned RL updates and attenuates disagreement.The normalized evidence reduces prompt-specific scale variation; the integrated advantage interpolates the base advantage with its hindsight-modulated counterpart while retaining the standard GRPO objective.
Experiments
Experiments show that TurnSight’s turn-level, multi-lookahead hindsight supervision consistently outperforms existing RL and self-distillation baselines across in-domain and out-of-domain tool-use benchmarks. Ablations further attribute its gains to turn-level aggregation, group normalization, multi-teacher selection, full-trajectory coverage, and balanced hindsight weighting.
- Main Results: Fine-grained tool- and turn-level credit assignment transfers better than outcome-only optimization, while GRPO generalizes poorly to out-of-domain BFCL and ToolHop.MatchTIR’s strong performance further establishes accurate turn-level supervision as an effective baseline for on-policy hindsight methods.
- Main Results: On-policy self-distillation alone is insufficient: SDPO, RLSD, and SDAR outperform vanilla GRPO but remain behind methods with explicit turn-level credit assignment.Privileged information from ground-truth trajectories or successful rollouts may not supervise every intermediate decision accurately, especially in long-horizon interactions.
- Main Results: TurnSight consistently outperforms all baselines across three benchmarks and improves the previous best method by 7.7% in overall average performance on Qwen3-8B.The largest improvements occur on challenging BFCL Long Context and Miss Parameter subsets.
- Ablation Studies: Removing turn-level aggregation, group normalization, or multi-lookahead teacher selection causes substantial degradation, showing that TurnSight needs coherent, calibrated, and temporally complementary hindsight signals.Shorter lookaheads capture immediate interaction quality, whereas longer horizons expose delayed consequences; normalization makes signal magnitudes comparable across rollout groups.
- Ablation Studies: Tool-result-only teacher contexts perform best, while adding or using ground-truth answers degrades hindsight supervision; among fixed horizons, one-step lookahead is strongest.Longer fixed horizons introduce irrelevant interactions, and teacher fusion cannot fully resolve conflicting signals, whereas direction-consistent selection performs best.
- Ablation Studies: Full-trajectory supervision improves average performance by 2.36% over first-five-turn supervision and 3.88% over last-five-turn supervision, while λ = 0.5 and ϵw = 0.5 are optimal.The unimodal trends indicate that hindsight must balance local interaction credit with global trajectory-level guidance.
Conclusion
The conclusion identifies on-policy state alignment and interaction-level coherence as key requirements for TIR credit assignment, and presents TurnSight as a turn-level hindsight self-distillation framework addressing them.
- TurnSight targets two credit-assignment requirements in TIR: alignment with on-policy execution states and interaction-level coherence.
- TurnSight derives execution-conditioned, multi-horizon hindsight assessments from on-policy tool execution outcomes.
- These hindsight assessments modulate RL advantages without altering their optimization direction.
Appendix · Benchmark and Evaluation Details
The evaluation combines FTRL for post-training and in-domain assessment with BFCL and ToolHop for generalization to unseen tools and interaction patterns. It specifies benchmark composition, task structures, correctness criteria, and metrics for executable multi-turn tool reasoning.
- Benchmark and Evaluation Details: FTRL uses 2,215 executable training instances and four 50-query test categories with different dependency structures.The categories include Single-Hop and Parallel Single-Hop tasks, among others described in the benchmark setup.
- Benchmark and Evaluation Details: FTRL provides automatically constructed queries, local tool environments, and programmatically verifiable feedback for executable tool-use tasks.These properties support post-training and evaluation within the same environment family.
- Benchmark and Evaluation Details: FTRL measures subtask-level behavior with Solve-P, Solve-R, and Solve-F1 based on generated calls, solved subtasks, and required subtasks.The metrics use Ncall, Nsolved, and Nreq to characterize tool-call efficiency and subtask resolution.
- Benchmark and Evaluation Details: BFCL evaluates function-calling and agentic behavior across four multi-turn subsets and two agentic subsets.The multi-turn subsets are Base, Missing Function, Missing Parameter, and Long Context.
- Benchmark and Evaluation Details: BFCL tests stateful calling, unavailable tools, incomplete arguments, and tool selection amid lengthy specifications with distracting information.These conditions correspond to the benchmark’s four multi-turn subsets.
- Benchmark and Evaluation Details: BFCL correctness relies on verified environment states, response-path validity when applicable, response requirements for read requests, and exact matching for designated agentic answers.Official accuracy is reported for each subset and as their average.
- Benchmark and Evaluation Details: ToolHop contains 995 multi-hop queries and 3,912 locally executable tools requiring dependent calls over a collection different from FTRL’s training environment.It evaluates transfer to new tools and interaction structures using final-answer accuracy, counting predictions correct only when they match the verified target.
Additional Implementation Details
Experiments use a shared verl-based training setup, turn-level hindsight implementation, and controlled evaluation environments. The implementation includes prompt-group normalization, multi-horizon teacher evaluation, and standardized tool-execution protocols.
- Training Configuration: All experiments use verl, initialize Qwen3-4B and Qwen3-8B from corresponding base checkpoints, train for three epochs, and omit an explicit KL penalty.Table 4 reports the shared training and rollout configuration.
- Prompt-Group Normalization: Prompt-group normalization pools selected hindsight gaps across 16 rollouts at valid policy-token positions and shares statistics across all turns.Masked positions, including environment-generated observation tokens, are excluded from the mean and standard deviation.
- Multi-Horizon Teacher Evaluation: Multi-horizon teachers share a frozen reference policy while using lookahead depths 1, 2, and 3 to produce turn-level hindsight assessments.Teacher-forced token-level gaps are aggregated into turn-level assessments, alongside teacher-selection outputs and advantage-modulation weights.
- Evaluation Protocol: All compared methods use identical benchmark environments and decoding settings, with BFCL Web Search using Serper’s Google Search endpoint.Other BFCL subsets, FTRL, and ToolHop use their released local execution environments.