Source-linked AI summary
HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
Woongyeng Yeo, Yumin Choi, Taekyung Ki, Sung Ju Hwang
TL;DR
Long-horizon agents receive sparse rewards that do not identify which intermediate actions need correction, while uniform or misaligned feedback can waste supervision. HINT-SD uses full-trajectory hindsight to target failure-relevant action spans, improving performance by up to 18.80% over dense per-turn feedback while reducing time per training step by 2.26×.
Problem
Sparse rewards and misaligned corrective supervision leave unclear which intermediate actions in long-horizon agent trajectories should be corrected.
Method
HINT-SD analyzes failed trajectories hindsightfully, selects failure-relevant actions, and distills a feedback-conditioned teacher only on their token spans.
Results
HINT-SD improves performance over dense per-turn feedback by up to 18.80% while achieving 2.26× lower time per training step.
Takeaways & Limitations
Deciding where to apply feedback is a central design choice for long-horizon agent post-training.
Takeaways & Limitations
Training still depends on feedback correctly identifying actionable failures and proposing corrections, requiring sufficient initial instruction-following and task-solving capability.
Abstract
from arXiv · showhide
Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only on targeted action spans. Experiments on BFCL v3 and AppWorld show that our method improves over the dense per-turn feedback baseline by up to 18.80 percent while achieving 2.26$\times$ lower time per training step, suggesting that selecting where to distill is a key factor for both effective and efficient long-horizon agent training.
1. Introduction
The introduction frames long-horizon agent training as a relevance-sparsity problem: sparse rewards and misaligned or overly broad feedback fail to identify which actions need correction. HINT-SD addresses this by selecting failure-relevant actions from full trajectories and applying feedback-conditioned self-distillation only to those spans.
- Problem: Long-horizon agent tasks commonly use sparse binary rewards that show whether a task succeeded but provide limited guidance about intermediate actions.This makes reinforcement-learning-based post-training difficult for complex tool-using workflows.
- Related Work: Recent methods densify learning signals with process-level rewards or feedback-conditioned teacher supervision, but they do not fully resolve where corrective supervision should apply.Self-attribution remains scalar, while feedback distillation can misalign supervision or cover irrelevant trajectory turns.
- Motivation: Relevance sparsity means only a small subset of actions in a failed trajectory may require correction, so supervising other turns wastes training budget and can introduce noisy updates.Failure explanations may also appear after the relevant decision, making feedback easy to misplace.
- Method: HINT-SD analyzes full failed rollouts to select failure-relevant steps, then distills a hindsight-conditioned teacher into the student only at those selected actions.The teacher sees the original prefix plus generated feedback, whereas the student sees only the original prefix.
- Contributions: 18.80% improvement over the dense per-turn feedback baseline and 2.26× lower time per training step are reported on BFCL v3 and AppWorld.The method is presented as both effective and more efficient than dense per-turn feedback.
2. Related Work
Prior work improves long-horizon agent training through process-level credit assignment and feedback-conditioned distillation. HINT-SD targets the remaining gap by selecting failure-relevant action spans instead of supervising every turn or only the trajectory as a whole.
- Credit assignment and selective training: Intermediate labels can provide more informative supervision than final outcomes alone for assigning sparse outcome signals to long-horizon agent decisions.AgentEvolver further uses LLM-based self-attribution to score each action’s contribution and convert those scores into process-level rewards for GRPO optimization.
- Feedback-Conditioned Distillation: Reflexion, Self-Refine, CRITIC, OpenClaw-RL, and Skill-SD use verbal, tool-grounded, environment, next-state, or retrieved-skill feedback to revise outputs or provide training signals.The cited agent methods include turn-level rewards or textual hints and teacher conditioning on retrieved skill descriptions.
- Feedback-Conditioned Distillation: HINT-SD addresses inefficient or misaligned supervision by using full-trajectory hindsight to select failure-relevant action spans and distill feedback only at targeted turns.Earlier methods either treat feedback as trajectory-level supervision or analyze it at every turn, even when many turns are correct or irrelevant.
3. Method
HINT-SD uses full-trajectory hindsight analysis to identify failure-relevant actions in failed rollouts, then applies feedback-conditioned self-distillation only to their action spans. This targets corrective supervision precisely where errors occurred while avoiding unnecessary updates to unrelated behavior.
- Hindsight feedback generation: HINT-SD analyzes each failed trajectory globally to select a sparse set of failure-relevant steps and generate corrective feedback for them.The current policy serves as a hindsight analyzer, using the task and complete rollout to explain why selected actions contributed to failure and how they should be corrected.
- Targeted self-distillation: The method improves outcomes by supervising only failure-responsible actions while preserving useful behavior elsewhere.The agent interacts through trajectories of states and actions, with states potentially containing tool outputs, error messages, and other feedback.
- Hindsight feedback generation: Global trajectory context avoids supervising locally noisy actions or later consequences of earlier wrong decisions.Local evidence can be misleading when an apparently valid tool call encodes an assumption whose harmful effect emerges several turns later.
- Targeted self-distillation: For each selected step, feedback-conditioned self-distillation updates the corresponding action span while leaving unrelated rollout regions untouched.The policy is exposed to hindsight feedback as a localized expert, with stop-gradient applied to the feedback-conditioned distribution.
4. Experiments
HINT-SD is evaluated on BFCL v3 and AppWorld against sparse-reward, full-trajectory distillation, and dense local-feedback baselines. HINT-SD-Multi achieves the strongest reported performance, while targeted feedback improves training efficiency and rollout success.
- Benchmarks & Metrics: The evaluation uses BFCL v3 BASE and LONG CONTEXT categories and AppWorld stateful application workflows, with Qwen3-4B-Instruct-2507, four rollouts per task, and 15 training epochs.Hindsight feedback is generated for at most three failure-relevant steps per failed trajectory.
- Main Results: HINT-SD-Multi improves BFCL v3 Avg@4 from 31.56 to 41.88 and Best@4 from 45.00 to 48.75.These gains are measured against the strongest baseline scores.
- Main Results: HINT-SD-Multi improves AppWorld Avg@4 from 9.74 to 18.46 and Best@4 from 19.32 to 31.11.AppWorld scores agents by unit tests over the final application environment.
- Training Dynamics and Efficiency: HINT-SD improves more rapidly and reaches the highest evaluation accuracy, while GRPO and SDPO saturate lower and OpenClaw-RL is less stable.The comparison covers methods using sparse terminal rewards, whole-trajectory supervision, or per-step feedback and rewards.
- Analysis on Target Turn Distribution: Feedback targets span the trajectory: 36.7% occur in turns 1–3, 44.8% in turns 4–8, and 18.5% in turn 9 or later.Later targets increase from 14.0% to 24.5% over training as corrections shift toward later-stage errors.
- Feedback Placement Analysis: Target-turn feedback yields Target - Start gains of +5.99 points on BFCL v3 and +1.72 points on AppWorld.The paired interventions use the same feedback and compare insertion at rollout start versus immediately before the selected target action.
5. Conclusion
HINT-SD is a targeted hindsight self-distillation framework that identifies failure-relevant turns and applies feedback-conditioned distillation only to selected action spans. Experiments show improved performance and training efficiency across BFCL v3 and AppWorld, with analyses indicating that relevant-turn feedback provides more actionable supervision.
- Framework: HINT-SD uses full-trajectory hindsight to identify failure-relevant turns and distills a feedback-conditioned teacher only at selected action spans.This replaces dense feedback applied uniformly across entire failed trajectories.
- Results: HINT-SD improves performance and training efficiency over reward-only optimization, full-trajectory distillation, and dense turn-level feedback baselines on BFCL v3 and AppWorld.The conclusion reports gains across both evaluation settings and all three baseline categories.
- Analysis: Target-turn and feedback-placement analyses show that hindsight targets are distributed across trajectories and provide more actionable supervision when applied at the relevant turn.These analyses support selecting where feedback is applied rather than using a fixed or uniformly dense placement.
Limitations · Appendix
HINT-SD’s training signal depends on correctly identifying actionable failures and proposing corrections that improve task completion. This requires the initial model to have sufficient instruction-following and task-solving capability, although Qwen3-4B-Instruct-2507 shows that a small model can still provide effective feedback.
- Limitations: HINT-SD depends on feedback that correctly identifies actionable failures and proposes corrections that improve task completion.Incorrect hindsight feedback can weaken the training signal.
- Limitations: The method requires an initial model with sufficient instruction-following and task-solving capability to reason about failed trajectories.This capability supports generating useful hindsight feedback for targeted self-distillation.
- Limitations: Qwen3-4B-Instruct-2507 shows that a small model can serve as an effective feedback generator.The result indicates that the required capability is not limited to large models.
A. Additional Experimental Details · A.1. Additional Details on Baselines
The baselines use distinct supervision strategies: selected teacher demonstrations for SFT, terminal rewards for GRPO, trajectory-conditioned feedback for SDPO, and dense next-state signals for OpenClaw-RL. These implementations define the comparison methods against which HINT-SD is evaluated.
- A.1. Additional Details on Baselines: SFT samples up to 10 candidate trajectories from GPT-5.4-mini for each training task.Candidates are executed in the corresponding environment before selecting a demonstration.
- A.1. Additional Details on Baselines: SFT retains a successful trajectory when any candidate satisfies the benchmark-specific success criterion.This selection uses demonstrated task success as the primary criterion.
- A.1. Additional Details on Baselines: If no SFT candidate succeeds, the trajectory with the highest reward provides the strongest available supervision.The fallback selection is applied when all sampled candidates fail the benchmark-specific criterion.
- A.1. Additional Details on Baselines: GRPO computes rewards solely from the final environment state and optimizes a trajectory-level objective.Because intermediate interaction steps cannot be directly evaluated, the method uses the global terminal reward.
- A.1. Additional Details on Baselines: SDPO extends its original single-turn formulation to multi-turn settings using teacher-generated feedback conditioned on the final trajectory and outcome.The global feedback is prepended to the initial prompt as privileged context.
- A.1. Additional Details on Baselines: OpenClaw-RL converts immediate next-state observations into scalar rewards and textual feedback after each action.A judge produces these evaluative signals; rewards guide policy optimization, while feedback enters only the teacher context for on-policy distillation.
- A.1. Additional Details on Baselines: Unlike HINT-SD, OpenClaw-RL provides dense local supervision rather than selecting actions through full-trajectory hindsight.Its supervision is derived from next-state signals observed immediately after each action.
A.2. Additional Implementation Details
The implementation uses dataset-specific optimization settings, LoRA-based parameter-efficient adaptation, and EMA-updated teacher models. Training selects the highest-reward checkpoint and uses TRL, vLLM, and a single NVIDIA H200 GPU.
- Optimization: BFCL uses a 5:1:4 train/eval/test split, while AdamW learning rates are 5 × 10−6 for BFCL and 3 × 10−6 for AppWorld.Both settings use a linear scheduler with warm-up over the first 5% of training steps.
- Model adaptation: LoRA is applied to query and value projection layers with rank r = 32, and teacher parameters are initialized from the student and updated via EMA.The EMA update rate is 0.001.
- Training infrastructure: The highest-reward checkpoint is selected, optimization-based methods use TRL, and on-policy generation uses vLLM.These choices support checkpoint selection and efficient generation during training.
- Training infrastructure: All experiments are conducted on a single NVIDIA H200 GPU, with HINT-SD Single and Multi feedback-generation prompts provided in Figures 4 and 5.The prompts are referenced as implementation details for the two HINT-SD variants.
B. Additional Experimental Results · B.1. Distribution of Selected Hindsight Targets
HINT-SD selects hindsight targets primarily in early and middle interactions, while still identifying later failure-relevant turns. This distribution supports targeted supervision instead of uniform full-trajectory or always-initial feedback.
- B.1. Distribution of Selected Hindsight Targets: Targets concentrate in early and middle interactions, with a mean selected turn of 5.32.The distribution is reported across selected hindsight target turns.
- B.1. Distribution of Selected Hindsight Targets: 10.0% of selected targets occur after turn 10, showing that targets are not restricted to the start.Later turns remain part of the selected hindsight-target distribution.
- B.1. Distribution of Selected Hindsight Targets: Figure 3 aggregates selected hindsight target turns to complement the epoch-wise regions shown in Figure 2.The figure summarizes target-turn counts over training.
- B.1. Distribution of Selected Hindsight Targets: The distribution supports attaching hindsight supervision to selected failure-relevant turns rather than uniformly to the full trajectory.This is the relevance-sparsity view motivating HINT-SD’s targeted supervision.
- B.1. Distribution of Selected Hindsight Targets: The BFCL target-turn distribution covers the first 15 training epochs.Figure 3 reports the count distribution over this training interval.
- B.1. Distribution of Selected Hindsight Targets: Turns after 10 are aggregated into the 11+ bin in Figure 3.The late-turn portion of the distribution is grouped rather than individually separated.
B.2. Qualitative Example of Targeted Hindsight Feedback
HINT-SD localizes hindsight feedback to failure-relevant action spans rather than applying generic episode-level corrections. Qualitative examples show actionable corrections for authentication errors and propagate root-cause feedback across dependent failures.
- Targeted Hindsight Feedback: Target-turn selection makes hindsight feedback action-specific by specifying how each selected action should change.The analyzer identifies a missing Spotify access token at Turn 14 and an undefined token variable at Turn 15.
- Feedback Generation: HINT-SD-Multi selects up to {max_steps} failure-relevant steps and returns localized corrective feedback for each selected step.HINT-SD-Single instead identifies the first mistake and returns one concise correction.
- Qualitative Comparisons: Global hindsight gives one episode-level correction, whereas HINT-SD attaches the root cause to concrete turns and propagates it across later booking-dependent failures.In the BFCL example, the booking is never created, so subsequent insurance and invoice calls fail.
- Qualitative Comparisons: HINT-SD exposes early actionable API-use and authentication errors instead of only summarizing a later episode-level failure.This comparison is shown on an AppWorld file-system task.