PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang

arXiv:2608.05987v1cs.AIcs.LG

TL;DR

Long-horizon agents receive sparse trajectory-level rewards that obscure pivotal turns. AgentOPSD recursively converts self-distillation gaps into belief revisions for turn-level credit, outperforming GRPO and strong self-distillation baselines across evaluated comparisons.

  • Problem

    Sparse terminal rewards make it difficult to distinguish pivotal decisions from routine, redundant, or misleading actions in long-horizon interactive trajectories.

  • Method

    AgentOPSD aggregates token-level teacher–student gaps at turn boundaries and recursively updates a trajectory-success belief in log-odds space without extra rollouts or a learned critic.

  • Results

    AgentOPSD consistently outperforms GRPO and strong self-distillation baselines across eight aggregate comparisons at two model scales.

  • Takeaways & Limitations

    Belief revision over accumulated history provides a more effective turn-level credit signal than isolated local teacher–student gaps.

  • Takeaways & Limitations

    The turn-evidence construction is a tractable hindsight approximation to ideal Bayesian evidence under conditions detailed in Appendix A.1.

Abstract

from arXiv · show

Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.

1 INTRODUCTION

Agentic RL struggles to assign credit when only trajectory-level rewards are available and uniform advantages obscure pivotal turns. AgentOPSD addresses this by recursively converting turn-level self-distillation evidence into sequential success-belief revisions without extra rollouts or a learned critic.

  • Motivation: Interactive agents receive observations and act across turns, but many environments reveal verifiable rewards only after trajectory termination.This makes intermediate decision contributions difficult to infer in partially observable, long-horizon interactions.
  • Motivation: GRPO broadcasts a trajectory-level outcome advantage uniformly, preventing it from distinguishing pivotal decisions from routine operations as horizons grow.The introduction identifies turn-level credit assignment as essential for locating decisions that determine outcomes.
  • Motivation: OPSD signals mismatch with agentic interaction because multiple tokens form one action, while existing step-aware methods ignore evidence accumulated across preceding turns.These mismatches motivate environment-aligned aggregation and history-dependent credit assignment.
  • Key Insight: AgentOPSD defines turn credit by how much each turn changes the estimated probability of eventual success, treating its self-distillation gap as Bayesian evidence.The belief state represents the probability that the trajectory ultimately succeeds given the interaction history.
  • Method Overview: AgentOPSD aggregates token-level teacher–student log-probability gaps into turn-level evidence and recursively updates a success belief in log-odds space.The belief starts from the average group success rate and requires neither additional rollouts nor a learned critic.

2 METHODOLOGY

AgentOPSD replaces unresolved token-level GRPO credit with a tractable, hindsight-based Bayesian evidence proxy aggregated at the turn level. It recursively updates success beliefs in log-odds space and uses marginal support revisions to reweight verifier-derived advantages without reversing policy-update directions.

  • GRPO assigns one sequence-level advantage to every trajectory token, leaving turn-level credit unresolved.
  • AgentOPSD approximates each turn’s counterfactual contribution using a self-distillation contrast between success-associated teacher behavior and the standard student policy.The approach avoids intractable marginalization over all possible continuations in long-horizon interactions.
  • Summing token-level likelihood contrasts produces turn evidence e_k whose sign indicates whether an action raises or lowers support for eventual success.The resulting e_k is treated as a tractable Bayesian-inspired proxy for ideal hindsight evidence.
  • The recursive belief state uses c_k = γc_{k−1} + e_k and ℓ_k = logit(B_0) + c_k, with γ ∈ (0, 1] geometrically decaying older evidence while retaining the prior logit.B_0 clips the GRPO group success fraction to [ε_0, 1 − ε_0], using ε_0 = 10^-4 to keep log-odds finite.
  • Turn importance is determined by the marginal belief revision, whose sensitivity is greatest under uncertainty and suppressed as support saturates.At γ = 1, the revision reduces to B_{k−1}(1 − B_{k−1})e_k.
  • Normalized recursive credits modulate verifier-derived advantage magnitude through a strictly positive bounded multiplier, preserving the GRPO update direction without a separate distillation loss.The detached self-teacher signal acts only through eA.

3 EXPERIMENTS

Experiments evaluate AgentOPSD across ALFWorld, Search-QA, and WebShop with Qwen2.5-3B/7B, showing gains from its credit construction under matched privileged information. Ablations and sensitivity analyses identify turn-level recursive credit, signed belief revision, prior anchoring, and reshaping weight as important design choices.

  • Experimental setup: AgentOPSD is evaluated on ALFWorld, Search-QA, and WebShop using Qwen2.5-3B/7B-Instruct, with privileged skills used only during training and no external skills at inference.The evaluation reports ALFWorld success rate, Search-QA accuracy, and WebShop Score/Acc.
  • Main comparisons: Under unified retrieved-skill access, AgentOPSD outperforms GRPO+OPSD, Skill-SD, and RLSD on all eight aggregate comparisons and exceeds SDAR on six of eight.The comparison attributes the gain to how the skill-induced teacher–student discrepancy enters learning rather than privileged access.
  • Horizon analysis: −3.59 for RLSD and −2.91 for GRPO points per turn are the reported success points lost as ALFWorld interaction horizons increase.The analysis uses per-sub-task success versus the measured mean number of turns of successful Qwen2.5-7B episodes.
  • Ablation study: 89.1% is AgentOPSD’s full success rate on ALFWorld with Qwen2.5-7B, while turn-level and recursive alternatives reach 85.9% and 82.8%.Per-token accumulation fragments complete decisions, whereas raw local gaps score turns in isolation.
  • Ablation study: 80.5% results from dropping the sign, and 78.9% from removing the empirical state prior, showing that outcome-aligned direction and prior anchoring materially affect success rate.The signed revision distinguishes outcome-consistent from inconsistent belief changes, while the prior sets initial log-odds and the gate’s operating region.
  • Hyperparameter sensitivity: 89.1 at λ=0.5 exceeds 84.4/85.9/83.6 at smaller λ values, while γ and ϵhigh produce only limited or non-systematic changes across the reported tasks.The study uses λ=0.5, γ=0.95, and ϵhigh=0.24 for the full setting; policy clipping is largely unaffected.

4 RELATED WORK

Related work frames AgentOPSD within long-horizon RL, where sparse terminal rewards make turn-level credit assignment difficult. Existing approaches use trajectory-, step-, value-, rollout-, or self-distillation-based signals, whereas AgentOPSD recursively assigns credit from privileged teacher–student evidence.

  • Credit assignment: Sparse terminal rewards make turn-level credit assignment particularly challenging in long-horizon interactive agents.These settings include embodied text worlds, web shopping, and retrieval-augmented question answering.
  • Credit assignment: Standard GRPO broadcasts a trajectory-level advantage uniformly across turns, while GiGPO combines episode-level and step-level advantages from repeated anchor states.GiGPO estimates step-level advantages from repeated anchor states across trajectories.
  • Credit assignment: AgentOPSD derives turn-level evidence from privileged teacher–student likelihood gaps and assigns credit through recursive belief revision.GiGPO and AgentOPSD use complementary signal sources: environment rewards and self-distillation evidence, respectively.
  • Self-distillation: Self-distillation variants remove the need for a separate teacher by conditioning the teacher branch on privileged information available only during training.Recent RLVR studies use the teacher–student log-probability gap to scale or reshape the advantage.
  • Classical credit assignment: Classical credit-assignment methods include PPO with GAE-derived per-step temporal-difference signals, RUDDER’s return decomposition, and rollout- or learned-scorer-based intermediate-value estimation.These approaches address sparse or delayed rewards through value functions, reward redistribution, additional rollouts, or learned scoring.

5 CONCLUSION

The conclusion frames long-horizon agent credit assignment as a challenge of distinguishing pivotal decisions from routine actions under sparse trajectory-level rewards. AgentOPSD addresses this by combining local turn-level signals with their revisions of accumulated belief in eventual trajectory success.

  • Conclusion: AgentOPSD targets long-horizon language-agent credit assignment, where trajectory-level rewards poorly distinguish pivotal decisions from routine or redundant actions.The method is motivated by limited supervision at the trajectory level.
  • Conclusion: Its key insight is that turn-level credit should reflect both a local signal and how that signal revises accumulated belief in eventual trajectory success.This makes credit dependent on sequential belief revision rather than local evidence alone.
  • Conclusion: AgentOPSD operationalizes this insight by aggregating token-level self-distillation gaps at environment-aligned turn boundaries.The supplied conclusion introduces this aggregation as the method’s basis.

A THEORETICAL ANALYSIS … B ALGORITHM

AgentOPSD derives turn-level credit from a self-teacher contrast that approximates Bayes-factor evidence, then recursively reshapes GRPO advantages using belief revisions. Its algorithm adds one teacher forward pass per trajectory without extra rollouts or learned parameters, while preserving boundedness, advantage signs, and GRPO recovery at zero mixing.

  • A.1 FROM THE BAYES FACTOR TO THE SELF-TEACHER CONTRAST: Under success-conditional skill branches and rare-success failure-dominated marginals, the self-teacher contrast recovers the per-turn Bayes factor in the ρ_k → 0 limit.Under the success-conditional assumption alone, the contrast is pointwise mutual information; its sign matches the Bayes factor and preserves turn ranking.
  • A.1 FROM THE BAYES FACTOR TO THE SELF-TEACHER CONTRAST: The marginal action distribution is explicitly modeled as a success/failure mixture weighted by the current success probability ρ_k.This mixture supplies the distributional basis for relating the self-teacher contrast to Bayes-factor evidence.
  • A.2 PROPERTIES OF THE RESHAPING: The reshaping is bounded: with m_k ∈ [1 − b, 1 + b], the modified advantage remains controlled relative to the group-relative advantage.The proof bounds the perturbation by |m_k − 1| ≤ b.
  • A.2 PROPERTIES OF THE RESHAPING: The reshaped advantage preserves the sign of the group-relative advantage, and setting λ = 0 exactly recovers the GRPO gradient.A strictly positive multiplier preserves sign, while λ = 0 makes the reshaped advantage identical to A(i).
  • A.2 PROPERTIES OF THE RESHAPING: Per-turn credit is not identifiable from trajectory return alone: identical-reward trajectories can differ in whether progress is concentrated in one decisive turn or spread across turns.The construction motivates adding an additional per-turn signal beyond the broadcast GRPO advantage.
  • B ALGORITHM: AgentOPSD initializes B_0 from the clipped group success fraction, then recursively updates log-odds with decayed evidence and uses consecutive belief revisions for outcome-aligned turn credit.The algorithm samples G on-policy trajectories, computes verifier rewards and group-relative advantages, standardizes turn signals, clips multipliers, and assigns each turn’s reshaped advantage to its tokens.
  • B ALGORITHM: The method’s overhead is one teacher forward pass per trajectory; belief reshaping is elementwise and requires no additional rollouts or learned parameters.The policy is then updated using the clipped GRPO objective with the reshaped advantages.

C DATASETS · D BASELINE DETAILS

The evaluation spans three multi-turn agentic environments covering embodied household reasoning, web navigation, and search-augmented question answering. Baselines include vanilla, skill-augmented, GRPO-based, and self-distillation methods, matched to AgentOPSD in backbone, interfaces, data, and training budget.

  • C DATASETS: Experiments cover ALFWorld, WebShop, and Search-QA, representing embodied household reasoning, web navigation, and search-augmented question answering.These are all multi-turn agentic environments.
  • C DATASETS: ALFWorld presents six text-based household task categories where agents act from language goals and textual observations until satisfying the goal.Categories include Pick and Place, object inspection, cleaning, heating, cooling, and two-object placement.
  • C DATASETS: WebShop requires agents to search catalogs, inspect products, select attributes, and purchase items satisfying user constraints on 128 fixed validation tasks.The validation tasks follow Feng et al. (2025).
  • C DATASETS: Search-QA follows Search-R1 across seven datasets, with NQ and HotpotQA in-domain and the remaining datasets held out.Agents issue queries, inspect E5-retrieved documents, and synthesize answers.
  • D BASELINE DETAILS: Baseline groups include Vanilla, Skill-Prompt∗, GRPO, Skill-GRPO, Skill-GRPO∗, and several self-distillation variants.Unless marked ∗, methods use only the standard task prompt and environment interaction history; ∗ methods additionally receive retrieved skills during validation and testing.
  • D BASELINE DETAILS: Vanilla uses the instruction-tuned backbone without post-training, while Skill-Prompt∗ prepends a retrieved task-relevant skill only during validation and testing.Skill-Prompt∗ keeps parameters frozen and measures inference-time skill value without parameter updates.
  • D BASELINE DETAILS: GRPO normalizes grouped terminal rewards into relative advantages for a clipped surrogate objective, whereas Skill-GRPO injects retrieved skills during training and varies their availability at inference.Skill-GRPO removes the skill at inference; Skill-GRPO∗ retains it.
  • D BASELINE DETAILS: Self-distillation baselines differ in signal granularity and optimization: OPSD and Skill-SD provide teacher guidance, RLSD and SDAR modulate GRPO-related updates, and StepOPSD applies local signals per turn.GRPO+OPSD jointly optimizes trajectory-level GRPO and token-level OPSD losses; StepOPSD does not use history across steps.

E EVALUATION METRICS

The evaluation uses task-specific metrics: overall success for ALFWorld, aggregated accuracy for Search-QA, and normalized completion and exact-completion rates for WebShop.

  • ALFWorld: ALFWorld reports overall success as the sample-weighted average of six per-category success rates.Success is the fraction of episodes reaching the specified goal.
  • Search-QA: Search-QA reports overall accuracy aggregated across evaluation questions from seven datasets.
  • WebShop: WebShop reports normalized completion Score and exact-completion success rate Succ.Score averages partial-constraint satisfaction and is scaled by 100; Succ. is the percentage of episodes satisfying all specified requirements.

F HYPERPARAMETERS

AgentOPSD uses one shared hyperparameter setting across environments and model scales, with turn-level credit reshaping, decayed gap accumulation, and specified PPO clipping. The optimization recipe also fixes dual clipping, gradient clipping, entropy regularization, PPO epochs, and distributed training, while remaining settings vary by environment.

  • AgentOPSD hyperparameters: AgentOPSD uses a single shared setting across environments and model scales: λ=0.5, b=0.2, γ=0.95, and PPO clipping ϵlow=0.2 / ϵhigh=0.24.It operates at turn-level granularity and initializes the state prior with the clipped empirical group success rate ¯R = S/G.
  • Optimization recipe: The shared optimization recipe includes dual-clip constant c=3.0, gradient clipping 1.0, entropy coefficient 0.001, one PPO epoch per update, and FSDP on a single node.Learning rate, group size, PPO clip range, and KL coefficient are also shared through Table 3.
  • Environment-specific settings: Remaining hyperparameters are environment-specific and are summarized separately in Table 4.The shared-setting policy therefore applies to the listed common optimization and AgentOPSD-specific parameters, not every experimental setting.

G TRAINING DYNAMICS · H PROMPT

The paper reports AgentOPSD’s training dynamics across model scales and environments, tracking teacher–student gaps and rewards, and specifies full environment-specific prompt templates for training and inference.

  • G TRAINING DYNAMICS: Per-environment training configurations are listed separately, while optimization settings shared across environments appear in Table 3.Table 4 provides the environment-specific settings.
  • G TRAINING DYNAMICS: The teacher–student gap ¯δ is examined for both Qwen2.5 model scales across all three evaluation environments.Figure 4 covers ALFWorld, WebShop, and Search-QA.
  • H PROMPT: The {skill context} placeholder is filled with the retrieved skill during training and left empty at inference time.This behavior applies across the full prompt templates.
  • G TRAINING DYNAMICS: Reward curves are reported for Qwen2.5-3B and Qwen2.5-7B training on ALFWorld, WebShop, and Search-QA.Figure 5 provides the corresponding reward trajectories.
  • H PROMPT: The Search-based QA prompt frames the agent as an expert answering step-by-step and includes task details, step count, and tagged search history.Past queries are wrapped in <search> </search>, while returned results are wrapped in <information> </information>.
Loading 2608.05987v1…