PaperlayerView exact revision on arXiv

Source-linked AI summary

Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL

Qizhen Lan, Xi Xiao, Xiangchen Guan, Mengchen Fan, Moule Lin, Jung Im Choi, Lijing Zhu

arXiv:2608.14945v1cs.AIcs.CL

TL;DR

OPSD provides dense teacher supervision, but teacher trust does not indicate whether emphasizing a token serves the current RL objective. ICSD calibrates teacher-directed objective influence into bounded allocation weights and consistently improves trust-only allocation across benchmarks, optimizers, and model configurations.

  • Problem

    OPSD provides dense teacher supervision, but existing trust-based selection does not determine whether an auxiliary update supports the current RL objective.

  • Method

    ICSD estimates teacher-directed first-order objective influence, batch-calibrates it into bounded multipliers, and redistributes supervision without increasing auxiliary-loss mass or requiring another model pass.

  • Results

    ICSD consistently improves trust-only allocation across three benchmarks, two optimizers, and five configurations; at 7B, ALFWorld success rises from 94.5 to 96.1.

  • Takeaways & Limitations

    ICSD reallocates trusted supervision toward objective-supported corrections while preserving the RL objective and auxiliary budget.

  • Takeaways & Limitations

    ICSD’s local sensitivity scores inherit drifting scales and heavy tails from advantages, ratios, and teacher gaps.

Abstract

from arXiv · show

On-policy self-distillation (OPSD) gives language agents dense token-level supervision from a privileged self-teacher on the policy's own trajectories. Existing methods allocate this supervision mainly by teacher trust, but trust does not reveal whether emphasizing a token supports the current policy objective. We call this the trust-utility mismatch and introduce Influence Calibration for Self-Distillation (ICSD). For each supervised token, ICSD measures the first-order response of its importance-weighted RL surrogate contribution to a teacher-directed output perturbation. Batch-adaptive calibration converts this non-stationary signal into a bounded allocation weight while preserving the original auxiliary-loss mass within each action turn. These detached weights affect only the distillation loss and require no additional model pass. Across ALFWorld, WebShop, and Search-QA, ICSD improves all matched aggregate metrics over trust-only allocation under Group Relative Policy Optimization (GRPO) and Group-in-Group Policy Optimization (GiGPO), across two model families spanning 1.5B to 7B. At 7B, it reaches 96.1% ALFWorld success and a WebShop score of 93.1. Frozen-batch analyses show that ICSD reduces teacher-supported mass assigned to objective-opposed tokens from 60.1% to 37.8% and raises cosine compatibility with the RL gradient by 0.192. A companion repository is avail- able at https://github.com/lanqz7766/Influence-Calibration-for-On-Policy-Self-Distillation-in-Agentic-RL.

Introduction

On-policy self-distillation (OPSD) provides dense teacher supervision on the policy’s own rollouts, but existing allocation rules estimate trust rather than utility for the current RL objective. ICSD addresses this trust–utility mismatch with objective influence, batch-adaptive calibration, and mass-preserving allocation without an additional model pass.

  • Motivation: OPSD supplies token-level teacher guidance on states visited by the current policy, narrowing distribution mismatch and complementing sparse trajectory-level rewards.Skill-conditioned variants expose privileged trajectory-derived context to the teacher but not the student.
  • Problem: Existing allocation rules use teacher agreement, uncertainty, disagreement, position, or outcome evidence, but equally trusted tokens can differ in utility for the RL objective.This is the paper’s trust–utility mismatch: teacher support alone does not determine whether an auxiliary update helps the current objective.
  • Method: ICSD estimates each token’s objective influence from a teacher-directed output perturbation using the first-order Taylor response of its local RL-surrogate contribution.The signal uses quantities already available from the policy update, avoiding parameter-space inverse-curvature calculations and per-item gradients.
  • Method: Batch-adaptive calibration converts drifting, heavy-tailed influence scores into bounded relative weights while a conservative fallback preserves signal validity where the deployed loss cannot realize the analyzed direction.The calibration addresses variation from advantages, importance ratios, and teacher gaps across turns and training stages.
  • Results: ICSD preserves exact auxiliary-loss mass within each action turn and yields consistent aggregate gains over trust-only allocation across three benchmarks, two optimizers, and five model configurations.Frozen-batch analyses verify reallocation from objective-opposed toward objective-supported teacher corrections.

Related Work

Prior work on on-policy distillation has developed privileged teachers, selective supervision, and interfaces for incorporating RL feedback. These approaches include trust- and discrepancy-based selectors, advantage modification, gating, filtering, and alternative supervision or credit signals.

  • Privileged and Selective On-Policy Distillation: OPSD uses a stop-gradient privileged self-teacher evaluated on student rollouts, while Skill-SD supplies trajectory-derived skills to the privileged branch.SDAR gates its loss by the teacher–student gap; SAGE-OPD and TurnOPD account for where supervision occurs in long interactions.
  • Privileged and Selective On-Policy Distillation: Other selectors estimate teacher trust from entropy and divergence, token position, or prefix discrepancy.
  • Coupling Distillation to the RL Objective: Prior methods couple distillation to RL by modifying advantages, applying sequence-level gates, changing supervision sources, filtering trajectories, or adding sibling-rollout credit.RLSD, EGRSD, and PBSD modify RL advantages; SG-OPD uses a binary sequence-level gate; DOPD changes supervision; RG-OPD filters by reward–teacher agreement; CRAFT adds an actor loss from sibling-rollout credit.

Method

ICSD augments inherited teacher-trust allocation with first-order influence of teacher-directed output perturbations on the current RL surrogate. It calibrates this influence by action-turn and preserves inherited auxiliary-loss mass while using detached coefficients without an additional model evaluation.

  • Influence-calibrated allocation: ICSD combines privileged teacher scoring with token-level RL objective influence to redistribute inherited trust allocation before the actor update.The privileged branch evaluates on-policy action tokens, while the RL objective supplies token-level influence.
  • Influence-calibrated allocation: Trust weights depend only on the teacher–student gap, whereas ICSD additionally measures each token’s response in the current policy surrogate.The inherited coefficient is gt = Gtrust(∆t), while the surrogate contribution is Jt = bAtρt.
  • Influence-calibrated allocation: ICSD defines influence by perturbing the sampled-token output coordinate according to the teacher–student gap, avoiding per-token inverse-curvature computation.The intervention uses log ρt(ϵ) = log ρt + ϵ∆t and ρt(ϵ) = ρt exp(ϵ∆t).
  • Calibration and composition: Median-based, two-sided turn-conditioned calibration converts variable-scale influence into a bounded relative score, while retaining inherited trust for sign-disagreeing fallback tokens.The calibrated score lies in (0, 1), and tokens with bAt < 0 and ∆t < 0 retain the inherited SDAR coefficient.
  • Actor update: Detached coefficients preserve SDAR’s auxiliary-loss mass within each action turn, and computing them requires no additional model evaluation.Auxiliary gradients pass only through the student log probability.

Experiments and Results

Across matched experiments on ALFWorld, WebShop, and Search-QA, ICSD consistently improves trust-only self-distillation across model families and policy optimizers. Ablations and frozen-batch analyses attribute the gains to combining teacher trust with signed, graded objective relevance.

  • Experimental setup: ICSD is evaluated on ALFWorld, WebShop, seven Search-QA subsets, Qwen2.5 models from 1.5B to 7B, Qwen3 models, GRPO, and GiGPO.Task splits and metrics follow SDAR.
  • Main results: 96.1% ALFWorld success and 93.1 WebShop score/accuracy are reached by Qwen2.5-7B with ICSD, versus 94.5 and 88.4 under SDAR.WebShop accuracy also rises from 78.9 to 84.4.
  • Extended baselines: GiGPO+ICSD gives the strongest ALFWorld and Search-QA averages and highest WebShop accuracy among the extended baselines, without privileged context at test time.Skill-Prompt* and Skill-GRPO* retain retrieved skills during evaluation.
  • Robustness: +11.7 WebShop-accuracy points with GRPO and +7.0 with GiGPO are obtained over SDAR at Qwen2.5-1.5B, while Qwen3-4B ALFWorld success rises from 89.8 to 95.3.The results show the benefit is not specific to GiGPO or Qwen2.5.
  • Allocation ablation: 93.8% is achieved by combining teacher trust with continuous signed influence, exceeding influence-only allocation at 93.0%, Fisher magnitude at 91.4%, and sign-only filtering at 92.2%.Sign-only filtering ties SDAR and remains 1.6 points below ICSD.

Conclusion

ICSD addresses the trust–utility mismatch by reallocating trusted token supervision according to teacher-directed objective influence. Its calibration and mass-matching mechanisms preserve the RL objective and auxiliary budget while improving over trust-only allocation across evaluated settings.

  • Method: ICSD reallocates trusted token supervision according to teacher-directed objective influence.The method targets the trust–utility mismatch in on-policy self-distillation.
  • Method: Batch-adaptive calibration, conservative fallback, and exact action-turn mass matching keep the RL objective and auxiliary budget unchanged.
  • Empirical findings: Across three benchmarks, two optimizers, and five model configurations, ICSD consistently improves over trust-only allocation.

A Derivations and Proofs … B Idealized Natural-Gradient Interpretation

The appendices formalize ICSD’s objective-influence score, calibration invariances, and turn-level conservation properties, then interpret its sign through an idealized natural-gradient argument. They also delimit the interpretation’s assumptions and fallback behavior.

  • A Derivations and Proofs: ICSD uses an output-coordinate influence score instead of classical parameter-space influence, avoiding per-token parameter gradients and inverse-curvature solves inside every actor update.Classical influence requires repeated Hessian–vector-product approximations, whereas ICSD differentiates a scalar output coordinate using quantities already available.
  • A.1 Taylor Remainder for Objective Influence: The perturbed surrogate contribution J(ϵ) = Aρe^(ϵ∆) is smooth, and its first-order coefficient is exact despite an explicit O(ϵ^2) Taylor remainder.The supplied derivation identifies the remainder’s explicit constant as 1/2|A|ρ∆^2e^|ϵ∆|.
  • A.2 Monotonicity and Affine Invariance: The asymmetric-Laplace calibration map is continuous and strictly increasing because both branch derivatives are positive and the branches meet at ΦG(µ̂) = η.This establishes monotonicity across the two branches at the median.
  • A.2 Monotonicity and Affine Invariance: Under positive affine transformations of influence scores, the median and side assignments transform equivariantly, standardized residuals remain invariant, and the calibrated scores are unchanged.The derivation also states that the turnwise sums remain unchanged when the relevant quantities are fixed; a binding scale floor is excluded from the invariance statement.
  • A.3 Turn-Level Conservation and Ordering: ICSD preserves the exact auxiliary coefficient mass within each action turn and globally, so it does not increase the total coefficient budget relative to inherited trust allocation.The degenerate case directly satisfies the same turn-mass identity.
  • A.3 Turn-Level Conservation and Ordering: At equal trust, within-turn allocation orders tokens by the monotone calibrated influence score; opposing signs of bAt and ∆t produce ut < 0 and lower calibrated scores than positive-influence items.The shared positive turn factor preserves this ordering, but negative-influence coefficients need not all fall below their base trust coefficients.
  • B Idealized Natural-Gradient Interpretation: Under positive-definite Fisher geometry, locally active unclipped surrogates, and neglected cross-token correlations, κt > 0 makes the same-token contribution change with the sign of ut.The argument establishes local sign compatibility, not full score-ranking preservation or finite-horizon return improvement.
  • B Idealized Natural-Gradient Interpretation: When ∆t < 0, the analyzed intervention and sampled-token auxiliary update point in different directions; the fallback retains base trust in the double-negative region where ut can falsely appear supportive.The supplied passage identifies this as a support restriction inherited from the teacher-directed intervention.

C Implementation Details · C.1 Training Protocol

The training protocol uses standard SDAR resources and a privileged self-teacher, while specifying GRPO/GiGPO settings and ICSD calibration safeguards. It also adapts minibatch statistics for sparse groups.

  • C.1 Training Protocol: The protocol uses SDAR data splits, SkillBank, keyword-matching retrieval, and a privileged self-teacher.
  • C.1 Training Protocol: GRPO and GiGPO retain their original advantage construction, with GiGPO using γ = 0.95 and unit step-advantage weight.
  • C.1 Training Protocol: ICSD uses trust and influence split parameters of 0.4 and 0.5, a scale floor of 10−4, and a minimum calibration group of eight tokens.
  • C.1 Training Protocol: Sparse groups use minibatch statistics before reverting to the fixed monotone map, alongside model-dependent micro-batches.

C.2 Runtime and Approximation Audit

ICSD adds minimal runtime cost because it requires no additional forward or backward pass. Its unclipped influence approximation is exposed to clipping effects for a small fraction of tokens, which receive disproportionate weighted distillation mass.

  • Compute overhead: 1.7% higher per-token actor-update time and about 0.4% normalized end-to-end overhead occur because ICSD uses detached arithmetic and CDF fitting without extra passes.The measurement uses matched Qwen2.5-1.5B WebShop runs on four H100 GPUs per method; actor updates account for roughly one fifth of step time.
  • Clipping exposure: 0.128% of valid response tokens fall outside the clip interval, yet receive 0.984% of absolute weighted distillation mass because influence is computed on the unclipped surrogate.Across four frozen ALFWorld evidence batches, 0.069% of tokens lie where the clipped surrogate is locally flat, and these tokens receive 0.103% of composite coefficient mass.

C.3 Allocator Implementation

ICSD maps raw influence values with a robust asymmetric location–scale CDF, accounting for heavy tails, drifting scale, and differing spreads between supported and opposed sides.

  • Trust and influence calibration: ICSD uses a fitted asymmetric location–scale CDF for influence calibration, with a robust location estimate and separate one-sided scales.The map Φ_G shares the trust map’s distribution family, while each scale is floored away from zero.

D Full Signal-Combination Audit · E Extended Learning Curves · F Continuous-Landscape Construction Details

The audit shows that ICSD reallocates teacher supervision toward tokens jointly supported by teacher–student gaps and policy advantages, improving objective alignment over base trust. Extended curves and construction details document how this allocation is measured, normalized, and tracked across training.

  • D Full Signal-Combination Audit: ICSD computes detached influence as u_t = bA_tρ_t∆t, calibrates it with group-adaptive statistics, and applies bounded weights while retaining base trust for degenerate turns.Undersized groups fall back to minibatch statistics, and the detached allocation coefficients affect the update after calibration.
  • D Full Signal-Combination Audit: 60.1% to 37.8%: ICSD reduces trusted-conflict mass versus SDAR across four matched ALFWorld batches.The paired reduction is 22.3 points, with a 95% action-turn bootstrap CI of [22.1, 22.6], appearing in every batch.
  • D Full Signal-Combination Audit: 0.192: ICSD increases cosine compatibility with the RL gradient across sixteen frozen batches.The pairing-specific difference versus permuted multipliers is 0.172, tying the improvement to token assignment rather than aggregate scale.
  • D Full Signal-Combination Audit: ICSD assigns 6.2 points more mass to jointly positive tokens and 7.5 points less to positive-gap, negative-advantage tokens.It also removes 1.5 points from negative-gap, positive-advantage tokens, while leaving the double-negative region unchanged.
  • E Extended Learning Curves: The extended appendix includes component-variant validation curves, training-dynamics plots, and full per-task ALFWorld, per-subset Search-QA, and WebShop results.The validation curves use Qwen2.5-1.5B with GiGPO and 150 training updates under identical training conditions.
  • E Extended Learning Curves: 46.6% versus 56.1%: under GRPO, ICSD’s five-step allocation trend ends below the corresponding SDAR trace, while GiGPO ends at 47.8% versus 49.2%.The allocation signal is noisy at minibatch level, the difference widens over GRPO’s final 30 steps, and it narrows under GiGPO.
  • F Continuous-Landscape Construction Details: The continuous mechanism plots use locally smoothed normalized allocation intensity over environment-wise empirical mid-ranks of ∆t and bA_t.The normalized intensity is relative to environment-average deployed allocation; density contours are descriptive only and do not enter the allocator.
  • F Continuous-Landscape Construction Details: Within the teacher-supported set S+ = {t : ∆t > 0}, redistribution is measured by separately ranking negative and positive advantages into five equal-frequency bins.Exact zero advantages remain a separate neutral point, and confidence intervals use action-turn bootstrap resampling over four evidence batches.
Loading 2608.14945v1…