PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent On-Policy Self-Distillation

Guibin Zhang, Jiayang Lyu, Ran Sun, Xinlei Yu, Haoyu Zhao, Qibing Ren, Shuicheng Yan

arXiv:2608.13040v1cs.LGcs.CL

TL;DR

Existing on-policy self-distillation methods depend on hand-designed privileged context, limiting its end-to-end learnability for scalable continual improvement. LOPD learns latent context from retrieved experience to supervise the student, achieving stronger results across agentic tool use and code generation with less than 30% of the rollout budget used by GRPO and Skill-SD.

  • Problem

    Existing OPSD methods rely on designer-specified privileged context, leaving how to learn useful context end-to-end from experience unresolved.

  • Method

    LOPD transforms retrieved experiences into learnable continuous latent context that conditions a self-teacher supervising the student’s visited prefixes, with a privileged-margin constraint.

  • Results

    LOPD achieves the best aggregate result in all ten backbone–benchmark comparisons and surpasses GRPO and Skill-SD with less than 30% of their rollout budget.

  • Takeaways & Limitations

    The results support optimizing experience representations end-to-end for the policies they are intended to improve.

  • Takeaways & Limitations

    Hand-crafted privileged contexts can improve or degrade performance depending on the task, benchmark, and model setting.

Abstract

from arXiv · show

Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher's privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.

LATENT ON-POLICY SELF-DISTILLATION

The section identifies the paper’s authors: Guibin Zhang, Jiayang Lyu, Ran Sun, Xinlei Yu, Haoyu Zhao, Qibing Ren, and Shuicheng Yan.

  • LATENT ON-POLICY SELF-DISTILLATION: The paper is authored by Guibin Zhang, Jiayang Lyu, Ran Sun, Xinlei Yu, Haoyu Zhao, Qibing Ren, and Shuicheng Yan.Guibin Zhang and Jiayang Lyu are marked with asterisks, while Qibing Ren and Shuicheng Yan carry dagger symbols.

1 INTRODUCTION

LOPD reframes on-policy self-distillation by making the self-teacher’s privileged context learnable from experience rather than fixing it through designer-specified artifacts. It retrieves and composes experience into latent context, enabling dense supervision while jointly optimizing the context and on-policy student.

  • Motivation: On-policy distillation trains a student on its own visited states using dense token-level supervision from a teacher.The student first samples its trajectory, after which the teacher supervises the states it actually visits.
  • Existing Paradigm: OPSD instantiates teacher and student from the same model under different contexts, with the teacher receiving privileged information unavailable to the student.Privileged information can include verified reasoning traces, final answers, environment feedback, or successful prior rollouts.
  • Problem: Existing OPSD methods use hand-designed or rule-extracted privileged contexts, motivating end-to-end learning of what experience to retain and how to encode it.The context must sharpen token-level supervision while remaining compatible with inference without privileged information.
  • LOPD: LOPD composes retrieved experiences into continuous latent context for the self-teacher and jointly optimizes that context with the student through on-policy distillation.Its training otherwise follows standard OPSD: the student rolls out from task and interaction history, while the teacher re-evaluates visited prefixes and supplies dense token-level distributions optimized with reverse-KL distillation.
  • Positioning: LOPD’s central claim is that scalable self-evolving AI should learn both the useful content of experience and its representation instead of fixing them with designer heuristics.The work positions LOPD as a substrate-level reformulation rather than another hand-designed OPSD variant.

2 RELATED WORK

Related work establishes on-policy self-distillation as dense teacher supervision over student-generated trajectories, while latent computation uses continuous hidden representations to carry computation instead of natural language. These foundations motivate combining trajectory-based supervision with learnable latent representations.

  • On-Policy (Self-)Distillation: On-policy self-distillation trains students on their own-policy trajectories while teachers provide dense token-level supervision over visited states.This paradigm has been applied to reasoning, efficient post-training, long-context modeling, and context or knowledge distillation.
  • Latent Computation: Latent computation represents LLM computation through continuous latent tokens, embeddings, or hidden states rather than natural language.These representations can expand internal reasoning compute and compactly carry procedural memory.

3 METHOD

LOPD makes OPSD’s privileged context learnable by retrieving experiences, compressing them into continuous latent tokens, and conditioning a frozen self-teacher while the student learns on its own trajectories. A privileged-margin objective prevents uninformative context collapse and directs supervision toward successful behavior.

  • Learnable Privileged Context: LOPD replaces OPSD’s fixed, designer-specified context constructor with a learnable composer that produces privileged context from retrieved experiences.The student conditions only on its interaction state, while the teacher additionally observes the learned context.
  • Experience Retrieval: The experience pipeline retains successful rollouts, retrieves the top-J task–trajectory pairs by cosine similarity, and uses them as the composer’s input.Stored entries contain task descriptions and compact action–result traces that omit verbose observations.
  • Latent Context Composition: Each retrieved experience is encoded and compressed into K continuous latent tokens using a lightweight QFormer-style cross-attention compressor with learned queries.The encoder LoRA parameters ψ and compressor parameters χ are trainable, while the encoder backbone remains frozen.
  • On-Policy Self-Distillation: The student generates on-policy trajectories, and a frozen reference teacher re-evaluates the same prefixes with latent context while retaining differentiable activations with respect to the composer.Distillation uses teacher top-M vocabulary entries plus a tail bucket and reverse KL over supervised action tokens.
  • Privileged-Margin Constraint: The privileged-margin constraint enforces a minimum privilege level m > 0, using outcome-weighted advantages and a dual penalty to exclude degenerate context.When the teacher collapses toward the student, Δ→0 < m and the dual penalty activates; an anchor term also penalizes drift from ϕ0.

4 EXPERIMENTS

Across tool-use and coding experiments, LOPD achieves the strongest aggregate performance while learning privileged context end-to-end rather than relying on fixed hand-crafted artifacts. Ablations show that joint context optimization, early sample-efficient learning, and policy internalization contribute to its effectiveness.

  • Experimental setup: LOPD is evaluated on agentic tool use and coding using disjoint test sets, including EnvScaler, BFCL-v3, ACEBench, LiveCodeBench, and EvalPlus.Training uses 2,349 tool-interactive tasks and 7K verified Python problems across QWEN3-4B, QWEN3-8B, and OLMO3-7B backbones.
  • Performance: LOPD obtains the best aggregate result in all ten backbone–benchmark comparisons.With QWEN3-8B, it reaches 66.4/29.88/62.7 on EnvScaler, BFCL-v3, and ACEBench, versus strongest baselines at 60.2/29.00/58.0.
  • Performance: LOPD improves QWEN3-4B aggregates to 48.78 on LiveCodeBench and 81.36 on EvalPlus, while OLMO3-7B reaches 50.98 and 78.41.The reported gains extend beyond interactive-agent benchmarks to code generation.
  • Learnable context: Fixed privileged contexts can hurt performance or reverse in utility across settings, whereas LOPD remains above vanilla in all ten aggregate backbone–benchmark settings.Examples include SDPO declining from 22.88 to 15.75 on BFCL-v3 and from 50.6 to 38.0 on ACEBench with QWEN3-4B.
  • Effect of joint optimization: Jointly optimized context transfers into the policy: frozen-composer training achieves 0.573, while the ablation evaluates students without retrieved experience or latent context.The setup isolates whether learned context improves the resulting student rather than merely adding trainable parameters.
  • Training dynamics and sample efficiency: LOPD exceeds 0.61 mean reward after 320 generations and reaches 0.637 by generation 576, then remains within 0.63–0.64 through generation 1,600.This demonstrates sustained early improvement under the same on-policy generation budget used to track strongest baselines.

5 CONCLUSION

LOPD shows that an OPSD teacher’s privileged context can be learned from experience rather than prescribed as a fixed artifact. Its design optimizes experience representations end-to-end for the policies they improve while constraining learned context to preserve a verifiable teacher advantage.

  • Conclusion: LOPD transforms retrieved raw trajectories into differentiable latent privileged context that supervises the student’s visited prefixes.The learned context is constrained to preserve a verifiable teacher advantage.
  • Conclusion: The decoded latent-token projections remain fragmented and do not reproduce retrieved procedures verbatim.Figure 6 pairs current tasks with structurally related rank-2 retrievals and excerpts from LM-head projections of 32 latent tokens, with and without task conditioning.
  • Conclusion: LOPD supports a design principle in which experience representations are optimized end-to-end for the policies they are meant to improve.Learning, rather than another privileged-context recipe, determines what becomes useful guidance.

A IMPLEMENTATION DETAILS … B EXPERIMENT DETAILS

LOPD combines a task-conditional latent composer, offline retrieval from successful training rollouts, and production-compatible latent injection while keeping the backbone frozen. Experiments verify numerical equivalence and evaluate task-specific interaction formats across agentic and coding benchmarks.

  • A.1 COMPOSER ARCHITECTURE: The composer uses a frozen backbone with trainable rank-8 LoRA adapters and an 8-layer shared-parameter QFormer compressor for task-conditional experience encoding.LoRA uses α=16, dropout 0, and all seven linear projections per transformer block; QFormer feedforward multiplier is 4.
  • A.1 COMPOSER ARCHITECTURE: K=32 latent tokens are produced per retrieved experience, concatenated into the latent context, while the encoder LoRA and QFormer remain trainable and backbone weights stay frozen.The same trainable components are updated during cold-start and joint optimization.
  • A.2 COLD-START INITIALIZATION: Cold-start training uses successful base-model rollouts, task-conditional compression, nret=3 retrieved experiences, AdamW at learning rate 10^-5, batch size 8, and gradient clipping 3.0.No external expert or stronger model is required, and training data volume and steps vary by domain.
  • A.3 RETRIEVAL CONFIGURATION: Retrieval uses asymmetric task-and-trajectory/document versus task-only/query embeddings and returns the nret entries with highest cosine similarity.QWEN3-EMBEDDING-8B produces 4,096-dimensional embeddings, indexed with exact FAISS inner-product search over successful training-split rollouts only.
  • A.4 LATENT INJECTION MECHANISM: Latent tokens are injected through sentinel-based placeholder spans, whose embeddings are replaced before the first transformer layer without modifying the backbone architecture.The implementation supports SGLang positional embed overrides and vLLM prompt embeds while preserving engine batching, attention-mask, and KV-cache logic.
  • A.4 LATENT INJECTION MECHANISM: Known-token equivalence tests produce identical greedy-decoded sequences with negligible log-probability differences, validating the placeholder injection path.The encoder LoRA is active for compression but disabled during inference and teacher evaluation so generation behaves as a pure base actor conditioned on latent context.
  • A.5 PROMPT AND INTERACTION FORMAT: Agentic prompts require step-by-step tool use with valid parameters, no parallel calls, and a final ‘Task Completed’ response, while coding prompts target correct Python programs.HumanEval+ and MBPP+ use no system prompt under the official EvalPlus protocol; all coding benchmarks use Python.
  • A.5 PROMPT AND INTERACTION FORMAT: Latent context is framed as a reference from another task or similar programming problem, and the encoder formats inputs as the current task followed by a retrieved trajectory.This framing distinguishes privileged context from instructions and enables task-relevant QFormer attention.

B.1 BASELINE SETUP

The baseline comparison standardizes model, task distribution, rollout budget, optimization, and evaluation while distinguishing methods by their teacher context, teacher updates, and distillation objectives. GRPO provides a reward-only optimization baseline without teacher or privileged context.

  • Shared setup: All trainable methods use the same base model, task distribution, 32 rollouts per step, AdamW with learning rate 10−5, gradient clipping 1.0, and evaluation protocol.Distillation methods retain their original-paper KL directions.
  • GRPO: GRPO samples G=4 stochastic rollouts per task, applies group-normalized advantages to a PPO-clip objective with ϵlo=ϵhi=0.2, and uses only environment rewards.It has no teacher or privileged context.
  • Distillation baselines: SDFT conditions an EMA teacher on oracle trajectories as textual demonstrations, while OPSD uses a frozen teacher receiving oracle trajectories as system-message privileged context.SDFT uses αEMA=0.01 and forward KL; OPSD uses forward KL over the full vocabulary.
  • SDPO: SDPO uses an EMA teacher conditioned on successful sibling trajectories from the same G=4 rollouts, filtering with domain-specific success criteria rather than an external trajectory pool.The teacher update rate is 0.05.
  • Skill-SD: Skill-SD selects one skill per task via UCB1 (c=2), injects structured rollout summaries into the teacher, and combines PPO-clip with importance-weighted sampled-token reverse KL.Its PPO-clip parameters are ϵlo=0.2 and ϵhi=0.28, with λsdl=0.001.

B.2 TRAINING DATA AND DOMAIN SETUP

Training uses only two domain-specific datasets, with LOPD’s experience bank restricted to training-split rollouts and all evaluation tasks and trajectories excluded. The agentic corpus spans diverse multi-turn API interaction scenarios across insurance, logistics, e-commerce, and healthcare.

  • Training data: All methods train exclusively on the two datasets listed in Table 4, without any additional task dataset.
  • Training data: LOPD’s experience bank contains only rollouts from the corresponding training split and excludes every evaluation task and trajectory.
  • Agentic domain: EnvScaler covers insurance, logistics, e-commerce, and healthcare scenarios requiring multi-turn API interactions to complete each task.

B.3 EVALUATION PROTOCOLS

The evaluation uses thinking mode across all benchmarks and applies benchmark-specific protocols spanning held-out agent tasks, function calling, simulated dialogues, and code-generation tests. Scores are reported using each benchmark’s prescribed subsets, releases, and metrics.

  • Inference configuration: Thinking mode is enabled for all benchmarks, with inference configurations summarized per benchmark.Table 5 provides the benchmark-specific inference settings.
  • EnvScaler: 200 held-out EnvScaler tasks are randomly sampled from the full task pool and fully disjoint from the training split.EnvScaler evaluates generalization on a held-out task set.
  • BFCL-v3: 4 multi-turn BFCL-v3 subsets—base, missing-function, missing-parameter, and long-context—are evaluated in function-calling (FC) mode with per-subset and average scores.The benchmark uses the official Berkeley Function Calling Leaderboard V3 codebase.
  • ACEBench: 50 ACEBench tasks comprise multi-step (20) and multi-turn (30) categories, with a user simulator driving multi-turn dialogues.The user simulator is deepseek-v4-flash via API.
  • Code generation: 279 tasks in LiveCodeBench release v5 and 131 tasks in release v6 are scored with pass@1 using the official code generation lite harness.Release v5 covers Aug 2024–Feb 2025, while release v6 covers Feb–May 2025.
  • Code generation: 164 HumanEval+ v0.1.10 tasks and 378 MBPP+ v0.2.0 tasks are evaluated using the official EvalPlus codebase.These are the two EvalPlus benchmark suites used in the protocol.

B.4 ABLATION AND ANALYSIS SETUP

This section defines the ablation and analysis configurations used to study joint optimization, training dynamics, latent-token and retrieval sensitivity, and behavioral internalization. Each experiment evaluates LOPD-derived students under specified retrieval, composer, and latent-context conditions.

  • Effect of Joint Optimization: Joint-optimization experiments vary margin m while evaluating students without retrieval, the composer, or latent privileged context; “Frozen ϕ0” removes joint optimization.All jointly optimized rows share the same training configuration except for m.
  • Training Dynamics: Training dynamics periodically evaluate mean reward on the EnvScaler test set throughout training using QWEN3-4B.
  • Sensitivity Analysis: Sensitivity analysis varies latent-token capacity K ∈{8, 16, 32, 64, 128} and retrieval count nret ∈{1, . . . , 10} across separate LOPD variants.Capacity variants use nret=3 during training; retrieval variants use K=32 and m=0.05, with evaluation on EnvScaler and ACEBench without retrieval or latent privileged context.
  • Behavioral Internalization: Behavioral internalization compares Vanilla, Base + Composer, and LOPD under otherwise identical EnvScaler inference settings.Vanilla is the unadapted QWEN3-4B backbone; Base + Composer uses m=0.05, nret=3, and K=32, while LOPD omits retrieval, the composer, and latent privileged context.
Loading 2608.13040v1…