PaperlayerView exact revision on arXiv

Source-linked AI summary

DAPD: Dual-Anchored Policy Distillation

Jianyu Wu, Yizhou Wang, Encheng Su, Chen Tang, Shixiang Tang

arXiv:2608.01735v2cs.AI

TL;DR

OPSD can create an information mismatch between privileged teachers and inference-time students, producing privilege-dependent behavior that degrades performance. DAPD addresses this mismatch with matched-information anchoring and outperforms OPSD by +2.00 points on average on Qwen3-4B, with gains persisting across scales.

  • Problem

    OPSD mixes useful reference guidance with non-reproducible teacher changes because the privileged teacher has information unavailable to the student at inference.

  • Method

    DAPD uses Dual-Path and Dual-Source Anchoring to align reference and rollout behavior under matched information availability while balancing privileged and student-reachable guidance.

  • Results

    +2.00 points is DAPD’s average improvement over OPSD on Qwen3-4B across six tasks, with gains persisting across model scales.

  • Takeaways & Limitations

    DAPD consistently alleviates privilege illusion and outperforms OPSD across benchmarks and Qwen3 models from 1.7B to 32B.

  • Takeaways & Limitations

    DAPD increases training-time computation, and its weights may require recalibration across scales or architectures.

Abstract

from arXiv · show

On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.

1 Introduction

OPSD suffers from privilege illusion because a privileged teacher supervises a student lacking the same inference-time information. DAPD addresses this asymmetry through matched-information anchoring and improves performance over OPSD across model scales.

  • Problem: Information asymmetry between the privileged teacher and inference-time student causes privilege illusion, leading to unsupported claims and degraded task performance.The student behaves as though an unseen reference remains available during inference.
  • Problem: Existing methods modify or filter privileged teacher signals but do not match the information available to teacher and student, while relying only on reference guidance.On-policy rollouts also contain reasoning signals that can matter as the student improves.
  • Diagnosis: Replacing the student’s unprivileged distribution with a self-conditioned distribution reduces wrong claims and improves reasoning performance while keeping the privileged teacher unchanged.The self-conditioned distribution conditions on the full completion being predicted, matching teacher and student information.
  • Method: DAPD combines Dual-Path Anchoring and Dual-Source Anchoring to align reference and rollout behavior along matched-information paths and reduce reliance on privileged reference guidance.DPA uses a self-conditioned trainable bridge; DSA applies the paths in both reference-to-rollout and rollout-to-reference directions.
  • Results: +2.00 points is DAPD’s average improvement over OPSD on Qwen3-4B across six benchmarks.The benchmarks span reasoning, coding, and instruction following tasks.
  • Results: +2.69 points at 4B and +2.78 points at 32B are DAPD’s gains over OPSD, which remain stable across Qwen3-1.7B–32B.The paper reports that OPSD’s gains largely disappear as model scale increases, whereas DAPD maintains improvements.

2 Background and Analysis

The background identifies information asymmetry between OPSD’s privileged teacher and inference-time student as the source of privilege illusion. It introduces Self as an information-matched, trainable bridge that reduces this asymmetry and improves behavior.

  • OPSD preliminaries: OPSD trains None, which lacks privileged information, under supervision from detached Cross, which conditions on the reference y∗.Both distributions are evaluated on on-policy rollout prefixes, while Cross makes the same-model teacher more informative.
  • Privilege illusion: Privilege illusion arises when the student behaves as though the teacher’s unavailable training-time information were present at inference.The teacher receives privileged information during training, but the student must predict without it at inference.
  • Behavioral Probe: Across five Qwen3 scales, wrong claims under OPSD rise from 13.0 to 37.0 per 10,000 generations, while mean Avg@12 falls from 59.56 to 53.24.The Behavioral Probe tracks unsupported-answer claims and reasoning Avg@12 across training steps under the same OPSD setup.
  • Information asymmetry: Information asymmetry combines recoverable reference guidance with differences that rely on y∗ and cannot be reproduced by the student.This distinction motivates separating useful supervision from privilege-dependent behavior.
  • Privileged Anchor: 45% fewer late-stage wrong claims and +6.22 points in mean Avg@12 result when OPSD’s None distribution is replaced by Self.Self conditions on the full completion, matching Cross’s information while retaining the privileged-teacher construction; the result identifies information asymmetry rather than teacher quality alone as the source.
  • Self distribution: Self is trainable, information-matched with Cross, and intermediate between inference-time None and privileged Cross.Self shares policy parameters with None and conditions on the completion being predicted, allowing updates to affect the policy used at inference.

3 Dual-Anchored Policy Distillation

DAPD addresses OPSD’s information asymmetry through Dual-Path Anchoring, which matches information availability, and Dual-Source Anchoring, which balances reference- and rollout-guided supervision. It constructs aligned distillation paths using Self as a bridge between inference-relevant and privileged conditions.

  • Framework overview: DAPD combines Dual-Path Anchoring (DPA) for matched-information distillation with Dual-Source Anchoring (DSA) for balanced reference- and rollout-guided supervision.The framework defines three directed objectives over None, Cross, and Self distributions.
  • Directed objectives: Inference Anchor trains Self toward None, while Privileged Anchor trains Self toward Cross, creating a self-conditioned bridge across information conditions.Entangled Distillation remains the original None-to-Cross OPSD term and retains privileged guidance transfer.
  • Dual-Path Anchoring: DPA combines an unconditioned path aligning None distributions with a privileged path aligning Cross and Self under matched privileged information.The unconditioned path uses Entangled Distillation and Inference Anchor; the privileged path directly reuses Privileged Anchor.
  • Dual-Source Anchoring: DSA applies DPA in both Rollout-to-Reference and Reference-to-Rollout directions to balance complementary supervision sources.Reference guidance is correctness-oriented but off-policy, whereas rollout guidance is student-reachable and on-policy but less reliable.
  • Dual-Source Anchoring: λ ∈[0, 1] balances reference- and rollout-guided supervision with weights λ and 1 −λ, respectively.This weighting preserves both directional DPA objectives in DSA.

4 Experiments

Experiments across Qwen3 scales and capability families show that DAPD consistently outperforms OPSD and related baselines, including under out-of-distribution evaluation. Ablations further support both dual-path and dual-source anchoring, while guidance preferences vary with model scale.

  • Overall performance: DAPD achieves the best six-task average of 57.34, exceeding OPSD by +2.00 points and improving over Purified OPSD and DOPD by +1.09 and +3.85 points overall.Unless otherwise specified, these results use separately trained Qwen3-4B models for each task family.
  • Scalability and robustness: DAPD retains gains of +2.41, +2.13, and +3.06 points at 8B, 14B, and 32B, respectively, while OPSD’s gain over Base falls to at most +0.28 from 8B through 32B.The comparison spans Qwen3 models at 1.7B, 4B, 8B, 14B, and 32B scales.
  • Scalability and robustness: DAPD obtains the best OOD average of 49.64, gaining +4.82 points on LCB v5 and +1.37 points on average over OPSD.Students are optimized on reasoning data and evaluated on Coding and Instruct benchmarks, including LCB v5, BFCL v3, and IFBench.
  • Dual-Path Anchoring: DPA’s ordered component additions improve performance to 63.89 for reference guidance and 65.09 for rollout guidance, indicating that both matched-information paths are needed.Completing the unconditioned path with the Inference Anchor improves over Entangled Distillation alone, and adding the Privileged Anchor further improves performance.
  • Dual-Source Anchoring: Combining reference-guided and rollout-guided supervision reaches the best DSA result of 65.28, while dual rollouts improve over OPSD by +0.46, +2.41, and +2.41 points at 1.7B, 4B, and 8B.Verification further raises the best scores by +0.65, +1.11, and +0.18 points to 43.15, 66.11, and 67.59.
  • Guidance-weight analysis: The optimal reference-guidance weight decreases from λ = 0.5 at 1.7B to λ = 0.2 at 4B, 8B, and 14B, while larger scales prefer stronger Privileged Anchor weights.The reference-guided Privileged Anchor weight increases from 0.5 at 1.7B to 1 at 4B, and 2 at 8B and 14B.

5 Related Work

Reasoning language models are post-trained using curated references, self-generated rollouts, or reward feedback, each offering different tradeoffs. Related distillation work modifies teacher contexts, objectives, or token weighting, while OPSD uses privileged information from the same model and auxiliary guidance such as reference solutions, environmental feedback, or successful rollouts.

  • Reasoning language models: Reasoning models commonly use curated reference completions, self-generated rollouts, or reward feedback during post-training.The passage presents these sources as distinct approaches with different tradeoffs.
  • Reasoning language models: References provide reliable correctness-oriented guidance, while rollouts and reward feedback represent alternative post-training sources.
  • Reasoning language models: Later work improves OPD by changing teacher contexts, training objectives, or token weights.
  • Reasoning language models: OPSD eliminates a separate teacher by conditioning the same model on privileged information and uses auxiliary guidance such as reference solutions, environmental feedback, or successful rollouts.

6 Conclusion

The paper identifies information asymmetry as the root cause of privilege illusion in OPSD and proposes DAPD, which uses two anchoring mechanisms to address it. Across Qwen3 models from 1.7B to 32B and benchmarks, DAPD alleviates privilege illusion and outperforms OPSD, with limitations in training cost, calibration, and correctness-signal availability.

  • Contributions: DAPD addresses OPSD’s privilege illusion by using DPA to match reference and rollout information and DSA to combine correctness and student-reachable guidance.DPA aligns behaviors under matched information availability, while DSA combines correctness-oriented reference guidance with rollout guidance reachable by the student.
  • Results: DAPD consistently outperforms OPSD across benchmarks and Qwen3 models from 1.7B to 32B.The conclusion reports that these results validate the benefits of both anchoring mechanisms.
  • Limitations: DAPD increases training-time computation by constructing multiple anchored distributions but adds no inference-time cost.Its weights may require recalibration across scales or architectures.
  • Limitations: The verified-rollout extension requires an automatic correctness signal, which may be unavailable for open-ended tasks.This limits its applicability when such a signal cannot be obtained.

A Experimental Details

The experimental-details section describes the data, evaluation, privileged-information construction, training procedure, baseline implementations, and final DAPD configurations used in the paper.

  • Experimental Details: The section covers the data and evaluation setup.It identifies these as components of the paper’s experimental specifications.
  • Experimental Details: It explains how privileged information is constructed and how training is conducted.Both privileged-information construction and the training procedure are included in the section’s scope.
  • Experimental Details: It describes baseline implementations and the final DAPD configurations.These are specified alongside the other experimental components.

A.1 Benchmarks and Evaluation … A.5 Baseline Methods

The paper evaluates DAPD across mathematics, coding, function calling, and instruction following using standardized benchmark protocols, capability-specific OpenThoughts data, and privileged reference completions. Training samples on-policy student rollouts, constructs self- and cross-conditioned supervision, and compares DAPD with established baselines under matched student backbones and data budgets.

  • A.1 Benchmarks and Evaluation: A.1 Benchmarks and Evaluation: Mathematical reasoning uses Avg@12 over 12 sampled solutions for AIME 2024, AIME 2025, and HMMT February 2025.Solutions use temperature 1.0, top-p 0.95, up to 38,912 new tokens, and final-answer verification; the reasoning aggregate is the unweighted mean of benchmark-level scores.
  • A.1 Benchmarks and Evaluation: A.1 Benchmarks and Evaluation: Broader evaluation uses official protocols for LiveCodeBench v5 Pass@1, BFCL v3 overall accuracy, and the released verifier-based IFBench score.The evaluations retain each benchmark’s official prompts and parsing rules and use models trained under capability-specific data protocols.
  • A.2 Training Data: A.2 Training Data: All task-specific models use OpenThoughts data, with math, code, or released math-code-science mixtures assigned to reasoning, coding, and instruct protocols.Baselines within each protocol share the same data split and prompt format, and benchmark evaluation examples are excluded from training corpora.
  • A.3 Construction of Privileged Information: A.3 Construction of Privileged Information: OPSD-style reference injection places reference completion y∗ in the teacher’s user message, while the student rollout prompt contains only x.Both messages use the same chat template, with explicit delimiters around the inserted reference completion.
  • A.3 Construction of Privileged Information: A.3 Construction of Privileged Information: DAPD computes token-level supervision for each completion using Cross with the other completion and Self with the completion itself.Both distributions are evaluated on prefixes and target tokens, and auxiliary completions are used only during training-distribution construction, never inference.
  • A.4 DAPD Training Procedure: A.4 DAPD Training Procedure: Each minibatch samples an on-policy rollout, builds None, Cross, and Self distributions for both completion directions, and combines detached-teacher losses to update LoRA parameters.The current LoRA-on policy generates rollouts and trainable distributions; snapshot teachers and the LoRA-off base policy provide the specified teacher distributions.
  • A.5 Baseline Methods: A.5 Baseline Methods: Comparisons use the same student backbone and on-policy data budget while varying supervision construction and transfer.Baselines include Base, OPSD, SDFT, SDPO, and Purified OPSD, with OPSD distilling same-model Cross supervision into None on sampled prefixes.

A.6 Training and Loss Implementation · A.7 Main DAPD Configurations

A.6 specifies the optimization, reproducibility, compute, rollout, and divergence implementations used for training. A.7 defines the bidirectional DAPD objective and its coefficient and teacher–student configurations across model scales and task-specific models.

  • A.6 Training and Loss Implementation: LoRA adapts all attention and multilayerperceptron projections with rank 64 and scale 128, using a 5 × 10−6 learning rate and effective batch size 32.Training uses a linear 500-step schedule without warmup, gradient-norm clipping at 0.1, bfloat16, and gradient checkpointing.
  • A.6 Training and Loss Implementation: Seed 42 is set before initialization, shuffling, and rollout sampling; matched methods share it so headline numbers use aligned randomness.This aligns initialization, data order, and rollout-generation streams.
  • A.6 Training and Loss Implementation: Training uses eight NVIDIA A100-80GB GPUs with PyTorch 2.8.0, Transformers 4.57.1, DeepSpeed 0.18.2, and vLLM 0.11.0.Data parallelism adjusts per-device microbatch size while preserving the effective batch size.
  • A.6 Training and Loss Implementation: Coding and Instruct comparisons preserve LoRA, optimizer, update budget, and DAPD objective within each scale, while adapting parsing and formatting to OpenThoughts examples.Every baseline is rerun under the corresponding capability-specific protocol.
  • A.6 Training and Loss Implementation: The student samples one rollout per problem at temperature 1.1, top-p 0.95, top-k 20, and up to 1,024 new tokens within a 20,000-token context.These settings define rollout generation during training.
  • A.6 Training and Loss Implementation: Each directed divergence evaluates the full vocabulary at every non-padding completion token, with logits divided by T = 1.1 and component cap c = 0.05.The implementation uses a component-clipped forward-KL surrogate evaluated directly at sampled prefixes, without importance-sampling ratios.
  • A.7 Main DAPD Configurations: The main DAPD objective applies all three directed losses in both reference-guided Rollout-to-Reference and rollout-guided Reference-to-Rollout directions.κ sets total loss weight, λ allocates weight to the reference-guided direction, and 1 −λ to the rollout-guided direction; β values redistribute terms within each direction.
  • A.7 Main DAPD Configurations: Teacher distributions are detached; Entangled-Distillation teachers use a periodically updated shared LoRA-on snapshot, while anchor teachers use the LoRA-off base model.All student distributions and rollout samples come from the current LoRA-on model; Table 5 provides scale-wise coefficient configurations, and task-specific Qwen3-4B models use the 4B allocation.

B Theoretical Analysis of the Unconditioned Path … C Additional Experimental Results

The theoretical analysis formalizes how the unconditioned path aligns None distributions, transfers Cross behavior to inference-side rollouts, and propagates Inference Anchor updates through shared parameters. Additional experiments report scale-wise scores, privilege-illusion measurements, and implementation sensitivity.

  • B Theoretical Analysis of the Unconditioned Path: DAPD’s unconditioned path combines Inference Anchor and Entangled Distillation, connected by Self–Cross alignment across matched reference and rollout prefixes.The analysis is organized around paired-prefix alignment, sequence-level transfer, and shared-parameter transfer.
  • B.1 Conditional Alignment of the None Distributions: The conditional alignment argument relies on a bridge-consistency assumption in which Self and Cross share the privileged condition and policy.This proxy relation links the intermediate distributions used to upper-bound None-to-None distance.
  • B.1 Conditional Alignment of the None Distributions: Proposition 1 bounds reference-side and rollout-side None-distribution distance using the two objective terms and a shared Self–Cross bridge.Inference Anchor controls the reference-side term, Entangled Distillation controls the rollout-side term, and the bridge controls their intermediate distance.
  • B.2 Sequence-Level Transfer from Entangled Distillation: Proposition 2 lifts token-level Entangled Distillation to a sequence-level bound between rollout-side None and Cross distributions on on-policy prefixes.The proof uses an autoregressive hybrid bound and converts local KL terms into a sequence-level total-variation bound.
  • B.3 Shared-Parameter Transfer from Inference Anchor: Shared parameters provide the optimization channel that transfers an Inference Anchor update from reference-side Self to rollout-side Cross.Refreshing the Cross teacher then carries the anchored change into rollout-side None through Entangled Distillation.
  • B.3 Shared-Parameter Transfer from Inference Anchor: 256.62 was the mean gradient dot product across 128 sampled positions, with a bootstrap 95% confidence interval of [147.55, 393.53].The empirical compatibility check used Qwen3-4B, four LoRA initializations, 32 math examples, and four interior token positions per example.
  • B.4 Implication for DAPD: Together, Propositions 1 and 2 and Eq. (21) formalize why the joint update moves rollout-side None under bridge-consistency and local compatibility conditions.The results respectively establish None-to-None alignment, Cross-to-None sequence transfer, and parameter-mediated teacher shaping.
  • C Additional Experimental Results: The additional experimental section reports absolute scale-wise scores, measurements supporting the privilege-illusion analysis, and implementation sensitivity.These experiments complement the theoretical analysis with performance, diagnostic, and configuration results.

C.1 Complete Scale-Wise Results · C.2 Privilege-Illusion Dynamics · C.3 Divergence and Clipping

Across Qwen3 scales, DAPD consistently outperforms scale-matched OPSD while reducing detected privilege-illusion claims and preserving late-stage reasoning performance. Component-clipped forward KL performs best among the evaluated divergence and clipping implementations.

  • C.1 Complete Scale-Wise Results: DAPD improves over scale-matched OPSD by 1.94, 2.69, 2.41, 2.04, and 2.78 points from 1.7B through 32B, respectively.Table 6 reports per-benchmark scores using DAPD’s complete objective and Table 5 configuration.
  • C.1 Complete Scale-Wise Results: DAPD improves AIME25 at every scale and typically improves HMMT25, indicating gains are not concentrated in one benchmark.The comparison uses scale-matched OPSD policies.
  • C.2 Privilege-Illusion Dynamics: The behavioral probe detects failed derivations followed by attribution of concrete answers to memory, references, official solutions, or external sources with incorrect final answers.Generic uncertainty, ordinary guesses, recalled formulas, and broad phrases such as “after research” are excluded.
  • C.2 Privilege-Illusion Dynamics: Each scale–checkpoint cell contains 90 problems and 12 generations per problem, with counts reported per 10,000 pooled generations and cluster-bootstrap confidence intervals.The bootstrap keeps all 12 generations from each problem together.
  • C.2 Privilege-Illusion Dynamics: 9 detected claims versus 33 for OPSD at steps 250–300 represents a 73% reduction, while DAPD’s step-300 Avg@12 is 7.66 points higher.Together with the isolated Privileged-Anchor intervention, the dynamics support matched-information anchoring as reducing privilege illusion while preserving late-stage reasoning performance.
  • C.3 Divergence and Clipping: Component-clipped forward KL performs best among the evaluated Qwen3-4B divergence and clipping implementations and is used as the default DAPD implementation.The comparison summarizes completed Qwen3-4B results.

D Additional Qualitative Evidence · E Future Directions

Additional inference-time cases show OPSD producing unsupported answer guesses after unresolved or invalid reasoning, whereas DAPD’s matched-information design motivates reference-free and adaptive extensions. Future directions target verified rollout sources, online source weighting, and broader privileged information.

  • D Additional Qualitative Evidence: Table 9 compares three inference-time cases under identical reference-free prompts and decoding, counting correctness across all 12 samples.Neither method receives a reference at inference.
  • D Additional Qualitative Evidence: 6/12: OPSD leaves arc relations unresolved, then guesses 360° because the competition answer is likely to be a nice number.The output returns 360.
  • D Additional Qualitative Evidence: 8/12: OPSD identifies a candidate violating the circle equation, then guesses that the answer is likely 1/7 or 2/7.The supplied passage continues with the resulting value 7/18.
  • D Additional Qualitative Evidence: 7/12: OPSD obtains a sign incompatible with the required answer form, then proceeds with the guess that the answer is likely 50.This case illustrates unsupported guessing after an invalid intermediate result.
  • E Future Directions: Reference-free DAPD could replace the fixed reference with a verified or consensus-selected rollout updated as the policy improves.This preserves matched-information paths while reducing dependence on curated solutions.
  • E Future Directions: Adaptive source trust could estimate reliability online and assign continuous source and anchor weights instead of using a global coefficient or brittle hard gate.Reliability may vary across policies, examples, and tokens.
  • E Future Directions: The framework could extend beyond references to retrieved documents, tool traces, verifier feedback, and intermediate plans unavailable at inference.Studying these sources may clarify how anchor design should adapt to privileged-information form and reliability.
Loading 2608.01735v2…