DAPD: Dual-Anchored Policy Distillation

• Existing direct-use and selective-transfer approaches do not reliably separate reproducible guidance from privilege-dependent behavior because they alter privileged signals rather than match teacher–student information.They also rely only on references, neglecting reasoning signals in on-policy rollouts as the student improves (Zhao et al. 2026a; Penaloza et al. 2026; Nguyen et al. 2026; Zheng et al. 2026; Yu et al. 2026; Tu et al. 2026; Shen et al. 2026).Source: §1 · pp. 1–2
• The mismatch makes students behave as if unseen references were available, and a self-conditioned student distribution reduces wrong claims and improves reasoning.The intervention keeps the privileged teacher unchanged while conditioning the student on the full completion being predicted; the paper calls the original mixed update Entangled Distillation.Source: §1 · pp. 1–2
• DPA adds a self-conditioned bridge and aligns reference and rollout behavior along unconditioned and privileged paths, retaining OPSD supervision while matching information availability.The unconditioned path combines Entangled Distillation with an Inference Anchor; the privileged path uses a Privileged Anchor.Source: §1 · p. 2
• DSA applies both paths in reference-to-rollout and rollout-to-reference directions, balancing reliable correctness-oriented reference guidance with student-reachable on-policy rollout guidance.This addresses the limitation of relying on a single guidance source.Source: §1 · p. 2
• On Qwen3-4B, DAPD improves over OPSD by +2.00 points on average across six tasks; across Qwen3-1.7B–32B, it maintains stable improvements, gaining +2.69 points at 4B and +2.78 points at 32B.Source: Figure 1 · p. 1, §1 · p. 2

2.1 Privilege Illusion in OPSD

• OPSD uses the detached Cross distribution, conditioned on the reference, to supervise the None student on prefixes from its own rollout.Cross conditions on the prompt, rollout prefix, and reference; None conditions only on the prompt and prefix.Source: §2.1 · pp. 2–3
• Across five Qwen3 scales, OPSD wrong claims rose from 13.0 to 37.0 per 10,000 generations as mean Avg@12 fell from 59.56 to 53.24.The probe counts unsupported answers presented with reasoning as though the answer were known; this pattern indicates privilege illusion emerging during training and declining inference performance.Source: §2.1 · p. 3

2.2 Information Asymmetry as the Cause

• Information asymmetry mixes useful reference guidance recoverable from the prompt with teacher changes reliant on unavailable reference information, encouraging confident unsupported predictions.The paper defines this mixed update as Entangled Distillation and identifies information asymmetry, rather than teacher quality alone, as the cause of privilege illusion.Source: §2.2 · p. 3
• Matching the privileged teacher’s information with Self reduced late-stage wrong claims by 45% and improved mean Avg@12 by +6.22 points versus OPSD.The Privileged Anchor intervention kept the teacher unchanged while replacing the student’s None distribution with Self.Source: §2.2 · p. 3

2.3 Self Distribution as a Bridging Anchor

• Self is trainable like None yet information-matched with Cross, making it a bridging anchor for aligning reference and rollout behavior without directly transferring privileged-teacher behavior.Self conditions on the full completion being predicted, while updates through Self affect the policy that produces None at inference.Source: §2.3 · p. 3

3.1 Distillation Beyond OPSD

• For each completion source, DAPD defines three directed objectives: Entangled Distillation retains None-to-Cross privileged guidance, while Inference and Privileged Anchors train Self toward None and Cross, respectively.The detached first distribution is the teacher and the second is the trainable student.Source: §3.1 · p. 4

3.2 Dual-Path Anchoring

• DPA combines an unconditioned path aligning the two None distributions through Self with a privileged path aligning Self and Cross when both receive privileged information.The unconditioned path combines Entangled Distillation on one completion with the Inference Anchor on the other; the privileged path reuses the Privileged Anchor.Source: §3.2 · p. 4

3.3 Dual-Source Anchoring

• DSA combines both DPA directions: reference guidance emphasizes correctness, while on-policy rollout guidance is student-reachable but less reliable, with λ and 1 − λ balancing their weights.Rollouts may contain useful reasoning signals and may themselves be correct, motivating their use as guidance as well as prediction targets.Source: §3.3 · pp. 4–5

4.1 Experimental Setup

• Experiments use Qwen3 models from 1.7B to 32B, task-specific training data and held-out benchmark examples, with LoRA and a component-clipped, full-vocabulary forward KL for DAPD.Reasoning uses math data, coding uses code data, and instruct uses combined math, code, and science data; evaluations cover AIME24, AIME25, HMMT25, LCB v5, BFCL v3, and IFBench. Baselines include Base and OPSD-style methods, using the same task-specific student and data.Source: §4.1 · p. 5

4.2 Main Results

• DAPD achieves a six-task average of 57.34, beating OPSD by +2.00 points, Purified OPSD by +1.09, and DOPD by +3.85 overall.The comparisons suggest that matching teacher and student information conditions is more effective than filtering or routing an asymmetric privileged signal.Source: §4.2 · p. 5
• DAPD records the best OOD average of 49.64, gains +4.82 on LCB v5 and +1.37 average points over OPSD, and remains competitive on BFCL v3 and IFBench.A reasoning-trained student is evaluated on coding and instruction benchmarks; the result supports robustness beyond its optimization domain.Source: Table 2 · p. 6, §4.2 · p. 6
• At 8B, 14B, and 32B, DAPD gains +2.41, +2.13, and +3.06 points, while OPSD’s gain over Base falls to at most +0.28 from 8B through 32B.OPSD’s gain also falls from +5.19 at 1.7B to +1.39 at 4B; the authors attribute its scaling difficulty to privilege illusion bypassing rollout reasoning and self-correction.Source: §4.2 · pp. 5–6
• In an AIME24 case, DAPD derives the correct answer while OPSD recalls an unsupported answer; across steps 250–300, DAPD reduces late-stage wrong claims by 73%.The case uses the same decoding setting; the aggregate reduction is relative to OPSD.Source: Figure 4 · p. 6, §4.2 · p. 6

4.3 Dual-Path and Dual-Source Ablations

• Both DPA paths improve performance for each guidance source, reaching 63.89 with reference guidance and 65.09 with rollout guidance; combining sources under DSA reaches 65.28.The ablations indicate that both matched-information paths are useful and that reference and rollout guidance contribute complementary information.Source: §4.3 · p. 6, Table 3 · p. 7

4.4 Sensitivity and Robustness

• The preferred reference-guidance weight falls from λ = 0.5 at 1.7B to λ = 0.2 at 4B, 8B, and 14B, while source-specific anchor weights vary by scale.Reference-guided Privileged Anchor weights are 0.5, 1, 2, and 2 across those scales; rollout-guided weights are 0.5, 2, 0.5, and 1. The authors describe a reliability–reachability trade-off: smaller policies need more corrective reference guidance, while larger policies can rely more on their rollouts.Source: §4.4 · pp. 6–7, Table 3 · p. 7

4.5 Toward Reference-Free DAPD

• Dual rollouts beat OPSD by +0.46, +2.41, and +2.41 at 1.7B, 4B, and 8B; verification adds +0.65, +1.11, and +0.18, reaching 43.15, 66.11, and 67.59.Dual rollouts use independent on-policy completions; verification selects a verifier-approved reference-side completion without exposing a curated reference to the policy.Source: Table 3 · p. 7, §4.5 · p. 7

Reasoning language models

• References offer reliable correctness-oriented guidance but may be off-policy, whereas rollouts reflect current behavior but may be incorrect; DAPD uses this complementarity for dense token-level post-training.Reasoning models are post-trained with curated reference completions, self-generated rollouts, or reward feedback (Wei et al. 2022; Kojima et al. 2022; Wang et al. 2023; Cobbe et al. 2021; Lewkowycz et al. 2022; Lightman et al. 2024; Hendrycks et al. 2021; Zelikman et al. 2022; Shao et al. 2024; DeepSeek-AI 2025; Guha et al. 2025; Tang et al. 2026).Source: §5 · p. 7

On-policy (self) distillation

• DAPD retains on-policy sampling and privileged-teacher construction while reorganizing supervision around matched information availability and both completion sources.This distinguishes its design from OPSD and related self-distillation methods that use reference solutions, environmental feedback, or successful rollouts as auxiliary guidance (Zhao et al. 2026a; Penaloza et al. 2026; Shenfeld et al. 2026; Hübotter et al. 2026).Source: §5 · p. 7
• OPD provides dense teacher supervision on student-sampled rollouts to reduce rollout mismatch, while later work changes teacher contexts, objectives, or token weights.OPD is discussed by Hinton, Vinyals, and Dean 2015; Kim and Rush 2016; Agarwal et al. 2024; Thinking Machines Lab 2025; later variants by Ye et al. 2026; Yang et al. 2026; Hou et al. 2026; Jin et al. 2026; Xu et al. 2026.Source: §5 · p. 7
• OSPD conditions the same model on privileged information, and related extensions calibrate, filter, route, relax, or localize supervision; analyses examine scale, rollout quality, and optimization.The passage cites Zhao et al. 2026a; Penaloza et al. 2026; Shenfeld et al. 2026; Hübotter et al. 2026 for related self-distillation guidance; Nguyen et al. 2026; Zheng et al. 2026; Yu et al. 2026; Tu et al. 2026; Ko et al. 2026; Shen et al. 2026; Zhao et al. 2026b for extensions; and Li et al. 2026; Kaur et al. 2026 for analyses.Source: §5 · p. 7

6 Conclusion

• DAPD significantly alleviates privilege illusion and consistently outperforms OPSD across benchmarks and Qwen3 models from 1.7B to 32B.The results validate the benefits of both matched-information paths and guidance sources.Source: §6 · p. 7
• DPA aligns reference and rollout behavior under matched information, while DSA combines correctness-oriented reference guidance with student-reachable rollout guidance.The framework targets information asymmetry, identified as the root cause of privilege illusion in OPSD.Source: §6 · p. 7

Limitations

• DAPD increases training-time computation but adds no inference-time cost; its weights may need recalibration across scales or architectures.The verified-rollout extension requires an automatic correctness signal, which may be unavailable for open-ended tasks.Source: §6 · p. 7

A Experimental Details

• It specifies the data, evaluation, privileged-information construction, training procedure, baseline implementations, and final DAPD configurations.Source: A Experimental Details · p. 8

A.1 Benchmarks and Evaluation

• AIME 2024, AIME 2025, and HMMT February 2025 use 12 samples per problem; Avg@12 averages correctness, and the reasoning aggregate is the unweighted mean of benchmark scores.Sampling uses temperature 1.0, top-p 0.95, up to 38,912 new tokens, and final-answer verification.Source: §A.1 · p. 8
• LiveCodeBench v5 uses official-split Pass@1, BFCL v3 uses its multi-turn harness and overall accuracy, and IFBench uses its released verifier-based score.Each benchmark retains its official prompt and parsing rules; evaluation uses the model trained under that benchmark’s capability-specific data protocol.Source: §A.1 · p. 8

A.2 Training Data

• Task-specific models use OpenThoughts math data for Reasoning, code data for Coding, and a math-code-science mixture for Instruct; evaluation examples are excluded from training.Within each protocol, baselines share the same data split and prompt format (Guha et al. 2025).Source: §A.2 · p. 8

A.3 Construction of Privileged Information

• For each problem, the OPSD-style teacher message includes the reference completion between delimiters, while the student rollout prompt contains only the problem.The same chat template is applied to both messages; Table 4 presents the None and reference-conditioned Cross message templates. The reference is an explicit worked completion, not a label or latent teacher state.Source: §A.3 · p. 8, Table 4 · p. 8
• DAPD uses token-level supervision on either rollout or reference tokens, evaluating Cross and Self distributions on those tokens while keeping auxiliary completions out of inference prompts.Cross inserts the other completion and Self inserts the target completion; both distributions are evaluated on the prefixes and target tokens of that completion.Source: §A.3 · p. 8

A.4 DAPD Training Procedure

• DAPD samples on-policy rollouts, constructs None, Cross, and Self distributions, evaluates three objectives in both completion directions, detaches teachers, and updates LoRA parameters.The current LoRA-on policy supplies rollouts and trainable distributions; Entangled-Distillation teachers use a shared snapshot, while Inference-Anchor and Privileged-Anchor teachers use the detached LoRA-off base policy.Source: §A.4 · p. 8
• The training procedure periodically copies current LoRA parameters into the shared snapshot used by both Entangled-Distillation teachers.Losses are combined using Equation (9), then backpropagated to update the current LoRA parameters.Source: §A.4 · p. 8

A.5 Baseline Methods

• Comparisons use the same student backbone and on-policy data budget; Base evaluates the original Qwen3 policy, while OPSD distills its Cross distribution into None (Yang et al. 2025; Zhao et al. 2026a).OPSD samples from the current student and distills on sampled prefixes.Source: §A.5 · pp. 8–9
• SDFT uses an EMA self-teacher conditioned on an expert demonstration, whereas SDPO conditions an EMA teacher on successful peer rollouts or environment feedback (Shenfeld et al. 2026; Hübotter et al. 2026).SDPO provides dense token-level supervision.Source: §A.5 · p. 9
• Purified OPSD corrects privileged supervision using pointwise mutual information, while DOPD routes token-level supervision using advantage gaps and relative probabilities (Shen et al. 2026; Yu et al. 2026).Purified OPSD removes the component predictable from the reference alone; DOPD routes between privileged-teacher and student policies.Source: §A.5 · p. 9

A.6 Training and Loss Implementation

• 64-rank LoRA adapts attention and MLP projections; training uses a 5 × 10−6 learning rate, 500-step linear schedule, 0.1 gradient clipping, bfloat16, checkpointing, and effective batch size 32.Source: §A.6 · p. 9
• Seed 42 is set before initialization, shuffling, and rollout sampling; matched comparisons align those streams, and headline numbers use this protocol.Source: §A.6 · p. 9
• Eight A100-80GB GPUs run the specified PyTorch, Transformers, DeepSpeed, and vLLM stack, while per-device microbatch size varies to preserve effective batch size.Source: §A.6 · p. 9

A.7 Main DAPD Configurations

• Coding and Instruct comparisons retain the same LoRA setup, optimizer, update budget, and DAPD objective within each scale, rerunning baselines under capability-specific data protocols; Qwen3-4B task models use the 4B allocation.Source: §A.6 · p. 9, §A.7 · p. 9

A.6 Training and Loss Implementation

• One rollout per problem is sampled at temperature 1.1, top-p 0.95, top-k 20, and a 1,024-token generation limit, with training context capped at 20,000 tokens.Source: §A.6 · p. 9
• Each directed loss evaluates the full vocabulary at non-padding completion tokens with logits scaled by T = 1.1 and component cap c = 0.05; the surrogate is component-clipped forward KL without importance sampling.Source: §A.6 · p. 9

A.7 Main DAPD Configurations

• DAPD applies all three directed losses in both reference-guided Rollout-to-Reference and rollout-guided Reference-to-Rollout directions, weighting them by κ, λ, 1 − λ, and four β coefficients.Source: §A.7 · p. 9
• Teacher distributions are detached; a periodically updated shared LoRA-on student snapshot supplies the two Entangled-Distillation instances, while LoRA-off base-model teachers supply the anchor terms.Source: §A.7 · p. 9

B Theoretical Analysis of the Unconditioned Path

• The rollout-to-reference unconditioned path combines Inference Anchor on reference prefixes with Entangled Distillation on rollout prefixes, linked by Self.The analysis treats paired-prefix alignment, sequence-level transfer on inference-policy states, and shared-parameter transfer of Inference Anchor updates.Source: B Theoretical Analysis of the… · pp. 9–10

B.1 Conditional Alignment of the None Distributions

• The paired-prefix bound connects Inference Anchor, Entangled Distillation, and Self–Cross bridge consistency to the desired None–None alignment.Inference Anchor reduces the Self-side term, Entangled Distillation reduces the Cross-side term, and the shared bridge controls the intermediate term.Source: §B.1 · p. 10

B.2 Sequence-Level Transfer from Entangled Distillation

• Entangled Distillation bounds the sequence-level distance between None and Cross distributions using token-level alignment on on-policy prefixes.The passage identifies the local KL as KL(Cross∥None), while the student-to-teacher alignment arrow is None →Cross.Source: §B.2 · p. 10

B.3 Shared-Parameter Transfer from Inference Anchor

• A compatible Inference Anchor step moves Cross in expectation toward the reference-side None anchor, and refreshed Cross guidance transfers that change to rollout-side None.Shared parameters provide the optimization channel connecting reference-side Self updates to rollout-side Cross behavior.Source: §B.3 · pp. 10–11
• 256.62 was the mean gradient dot product (95% CI [147.55, 393.53]) and 0.101 the mean gradient cosine (95% CI [0.058, 0.144]) across 128 positions.Averaging positions within examples yielded dot product 225.30 [122.94, 352.53] and cosine 0.119 [0.054, 0.185]; the positive intervals support local compatibility.Source: §B.3 · p. 11

B.4 Implication for DAPD

• The joint results formalize how the update moves rollout-side None toward reference-side None under bridge-consistency and local-compatibility conditions.The argument combines conditional None-to-None alignment, on-policy sequence transfer, and shared-parameter transfer from Inference Anchor.Source: §B.4 · p. 11

C Additional Experimental Results

• Additional experiments report absolute scale-wise scores, measurements behind the privilege-illusion analysis, and implementation sensitivity.The supplied passage gives the scope of these results but no scores or comparisons.Source: C Additional Experimental Results · p. 11

C.2 Privilege-Illusion Dynamics

• The probe counts generations that report a failed derivation, attribute a concrete answer to memory, references, or an external source, and return an incorrect answer.It excludes generic uncertainty, ordinary guesses, recalled formulas, and broad phrases such as “after research”; each scale-checkpoint cell has 90 problems and 12 generations per problem, pooled across five scales with problem-cluster bootstrap intervals.Source: §C.2 · p. 11
• The Privileged Anchor has 41 detected claims across checkpoints versus OPSD’s 58, and 18 versus 33 at steps 250–300.The comparison isolates replacing the trainable None distribution with Self while retaining the Cross teacher, as described in Table 7.Source: §C.2 · p. 11, Table 7 · p. 12
• 9 versus 33 detected claims at steps 250–300 means DAPD reduces claims by 73% relative to OPSD, while its step-300 Avg@12 is 7.66 points higher.Across all checkpoints, OPSD yields 58 detected claims in 32,400 generations versus 41 for the Privileged Anchor; at steps 250–300, their counts are 33 and 18, respectively.Source: §C.2 · p. 11

C.3 Divergence and Clipping

• Component-clipped forward KL performs best among the evaluated Qwen3-4B divergence and clipping implementations.Table 8 presents the divergence and component-clipping ablation.Source: §C.3 · p. 11, Table 8 · p. 12

C.1 Complete Scale-Wise Results

• 1.94, 2.69, 2.41, 2.04, and 2.78 points are DAPD’s respective gains over scale-matched OPSD from 1.7B through 32B.DAPD improves AIME25 at every scale and typically improves HMMT25; Table 6 reports per-benchmark scores and Avg@12, the unweighted mean of AIME24, AIME25, and HMMT25.Source: §C.1 · p. 11, Table 6 · p. 12

D Additional Qualitative Evidence

• In one reference-free comparison, OPSD is labeled 6/12, and the shown output guesses 360° after leaving arc relations unresolved.Table 9 reports correct outputs among 12 generations; the compared methods use the same reference-free prompt and decoding protocol and receive no reference at inference.Source: D Additional Qualitative Evidence · pp. 11–12, Table 9 · p. 12

E Future Directions

• Future work proposes verified or consensus-selected rollouts, online continuous source and anchor weighting, and tests with other privileged information such as documents, tool traces, verifier feedback, or plans.These directions aim to reduce dependence on curated solutions, adapt trust to source reliability across examples and tokens, and examine how anchor design should vary with the information source.Source: E Future Directions · p. 13