FocusOPD: Structure-Guided On-Policy Distillation for Diffusion Models

1 INTRODUCTION

• The paper identifies an open question about how the value of feature supervision varies across network depth and denoising stages.Prior work selects denoising states or aligns representation changes across blocks, but the paper says their relative supervision value remains open.Source: §1 · p. 1
• Teacher–student mismatch is larger in deeper blocks and early, high-noise denoising, and its relative organization persists before and after output-level OPD.The paper calls this a persistent depth–temporal structure; persistence concerns relative organization, not unchanged absolute mismatch.Source: §1 · p. 1
• Under local smoothness and gradient-compatibility conditions, sufficiently larger discrepancies can yield greater output-loss reduction for small auxiliary updates.The analysis also reports diagnostics in which upstream differences dominate at selected deep blocks and noise-dependent scaling improves held-out prediction over a timestep control.Source: §1 · p. 2
• FocusOPD improves capability-transfer metrics on SD3.5-M with DiffusionOPD, including GenEval 0.953 to 0.967 and PickScore 23.797 to 23.944.It adds tokenwise cosine alignment of conditional image features in deep layers during early denoising.Source: §1 · p. 2
• Selective supervision of deep layers during early denoising improves capability transfer, while broader temporal or depth coverage adds cost without consistent gains.The paper reports transfer across composition, text rendering, and preference metrics.Source: §1 · p. 2

2 A PERSISTENT STRUCTURE OF TEACHER–STUDENT MISMATCH

• On SD3.5-M, corresponding conditional image-token features are compared on the same noisy latent, timestep, and prompt; mismatch is larger in deeper blocks and early, high-noise steps.The section also examines joint mismatch profiles before and after output-level OPD and extends the analysis to Z-Image.Source: §2 · pp. 2–3
• Marginal trends locate larger discrepancies but do not explain how they arise; joint profiles show their organization persists through training.The section poses the origins and persistence of this structure as questions for the mechanistic account that follows.Source: §2 · p. 3
• Across SD3.5-M and Z-Image, larger discrepancies concentrate in deeper blocks and earlier denoising steps, and this relative organization persists after training.Figure 2 compares checkpoints 0, 500, and 1000; persistence refers to mismatch distribution, not unchanged absolute MSE.Source: §2 · p. 3, Figure 2 · p. 3

3 FROM MISMATCH MECHANISMS TO SELECTIVE SUPERVISION

• Deep mismatch combines local discrepancies with differences inherited from upstream states, whose interaction determines how mismatch accumulates.The depth-wise transport account is stated for differentiable student blocks.Source: §3 · p. 3
• In the Gaussian denoising model, lower signal-to-noise ratios preserve more of the difference between teacher and student priors under the same text.The priors encode model-specific expectations under that text condition.Source: §3 · p. 3
• Under local smoothness and gradient-compatibility conditions, sufficiently larger mismatch can produce greater single-step output-loss reduction at a matched supervision budget.The proposition compares supervision distributions when correction efficiencies are positive and bounded; it motivates selection by mismatch rather than uniform allocation.Source: §3 · p. 4

• The diagnostics use shared teacher–student inputs for 32 prompts per pair, fixing each prompt’s clean latent and noise across five σ values.The tested σ values are 0.1, 0.3, 0.5, 0.7, and 0.9.Source: §4 · p. 4
• At selected deep positions, propagated energy dominates discrepancy energy, indicating that upstream state differences dominate mismatch there.The analysis separates local discrepancy from the propagated term using native and complete teacher inputs.Source: §4 · p. 4
• Noise-dependent scaling yields lower held-out NMSE than timestep control in all six teacher–student pairs.The comparison fits τ^2 on 16 prompts and evaluates on the remaining 16.Source: Table 1 · p. 4, §4 · p. 5

5 FOCUSOPD: STRUCTURE-GUIDED FEATURE DISTILLATION

• FocusOPD augments multi-teacher OPD with directional feature alignment in deep layers and early denoising, where mismatch is concentrated.The feature-alignment region is fixed and motivated by the observed persistent depth–temporal structure and Proposition 3.3.Source: §5 · p. 5, §5.2 · p. 5

5.1 DISTILLATION ON STUDENT TRAJECTORIES

• The student generates trajectories, and teacher and student receive the same detached trajectory states while the base OPD output objective is retained.Teacher feedback is applied at the base objective’s output-query set.Source: §5.1 · p. 5

5.2 SELECTING THE SUPERVISION REGION

• The feature window selects a contiguous deep-layer region and early steps measured from the trajectory’s high-noise start, independently of output queries.Output supervision continues beyond the feature window according to the base protocol.Source: §5.2 · pp. 5–6

5.3 DIRECTIONAL FEATURE SUPERVISION

• Directional feature loss aligns normalized conditional image-token feature directions, averaging over selected blocks and tokens, with its weight and region held fixed during training.Training replays states at output or feature queries, shares teacher forward passes when queries overlap, and updates only student parameters; inference adds no components.Source: §5.3 · p. 6

6.1 EXPERIMENTAL SETUP

• Experiments use SD3.5-M and Z-Image with three task-specialized teachers, ten-step deterministic-flow trajectories, LoRA-only optimization, and 1000 training epochs.Evaluation covers GenEval, held-out OCR accuracy, and four DrawBench preference or visual-quality metrics.Source: §6.1 · p. 6

6.2 MAIN RESULTS

• FocusOPD consistently improves DiffusionOPD, DanceOPD, and STEP-OPD across SD3.5-M and Z-Image; selective deep/early supervision also outperforms uniform supervision.Table 2 reports GenEval, OCR, and four DrawBench metrics; the backbone-specific depth and denoising windows are calibrated independently.Source: Table 2 · p. 7, §6.2 · p. 8

6.3 ABLATION STUDIES

• All-24/Full yields only modest gains on some metrics and costs 21.77 seconds per update and 2160 MiB per rank; Last-12/Front-40% uses less time and cache.Last-12/Front-40% reaches GenEval 0.960 and LAION-Aesthetic 6.214; full-trajectory supervision gives no GenEval improvement over output-only.Source: §6.3 · p. 8, Table 3 · p. 8
• Last-6 reaches GenEval 0.962 and OCR 0.962; All-24 reaches GenEval 0.965 but requires double Last-12's feature computation for only +0.5% GenEval gain.Middle windows improve moderately, while shallow supervision is minimal; non-contiguous First-6+Last-6 does not outperform Last-12.Source: §6.3 · pp. 8–9, Table 4 · p. 8
• Front-20% raises GenEval from 0.951 to 0.959, while Front-40% reaches 0.960 and LAION-Aesthetic 6.214; full-trajectory supervision returns GenEval to 0.951.Middle and late windows underperform, and extending early supervision beyond Front-50% degrades performance.Source: Table 5 · p. 8, §6.3 · p. 9
• At Last-12/Front-40%, cosine distance gives the strongest GenEval and LAION-Aesthetic scores versus normalized MSE, with λ fixed rather than matched gradient magnitudes.The comparison uses SD3.5-M with DiffusionOPD.Source: §6.3 · p. 9, Table 6 · p. 9

7 RELATED WORK

• Multi-teacher diffusion OPD integrates capabilities but primarily uses output-level supervision; feature distillation adds intermediate representations, while selective KD adapts supervised content or location.The paper studies conditional image-token representations in selected deep blocks during early denoising.Source: §7 · p. 9

8 CONCLUSION

• FocusOPD complements output-level distillation with directional feature alignment in deep layers during early denoising, with consistent gains across two backbones and multiple OPD baselines.The paper reports theoretical and empirical evidence linking persistent depth–temporal mismatch structure to selective supervision.Source: §8 · p. 10

AI USE STATEMENT

• Generative AI supported manuscript editing, related-work retrieval, and experiment environment construction, testing, and monitoring; the authors reviewed the work and accept responsibility for the final content.The named tools include GPT-6 and Claude Opus 5.Source: AI USE STATEMENT · p. 10

A.1 DEPTH-WISE DISCREPANCY TRANSPORT

• The depth-wise transport identity includes an initial embedded-state difference, and its cross term allows reinforcement or cancellation rather than guaranteeing monotonic mismatch growth.Shared noisy latents do not necessarily make the initial difference zero when input embeddings differ.Source: §A.1 · p. 12

A.2 NOISE-DEPENDENT PRIOR SENSITIVITY

• For clean-latent estimates with shared covariance, weaker observations preserve more of the prior-mean difference, but the resulting coefficient need not grow monotonically with noise.The lemma does not impose the same response on hidden features or velocity predictions.Source: §A.2 · p. 12

A.3 LOCAL UTILITY OF SELECTIVE SUPERVISION

• Under local smoothness and positive, comparable correction efficiencies, sufficiently larger mismatch can yield greater single-step output-loss reduction at a matched supervision budget.This is a local allocation principle; persistence of the selected region is an empirical observation.Source: §A.3 · p. 13

A.4 DIRECT CONNECTION TO COSINE SUPERVISION

• The local-utility proposition applies to actual cosine-loss gradients through an angular-discrepancy identity, without feature-response linearization.The identity assumes nonzero tokens with the numerical norm floor inactive; raw-MSE rankings alone do not establish the same ordering.Source: §A.4 · p. 13

B FULL RELATED WORK.

• Prior work covers multi-teacher OPD, intermediate-feature alignment, and selective distillation; this work studies which blocks and denoising stages benefit from feature supervision in diffusion OPD.Related methods include output-level OPD, representation-change alignment, and selection of feature content or locations.Source: B FULL RELATED WORK · p. 13

• The cumulative effect dominates teacher–student mismatch across depths, stages, and configurations of different networks.Figure 6 presents the full version of Figure 3.Source: C FULL RESULTS OF EMPIRICAL EVIDENCE… · p. 14