Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

Shiyi Zhang; Mushui Liu; Yunze Tong; Wanggui He; Siyu Zou; Jinlong Liu; Yunlong Yu; Jian Song; Hao Jiang; Pipei Huang; Bo Zheng

Source-linked AI summary

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

Shiyi Zhang, Mushui Liu, Yunze Tong, Wanggui He, Siyu Zou, Jinlong Liu, Yunlong Yu, Jian Song, Hao Jiang, Pipei Huang, Bo Zheng

arXiv:2608.26872v1Aug 27, 2026cs.CV

TL;DR

Prior alignment methods either propagate terminal rewards through trajectories or depend on specialized teachers, creating variance, cost, and teacher-related limitations. Self-OPD instead uses reward-guided stochastic self-exploration, self-referenced advantages, and pull-push distillation, with reward-level fusion for multiple objectives. Across single- and mixed-reward benchmarks, it outperforms prior RL and OPD methods without task-specific teachers.

Problem
Terminal-reward RL has high-variance credit assignment, while teacher-based OPD requires specialized teachers and can suffer from teacher bias and conflicting field-level updates.
Method
Self-OPD branches the student trajectory into stochastic SDE candidates, compares their rewards with a deterministic self-reference, and applies all-branch advantage-weighted pull-push distillation with reward-level fusion.
Results
Self-OPD outperforms prior RL and teacher-based OPD methods across single-reward and mixed-reward benchmarks.
Takeaways & Limitations
Self-OPD can match or surpass teacher-based OPD while avoiding task-specific teacher training and enabling reward-level multi-objective fusion.

Abstract

from arXiv · show

On-policy distillation (OPD), which leverages a pre-trained, specialized teacher model to provide dense supervisory signals, has achieved significant success in Large Language Models (LLMs) and has recently been adapted to flow matching models. However, this paradigm suffers from two major issues: First, training a separate, task-specific teacher for every new objective incurs high computational costs. Second, the discrepancy between teacher and student distributions often leads to compounding errors along the generation trajectory. In this paper, we introduce \textbf{Self-OPD}, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision. At each timestep, Self-OPD branches the deterministic next-state prediction into $K$ stochastic SDE candidates, rolls them out with the ODE sampler, and compares their rewards against a deterministic self-reference baseline to obtain normalized advantages. The velocity field is optimized with an all-branch pull-push objective, where high-advantage branches attract the student and low-advantage branches repel it under direction-aware attenuation and SDE-variance normalization. For multi-objective alignment, Self-OPD fuses normalized scores at the reward level, avoiding direct gradient conflict. Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without task-specific teachers.

1 Introduction

Self-OPD addresses the cost, teacher dependence, and distribution-gap limitations of prior alignment approaches by turning student self-exploration into dense step-wise supervision. It combines self-referenced reward advantages with pull-push distillation and reward-level multi-objective fusion.

Flow Matching models provide a strong visual-generation backbone, but downstream alignment for rendering, composition, and preference remains challenging.
Terminal-reward RL assigns credit across denoising trajectories, producing high-variance gradients and fragile multi-objective alignment.
Teacher-based OPD supplies dense per-step supervision but requires specialized teachers for new objectives, inherits teacher quality and bias, and can create conflicting field-level updates.
Self-OPD replaces teacher velocity targets with reward-weighted targets from K stochastic SDE branches, deterministic ODE rollouts, and a self-reference baseline.
The method converts reward-guided self-exploration and a self-reference baseline into dense step-wise supervision through an all-branch pull-push objective.
Reward-level fusion supports black-box multi-objective alignment without field-level teacher routing or gradient conflicts.
Self-OPD empirically outperforms prior RL and teacher-based OPD methods across single-reward and mixed-reward benchmarks.

2 Related Work

Prior work aligns diffusion and flow models through reinforcement learning or on-policy distillation. These approaches differ in where rewards are estimated and whether step-wise supervision depends on a pretrained teacher.

RL in Diffusion Models: RL methods reformulate denoising as sequential decision-making to optimize rewards with policy gradients over diffusion timesteps.
On-Policy Distillation: OPD trains models on student-generated outputs to reduce exposure bias and improve training efficiency, with Flow-OPD and Diffusion-OPD extending the paradigm to generative flow and diffusion models.
Self-OPD’s pipeline uses stochastic SDE branching, ODE rollouts, reward scoring, and advantage-weighted pull-push distillation.

3 Methodology

Self-OPD converts the student's own stochastic local exploration into dense, step-wise supervision, then distills it through advantage-weighted pull-push updates. It also combines multiple objectives at the reward level rather than merging potentially conflicting gradients.

3.1 Preliminaries: Flow Matching learns a continuous velocity field that transports noise to data along a discretized reverse-time ODE trajectory.The trajectory uses timesteps 1 = t_0 > t_1 > ··· > t_S ≈ 0 and increments Δt_j < 0.
3.1 Preliminaries: The reverse ODE can be augmented into an SDE whose Euler-Maruyama transitions combine deterministic predictions with isotropic stochastic perturbations.The noise schedule is controlled by η, while setting η = 0 recovers the deterministic Euler ODE step.
3.2 Self-Referenced Evaluation and Self-Exploration: Self-OPD samples K stochastic next states around each deterministic prediction, rolls branches out with deterministic ODE integration, and scores their decoded outputs.The transformer computes the shared base prediction once; additional branch diversity comes from lightweight Gaussian SDE perturbations.
3.2 Self-Referenced Evaluation and Self-Exploration: A deterministic ODE rollout from the same parent state supplies a self-reference baseline, converting terminal branch rewards into normalized advantages.Positive advantages identify branches outperforming the default trajectory, whereas negative advantages identify directions to avoid.
3.3 All-Branch Pull-Push Distillation: The all-branch objective pulls the student toward high-advantage branch velocities and pushes it away from low-advantage velocities.Direction-aware attenuation suppresses repulsion when a negative branch aligns with the best direction, while transition-variance normalization avoids timestep-dependent misweighting.
3.4 Reward-Level Fusion for Multi-Objective Alignment: Self-OPD fuses multiple normalized rewards at the non-differentiable reward level, shifting joint optimization from parameter-space gradient combination to trajectory-space branch ranking.The composite score remains a branch-ranking signal and avoids direct inter-objective gradient conflicts.

4 Experiments

Experiments evaluate Self-OPD on single- and mixed-reward alignment across compositional generation, text rendering, and preference metrics. Self-OPD delivers strong quality, robustness across prompt families, stable ablations, and lower wall-clock cost than teacher-based OPD.

Experimental settings: Self-OPD is evaluated on text rendering, compositional generation, and aesthetic or human-preference alignment using OCR, GenEval, PickScore, and HPSv2.Mixed-reward variants fuse each task scorer with preference rewards.
Single-reward results: 0.95 GenEval, 97.5% OCR accuracy, 24.79 PickScore, and 0.3665 HPSv2 are achieved by Self-OPD in single-reward training.It outperforms Flow-GRPO and GRPO-Guard on every task with identical base models and reward signals.
Mixed-reward results: 0.95 GenEval, 96.0% OCR accuracy, 23.87 PickScore, and 0.3214 HPSv2 are achieved by mixed-reward Self-OPD, exceeding DiffusionOPD’s 22.72 PickScore and 0.2676 HPSv2.The comparison uses the same test images protocol for preference evaluation.
Mixed-reward results: DiffusionOPD preference scores fall by Δ=1.23 for PickScore and Δ=0.105 for HPSv2 on GenEval/OCR prompts, while Self-OPD distributions nearly coincide across prompt families.The figure contrasts aesthetic/preference prompts with GenEval/OCR task prompts.
Qualitative results: Self-OPD qualitatively improves text fidelity, spatial layouts, counting constraints, scene richness, and aesthetic quality compared with mixed-reward and single-reward baselines.Examples include exact phrases, laptop-on-teddy-bear placement, vase-right-of-horse placement, and exactly four benches.
Ablation study: All-branch advantage-weighted distillation stabilizes training compared with Best-of-K, while bounded gradient gating maintains stable, monotonic improvement.Unbounded repulsive gradients can dominate and trigger performance collapse; timestep weighting accelerates convergence, whereas early-step oversampling degrades performance below the base model.
Training efficiency: 97.0 h is DiffusionOPD’s total wall-clock time, versus approximately 48 h for warm-start OCR and 44 h for warm-start GenEval with Self-OPD.From-scratch Self-OPD reaches DiffusionOPD-level OCR in approximately 62 h and GenEval in approximately 90 h.

5 Conclusion

The paper introduces Self-OPD as a teacher-free framework that converts student self-exploration into dense per-step supervision. Across text rendering, compositional generation, and preference alignment, it matches or surpasses teacher-based OPD while avoiding task-specific teachers and supporting reward-level fusion.

Conclusion: Self-OPD branches the student trajectory, scores branches with task rewards, and applies all-branch advantage-weighted regression with direction-aware modulation.This turns self-exploration into dense per-step supervision.
Conclusion: Experiments show that Self-OPD can match or surpass teacher-based OPD across text rendering, compositional generation, and preference alignment.The framework also enables reward-level multi-objective fusion without task-specific teacher training.

A.1 GenEval: Strict vs. Continuous Scoring

GenEval reports either strict binary success or continuous partial credit over prompt sub-requirements. The example shows that continuous scoring distinguishes partial fulfillment that strict scoring records as failure.

Strict scoring: Strict GenEval scoring assigns 1 only when every prompt sub-requirement is satisfied and 0 otherwise.The reported score averages this binary per-image metric across test prompts.
Continuous scoring: Continuous GenEval scoring assigns each image the fraction of satisfied sub-requirements, ranging from 0 to 1.The average across prompts gives partial credit for incomplete fulfillment.
Illustration: 0 versus 0.5 separates strict and continuous scores for the base model’s partially correct handbag-and-giraffe image.Self-OPD satisfies both requirements, receiving strict score 1 and continuous score 1.0.

A.2 Preference Protocols: Same Test Images vs. Separate Test Set

The paper compares preference quality on the same task-generated images versus a separate aesthetic prompt set. Self-OPD is strongest under the primary same-test-images protocol, while separate-set gains may not transfer to structured tasks.

Same Test Images: The same test images protocol scores GenEval- and OCR-generated images with PickScore and HPSv2, averaging the two prompt-suite scores equally.It evaluates preference quality without introducing distribution shift from additional prompts.
Separate Test Set: The separate test set uses disjoint DrawBench aesthetic prompts to measure preference quality on a general-purpose prompt distribution.This protocol can favor aesthetic-prompt overfitting while structured-task performance degrades.
Protocol Comparison: Self-OPD attains the highest PickScore and HPSv2 among all methods under the same test images protocol.The protocol is treated as primary because it compares metrics on identical generations and reflects overall image quality without sacrificing task performance.
Protocol Comparison: DiffusionOPD’s lead on the separate set does not carry over to GenEval and OCR test images, whereas Self-OPD remains strong on both.The comparison links field-level fusion with prompt-family dependence, while Self-OPD maintains preference quality across task-specific prompts.

B Additional Qualitative Results

Qualitative comparisons show that Self-OPD combines task accuracy with richer, more realistic visual appearance on both GenEval and OCR prompts.

GenEval: On GenEval prompts, Self-OPD satisfies counting, spatial-relation, and attribute requirements while producing richer lighting, natural textures, and better composition.The figure comparison covers Self-OPD, Flow-GRPO, GRPO-Guard, DiffusionNFT, Flow-OPD, and DiffusionOPD.
OCR: On OCR prompts, Self-OPD renders target text accurately while generating appealing scenes with coherent backgrounds and realistic details.Other methods tend toward flatter or overly simplistic images, with some showing over-saturation and lost detail.