Source-linked AI summary
On-Policy Adversarial Flow Distillation for Autoregressive Video Generation
Yang Luo, Shengju Qian, Xiaohang Tang, Zirui Zhu, Yong Liu, Xin Wang, Yang You
TL;DR
Distilling heterogeneous black-box video teachers into causal autoregressive students is difficult because students must learn from their own rollouts while teachers may expose only completed videos. AFD combines prompt-paired adversarial discrepancy estimation with forward-process flow-matching updates on student noised states. Across two causal student families, it improves motion- and physics-sensitive generation while preserving general video quality.
Problem
Black-box teachers expose only completed videos and lack scores, latents, or denoising trajectories, making SFT off-policy and score-based distillation unavailable.
Method
AFD compares teacher videos with on-policy student rollouts using a prompt-conditioned discriminator, then converts its advantage into forward-process flow-matching updates.
Results
AFD consistently improves motion- and physics-sensitive generation across two causal autoregressive student families while keeping general video quality close to the strongest baselines.
Takeaways & Limitations
AFD provides a practical route for transferring capabilities from sampling-only teachers to efficient autoregressive students using completed teacher videos and student rollouts.
Takeaways & Limitations
Experiments cover two causal student backbones and a small set of target domains, while broader teacher, video-length, prompt-distribution, and discriminator-tuning studies remain future work.
Abstract
from arXiv · showhide
Autoregressive video generators are attractive for streaming, long-horizon, and interactive applications, but distilling strong black-box teachers into causal students remains difficult. The student must learn under its own rollout distribution, whereas practical teachers may expose only prompt-conditioned completed videos and may differ in architecture, capacity, temporal design, and sampling schedule. This interface makes supervised fine-tuning off-policy, score-based distillation inapplicable, and direct adversarial imitation too sparse for denoising-time credit assignment. We propose Adversarial Flow Distillation (AFD), an on-policy framework for heterogeneous black-box video distillation. AFD queries the teacher and rolls out the current student on the same prompts, trains a prompt-paired Bradley-Terry discriminator to estimate clean-sample teacher-student discrepancy, and converts the resulting on-policy advantage into forward-process flow-matching updates on the student's own noised states. Thus, AFD provides dense velocity-field supervision while requiring no teacher scores, latents, denoising trajectories, step alignment, or reverse-chain reinforcement learning. Experiments across two causal AR student families show that AFD consistently improves motion- and physics-sensitive generation while preserving general video quality, and ablations validate the importance of adaptive on-policy feedback and forward-process credit assignment. The method requires only clean teacher videos and student rollouts, providing a practical route for distilling proprietary or heterogeneous video generators into efficient autoregressive students.
1. Introduction
Black-box teacher access and student rollout mismatch make conventional distillation inadequate for causal autoregressive video generation. AFD addresses this by pairing on-policy teacher–student discrimination with forward-process flow-matching updates, improving motion-sensitive generation while preserving general quality.
- Motivation: Black-box teachers often provide only completed prompt-conditioned videos, while differing from students in architecture, temporal conditioning, capacity, and sampling schedule.This prevents access to teacher scores, latents, sampler states, or denoising trajectories.
- Motivation: SFT trains on teacher-induced prefixes rather than student rollouts, allowing local autoregressive errors to shift future denoising states.Figure 1 reports that VBench and VideoPhy-2 scores decrease rather than improve monotonically as SFT steps increase.
- Motivation: Black-box video evidence must be converted from completed-video feedback into dense supervision for the student’s time-dependent causal denoising vector field.Sample-level rewards alone do not identify which frames, motion patterns, or denoising states cause the discrepancy.
- AFD: AFD queries the teacher and rolls out the current student on shared prompts, then trains a prompt-conditioned discriminator to estimate their clean-sample discrepancy.The discriminator produces an on-policy advantage normalized against the current batch or prompt distribution.
- AFD: AFD uses reward-weighted forward-process flow matching on the student’s own noised rollouts rather than reverse-trajectory policy gradients.Higher-advantage rollouts provide positive examples whose forward velocities guide the student vector field.
- Results: AFD is a score-free framework evaluated on two causal autoregressive backbones, with consistent gains on motion- and physics-sensitive metrics and ablations of adaptation and discriminator design.The stated contributions emphasize heterogeneous black-box access and dense forward-process updates.
2. Related Work
Prior work develops diffusion and flow-based generation, on-policy distillation, and preference-guided alignment. AFD builds on these directions while targeting black-box video teachers and avoiding expensive reverse-trajectory optimization.
- Video diffusion and flow models: Diffusion, score-based SDEs, Flow Matching, and Rectified Flow established denoising-time generation and continuous-time vector-field learning for modern video systems.DiT improved transformer diffusion scalability and underlies several video-generation systems.
- Black-box on-policy distillation: On-policy distillation keeps training aligned with the student’s own trajectory distribution while using teacher supervision, with extensions to language, action, and temporal video settings.The related methods apply the on-policy idea across multiple generation and control domains.
- Adversarial and preference-guided alignment: Adversarial and preference-guided methods use learned or human feedback to guide diffusion models, while reverse-trajectory policy gradients remain expensive for video.DiffusionNFT instead optimizes on the forward process from clean generated samples, avoiding likelihood estimation and reverse-trajectory storage.
3. Method
AFD distills heterogeneous black-box video teachers by comparing teacher samples with current student rollouts, then transferring the resulting clean-sample discrepancy to dense updates on the student’s own noised states. Its design avoids teacher scores, hidden trajectories, shared latent spaces, and step alignment while retaining on-policy training.
- Black-Box Teacher Interface: The teacher and student need not share architecture, capacity, latent representation, sampling schedule, or denoising-step alignment.Because the teacher exposes no trajectories and may use a different number of denoising evaluations, no observable teacher-to-student step alignment operator is available or required.
- Adaptive Teacher–Student Discrimination: AFD queries completed teacher videos and autoregressively self-conditioned student rollouts, then trains a prompt-conditioned discriminator to estimate their distributional discrepancy.The discriminator uses teacher samples as preferred over current student samples in a Bradley–Terry objective and provides adaptive feedback under the shared prompt distribution.
- Diffusion-Native On-Policy Update: The discriminator’s sample-level advantage is converted into forward-process flow-matching updates on the student’s own noised rollouts.Student videos are forward-noised using the student’s schedule; normalized advantages weight positive and negative updates to the velocity field.
- Black-Box Teacher Interface: AFD addresses the black-box interface by using only completed teacher and student videos, rather than teacher scores, latents, reverse states, or transition kernels.The method transfers completed-video evidence to student noised states through the student’s known forward-noising kernel.
- Black-Box Teacher Interface: SFT and DMD are mismatched to this setting because SFT is off-policy, while DMD requires unavailable teacher scores or density ratios at student noised states.Student errors can shift future conditioning distributions, and the teacher exposes only completed prompt-conditioned samples.
- Forward-Process Credit Assignment: Forward-process credit assignment avoids treating the long, high-resolution reverse denoising chain as a reinforcement-learning trajectory.AFD separates clean-sample density-ratio estimation from denoising-time credit assignment and propagates sample-level preferences through a student-side forward-noising map.
4. Experiments
AFD is evaluated on two causal autoregressive student families under black-box continual adaptation, including physics-oriented and shifted-style domains. It improves physics-sensitive generation and motion while largely preserving general video quality, with ablations showing that adaptive discriminator feedback and prompt-paired objectives matter.
- Experimental setup: AFD is evaluated on Self-Forcing and Causal-Forcing students using a prompt-conditioned Seedance 2.0 sampling API and limited adaptation data.The main experiment samples 200 VideoPhy-2 examples, while discriminators are updated online against current student rollouts.
- Main results: AFD achieves the best Physics VBench Total on both Self-Forcing (87.55) and Causal-Forcing (88.52).It also reaches the best reported physics-domain scores on several accompanying evaluations, including VideoPhy-2-PC.
- Main results: AFD preserves general video quality, with General Total scores of 60.83 versus 60.95 for Self-Forcing and 59.83 versus 60.32 for Causal-Forcing.These comparisons place AFD close to the strongest baseline while improving physics-sensitive metrics.
- Main results: AFD produces the strongest dynamic degree on both student families, indicating improved motion and physical behavior without a large general-quality drop.Qualitative examples likewise show preserved prompt content and visual quality alongside improved motion-related behavior.
- Continual learning: AFD adapts students to a Disney-style animation domain while preserving prompt alignment and coherent motion.This continual-learning result extends evaluation beyond the physics-oriented prompt distribution.
- Ablations: Intermediate discriminator learning rates, 5×10^-6 and 1×10^-5, avoid reward saturation caused by under-updating or over-updating.Rates at or below 1×10^-6 produce stale reward saturation, whereas 5×10^-5 makes the discriminator overly strong early.
- Ablations: Replacing the prompt-paired Bradley–Terry loss with a GAN-style binary loss mainly reduces dynamic degree while leaving other motion and physics dimensions close.The Bradley–Terry objective supplies a relative advantage under shared prompts, whereas absolute classification can rely on easy distributional cues and saturate.
5. Conclusion
The conclusion frames AFD as a practical black-box distillation route that uses observable completed videos rather than hidden teacher generation processes. By combining adaptive discrimination with forward-process credit assignment, it improves motion and physical behavior without sacrificing general quality.
- 5. Conclusion: AFD formulates black-box video distillation around the available sample interface rather than the teacher’s hidden generation process.For causal autoregressive students, it connects completed-video discrepancies to the student’s own denoising-time states.
- 5. Conclusion: Separating adaptive discrimination from forward-process credit assignment turns completed-video supervision into an on-policy training signal.The conclusion identifies this decomposition as a practical route for transferring capabilities from sampling-only teachers to efficient autoregressive generators.
6. Limitations
AFD is evaluated on two causal autoregressive student backbones and a small set of target domains, limiting evidence about its broader generality. Performance may also depend on online discriminator update rate and reward calibration.
- Experiments cover two causal autoregressive student backbones and mainly physics-oriented and stylized animation prompts.
- Broader studies across more teachers, longer videos, and more diverse prompts are needed to further test AFD’s generality.
- AFD performance can depend on discriminator update rate and reward calibration, with systematic tuning left for future work.
- Evaluation on broader reasoning-oriented benchmarks such as V-ReasonBench is left for future work.
7. Hyperparameters
The section provides comprehensive AFD training configurations across model, optimization, diffusion, regularization, and rollout settings.
- Table 3 lists AFD’s model, optimization, diffusion, regularization, and rollout settings.
- The configurations include model and video specifications, LoRA fine-tuning, selective regularization, and streaming rollout.
8. Additional Qualitative Visualizations
Additional visualizations illustrate the main experimental pattern: AFD improves motion evolution and physical plausibility while preserving prompt semantics and visual fidelity.
- AFD improves motion evolution and physical plausibility while preserving the base student’s prompt-level semantics and visual fidelity.
9. Prompts for Qualitative Visualizations
The section presents the prompts used for qualitative examples in images 3 and 4.
- The prompts used for qualitative examples are provided for images 3 and 4.
10. Theoretical Insights
The theoretical analysis connects AFD’s reward-tilting step to teacher distillation under an on-policy setting. It shows that, with an optimal discriminator, the resulting policy improvement is equivalent to reverse-KL distillation from the teacher.
- Formal setup: AFD models the teacher as a clean-sample distribution and the student through a velocity field, with an EMA data-collection policy inducing the rollout distribution.The forward process and sample-level forward velocity provide the formal setting for the analysis.
- Adversarial policy improvement: The discriminator estimates teacher–student density differences from clean samples, which are converted into an advantage-form reward for policy improvement.The reward is defined from the density ratio estimated by the discriminator.
- Theoretical result: Under an optimal discriminator, the reward-tilted policy recovers the teacher policy.The analysis identifies the tilted policy as the conditional distribution induced by reward reweighting and states that Equation 20 recovers the teacher.
- Theoretical result: Proposition 1 establishes that the policy improvement step from π_old to π+ is equivalent to on-policy reverse-KL distillation from the teacher π_T.The proof concludes that the resulting optimization problem is reverse-KL distillation from the teacher model.