PaperlayerView exact revision on arXiv

Source-linked AI summary

DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation

Jiaxing Li, Kai Zou, Cindy Zhou, Kaichen Huang, Junyao Gao, Zile Wang, Yang Liu, Bin Liu, Bo An, Yangguang Li

arXiv:2607.26811v1cs.CV

TL;DR

Autoregressive video distillation often evaluates initialization visually despite differing distributional targets across stages. DistillAlign measures mode coverage and combines mode-seeking DMD with mode-covering consistency distillation, improving quality, coverage, and diversity, even with a smaller DMD teacher.

  • Problem

    Existing pipelines decouple initialization from DMD refinement and mainly use visual scores, leaving matched mode coverage between student initialization and DMD teacher insufficiently evaluated.

  • Method

    DistillAlign evaluates student–teacher precision and coverage in shared latent space and jointly optimizes DMD mode seeking with consistency-based mode covering.

  • Results

    The method improves generation quality, teacher-distribution coverage, and diversity; with a Wan-1.3B DMD teacher, it surpasses all Wan-14B teacher baselines.

  • Takeaways & Limitations

    Distributional alignment and coordinated mode covering and seeking are important for balancing visual quality, coverage, and diversity in autoregressive video distillation.

  • Takeaways & Limitations

    Mode-seeking refinement can suppress rare, high-motion, detailed, or dynamically rich videos, trading diversity and sometimes overall quality for local realism.

Abstract

from arXiv · show

Existing autoregressive video distillation methods commonly adopt a Distribution Matching Distillation (DMD)-based multi-stage pipeline. However, they typically decouple the initialization and DMD stages -- which then pursue different target distributions -- and judge the intermediate student mainly by visual scores such as VBench. In this paper, we revisit this design from a distributional perspective. Given the mode-seeking nature of the distribution matching loss, a good initialization should match the mode coverage of the target DMD teacher, rather than merely pursuing high quality. To analyze this, we introduce a distributional evaluation protocol that measures precision and coverage between student and teacher distributions in a shared latent space. It exposes differences hidden by visual scores: some initializations reach high precision but low coverage, leading to suboptimal refinement, while mode-covering ones preserve broader support. Furthermore, even when the target distributions are aligned, DMD's reverse-KL objective can still drive the student toward high-probability teacher regions in late training, reducing coverage and diversity. To address this, we propose joint distillation, which combines DMD's mode-seeking objective with a Consistency Distillation-based mode-covering constraint. Experiments show that our method improves generation quality, coverage, and diversity; notably, even with a Wan-1.3B DMD teacher, it outperforms baselines refined with Wan-14B, underscoring the importance of distributional alignment in autoregressive video distillation.

1 INTRODUCTION

The paper argues that autoregressive video distillation should align initialization and DMD target distributions, evaluating mode coverage alongside precision rather than relying on visual scores alone. It proposes joint distillation to combine DMD’s mode-seeking objective with CD’s mode-covering constraint, improving quality, coverage, and diversity.

  • Problem: Existing pipelines decouple initialization from DMD refinement and commonly judge intermediate students using visual scores such as VBench.Self Forcing uses ODE distillation before DMD, while Causal Forcing uses causal training and causal ODE or Consistency Distillation before DMD.
  • Distributional alignment: DMD’s mode-seeking refinement works best when initialization already covers the target modes; misalignment can cause unsupported gradients, coverage collapse, and suboptimal refinement.A controlled target-distribution swap shows that matched distributions can outperform settings with stronger initialization data or DMD teachers.
  • Distributional evaluation: The proposed evaluation projects teacher and student samples into a shared latent space and measures distribution agreement using precision and coverage.It exposes initializations with high visual quality and precision but low coverage, which may have collapsed onto a small set of high-quality modes.
  • Late-stage drift: Even with aligned targets, pure DMD can drift toward high-probability teacher regions, causing diversity to decrease while visual quality first rises and then drops.Table 2 reports that VBench score first increases and then decreases, while diversity consistently drops during late-stage pure DMD training.
  • Joint distillation: Joint distillation combines DMD’s mode-seeking objective with CD’s mode-covering constraint to improve generation quality, teacher-distribution coverage, and diversity.With a Wan-1.3B DMD teacher, the method surpasses all Wan-14B teacher baselines, showing that distributional alignment can rival teacher scale.

2 BACKGROUND

The background reviews flow matching and distillation approaches for learning generative dynamics. Flow matching and ODE/consistency distillation are mode-covering, whereas DMD uses reverse KL and is mode-seeking.

  • Flow Matching: Flow matching learns a velocity field connecting noise and data distributions, then generates samples by integrating the learned ODE from t = 1 to t = 0.The target velocity is x1 −x0, conditioned on signal c.
  • ODE and Consistency Distillation: ODE distillation directly predicts teacher ODE endpoints, while consistency distillation enforces local consistency across adjacent discretized time steps.Both methods recover the teacher flow map fθ(xt, t) = Ψt→0(xt) at convergence.
  • ODE and Consistency Distillation: As regressions onto teacher endpoints, ODE distillation and consistency distillation exhibit mode-covering behavior that preserves broad support at the cost of sharpness.Their shared endpoint-recovery formulation underlies this mode-covering characterization.
  • Distribution Matching Distillation: DMD distills a multi-step teacher into a few-step generator by minimizing a reverse-KL objective over perturbed generator and teacher marginals.Its score-difference update uses a frozen teacher score and an online critic estimating the fake score.
  • Distribution Matching Distillation: DMD's reverse-KL objective is inherently mode-seeking, concentrating the student on high-density teacher modes at the expense of coverage and diversity.This behavior contrasts with the broad-support preservation described for endpoint-regression methods.

3 METHODOLOGY

The methodology evaluates initialization by distributional coverage rather than visual quality, using teacher-normalized refinement and precision–coverage measurements in a shared video-feature space. It also analyzes mode-seeking drift during DMD and introduces joint distillation to preserve coverage while sharpening high-probability teacher regions.

  • Distributional evaluation: DMD sharpens modes already covered by initialization but cannot recover missed modes, making initialization coverage central to refinement quality.The protocol focuses on accessible scene layouts, object configurations, motions, and teacher basins rather than visual quality alone.
  • Distributional evaluation: Teacher-normalized refinement re-noises each initializer sample with a fixed teacher process and denoises it under identical normalization settings before comparison.Shared re-noising level, normalization teacher, and denoising schedule largely normalize low-level fidelity while retaining the initializer’s reached semantic region.
  • Distributional evaluation: Precision and coverage are computed by comparing refined initializer features with teacher samples in a frozen video-representation space, using k-nearest-neighbor teacher support.Precision tests whether refined samples lie inside estimated teacher support, while coverage measures how much teacher support the initializer reaches.
  • Target alignment: Matched initialization and DMD targets achieve higher coverage than mismatched settings both before and after DMD.This controlled teacher-source swap quantifies coverage under the corresponding DMD teacher distribution.
  • Initialization comparison: High initialization VBench does not ensure the best final result: AR Init and Causal DMD Init show high precision but low coverage, whereas Causal CD Init preserves broader coverage before and after DMD.Under the same initialization target, broader coverage is also reflected in higher sample diversity.
  • Joint distillation: Joint distillation combines DMD’s mode-seeking loss with a consistency-distillation constraint that preserves teacher coverage and suppresses excessive mode collapse.The consistency term anchors the student while DMD sharpens high-probability regions, balancing mode seeking and mode covering during late training.

4 EXPERIMENTS

Experiments show that joint distillation improves visual quality while preserving teacher coverage and diversity, and that mode-covering initialization benefits DMD refinement. Ablations further show that pure DMD contracts coverage over training, while teacher-normalized re-noising makes initializer comparisons more reliable.

  • Main comparison: Our method achieves the best overall VBench score and strongest coverage among distilled autoregressive models.The weak variant remains competitive with a Wan2.1-1.3B teacher, while the full variant further improves performance with Wan2.1-14B.
  • Main comparison: Joint distillation preserves sharper subjects, stronger motion, and broader appearance and motion variations without sacrificing visual quality.Qualitative comparisons report better object identity, motion details, and temporal coherence than prior distilled autoregressive baselines.
  • Ablation studies: Larger λ yields the highest coverage but lower VBench scores, whereas decreasing λ strengthens DMD, improving visual quality and precision; λ = 0 recovers pure DMD.The joint loss balances DMD’s mode-seeking objective against the CD-based mode-covering constraint over 2500-step trajectories.
  • Ablation studies: Pure DMD’s coverage and diversity continuously decrease during training, while joint distillation maintains higher coverage and diversity with comparable or better VBench scores.This indicates that mode-seeking refinement gradually contracts the student distribution.
  • Initializer evaluation: 0.020 raw coverage for ODE rises to 0.326 after teacher-normalized re-noising, resolving its misleading gap versus sharper initializers.ODE reaches 0.484 coverage after DMD versus 0.422 for Causal DMD, despite raw coverage of 0.020 versus 0.324.

5 RELATED WORK

Related work has advanced autoregressive video generation by modifying temporal denoising or converting bidirectional diffusion models into causal generators. Diffusion distillation objectives differ distributionally, with teacher- or data-trajectory training favoring coverage but potentially averaging modes and losing detail.

  • Autoregressive Video Generation: Autoregressive video generation work changes temporal denoising or converts pretrained bidirectional video diffusion models into causal generators.Diffusion Forcing and progressive autoregressive diffusion use per-frame or progressively scheduled noise levels.
  • Autoregressive Video Generation: Diffusion Forcing and progressive autoregressive diffusion introduce per-frame or progressively scheduled noise levels to bridge next-token prediction.
  • Diffusion Distillation: Offline teacher-supervised objectives exhibit forward-divergence or mean-seeking behavior, covering teacher samples while potentially averaging modes or losing fine detail.The passage includes progressive distillation, consistency distillation, consistency trajectory models, and continuous-time consistency models.

6 CONCLUSION

The paper revisits autoregressive video distillation from a distributional perspective, identifying a mismatch between mode-covering initialization and DMD’s mode-seeking refinement. Joint distillation addresses this by combining DMD with a CD-based mode-covering constraint, improving the balance among visual quality, coverage, and diversity.

  • Existing pipelines decouple pre-DMD initialization from DMD refinement and evaluate initialization mainly through visual scores.
  • Trajectory- or consistency-based initialization primarily provides mode coverage, whereas DMD performs mode-seeking reverse-KL refinement.
  • Joint distillation combines DMD’s mode-seeking objective with CD’s mode-covering constraint, achieving a better balance among visual quality, coverage, and diversity.The results highlight the importance of distributional alignment in autoregressive video distillation.

A MORE DETAILS OF COVERAGE METRICS

The protocol evaluates student and reference-teacher distributions on matched 256-sample sets using teacher-normalized V-JEPA2 features. It reports fixed-support precision and coverage, alongside a separate within-method Vendi diversity score.

  • Teacher normalization and reference support: The protocol distinguishes normalization teacher Tnorm from reference teacher Tref, applying shared normalization while defining target support and neighborhood radii from Tref.For Wan2.1-14B-targeted experiments, the shared 1.3B model performs normalization, whereas reported precision and coverage use Wan2.1-14B reference samples.
  • Evaluation sample sets: Evaluation uses 16 prompts and 16 seeds, producing exactly one video per prompt–seed pair and 256 pooled samples for both model and reference teacher.The same grid controls prompt mixture and sampling budget, while precision and coverage remain set-level statistics rather than matched-pair comparisons.
  • Temporal standardization and representation: Videos are restricted to the first five seconds, represented by eight uniformly spaced frames, and encoded as raw ℓ2-normalized V-JEPA2 features without PCA or dimensionality reduction.The default representation uses the first 81 frames at 16 fps and concatenates token-wise mean and sample standard deviation into one descriptor.
  • Local teacher support and metrics: Precision measures evaluated samples inside adaptive teacher neighborhoods, whereas coverage measures teacher samples whose neighborhoods contain evaluated samples, using globally fixed k = 5.Coverage uses only teacher radii, keeping target support fixed and preventing isolated student outliers from enlarging regions; with M = 256, it changes in increments of 1/256 before rounding.
  • Metric interpretation and diversity: High precision can coexist with low coverage for mode-seeking generators, while coverage is not a standalone perceptual-quality score and both metrics depend on the evaluation configuration.Diversity is reported separately as the raw Vendi effective-rank score on the same normalized V-JEPA2 descriptors, not as a teacher ratio.

B ADDITIONAL ABLATION · B.1 JOINT DISTILLATION

Under the weak Wan2.1-1.3B setting, joint distillation consistently preserves coverage and diversity better than pure DMD, with λ = 0.01 offering the best overall trade-off. Across training, joint distillation maintains these benefits while achieving comparable or better total scores later in training.

  • B.1 JOINT DISTILLATION: Joint distillation better preserves coverage and diversity than pure DMD when Wan2.1-1.3B supplies both initialization data and the DMD teacher.This weak-setting result is consistent with the main Wan2.1-14B setting.
  • B.1 JOINT DISTILLATION: λ = 0.01 provides the best overall trade-off in the weak-setting joint-distillation ablation.The comparison considers quality, precision, and coverage over the best checkpoint in a 2500-step trajectory.
  • B.1 JOINT DISTILLATION: As λ decreases, DMD’s mode-seeking effect strengthens, improving precision while reducing coverage and diversity.λ = 0 recovers pure DMD.
  • B.1 JOINT DISTILLATION: Larger λ strengthens the Consistency Distillation-based mode-covering constraint, whereas smaller λ gives greater weight to DMD refinement.Table 8 reports diversity as the raw Vendi score under the same V-JEPA2 protocol as Table 4.
  • B.1 JOINT DISTILLATION: Across training steps, pure DMD steadily loses coverage and diversity, whereas joint distillation maintains higher values for both.The comparison tracks both methods under the weak Wan2.1-1.3B setting.
  • B.1 JOINT DISTILLATION: In later training, joint distillation reaches comparable or better total scores while preserving its coverage and diversity advantage over pure DMD.This trend mirrors the Wan2.1-14B results.

C PROMPTS

This section lists the prompts used for qualitative examples and visual comparisons, with shared prompts and, when applicable, fixed random seeds ensuring fair comparisons across methods.

  • Prompt Reproducibility: All methods shown in the same figure use the same prompt and, when applicable, the same set of fixed random seeds.The reference column identifies the figures in which each prompt is used.
Loading 2607.26811v1…