PaperlayerView exact revision on arXiv

Source-linked AI summary

AAD-1: Asymmetric Adversarial Distillation for One-Step Autoregressive Video Generation

Haobo Li, Yanhong Zeng, Yunhong Lu, Jiapeng Zhu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Yujun Shen, Zhipeng Zhang

arXiv:2606.03972v2cs.CV

TL;DR

One-step autoregressive video generation is limited by motion collapse and unstable training. AAD-1 uses an asymmetric bidirectional holistic discriminator and distribution-matching warm-up, achieving state-of-the-art performance on VBench with superior visual quality and motion fidelity.

  • Problem

    Existing one-step autoregressive video methods face architectural and training limitations, including motion collapse and instability during coupled causal adaptation and distillation.

  • Method

    AAD-1 keeps the generator causal while using a bidirectional video-level discriminator and phased distribution-matching warm-up before adversarial distillation.

  • Results

    AAD-1 achieves state-of-the-art one-step autoregressive video generation on VBench with superior visual quality and motion fidelity.

  • Takeaways & Limitations

    AAD-1 provides a one-step autoregressive video generation framework that addresses motion collapse and training instability within the evaluated VBench setting.

  • Takeaways & Limitations

    The method remains limited in fast motion, complex structures, and long-horizon extrapolation, where rollouts may drift and degrade in quality.

Abstract

from arXiv · show

We present AAD-1, an Asymmetric Adversarial Distillation framework for One-step autoregressive image-to-video generation. State-of-the-art methods adopt adversarial distillation but suffer from motion collapse and training instability, resulting in static videos. AAD-1 addresses these challenges through two key designs in architecture and training strategy. Our key architectural insight is to break the symmetry between generator and discriminator. While the generator remains causal to preserve autoregressive sampling capability, the discriminator attends bidirectionally over the full spatiotemporal context and produces a single holistic realism score for the entire video sequence. This asymmetric design enables the discriminator to effectively detect global temporal failures and long-range drift that cause motion collapse in autoregressive generation. To stabilize training, we introduce a phased strategy that first uses distribution matching to bootstrap a stable one-step generator, providing a warm-up phase that brings the student distribution closer to the teacher before adversarial distillation begins. Extensive experiments on VBench demonstrate that AAD-1 achieves state-of-the-art performance in one-step autoregressive video generation.

1. Introduction

AAD-1 targets one-step autoregressive image-to-video generation, where existing approaches face motion collapse and training instability. It combines asymmetric bidirectional holistic discrimination with distribution-matching warm-up and achieves state-of-the-art VBench performance.

  • Background: Fast autoregressive video diffusion adapts pretrained bidirectional models into few-step autoregressive systems supporting indefinitely long video generation.The paradigm is motivated by its value for real-time streaming applications.
  • Challenges: Self-rollout training jointly learns autoregressive dynamics and accelerated sampling, making optimization difficult and requiring four or more sampling steps for acceptable quality.It addresses exposure bias but couples causal adaptation with step distillation.
  • Architecture: AAD-1 keeps the generator causal but uses a bidirectional discriminator with video-level holistic discrimination over full spatiotemporal context.This asymmetric architecture is designed to address existing architectural limitations in one-step adversarial distillation.
  • Training strategy: AAD-1 first applies distribution matching to bootstrap a stable one-step generator before adversarial refinement.The warm-up produces on-manifold predictions sufficiently close to real data for meaningful discriminator gradients and helps prevent training instability.
  • Results: AAD-1 achieves state-of-the-art performance in one-step autoregressive video generation on VBench with superior visual quality and motion fidelity.The paper identifies motion collapse and training instability as critical limitations of existing methods.

2. Related Work

Prior work studies autoregressive video diffusion, efficient attention, self-rollout training, and diffusion distillation. AAD-1 differs from APT2 through bidirectional video-level discrimination and a separate DMD warm-up stage for more stable long-horizon generation.

  • Autoregressive video diffusion models: Autoregressive video diffusion models synthesize frames sequentially, conditioning each frame on preceding frames through a diffusion process.Standard training includes Teacher Forcing with clean historical frames.
  • Autoregressive video diffusion models: Block-causal attention enables streaming inference by using bidirectional attention within local temporal windows while preserving causal dependencies across blocks.This pattern supports KV-cache reuse during sequential generation.
  • Autoregressive video diffusion models: Self-rollout methods address the train-test distribution gap by learning from model-generated trajectories, but typically distill autoregressive dynamics and accelerated sampling simultaneously.These approaches are also called Self Forcing or Student Forcing.
  • Accelerating video diffusion models: Diffusion distillation compresses multi-step sampling into fewer iterations and includes trajectory-level methods and distribution-level methods.Trajectory-level techniques include progressive distillation and consistency models.
  • Accelerating video diffusion models: Compared with APT2, AAD-1 uses a bidirectional discriminator with a video-level logit and separates one-step initialization from adversarial refinement through a DMD warm-up stage.Controlled ablations associate causal frame-wise discrimination with static-video collapse, whereas bidirectional video-level discrimination supports more stable long-horizon generation.

3. Preliminaries

The preliminaries define sliding-window causal streaming, where each frame uses recent frames and retained sink frames, and explain how DMD and adversarial distillation train a causal generator. They also note that DMD may degrade at one step and that discriminator causality can vary during training.

  • Video notation and streaming: A video clip x1:T consists of T frames, each frame xt ∈ R^H×W×C, with optional conditioning c such as text.The notation specifies height H, width W, and channels C.
  • Video notation and streaming: At step t, causal streaming generates ˆxt from the previous L frames and S permanently retained sink frames.The context is written as xctx,t = (x1:S, ˆxt−L:t−1), and the window advances after each generated frame.
  • Video notation and streaming: Sink frames anchor the sequence and sliding windows capture recent context, but errors can compound over long sequences and cause temporal drift.These mechanisms improve long-range consistency and recent temporal modeling without eliminating accumulated autoregressive errors.
  • Distribution matching distillation: DMD distills a strong teacher diffusion model pT into a fast causal generator Gθ through score-based distribution matching, but quality can degrade at one step.The student generates ˆxt = Gθ(zt, xctx,t, c) from Gaussian noise, visual context, and conditioning; DMD is stable for few-step distillation.
  • Adversarial distillation: Adversarial distillation jointly trains a causal generator and discriminator, whose design may be causal or bidirectional during training.The generator must remain strictly causal, while the discriminator distinguishes real from generated frames and may access future frames.

4. Asymmetric Adversarial Distillation

AAD-1 trains a one-step autoregressive generator through ODE initialization, self-rollout distribution-matching warmup, and asymmetric adversarial refinement. The causal generator is paired with a bidirectional full-video discriminator to improve temporal consistency and stabilize one-step training.

  • Training pipeline: The three-stage pipeline performs ODE initialization, one-step DMD warmup under self-rollout, and asymmetric adversarial refinement.The stages progressively initialize the causal student, align its distribution with the teacher, and refine visual quality and temporal coherence.
  • Stage I: ODE initialization: Stage I adapts pretrained bidirectional video models into causal generators and regresses teacher denoising trajectories at downstream few-step timesteps.Block-wise causal attention provides stable initialization for subsequent one-step distillation.
  • Stage II: distribution matching warmup: Stage II uses self-rollout distribution matching with real and fake score models to align the student autoregressive distribution with the teacher and bridge the train-test gap.The generated full clip is perturbed to a random noise level, while the fake-score model estimates the generated distribution for generator updates.
  • Stage III: asymmetric adversarial refinement: Stage III keeps the generator strictly causal while training a bidirectional discriminator on full noisy clips with a video-level scalar realism score.The discriminator aggregates full spatiotemporal context and provides temporal consistency signals sensitive to long-horizon drift.
  • Rationale for staged training design: Staged training is necessary because directly training the causal-generator and bidirectional-discriminator setup is empirically unstable in the one-step regime.ODE and DMD first move the student toward the teacher distribution, allowing adversarial refinement to focus on visual quality and temporal coherence.
  • Long-video generation mechanisms: Sink tokens, sliding-window attention, and Relative RoPE support stable infinite streaming by preserving global identity, recent motion context, and positional extrapolation.The implementation uses sink tokens alongside a local window and maintains relative query-key distances within the training distribution.

5. Experiments

Experiments show that AAD-1 delivers strong one-step video quality and stability, while ablations identify DMD warmup, bidirectional video-wise discrimination, and appropriate regularization as key to preventing degradation, drift, and collapse.

  • Benchmark comparison: Stage-III achieves the best autoregressive subject consistency (94.34), background consistency (95.08), and I2V subject faithfulness (98.65) with one-step generation.It also reaches 97.83 on I2V background faithfulness and 71.49 on imaging quality.
  • Benchmark comparison: Stage-II produces stronger motion magnitude, reaching Dynamic Degree 50.30, whereas Stage-III provides better consistency and faithfulness overall.The comparison reveals a trade-off between motion strength and fidelity.
  • Benchmark comparison: User studies prefer AAD-1 over Self Forcing and CausVid for motion realism and image quality.Qualitative comparisons show vivid motion, visual fidelity, and identity consistency over horizons up to 320 frames, while baselines exhibit identity drift.
  • DMD warmup ablation: Removing DMD warmup makes adversarial refinement unstable and causes severe visual degradation, while warmup preserves scene structure and object appearance.Warmup initializes the generator from a substantially better one-step solution before adversarial refinement.
  • Discriminator ablation: Bidirectional backbones provide stable supervision, with video-wise logits achieving better drift mitigation than frame-wise logits, 4.02 versus 4.38.Causal frame-wise discrimination produces static videos, while causal video-wise and bidirectional frame-wise settings remain prone to drift.
  • Causal teacher analysis: A converged full-step causal teacher still accumulates severe long-horizon error, causing geometric distortion and identity loss that can provide flawed discriminator supervision.The drifting trajectory remains high-likelihood under the teacher itself, motivating asymmetric adversarial distillation.
  • Regularization analysis: Setting λ = 0 causes rapid training collapse, whereas λ = 50 introduces visible grid-like artifacts from over-regularization.The regularization coefficient therefore critically affects training stability and frame quality.

6. Conclusion

AAD-1 is an asymmetric adversarial distillation framework for one-step autoregressive video generation that addresses motion collapse and training instability through architectural asymmetry and phased training. On VBench, it achieves state-of-the-art performance with superior visual quality and motion fidelity.

  • Conclusion: AAD-1 combines a bidirectional discriminator with video-level holistic discrimination and a phased training strategy using distribution matching warm-up.These designs target motion collapse and training instability in one-step autoregressive video generation.
  • Conclusion: Extensive VBench experiments show that AAD-1 achieves state-of-the-art performance with superior visual quality and motion fidelity.

Limitations

Despite strong chunk-wise one-step autoregressive generation, AAD-1 remains limited in fast motion, complex structures, and long-horizon extrapolation. These limitations arise from single-pass motion prediction, synthesizing multiple latent frames per chunk, and training adversarial refinement on 5-second clips.

  • Fast motion: Fast motion can produce blurry frames, distorted structures, or degraded temporal coherence when large inter-frame motion must be predicted in a single denoising pass.The one-step setting cannot refine fast-moving scenes across multiple sampling steps.
  • Complex structures: Chunk-wise one-step generation makes preserving fine-grained details and subtle local dynamics more challenging for complex content such as human faces and hands.Unlike APT2-style one-step-per-image generation, the method synthesizes multiple latent frames within each chunk in one forward pass.
  • Long-horizon extrapolation: Adversarial refinement is trained on 5-second clips because high-quality long-video training data remains scarce and expensive to curate.This training horizon is constrained by data and compute limitations.
  • Long-horizon extrapolation: Long rollouts beyond the training horizon may exhibit drift and quality degradation as errors accumulate over autoregressive chunks.The model can extrapolate beyond 5 seconds, but extended generation remains challenging.

Impact Statement · A. Theoretical Analysis of Ablation Settings

The paper targets efficient video generation by enabling faster autoregressive synthesis while acknowledging generative-model dual-use risks. Its theoretical analysis defines video, rollout, data-distribution, and KL-divergence notation.

  • Impact Statement: The work aims to advance machine learning through efficient video generation.
  • Impact Statement: The method enables faster autoregressive video synthesis.
  • Impact Statement: Generative-model dual-use concerns include creating misleading or harmful content.
  • Impact Statement: The paper encourages detection mechanisms and responsible deployment practices alongside the technology.
  • A. Theoretical Analysis of Ablation Settings: The notation defines x1:T as a video clip conditioned on context c, with p(x1:T) denoting the data distribution.
  • A. Theoretical Analysis of Ablation Settings: The causal generator’s rollout distribution is denoted q(x1:T), and x<t denotes the preceding frames x1:t−1.
  • A. Theoretical Analysis of Ablation Settings: For distributions P and Q with densities p and q, KL(P∥Q) is defined as Ex∼P[log(p(x)/q(x))].

A.1. On-Policy Error Accumulation in Causal Rollouts · A.2. Analysis of backbone visibility · A.3. Analysis of logit granularity

The appendix formalizes how on-policy one-step errors accumulate across causal rollouts and explains why bidirectional, video-wise discrimination provides stronger temporal supervision. It also shows that video-wise heads can reproduce frame-wise behavior while representing a strictly broader function class.

  • A.1. On-Policy Error Accumulation in Causal Rollouts: If the expected on-policy conditional KL divergence is at most ε at each step, the joint sequence KL divergence is bounded by Tε.This establishes linear accumulation of one-step error with sequence length.
  • A.1. On-Policy Error Accumulation in Causal Rollouts: Controlling one-step error ε on the generator’s own induced distribution is sufficient to bound sequence-level drift linearly in T.Stage III self-rollout training explicitly targets this on-policy minimization.
  • A.2. Analysis of backbone visibility: With a causal backbone, the discriminator representation H_t depends only on x≤t, so its logit has zero dependence on future frames x>t.Formally, ∂s_t/∂x>t = 0.
  • A.2. Analysis of backbone visibility: With a bidirectional backbone, H_t depends on the full sequence, so future-frame perturbations generally change s_t through nonzero gradients.This enables the discriminator to penalize step t when it conflicts with the ground-truth future during offline training.
  • A.2. Analysis of backbone visibility: A causal backbone with a video-wise head provides only late fusion, whereas a bidirectional backbone provides early fusion by incorporating future context directly into H_t.Pooling can make the final score depend on all frames without making each causal feature future-aware.
  • A.3. Analysis of logit granularity: Video-wise heads implement a strictly broader class of functions than frame-wise heads because they query H_1:T rather than only H_t.The frame-wise computation remains available as a special case of the video-wise architecture.
  • A.3. Analysis of logit granularity: A block-diagonal attention mask lets a video-wise head attend only to tokens from the corresponding frame, exactly recovering frame-wise computation.Setting cross-frame mask entries to −∞ causes softmax normalization over single-frame tokens.

B. Additional Quantitative Results

Additional experiments evaluate AAD-1 beyond the standard 1-NFE, 480p, 5-second setting, including drift measurement, 2-NFE inference, 20-second rollouts, and zero-shot 720p generation. These evaluations examine temporal stability, motion, visual quality, and image-to-video condition preservation under varied inference settings.

  • Drift score: Drift Score measures long-horizon visual drift as the standard deviation of imaging-quality scores across temporal segments, averaged over videos.Lower Drift Score indicates more stable visual quality over time.
  • Additional evaluation settings: Additional VBench-I2V evaluations cover a 2-NFE variant, 20-second rollouts, and zero-shot 720p generation beyond the standard 1-NFE, 480p, 5-second setting.These results complement the main paper’s evaluation settings.
  • 2-NFE variant: The 2-NFE variant retains AAD-1’s three-stage pipeline: ODE initialization, DMD warmup, and asymmetric adversarial refinement.It serves as an inference-budget reference.
  • 2-NFE variant: For adversarial stabilization, the 2-NFE variant adds timestep-dependent Gaussian noise to discriminator inputs, following Self Forcing.Noise is sampled from the associated timestep interval for generated rollouts corresponding to each generator output timestep.
  • Zero-shot settings: The 20-second and 720p evaluations use the standard AAD-1 model zero-shot, without additional training on longer videos or higher-resolution data.They illustrate the effects of inference settings on temporal consistency, motion dynamics, visual quality, and condition preservation.

C. Training Cost and Memory · D. Inference Efficiency

The paper details substantial but optimized training costs and evaluates inference efficiency across matched model scales. One-step inference reduces latency and improves throughput relative to four-step sampling, while bidirectional discrimination increases memory use but enables efficient full-sequence attention.

  • C. Training Cost and Memory: 3.5 days of full training are required on 64 NVIDIA H20 GPUs across three stages.Stage I takes about 0.5 day, Stage II 1 day, and Stage III 2 days.
  • C. Training Cost and Memory: Ulysses-style context parallelism with context parallel size 8 and PyTorch activation checkpointing reduces memory usage.
  • C. Training Cost and Memory: Additional VBench-I2V results compare Wan 2.1 I2V sampled with 100 NFE against AAD-1 variants under different inference settings.A bidirectional reference is included in the evaluation.
  • C. Training Cost and Memory: The causal-discriminator adversarial-training baseline uses 830 GB of memory and requires 65 hours of training.
  • C. Training Cost and Memory: The bidirectional discriminator costs more memory because it processes the full sequence jointly, but it uses FlashAttention-3 for efficient full-sequence attention.The causal discriminator instead relies on FlexAttention for causal masking.
  • D. Inference Efficiency: Inference latency and throughput are measured on a single H100 GPU using the Self-Forcing protocol at matched parameter scales.
  • D. Inference Efficiency: Reducing the sampling budget from 4 NFE to 1 NFE consistently lowers latency and improves throughput within each model scale.
Loading 2606.03972v2…