Source-linked AI summary
Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
Shanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang, Yuxi Ren, Xin Xia, Yang Zhao, Xuefeng Xiao, Lu Jiang
TL;DR
Large-scale video diffusion models are slow and expensive, while long-duration training data are scarce. AAPT transforms a pre-trained latent video diffusion model into an autoregressive generator using adversarial, student-forcing training, achieving performance comparable to leading methods with greater efficiency.
Problem
Diffusion video generation is slow and expensive, and scarce long-duration training videos limit temporal extrapolation for long-video generation.
Method
AAPT applies adversarial post-training to a pre-trained 8B latent video diffusion model, using generated long-video segments and student-forcing training for autoregressive generation.
Results
AAPT achieves performance comparable to the best methods while being significantly more efficient.
Takeaways & Limitations
Adversarial training provides a paradigm for converting video diffusion models into fast autoregressive generators suitable for real-time interactive applications.
Takeaways & Limitations
The model can struggle to maintain subjects and scenes, while segment-based discrimination cannot enforce long-range consistency and one-step generation can produce persistent defects.
Abstract
from arXiv · showhide
Existing large-scale video generation models are computationally intensive, preventing adoption in real-time and interactive applications. In this work, we propose autoregressive adversarial post-training (AAPT) to transform a pre-trained latent video diffusion model into a real-time, interactive video generator. Our model autoregressively generates a latent frame at a time using a single neural function evaluation (1NFE). The model can stream the result to the user in real time and receive interactive responses as controls to generate the next latent frame. Unlike existing approaches, our method explores adversarial training as an effective paradigm for autoregressive generation. This not only allows us to design an architecture that is more efficient for one-step generation while fully utilizing the KV cache, but also enables training the model in a student-forcing manner that proves to be effective in reducing error accumulation during long video generation. Our experiments demonstrate that our 8B model achieves real-time, 24fps, streaming video generation at 736x416 resolution on a single H100, or 1280x720 on 8xH100 up to a minute long (1440 frames). Visit our research website at https://seaweed-apt.com/2
1 Related Work
Prior video-generation systems trade quality, duration, or flexibility against the computation required for streaming and interaction. This work situates AAPT among diffusion-forcing and autoregressive approaches for real-time interactive video.
- Diffusion and streaming generation: Diffusion models provide high-quality video but remain expensive because iterative generation and repeated context processing burden real-time use.Diffusion forcing, causal attention, KV caching, and step distillation reduce inference costs, but existing systems retain duration and streaming constraints.
- Diffusion and streaming generation: Diffusion-forcing models support causal streaming with KV caching, yet are generally trained only on fixed-duration windows and can behave out of distribution when extrapolated.Naively dropping cached tokens does not remove the receptive field encoded in previously computed tokens.
- Autoregressive generation: Token-based autoregressive video generation can exploit KV caching, but token-by-token decoding limits parallelization and slows high-resolution generation.The proposed architecture instead generates all tokens for one frame in a single forward pass using an adversarial objective.
- Interactive applications: Interactive video applications require continuous next-frame prediction conditioned on user controls, including camera signals for world exploration and pose or multimodal signals for virtual humans.The paper highlights these settings as representative applications of real-time interactive generation.
- Interactive applications: Figure 1 depicts a causal generator that recycles generated frames, uses KV caching, and applies a sliding window to maintain constant speed and memory for arbitrary-length generation.The discriminator uses the same block-causal structure, with shifted conditions and frame inputs replacing noise channels after initialization from diffusion weights.
2 Method
AAPT converts a pretrained latent video diffusion transformer into a causal, one-step-per-frame generator and trains it adversarially under inference-matched conditions. Its design combines block-causal attention, KV-cache reuse, student forcing, and segmented long-video training.
- Causal architecture: AAPT transforms an 8B-parameter bidirectional diffusion transformer operating in a compressed latent space into a causal autoregressive generator.The pretrained model uses a 3D VAE and originally applies full bidirectional attention over text and video tokens.
- Causal architecture: Block-causal attention lets text attend to itself while visual tokens attend to text and visual tokens from previous and current frames.The model also receives the previously generated frame through channel concatenation, except at the first autoregressive step.
- Causal architecture: The generator produces all tokens for the next frame in one noise-sampled forward pass, requiring one frame of computation per autoregressive step instead of two for one-step diffusion forcing with KV caching.This input-recycling design is reported as more efficient than one-step diffusion forcing.
- Training procedure: Training proceeds through diffusion adaptation, consistency distillation, and adversarial training.Diffusion adaptation uses teacher forcing, while consistency distillation initializes the model before adversarial optimization.
- Training procedure: Student forcing recycles generated frames during training, matching inference behavior and mitigating error accumulation that appears under teacher-forcing adversarial training.The generator begins with the ground-truth first frame and then autoregressively uses its own outputs with KV caching.
- Training procedure: Segmented long-video training lets the generator produce long videos while the discriminator evaluates short overlapping segments, avoiding dependence on abundant long single-shot videos.Overlapping one-second segments encourage continuation, and the discriminator evaluates generated segments against real videos.
- Training procedure: Adversarial discrimination supports long-generator training without explicit ground-truth supervision for every input frame, although it increases training time.The discriminator learns to distinguish real from generated videos, allowing learning from every video sample.
- Experiments: The experiments train separate models for general image-to-video generation, pose-conditioned human generation, and camera-conditioned world exploration.These settings evaluate generation quality and controllability under distinct condition signals.
3 Evaluation
Evaluation covers short- and long-video VBench-I2V generation, interactive pose and camera control, and inference efficiency. AAPT maintains competitive quality while supporting extended streaming and faster generation.
- VBench-I2V results: AAPT improves frame quality and image-conditioning scores over the diffusion baseline for 120-frame I2V generation, while temporal quality declines slightly but remains above Wan and near Hunyuan.For 1440-frame I2V generation, AAPT achieves the best quality scores and improves conditioning over the diffusion baseline.
- Interactive applications: In camera-conditioned world exploration, AAPT reaches state-of-the-art performance on three of six metrics and closely follows CameraCtrl2 on the remaining metrics.The evaluation also includes pose-conditioned human video generation with pose accuracy, visual quality, and distributional metrics.
- Inference efficiency: AAPT is significantly faster than other streaming video-generation methods while achieving performance comparable to the state of the art.The reported inference comparison considers both throughput and latency.
4 Ablation Studies
The ablations examine throughput, long-video training, forcing strategy, and remaining limitations. Longer-duration training improves one-minute generation, while student-forcing is critical for avoiding rapid inference drift.
- Table 4 compares model latency and throughput across systems.
- Long Video Training: Training for 60 seconds significantly outperforms training for only 10 seconds on one-minute VBench-I2V generation.
- Teacher-Forcing and Student-Forcing: Teacher-forcing adversarial training fails at inference, with content drifting significantly after only a few generated frames.
- Teacher-Forcing and Student-Forcing: Student-forcing is critical for mitigating error accumulation during autoregressive generation.
- Limitations: The model can struggle to maintain subjects and scenes, while one-step defects may persist and five-minute generation produces artifacts.
5 Conclusion
The conclusion presents AAPT as a way to convert video diffusion models into efficient autoregressive generators for real-time interactive applications. It reports comparable performance with substantially greater efficiency while acknowledging unresolved limitations.
- Teacher-forcing adversarial objectives fail to generate proper content at inference.
- AAPT transforms video diffusion models into fast autoregressive generators for real-time interactive applications.
- The method achieves performance comparable to the best methods while being significantly more efficient.
A Model Architecture
The architecture uses an 8B-parameter diffusion transformer with causal attention and implementation choices aimed at efficient recurrent inference and scalable training.
- The diffusion transformer has 8B parameters and 36 transformer blocks, while the discriminator uses the same architecture.
- Block causal attention is implemented with Flash Attention 3 and supports recurrent autoregressive inference without a performance penalty.
- Temporal positional embeddings use fixed intervals to support arbitrary training and generation lengths, while spatial embeddings stretch dynamically across resolutions.
- FSDP, ZERO-2, ZERO-3, Ulysses, and gradient checkpointing distribute computation and reduce memory usage across 8 GPUs per video sample.
B Training Details
Training proceeds from diffusion adaptation through consistency distillation to adversarial student-forcing, followed by long-video extension and efficiency-focused decoder training.
- Diffusion Adaptation: Diffusion adaptation first modifies the architecture and recycled input channels before subsequent training stages.
- Diffusion Adaptation: Flow matching interpolates samples and noise with xt = (1 − t) · x0 + t · ϵ using uniformly sampled timesteps and shift factor s = 24.
- Diffusion Adaptation: The training curriculum progresses from 736×416 five-second videos to mixed 1280×720 data and then longer 15-second clips.
- Consistency Distillation: Consistency distillation creates a one-step generator and provides a better initialization for adversarial training despite producing blurry results.
- Adversarial Training: Adversarial training switches to student-forcing, recycling generated frames and using parallel per-frame discriminator logits for multi-duration discrimination.
- Long Video Training: Without long-video extension, the model generates only up to 10 seconds and drifts beyond that duration.
- Long Video Training: Long-video extension increases the maximum trained duration to 19 seconds and then 55 seconds using overlapping video segments.
- Training Infrastructure: ZERO-2 avoids repeated parameter gathering during recurrent student-forcing training, while ZERO-3 shards the discriminator and text encoder.
D Teacher-Forcing Adversarial Training
Teacher-forcing adversarial training evaluates independently generated outputs against their correct dependencies, while student forcing recycles generated frames autoregressively. However, teacher-forcing training fails to produce reasonable videos.
- Training modes: Student forcing recycles each generated frame as input for the next autoregressive step, while the discriminator evaluates results in parallel.Teacher forcing instead uses ground-truth past frames and predicts subsequent frames in parallel.
- Teacher-forcing design: In teacher-forcing mode, outputs O2, O3, and O4 are generated independently from inputs I1, I2, and I3, so the discriminator produces separate logits with causal KV-cache reuse.Because O3 depends on I2 rather than O2, the discriminator must preserve the correct dependencies when evaluating each output.
- Observed limitation: Teacher-forcing adversarial training fails to generate reasonable videos for continuous latent frame prediction.The authors attribute this to error accumulation from slight inaccuracies in continuous latent values.
E The Importance of Result Recycling
Result recycling is important for maintaining coherent motion in generated videos. Removing recycled inputs prevents large motion and makes some movements incohesive.
- Ablation result: Models trained without recycled inputs cannot generate large motion.The ablation masks recycled inputs as zero tensors while retaining the user image as the first frame.
- Ablation result: Removing result recycling makes some generated movements incohesive.The experiment keeps the architecture and training settings otherwise unchanged.
- Experimental design: The result-recycling experiment isolates the contribution of recycling by masking only the recycled inputs, except for the initial user image.This compares the same architecture and training settings with and without recycled inputs.
F I2V Evaluation
The I2V evaluation compares the model at 736×416 with other models using their default sampling configurations and trained resolutions, and additionally reports results at 1280×720. The higher-resolution setting uses a smaller attention window to fit memory.
- Evaluation setup: The main-text comparison evaluates the model at 736×416 resolution.Other models use their default sampling settings, including sampling steps and classifier-free guidance.
- High-resolution evaluation: 1280×720 evaluation metrics are additionally reported, using an attention window size of N = 15 to fit memory.The 1280×720 setting is trained and inferred with this smaller attention window.
G Camera-Conditioned World Exploration
The camera-conditioned model adapts camera representations and trains separately from the I2V model for camera-controlled generation. Evaluation measures video quality, movement, camera-motion errors, and geometric consistency against ground truth.
- Camera conditioning: Camera positions and orientations are represented relative to the previous frame rather than the first frame.This modification avoids unbounded values when camera displacement increases over time.
- Training: The camera-conditioned model is trained separately from the I2V model, with consistency distillation and adversarial training performed separately.Long-video extension training randomly samples new camera trajectories for the extended portions.
- Evaluation: Evaluation computes FVD, foreground movement strength, translational and rotational camera errors, and geometric consistency against ground-truth videos.Movement uses dense optical flow and segmentation masks, while camera errors compare estimated parameters with ground truth.
H Societal Impacts
The authors argue that real-time streaming video generation could support interactive applications, while visible generation imperfections constrain malicious use.
- Generated videos still contain identifiable imperfections, which the authors say prevents malicious use.