Source-linked AI summary
Diffusion Adversarial Post-Training for One-Step Video Generation
Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, Lu Jiang
TL;DR
One-step diffusion generation remains difficult because iterative denoising is important for quality, particularly in high-resolution video. The paper proposes Adversarial Post-Training, fine-tuning a pretrained diffusion model against real data with stability-oriented design changes. Seaweed-APT generates two-second 1280×720 24fps videos in real time with one forward step and produces 1024px images at comparable state-of-the-art quality.
Problem
One-step generation remains challenging because removing iterative denoising can harm fine details, artifacts, and structural integrity, while high-resolution video progress is limited.
Method
APT fine-tunes a pretrained diffusion model with a generative adversarial objective against real data, using architectural and approximated R1 regularization changes for stability.
Results
A ∼16B model generates two-second 1280×720 24fps video latents in two seconds on one H100 with one forward step, and the full pipeline runs in real time on 8×H100 GPUs.
Takeaways & Limitations
APT provides a one-step generator for both images and videos, with high-resolution video generation and 1024px image quality comparable to state-of-the-art methods.
Takeaways & Limitations
Training produced videos up to two seconds, and one-step generation still has degradation in structural integrity and text alignment.
Abstract
from arXiv · showhide
The diffusion models are widely used for image and video generation, but their iterative generation process is slow and expansive. While existing distillation approaches have demonstrated the potential for one-step generation in the image domain, they still suffer from significant quality degradation. In this work, we propose Adversarial Post-Training (APT) against real data following diffusion pre-training for one-step video generation. To improve the training stability and quality, we introduce several improvements to the model architecture and training procedures, along with an approximated R1 regularization objective. Empirically, our experiments show that our adversarial post-trained model, Seaweed-APT, can generate 2-second, 1280x720, 24fps videos in real time using a single forward evaluation step. Additionally, our model is capable of generating 1024px images in a single step, achieving quality comparable to state-of-the-art methods.
1. Introduction
The paper addresses the difficulty of high-quality one-step generation, especially for video, by adversarially post-training a pre-trained diffusion transformer on real data. APT avoids teacher-sample precomputation and enables high-resolution single-step image and video generation.
- Motivation: Diffusion step distillation reduces inference cost by training a student from a pretrained diffusion teacher that produces multistep targets.The paper frames this as the prevailing route for reducing diffusion generation cost.
- Motivation: One-step generation remains challenging because iterative denoising is fundamental to diffusion models, causing detail, artifact, and structural-integrity problems when removed.
- Motivation: Existing video distillation studies largely target small, low-resolution models, while a concurrent large-scale effort generates 640×352 12fps videos but still requires four diffusion steps.
- Approach: APT continues training a pretrained DiT with an adversarial objective against real data, using diffusion pretraining for initialization rather than teacher-generated targets.
- Benefits: APT eliminates precomputed video-teacher samples and can surpass the teacher on realism, exposure, and fine-detail criteria.
- Training: The method stabilizes large-transformer adversarial training through deterministic-distillation initialization, discriminator changes, timestep ensembling, and approximated R1 regularization.
- Results: A ∼16B model generates a two-second 1280×720 24fps video latent in two seconds on one H100 using one forward step, while the full pipeline runs in real time on 8×H100 GPUs.
2. Related Works
Related work accelerates diffusion through deterministic or distributional distillation, but one-step quality and high-resolution video remain limited. The paper also motivates an approximated R1 objective because standard R1 requires unsupported higher-order gradients in large transformer systems.
- Accelerating Diffusion Models: Diffusion step distillation is a common approach for reducing diffusion inference cost, using deterministic or distributional methods.
- Accelerating Diffusion Models: Deterministic methods predict exact teacher outputs with fewer steps, but can produce blurry samples and often require eight steps for desirable large-scale text-to-image generation.
- Accelerating Diffusion Models: Distributional methods approximate the teacher distribution, yet existing approaches often show severe one-step artifacts and still require multiple steps for desirable results.
- One-Step Video Generation: Prior one-step video methods include domain-restricted GANs, low-resolution 16-frame distillation models, and a concurrent 640×352 12fps method requiring four steps.
- Stable Adversarial Training: R1 regularization supports GAN convergence, but its higher-order gradients are computationally expensive and unsupported by several large-model training components.
3. Method
APT converts a pre-trained diffusion model into a one-step generator by adversarially training against real data, with architectural and optimization changes aimed at stable, high-quality generation.
- Adversarial Post-Training: APT fine-tunes a pre-trained text-to-video diffusion model using a generative adversarial objective against real data.The discriminator distinguishes real samples from generated samples while the generator learns to fool it.
- Architecture: The model uses diffusion-transformer backbones for both generator and discriminator, with the discriminator producing a scalar logit through additional output heads.The backbone is an MMDiT model trained in latent space over images and videos, comprising 36 transformer layers and 8 billion parameters.
- Generator Initialization: Consistency distillation initializes the generator before adversarial training because direct adversarial training on the diffusion model leads to collapse.The distilled model predicts a velocity field and provides the subsequent generator initialization despite producing blurry samples.
- Discriminator: The discriminator is stabilized with full-parameter diffusion initialization, multi-layer logit features, raw samples, and an ensemble of shifted timestep inputs.The discriminator adds cross-attention-only blocks at layers 16, 26, and 36; timestep shifting uses s = 1 for images and s = 12 for videos.
- Regularization: An approximated R1 objective perturbs real data with Gaussian noise to reduce discriminator gradients without higher-order backpropagation.The approximation addresses software and computational constraints that prevent standard R1 training in large-scale transformer models; experiments use λ = 100, σ = 0.01 for images, and σ = 0.1 for videos.
- Training Details: Training proceeds from 1024px images to 2-second 1280×720 24fps videos, using image initialization for the video generator and diffusion-weight initialization for the video discriminator.The video stage uses 1024 H100 GPUs, gradient accumulation to batch size 2048, a 3e−6 learning rate, and 300 updates.
4. Experimental Results
The experiments evaluate APT through qualitative comparisons and user studies for one-step image and video generation. APT improves visual fidelity and realism, while retaining weaknesses in structural integrity and text alignment, especially for video.
- Evaluation Protocol: APT is evaluated qualitatively and through user studies covering visual fidelity, structural integrity, and text alignment.Visual fidelity includes texture, details, color, exposure, and realism; structural integrity concerns object and body-part correctness.
- Image Generation: APT produces more realistic image tones than the original diffusion model, which can generate over-exposed images under classifier-free guidance.Qualitative comparisons report advantages in preserving details and structural integrity relative to baseline methods.
- Video Generation: For video, one- and two-step APT improve details and realism but remain worse than 25-step diffusion in structural integrity and text alignment.Despite these degradations, one-step videos retain decent quality at 1280×720 resolution; the structural degradation is more severe for video than for images because motion is involved.
- Evaluation Protocol: User studies compare generated samples pairwise, with raters selecting preferences or no preference for each criterion.The preference score is calculated as (G − B)/(G + S + B), where G, B, and S count preferred, non-preferred, and similar samples.
- Image Generation: All existing one-step image methods degrade across the three criteria relative to their corresponding 25-step diffusion models, while APT has the least structural degradation among almost all methods.APT is weaker in text alignment but is the only method reported to improve visual fidelity, consistent with better details and realism.
- Image Generation: APT achieves performance comparable to the state of the art for one-step image generation, ranking second in absolute preference and first in relative preference.Relative preference adjusts for differences in the base diffusion models used for comparison.
5. Ablation Study and Discussion
The ablations identify approximated R1 regularization and discriminator design as important for stable training, image quality, and structural correctness, while APT also addresses exposure issues but retains mode-switching limitations.
- Training Stability: Without approximated R1 regularization, discriminator loss reaches zero and training collapses; regularization prevents this behavior.The unregularized discriminator loss quickly approaches zero, while the regularized curve does not.
- Discriminator Design: A deeper discriminator that includes the full pre-trained network improves image quality.
- Discriminator Design: Using multilayer features improves structural correctness.
- Discussion: APT mainly resolves the over-exposure and over-saturating tone issue resulting from CFG.The authors also report that APT helps the generator learn distributions closer to real data.
- Discussion: Structural incorrectness often occurs during smooth transitions between modes, and the authors hypothesize that limited generator capacity contributes to this degradation.The one-step generator has limited capacity to make sharp switches between modes.
- Discussion: APT learns from the real distribution without CFG’s boosting effect, so text alignment remains acceptable but weaker than CFG.The dataset was re-captioned, and another identified issue also negatively affects text alignment.
6. Conclusion and Limitations
The paper proposes APT for single-step image and video generation, with architectural enhancements and approximate R1 regularization supporting training stability. Its high-resolution video proof of concept remains limited to two seconds and shows degradation in structural integrity and text alignment.
- Conclusion: APT enables single-step generation of both images and videos using architectural enhancements and approximate R1 regularization.The paper identifies these components as crucial for training stability.
- Limitations: The high-resolution video proof of concept is limited to videos of up to two seconds because of computational constraints.
- Limitations: One-step generation still degrades structural integrity and text alignment, which the authors identify as targets for future work.
A. Image Quantitative Metrics
The paper evaluates image generation with FID, PFID, and CLIP score, but reports that these metrics can substantially disagree with human judgments and actual observed model quality.
- Evaluation: FID, PFID, and CLIP score are computed on COCO5K and COCO-10K image-generation evaluations.Both dataset variants are reported because prior works use different COCO subsets.
- Metric Reliability: The reported metrics can be far off from actual model performance, contradicting human evaluations and qualitative observations.Examples include FLUX 1-step surpassing FLUX 25-step in all metrics despite visible quality degradation, and SDXL 25-step exceeding FLUX 25-step contrary to human evaluation.
- Metric Reliability: The metric mismatch persists even within the same model architecture family, where metrics wrongly identify FLUX 1-step as better than FLUX 25-step.
- Metric Reliability: The authors therefore primarily rely on human evaluations and caution that these metrics require careful interpretation.They leave exploration of better metrics to future work.
B. Video Quantitative Metrics
VBench results show that APT outperforms consistency-distillation baselines and achieves a total score comparable to the 50NFE base model despite using one forward evaluation.
- VBench Evaluation: APT significantly outperforms the consistency-distillation 1NFE, 2NFE, and 4NFE baselines on VBench.The authors report that the 1NFE baseline produces blurry videos whose issues persist even at 4NFE.
- VBench Evaluation: 82.00 vs. 82.15: APT 1NFE achieves a comparable VBench total score to the base model at 50NFE.The paper explicitly notes that this is an unfair comparison.
C. Inference Speed
The model generates 2-second, 1280×720, 24fps videos in 6.03 seconds on one H100 GPU and reaches real-time performance with eight H100 GPUs.
- 6.03s on a single H100 GPU generates a 2-second, 1280×720, 24fps video.
- 8 H100 GPUs enable real-time generation for the same one-step video task.
D. The Effect of Training Iterations and EMA
The model adapts rapidly during training, with EMA improving quality and excessive training causing structural degradation.
- 50 updates are sufficient for the non-EMA model to generate sharp images.
- The EMA model generally performs better than the non-EMA model.
- 350 updates produce peak quality for the EMA model, while longer training causes more structural degradation.
E. The Effect of the Batch Size
Larger batch sizes improve training stability and structural integrity, while small video batches can cause mode collapse; the model’s layers progress from coarse structure to fine detail.
- The Effect of the Batch Size: 1024 video samples per batch avoid the mode collapse observed with a batch size of 256.Final training uses batch sizes of 9062 for images and 2048 for videos.
- The Effect of the Batch Size: Training the entire discriminator is preferable to freezing its backbone or lowering the backbone learning rate.Freezing produces undesirable artifacts, while lowering the learning rate causes slow convergence.
- The Effect of the Batch Size: Shallow transformer layers generate coarse structure, whereas deeper layers generate high-frequency details.This compresses a progression resembling iterative diffusion generation into the one-step model.
- The Effect of the Batch Size: Adversarial training is hypothesized to help the generator learn distributions closer to real training data through an in-the-loop perceptual critic.
- The Effect of the Batch Size: High-dimensional noise interpolation produces smoother transitions between modes than diffusion models, but structural incorrectness often occurs during these transitions.