PaperlayerView exact revision on arXiv

Source-linked AI summary

Parallel Decoding Distillation for Fast Image and Video Generation

Neta Shaul, Chao Liu, Arash Vahdat, Julius Berner

arXiv:2607.26004v1cs.CVcs.LG

TL;DR

Iterative sampling makes diffusion and flow-based media generation costly, motivating few-step distillation. PDD predicts multiple denoising-step velocities per evaluation and achieves SOTA few-step performance while better preserving video diversity and motion.

  • Problem

    Iterative sampling in diffusion and flow models requires hundreds of network evaluations, creating major generation-cost and latency bottlenecks.

  • Method

    PDD trains a trajectory-based parallel decoder to predict mean velocities across multiple denoising intervals in one network evaluation.

  • Results

    PDD achieves SOTA video quality with 4 NFE and SOTA image-generation scores with 4 to 8 NFE across the reported benchmarks.

  • Takeaways & Limitations

    PDD supports high-quality few-step image, video, and audio generation while better preserving sample diversity and motion than distribution-based baselines.

  • Takeaways & Limitations

    Large-scale text-to-image and video experiments use data-free training, so data-dependent settings beyond ImageNet-256 remain future work.

Abstract

from arXiv · show

Generation in video diffusion or flow models is computationally expensive due to the slow and iterative sampling process. Current state-of-the-art (SOTA) acceleration methods heavily rely on variational score distillation (VSD) and adversarial losses to distill diffusion models into few-step generators. Albeit achieving high-quality video generation, these training losses are notoriously hard to optimize and suffer from mode collapse, leading to loss of video diversity and lack of motion. In this paper, we introduce Parallel Decoding Distillation (PDD), a simplified and scalable trajectory-based distillation method for fast inference of diffusion and flow matching models. Our architecture and training procedure are compatible with any pre-trained model and support sampling with a varying number of function evaluations (NFE). PDD accelerates generation by predicting multiple denoising steps per network evaluation. Conceptually, it learns a representation of the mean velocity without regressing its derivative using JVPs or finite-difference approximations. Our method achieves SOTA performance with 4-8 NFE on LTX-2.3 Text-to-Video/Audio, Wan 14B Text-to-Video, and Qwen-Image Text-to-Image. Moreover, PDD presents a significant improvement in generated video diversity.

1. Introduction

PDD is a scalable, trajectory-based distillation method that accelerates diffusion and flow-model inference by predicting multiple denoising steps per network evaluation. It uses a simplified regression-based objective and compatible architecture supporting variable NFE, while achieving strong quality and diversity across image and video generation tasks.

  • Contribution: PDD is formulated as a trajectory-based method for fast inference of flow matching and diffusion models and is supported by any pre-trained model.The approach is designed as a scalable alternative to distribution-based distillation.
  • Method: PDD predicts multiple denoising steps in a single network evaluation instead of merging them into one larger step.The method discretizes the flow trajectory into N intervals grouped into blocks of size L and predicts mean velocities for all intervals within each block.
  • Method: PDD supports variable NFE at inference and avoids JVPs, finite differences, additional policy heads, and multi-stage training.Its single regression-based objective produces high-quality and diverse samples without VSD or GAN losses.
  • Results: 4 NFE achieves SOTA video quality on VBench for Wan2.1 1.3B and 14B while preserving better diversity than distribution-based baselines.The evaluation covers text-to-video generation with Wan2.1 models.

2. Generative Flow Models

Generative flow models, including flow matching and diffusion models, are treated as deterministic flows defined by velocity fields and source distributions. Sampling solves the resulting ODE numerically, with Euler and higher-order Runge–Kutta methods approximating interval-wise mean velocities.

  • Flow Matching and Diffusion Models: Flow matching and diffusion models are treated as deterministic flows for this section.These are identified as the dominant approaches for training generative flow models.
  • Flow Matching and Diffusion Models: A flow is defined by a velocity field and source distribution, whose time-dependent marginals form the probability path.The source distribution sets the marginal at t = 0, while X_t follows the marginal p_t.
  • Flow Matching and Diffusion Models: Flow matching learns a velocity model that maps samples from source distribution p_0 to target distribution p_1.Training assumes i.i.d. target samples and an easy-to-sample source distribution.
  • Sampling with flows: Sampling from a trained flow solves its ODE after discretizing [0, 1] into N intervals, each characterized by a mean velocity.The trajectory is approximated sequentially across the intervals.
  • Sampling with flows: Euler assumes constant velocity within each interval, whereas Runge–Kutta methods use additional velocity evaluations for higher-order approximations.The section specifically uses the Midpoint method among Runge–Kutta solvers.

3. Parallel Decoding Distillation

Parallel Decoding Distillation trains a decoder to predict multiple consecutive mean velocities in one network evaluation, enabling block-wise sampling instead of one interval at a time. Its tractable on-policy objective, flexible architecture, and fused inference layers support trajectory-faithful, variable-block-size acceleration.

  • Parallel decoder: PDD predicts the mean velocities of multiple integration intervals in a single network evaluation, rather than approximating them sequentially.For a block of size L, the decoder outputs one mean velocity for each time step in the block.
  • Sampling: At inference, the decoder advances L intervals simultaneously through a recursive block-step rule, repeated N/L times.This replaces classical sampling, which advances one interval at each ODE step.
  • Training: The PD loss regresses student mean velocities against Runge-Kutta approximations of the pretrained teacher’s mean velocity using on-policy student states.A single Euler or Midpoint step requires 1-2 teacher evaluations, making the loss tractable at large scales.
  • Training: The PD-loss minimizer satisfies the parallel decoder condition and, up to controllable Runge-Kutta approximation error, samples exactly the teacher trajectories.Proposition 1 establishes that the learned parallel decoder recovers the target condition.
  • Architecture and variable block size: Repeating the pretrained backbone’s final linear layer N times enables one model to predict any block size without a second time coordinate.The design also permits initialization from the pretrained model’s final layer.
  • Architecture and variable block size: Mean-velocity representations use learnable linear maps instead of JVPs or finite differences, while inference can retain only one fused linear layer per block.This avoids the extra compute of an enlarged final layer during inference.

4. Related works

Related work spans trajectory-based, distribution-based, and from-scratch distillation approaches for few-step generation. The paper emphasizes their limitations on large video models and contrasts PDD’s simpler, more robust training with these alternatives.

  • Trajectory-based distillation: Trajectory-based methods include Progressive Distillation, consistency models, and approaches that distill mean velocity between trajectory states.Progressive Distillation uses multi-phase student-teacher training, while consistency models map intermediate states directly to the clean state.
  • Trajectory-based distillation: These trajectory-based methods struggle with high-quality few-step generation on large video models, while JVPs and finite differences are expensive or unstable.The limitation concerns both generation quality and training or evaluation dynamics at large scale.
  • Trajectory-based distillation: Pi-Flow delegates integration over a trajectory interval to a learnable policy head, whereas PDD uses the observation to motivate parallel prediction and simplify training and inference.The supplied passage also contrasts their emphasis on expressive parametrizations, but the excerpt is truncated after that comparison.
  • Training from scratch: From-scratch methods jointly use flow matching and trajectory-based distillation to learn flow maps without separate pre-training and distillation phases.They use the current model state as the teacher rather than a separate student-teacher scheme.
  • Training from scratch: Self-distillation could extend PDD, but it would require a different pipeline with larger datasets and significantly more compute.The passage presents this as a potential extension rather than the method used in PDD.
  • Distribution-based distillation: Distribution-based methods align marginals using GAN, VSD, or generalized f-divergence losses, but they are sensitive to hyperparameters and alternating objectives.PDD’s memory-efficient PD loss is reported as more robust to hyperparameter choices and more consistent across training iterations.

5. Experiments

PDD is evaluated on class-conditional image, text-to-image, and text-to-video generation across ImageNet-256, Qwen-Image, Wan2.1, and LTX-2.3. Across these experiments, midpoint approximation and multi-NFE inference support strong performance, including high-quality LTX-2.3 generation at 8 NFE without training data.

  • Experimental Tasks: PDD is evaluated on ImageNet-256, Qwen-Image, Wan2.1 1.3B and 14B, and LTX-2.3 across class-conditional image, text-to-image, and text-to-video generation.The text-to-image evaluation uses OneIG, DPG-Bench, and GenEval with NFE = 2, 4, 8.
  • Training Setup: PDD training varies time discretization, Runge-Kutta mean-velocity approximation, and block-size ranges to determine available inference NFEs.The method trains two models per task with different grid sizes, Runge-Kutta methods, and block-size ranges.
  • Training Findings: Larger batch sizes and midpoint approximation improve performance, with midpoint consistently outperforming Euler across the reported tables.The two most important training hyperparameters are time reparameterization and batch size; convergence is otherwise stable across hyperparameters.
  • Class-conditional ImageNet: ImageNet-256 FID generally improves as NFE increases, demonstrating that PDD shares weights across different inference budgets.The 8-NFE setting shows increased FID, which can be mitigated with a lower guidance scale at the cost of higher FID at lower NFEs.
  • Class-conditional ImageNet: PDD achieves competitive single-step ImageNet FID with a simpler objective than Pi-Flow and FreeFlow while supporting multiple NFE budgets.PDD avoids Pi-Flow’s Gaussian mixture and FreeFlow’s additional network.
  • Text-to-video LTX-2.3: 8 NFE produces LTX-2.3 generations on par with or better than the official distilled model after as few as 250 training iterations, without training data.The teacher uses 4×30 NFE per generation, whereas the comparison evaluates PDD and the official distilled model at 8 NFE.

6. Conclusion · A. Algorithms

PDD is a trajectory-based distillation method that predicts mean velocities for multiple consecutive denoising intervals in one evaluation, avoiding several difficult training components. It delivers strong few-step generation while preserving diversity and motion, with data-dependent training and adaptive block sizing identified as future directions.

  • 6. Conclusion: PDD accelerates diffusion and flow models by predicting mean velocities over multiple consecutive intervals in one parallel-decoder evaluation.This avoids collapsing all denoising steps into one large update.
  • 6. Conclusion: PDD uses a practical objective that avoids VSD, adversarial losses, JVPs, finite differences, and mult…
  • 6. Conclusion: PDD achieves strong few-step text-to-image, text-to-video, and text-to-audio performance on Qwen-Image, Wan2.1, and LTX-2.3.
  • 6. Conclusion: PDD better preserves sample diversity and motion than distribution-based baselines and is presented as the first pure trajectory-based method for few-step, high-resolution video generation.
  • 6. Conclusion: Data-free large-scale text-to-image and video experiments leave data-dependent settings beyond ImageNet-256 for future work.
  • 6. Conclusion: PDD permits inference-time block-size selection after evaluating the parallel decoder, motivating adaptive skipping based on a verifier or confidence criterion.
  • A. Algorithms: Algorithm 3 defines data-free parallel decoding loss using a student parallel decoder, a pre-trained flow-model teacher, Runge–Kutta mean-velocity approximation, and variable block sizes L_min through L_max.
  • A. Algorithms: The algorithm samples a carried state and time index, predicts parallel velocities, advances to a sampled block endpoint, matches the student velocity to the detached teacher estimate with MSE, then advances by L_min.

B. Experiments

The experiments use model-specific datasets and native VAEs, including data-free PDD training for Qwen-Image, Wan2.1, and LTX-2.3. PDD retains each teacher’s backbone, expands the final layer for parallel steps, and incorporates guidance without additional network evaluations.

  • Datasets: PDD training uses a distinct dataset and VAE for each model, with data-free training for Qwen-Image, Wan2.1, and LTX-2.3.Qwen-Image uses Pi-Flow’s text-prompt set at 1024 × 1024 with the native Qwen-Image VAE; Wan2.1 and LTX-2.3 use their native VAEs.
  • Architecture: All PDD models use the teacher’s exact backbone and repeat the final linear layer’s channel dimension N times to initialize parallel steps.The repeated layer is constructed from correctly reshaped pretrained weights so it is equivalent to initializing every parallel step with the teacher’s final layer.
  • Classifier free guidance (CFG): PDD replaces the teacher velocity with classifier-free-guided velocity and treats LTX-2.3’s cross-modal and spatiotemporal skip guidance analogously.These guidance methods are incorporated within the PDD formulation.
  • Classifier free guidance (CFG): PDD guidance alleviates the need for additional network evaluations required by different guidance methods.The result applies to the guidance procedures described for classifier-free, cross-modal, and spatiotemporal skip guidance.

B.1. ImageNet

On ImageNet, PDD is trained with a fixed AdamW setup and evaluated across guidance scales and NFE values. FID steadily decreases during training, while guidance-scale choice affects performance at different inference NFEs.

  • B.1. ImageNet: PDD training uses AdamW with constant learning rate 5e−5, weight decay 0, batch size 2048, EMA constant 0.99995, and 300k iterations.These settings define the ImageNet training configuration.
  • B.1. ImageNet: Three PDD models are trained for each Runge-Kutta method—Euler and Midpoint—with guidance scales w=2.7, 2.9, and 3.2.Each model uses a distinct guidance scale.
  • B.1. ImageNet: Different inference NFE values benefit from different guidance scales, while w=2.9 provides the most balanced results shown in the main paper.The guidance-scale comparison is reported for Euler and Midpoint methods.
  • B.1. ImageNet: FID is evaluated every 10K training iterations at NFE=1, 2, 4, and 8, showing a steady decrease during training.The trend is reported for PDD with Euler and Midpoint methods on Repa-ImageNet-256.

B.2. Qwen-Image · PDD

For Qwen-Image, PDD training uses a fixed AdamW setup and selects the best checkpoint by average performance across OneIG, DPG-Bench, and GenEval. Additional tables and figures report benchmark dimensions and qualitative comparisons among the teacher, PDD, and DMD2 (Lightning-v2).

  • B.2. Qwen-Image: PDD uses AdamW with a constant learning rate of 1e−5 and zero weight decay.Training uses batch size 2048 without EMA.
  • B.2. Qwen-Image: 3k iterations are trained, with OneIG, DPG-Bench, and GenEval evaluated every 250 iterations.The reported checkpoint is selected using the best average across the three benchmarks.
  • B.2. Qwen-Image: Iteration 1250 is selected for the model trained with Euler approximation.The passage identifies this checkpoint choice but does not provide its benchmark scores.
  • B.2. Qwen-Image: Figure 10 plots the overall metrics of OneIG, DPG-Bench, and GenEval against Qwen-Image PDD training iteration.The figure summarizes how the three benchmark metrics vary during training.
  • B.2. Qwen-Image: Tables 6, 7, and 8 provide all considered benchmark dimensions for OneIG-EN, DPG-Bench, and GenEval, respectively.The OneIG-EN overall score is the average of five dimensions, and asterisks denote re-evaluation of official checkpoints where indicated.
  • PDD: Figures 11 to 16 compare the Qwen-Image teacher, PDD, and DMD2 (Lightning-v2).These figures provide additional comparisons for the three model variants.
  • PDD: The teacher uses Euler sampling with 2×50 NFE, whereas PDD and DMD2 (Lightning-v2) use 4 NFE in Figures 11 to 16.This sampling configuration is repeated in the captions for each qualitative comparison figure.

B.3. Wan Text-to-Video

The Wan2.1 experiments evaluate PDD with Euler and Midpoint approximations on the 1.3B model and Midpoint alone on the 14B model. Training checkpoints are selected using periodic VBench evaluation, with additional curvature, dimension, and baseline comparisons reported for Wan2.1 14B.

  • Wan2.1 1.3B: 250 iterations without EMA produced the best overall VBench score at iteration 25 with Euler and iteration 225 with Midpoint on Wan2.1 1.3B.VBench was evaluated every 25 iterations.
  • Wan2.1 14B: 250 iterations without EMA produced the best overall VBench score at iteration 200 for the Wan2.1 14B short checkpoint trained with Midpoint.VBench was evaluated every 25 iterations.
  • Wan2.1 14B: 3.5k iterations with EMA and constant coefficient 0.99 produced the best VBench score at 3k for the Wan2.1 14B long checkpoint.VBench was evaluated every 250 iterations.
  • Additional results: PDD is compared with the Wan2.1 14B teacher through averaged trajectory curvature, with full VBench dimensions and 4 NFE comparisons against DMD2 (FastGen) and AnyFlow.Curvature is averaged over 10 trajectories and reports PDD intra-block curvature.

B.4. LTX-2.3 Text-to-Video/Audio · C. Connection to flow maps

On LTX-2.3 Text-to-Video/Audio, PDD is quantitatively compared with the official distilled model at 8 NFE using 300 paired prompt-seed comparisons, while additional figures compare both against the teacher. Relative to flow maps, PDD discretizes fixed intervals and directly regresses numerical velocity integrations rather than displacement derivatives.

  • B.4. LTX-2.3 Text-to-Video/Audio: Training uses AdamW with learning rate 1e−5, zero weight decay, batch size 2048, no EMA, 250 iterations, and shift s=10.These are the reported LTX-2.3 training details.
  • B.4. LTX-2.3 Text-to-Video/Audio: PDD, the official distilled model, and the teacher are qualitatively compared in Figures 23 to 25 for LTX-2.3 generation.The teacher uses 4×30 NFE, while PDD and the official distilled model use 8 NFE.
  • B.4. LTX-2.3 Text-to-Video/Audio: The quantitative evaluation uses Gemini 3.1 Pro Preview on 100 held-out ViMix and VidProm prompts, with three seeds per prompt and two VLM evaluations per clip.The four judge axes are prompt alignment, visual quality, motion quality, and audio quality.
  • B.4. LTX-2.3 Text-to-Video/Audio: 142 wins, 35 ties, and 123 losses summarize PDD’s averaged four-axis preference over the official distilled LTX-2.3 model, with mean scores of 2.62 versus 2.59.Both models use 8 NFE, and Gemini 3.1 Pro Preview evaluates 300 paired prompt-seed comparisons.
  • C. Connection to flow maps: Flow maps use continuous time and define mean velocity between any two flow-process times by conditioning on t and s.For a state X_t=x_t at time t∈[0,1], the formulation considers mean velocity toward a later time s>t.
  • C. Connection to flow maps: PDD and Lagrangian flow maps both use an on-policy training approximation, but PDD discretizes a fixed interval into L block intervals and regresses numerical velocity integrations.Lagrangian flow maps instead regress the derivative of displacement onto velocity.
  • C. Connection to flow maps: Integrating the flow-map expression over s produces a PDD-like objective in continuous time, although practical flow maps learn mean velocity for any subinterval [t,s]⊆[0,1].The fixed-interval equation is presented as an example before the broader practical objective.

D. Proofs

Under a realizable PD objective with zero global loss, Proposition 1 proves that parallel decoding exactly follows the teacher trajectory throughout each block. The proof establishes this by induction from the block’s base case.

  • Proposition 1: Under realizability and a zero global minimum, every state in a length-L block matches the teacher trajectory.The block starts at n ≤ N−L, with X_n ∼ p_tn and θ⋆ a global minimizer of the PD loss.
  • Proposition 1: Equation 22 implies the parallel decoding states equal the exact teacher states throughout k = n, …, n + L − 1.This equality is the parallel decoding condition along the teacher trajectory.
  • Proposition 1: The proof establishes trajectory equality by showing the base case X̄_n = X_n and propagating it inductively across the block.Substitution into the parallelized process and the exact solution justify the induction step.
Loading 2607.26004v1…