PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Video Generation with Pyramid Attention Broadcast

Xuanlei Zhao, Xiaolong Jin, Kai Wang, Yang You

arXiv:2408.12588v3cs.CVcs.DC

TL;DR

DiT-based video generation can improve quality over CNN-based methods, but at increased memory, computation, and inference-time costs, while training-free speedup for video DiTs remains unexplored. PAB broadcasts attention outputs across steps using ranges tailored to attention stability and adds broadcast sequence parallelism. The paper reports up to 10.5× speedup without compromising quality, with performance varying by input complexity and scene dynamics.

  • Problem

    DiT-based video generation improves quality over CNN-based methods but costs more memory, computation, and inference time, and training-free speedup methods for video DiTs remain unexplored.

    Source: §1 · p. 2

  • Method

    PAB broadcasts attention outputs to subsequent diffusion steps with ranges tailored to attention stability, and adds broadcast sequence parallelism for distributed inference.

    Source: §1 · pp. 2–3

  • Results

    Up to 10.5× speedup with negligible quality loss; PAB consistently outperforms baselines across various models.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The authors present PAB as a simple foundation for future research and practical applications in video generation.

    Source: §5 · p. 10

    Performance may vary with input complexity, especially in dynamic scenes; the fixed broadcast strategy may not suit all video types and tasks, and the work reduces attention redundancy only.

    Source: §5 · p. 10

Abstract

from arXiv · show

We present Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free approach for DiT-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style. It applies different broadcast strategies to each attention based on their variance for best efficiency. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates up to 10.5x speedup across three models compared to baselines, achieving real-time generation for up to 720p videos. We anticipate that our simple yet effective method will serve as a robust baseline and facilitate future research and application for video generation.

1 INTRODUCTION

PAB targets the costs and limited applicability of existing approaches to efficient DiT-based video generation. It uses differences in attention stability to guide broadcasting and adds a parallel inference strategy; the paper reports real-time generation with up to 10.5× acceleration without compromising quality.

1 INTRODUCTION

  • DiT-based video generation improves video quality over CNN-based methods but increases memory use, computation, and inference time.The paper identifies training-free acceleration as unexplored for video DiTs and says prior caching methods do not directly transfer across their architectures and diverse attention components.Source: §1 · p. 2
  • Attention-output differences follow a U-shaped pattern, with stability across the middle 70% of diffusion steps and different variation levels across attention types.Spatial attention varies most, temporal attention less, and cross-modal attention least.Source: §1 · p. 2
  • PAB broadcasts attention outputs to later steps using attention-specific ranges, and introduces broadcast sequence parallel for distributed inference with lower communication costs.The paper also reports that the broadcast strategy can be applied to MLP layers.Source: §1 · pp. 2–3
  • Up to 10.5× acceleration with real-time video generation and no reported quality compromise; PAB performs consistently across three open-source video DiTs.The models are Open-Sora, Open-Sora-Plan, and Latte.Source: §1 · p. 3

2 HOW TO ACHIEVE REAL-TIME VIDEO GENERATION

Video DiTs use spatial and temporal transformer blocks, with cross-attention incorporating conditioning information. The section identifies attention redundancy in the diffusion process and describes attention-specific broadcasting and broadcast sequence parallelism as ways to reduce redundant computation and communication.

2.1 PRELIMINARIES

  • Video DiTs use spatial blocks for tokens at the same time index and temporal blocks for cross-time information; cross-attention incorporates conditioning, though some models omit it from temporal blocks.The paper notes that the conditioning helps generated output align with context.Source: §2.1 · p. 3

2.2 ATTENTION REDUNDANCY IN VIDEO DITS

  • Attention consumes a larger share of computation in video DiTs than in CNN approaches, and its share grows with video size; output differences are low across about 70% of middle diffusion steps.Across attention types, variance is highest for spatial, then temporal, and lowest for cross-attention.Source: §2.2 · pp. 3–4, Figure 4 · p. 4

2.3 PYRAMID ATTENTION BROADCAST

  • PAB broadcasts entire attention-module outputs from selected steps across subsequent steps in the stable middle segment, bypassing redundant attention calculations.It assigns shorter broadcast ranges to attention types with more change and fluctuation, and applies a similar strategy to MLP modules.Source: §2.3 · pp. 4–5, Figure 5 · p. 4

2.4 BROADCAST SEQUENCE PARALLELISM

  • When temporal attention is broadcast, broadcast sequence parallelism avoids all communication compared with original sequence parallelism.The figure contrasts the original sequence-parallel approach with the proposed approach.Source: Figure 6 · p. 5

3 EXPERIMENTS

Experiments compare PAB with training-free caching baselines across three video DiTs, then examine quality, speed, broadcast choices, scaling, and generated-video appearance.

3.1 EXPERIMENTAL SETUP

  • The evaluation uses VBench, PSNR, LPIPS, and SSIM, with ∆-DiT and T-GATE as training-free caching baselines and FlashAttention enabled.Experiments run on NVIDIA H100 80GB GPUs.Source: §3.1 · p. 5

3.2 MAIN RESULTS

  • On a single GPU, PAB achieves superior quality versus two baselines with acceleration up to 1.58× and negligible quality loss, performing consistently across all three models.Table 1 reports PSNR, SSIM, and LPIPS against original-model results; FLOPs denotes floating-point operations per video generation.Source: §3.2 · p. 6, Table 1 · p. 6
  • 10.50× speedup on 8 GPUs accompanies almost-linear scaling across three models with broadcast sequence parallelism.Figure 7 compares multi-GPU speedup with single-GPU speed.Source: Figure 7 · p. 6, §3.2 · pp. 6–7

3.3 ABLATION STUDY

  • Broadcasting attention outputs preserves similar quality with better efficiency than broadcasting scores, which requires attention calculations and linear projection.The comparison is summarized in Table 3; output changes remain low despite pixel-level changes.Source: Table 3 · p. 7, §3.3 · p. 7
  • Spatial and temporal attention yield the largest computational savings; longer broadcast ranges reduce video quality, with effects varying by attention type.The component ablation reports negligible VBench impact when individual broadcasts are disabled.Source: §3.3 · p. 7, Figure 8 · p. 7

3.4 SCALING ABILITY

  • PAB maintains stable speedup as video size grows, and reaches real-time generation for 480p on 8 devices and 720p on 16.The 16-device setup splits the diffusion batch before applying sequence parallelism.Source: Figure 10 · p. 8, §3.4 · p. 8, Figure 9 · p. 8
  • Attention-related normalization and projection take more time than attention calculation itself and account for most of the method’s speedup.Figure 11 breaks down runtime for a 2-second 480p video.Source: §3.4 · pp. 8–9, Figure 11 · p. 9

3.5 VISUALIZATION

  • Visual comparisons report that PAB retains the original model’s video quality and details.Figure 12 shows randomly sampled frames using each model’s highest-quality strategy.Source: Figure 12 · p. 9, §3.5 · p. 9

4 RELATED WORK

Related work covers the shift toward transformer-based video generation and methods that accelerate diffusion through scheduling, compression, caching, and parallelism.

4.1 VIDEO GENERATION

  • Video generation has moved from GANs, VA-VAE, autoregressive transformers, and convolutional diffusion toward transformer-based diffusion models for scalability and efficiency.The cited transformer-based video models include Zheng et al. (2024), Lab & etc. (2024), and Ma et al. (2024a).Source: §4.1 · p. 10

4.2 DIFFUSION MODEL ACCELERATION

  • Diffusion acceleration includes schedulers that reduce sampling steps and compression methods such as distillation, quantization, and joint optimization, which require substantial training resources and data.The section also describes ODE and SDE solvers as scheduler approaches.Source: §4.2 · p. 10
  • Caching reuses features or attention without training, but convolution-based approaches do not transfer directly to video DiTs, and existing transformer caches miss video DiTs’ distinctive attention features.Sequence parallelism reduces latency but adds communication costs; cited communication-reduction approaches remain convolution-limited.Source: §4.2 · p. 10

5 DISCUSSION AND CONCLUSION

The paper presents PAB as a training-free efficiency approach, notes limits in dynamic scenes and scope, and outlines future work and appendix experiments.

5 DISCUSSION AND CONCLUSION

  • PAB reports up to 10.5× speedup with negligible quality loss; performance may vary with scene complexity, and future work includes broader models and MLP redundancy.The current method targets attention redundancy; fixed broadcast strategies may not suit every video type or task.Source: §5 · p. 10

Appendix

Additional Experimental Results and Findings:

A.1 MODELS

  • The appendix provides inference configurations for the three evaluated video models.Table 5 contains the configurations.Source: Table 5 · p. 15

A.2.1 ATTENTION

  • The attention appendix specifies broadcast configurations and the diffusion-timestep range, with 1000 marking the beginning and 0 the end.Source: §A.2.1 · p. 15, Table 6 · p. 16

A.2.2 MLP

  • MLP outputs also show low differences across about 70% of middle diffusion steps; skippable modules are selected empirically and configured by model.Table 7 lists the MLP broadcast configuration.Source: §A.2.2 · p. 16, Table 7 · p. 16

A.3 BASELINES GENERATION SETTINGS

  • The baseline appendix describes ∆-DiT’s offset caching and T-GATE’s phase-based attention caching, with configurations for both methods.The paper reports using ∆-DiT and T-GATE as evaluation baselines.Source: §A.3 · pp. 16–17, Table 8 · p. 16, Table 9 · p. 17

A.4 METRICS

The evaluation combines VBench, a human-perception-aligned video-generation benchmark, with frame-averaged metrics comparing accelerated videos against the original model’s outputs. The metrics cover pixel fidelity, perceptual similarity, and structural similarity.

A.4 METRICS

  • VBench evaluates video-generation quality across 16 dimensions, using the official framework’s weighted scores.The dimensions include subject and background consistency, motion smoothness, aesthetic and imaging quality, and overall consistency.Source: §A.4 · p. 17
  • The accelerated model is compared with the original model, with image-similarity metrics calculated per frame and averaged across the video.The evaluation also uses VBench to assess video-generation quality.Source: §A.4 · p. 17
  • PSNR measures pixel-level reconstruction fidelity; higher values indicate lower error, though pixel fidelity may not match perceived quality.For videos, scores are computed per frame and averaged.Source: §A.4 · p. 17
  • LPIPS measures perceptual similarity using distances between features from pretrained deep networks; lower scores indicate greater similarity.Video scores are calculated per frame and averaged.Source: §A.4 · pp. 17–18
  • SSIM assesses structural similarity through structural information, luminance, and contrast; video scores are averaged across frames.SSIM ranges from -1 to 1, with 1 indicating perfect structural similarity.Source: §A.4 · p. 18

B.1 ADDITIONAL QUANTITATIVE RESULTS.

The appendix reports additional PAB evaluations on WebVid and analyzes MLP-output differences across three video-generation models. The MLP analysis finds periodic low-difference intervals that can be reused, with broadcast choices varying by layer and model.

B.1 ADDITIONAL QUANTITATIVE RESULTS.

  • The additional WebVid evaluation uses 1,000 videos to assess PAB across three models and four metrics.Table 10 covers Open-Sora (2s, 480p), Open-Sora-Plan (2.7s, 512x512), and Latte (2s, 512x512); latency and speedup are measured on one GPU.Source: §B.1 · p. 18, Table 10 · p. 18

B.2 FINDINGS FOR MLP BROADCAST.

  • During intermediate diffusion stages, MLP-output MSE periodically spikes before remaining low, supporting reuse of peak outputs in subsequent low-difference steps.The analysis covers Latte, Open-Sora, and Open-Sora-Plan.Source: §B.2 · p. 19
  • Lower-layer MLP outputs differ relatively little, while upper-layer outputs differ more, informing model-specific choices of which MLPs to broadcast and their ranges.The findings are illustrated for spatial and temporal MLPs across selected blocks.Source: Figure 13 · p. 19, Figure 14 · p. 19, §B.2 · p. 20

B.3 RESULTS FOR LONG, COMPLEX AND DYNAMIC SCENES.

PAB is evaluated on 16-second, 480p Open-Sora videos using seven VBench dimensions selected for complex and dynamic scenes. The reported results show consistently good performance, including on human action and dynamic degree, alongside qualitative results for long, complex scenes.

B.3 RESULTS FOR LONG, COMPLEX AND DYNAMIC SCENES.

  • PAB performs consistently well on the selected long-video evaluation, with PAB246 showing comparable performance.Table 11 reports quantitative results for Open-Sora (16s, 480p) on a subset of VBench dimensions.Source: Table 11 · p. 20, §B.3 · p. 20
  • The evaluation uses Open-Sora at 16 seconds because other models are restricted to fixed short lengths, and selects seven VBench categories for complex and dynamic scenes.Scores are calculated exclusively from those seven categories.Source: §B.3 · p. 20
  • PAB maintains good scores on the tested difficult dimensions, including human action and dynamic degree.Qualitative results also describe robust processing of dynamic, complex scenes while maintaining high-quality output.Source: §B.3 · p. 21, Figure 16 · p. 21

B.4 BREAKDOWN OF PAB’S CONTRIBUTION WITH MULTIPLE GPUS

The multi-GPU breakdown attributes additional latency reductions to PAB’s computation and communication speedups. Further analysis identifies layer normalization and positional embedding as practical bottlenecks, while the workflow comparison explains why broadcasting outputs can skip more computation than broadcasting scores.

B.4 BREAKDOWN OF PAB’S CONTRIBUTION WITH MULTIPLE GPUS

  • PAB computation speedup reduces latency by 24% versus DSP alone, and its communication speedup reduces latency by a further 5.0% versus computation speedup alone.The evaluation uses PAB246, which has better quality but less speedup; more aggressive strategies could provide more speedup.Source: §B.4 · p. 21, Table 12 · p. 21

B.5 BREAKDOWN OF TIME COST WITHIN ATTENTION MODULE

  • Layer normalization and positional embedding, rather than attention operations, are the main attention-module time bottlenecks despite having fewer calculations and FLOPs.The analysis reports time-cost breakdowns for spatial, temporal, and cross attention.Source: §B.5 · p. 21, Table 13 · p. 22, Table 14 · p. 22

B.6 WORKFLOW COMPARISON FOR BROADCASTING DIFFERENT OBJECTS

  • Broadcasting attention outputs bypasses intermediate attention-module computations while retaining compatibility with FlashAttention; broadcasting scores still requires partial computation and may be slower than baseline.The score-broadcast workflow can also be incompatible with FlashAttention.Source: §B.6 · p. 22

B.7 VARIOUS METRICS FOR EVALUATING REDUNDANCY

This subsection evaluates multiple metrics for measuring redundancy and presents them in Figure 17.

B.7 VARIOUS METRICS FOR EVALUATING REDUNDANCY

B.8 EXTENSION TO TEXT-TO-IMAGE MODEL

The section explores extending PAB to the text-to-image model FLUX. It reports a 1.77× speedup over the original method and comparable results to the baseline.

B.8 EXTENSION TO TEXT-TO-IMAGE MODEL

  • PAB is explored as an extension to the text-to-image model FLUX.Source: §B.8 · p. 23
  • 1.77× speedup over the original method is reported for FLUX with PAB55.The speedup was measured on a single GPU.Source: Table 16 · p. 23, §B.8 · p. 24
  • PAB55 is selected for its reported balance between speedup and image quality.Source: §B.8 · p. 24
  • The text reports comparable results to the baseline and identifies Figure 18 as qualitative FLUX results.Source: Figure 18 · p. 23, §B.8 · p. 24
Loading 2408.12588v3…