PaperlayerView exact revision on arXiv

Source-linked AI summary

EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagation

Jiaqi Xu, Kunzhe Huang, Xinyi Zou, Yunkuo Chen, Bo Liu, MengLi Cheng, Jun Huang, Xing Shi

arXiv:2405.18991v3cs.CVcs.CLcs.MM

TL;DR

Video diffusion models face slow generation and training as sequences grow, alongside quality and alignment challenges. EasyAnimate combines hybrid attention, token-length training, reward backpropagation, and MLLM text encoding; its authors report improved generated-video performance and human-preference alignment. They also report limitations in color accuracy, dynamic degree, and supported video duration.

  • Problem

    Video diffusion models face slow generation and training as sequence length increases, as well as suboptimal human-preference alignment and prompt adherence.

    Source: §1 · p. 2

  • Method

    EasyAnimate combines multidirectional sliding-window and full attention with token-length training, reward-backpropagation post-training, and MLLM text encoding.

    Source: §1 · p. 3

  • Results

    Reward-backpropagation post-training is reported to improve generated videos and their alignment with human preferences.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The framework brings data processing, model training, and post-training together, with additional strategies for training efficiency and text understanding.

    Source: §1 · p. 3

    EasyAnimate has limitations in color accuracy and dynamic degree and currently supports video generation only up to 5 seconds.

    Source: §6 · p. 8

Abstract

from arXiv · show

This paper introduces EasyAnimate, an efficient and high quality video generation framework that leverages diffusion transformers to achieve high-quality video production, encompassing data processing, model training, and end-to-end inference. Despite substantial advancements achieved by video diffusion models, existing video generation models still struggles with slow generation speeds and less-than-ideal video quality. To improve training and inference efficiency without compromising performance, we propose Hybrid Window Attention. We design the multidirectional sliding window attention in Hybrid Window Attention, which provides stronger receptive capabilities in 3D dimensions compared to naive one, while reducing the model's computational complexity as the video sequence length increases. To enhance video generation quality, we optimize EasyAnimate using reward backpropagation to better align with human preferences. As a post-training method, it greatly enhances the model's performance while ensuring efficiency. In addition to the aforementioned improvements, EasyAnimate integrates a series of further refinements that significantly improve both computational efficiency and model performance. We introduce a new training strategy called Training with Token Length to resolve uneven GPU utilization in training videos of varying resolutions and lengths, thereby enhancing efficiency. Additionally, we use a multimodal large language model as the text encoder to improve text comprehension of the model. Experiments demonstrate significant enhancements resulting from the above improvements. The EasyAnimate achieves state-of-the-art performance on both the VBench leaderboard and human evaluation. Code and pre-trained models are available at https://github.com/aigc-apps/EasyAnimate.

1 Introduction

Video diffusion faces high training and inference costs as sequences grow, alongside quality gaps in human-preference alignment and prompt adherence. EasyAnimate addresses these challenges with hybrid attention, token-length training, reward backpropagation, and multimodal text encoding.

1 Introduction

  • Video diffusion transformers face high computational costs that grow quadratically with sequence length, while spatial-temporal decoupled attention can restrict receptive fields and miss large frame-to-frame changes.Three-dimensional full attention captures global video information but requires substantial computational resources.Source: §1 · p. 2
  • Training with Token Length combines videos with varying resolutions and frame counts while keeping each sample's maximum token count equal to reduce GPU idle time.The strategy is intended to balance GPU processing speed across samples.Source: §1 · pp. 2–3
  • Reward backpropagation uses human-preference models to steer post-training, while MLLMs strengthen understanding of detailed descriptions and complex object relationships.The paper reports that combining reward models can achieve superior performance.Source: §1 · pp. 2–3
  • EasyAnimate combines Hybrid Windows Attention, reward-backpropagation post-training, token-length training, and MLLM text encoding in a video-generation framework.The framework spans data preprocessing, VAE training, diffusion-transformer training, and post-training.Source: §1 · pp. 2–3
  • Hybrid Windows Attention interleaves multidirectional sliding-window and full attention to improve video-generation and training efficiency.The multidirectional module is designed to enlarge the receptive field across 3D dimensions while balancing efficiency and complexity.Source: §1 · pp. 2–3

2 Related Work

Prior video-generation work has moved from U-Net diffusion toward diffusion transformers, while longer sequences and conventional text encoders remain challenges. For preference alignment, reward backpropagation is established in other diffusion settings, but its use with rectified-flow video diffusion transformers presents unresolved challenges.

2.1 Video Diffusion Model

  • Video diffusion research has shifted from U-Net-based synthesis toward diffusion transformers, while longer sequences increase computational complexity and CLIP/T5 can limit language understanding.EasyAnimate responds with Hybrid Window Attention, token-length training, and MLLM text encoding.Source: §2.1 · p. 3

2.2 Human Preference Alignment

  • Reward backpropagation uses differentiable rewards to guide diffusion sampling, but its application to rectified-flow video diffusion transformers remains unexplored and faces memory and training-stability challenges.Prior alignment work focused on 2D VAE and U-Net video diffusion with DDPM sampling; the paper modifies reward backpropagation for efficient training and convergence in the newer setting.Source: §2.2 · p. 3

3 Architecture

EasyAnimate combines a text encoder, diffusion transformer, and video VAE, with architecture choices targeting efficient video attention, text–video alignment, and high-resolution processing. Its components include multidirectional sliding-window attention, Qwen2-VL-7B text encoding, MMDiT, and a 3D causal VAE.

3 Architecture

  • EasyAnimate comprises a text encoder, diffusion transformer, and video VAE, and presents Hybrid Windows Attention as its first architecture component.Figure 4 illustrates full attention and multidirectional sliding-window attention.Source: §3 · pp. 3–4, Figure 4 · p. 4

3.1 Hybrid Window Attention

  • Hybrid Window Attention interleaves 3D full attention with multidirectional sliding-window attention to reduce training and inference time, with gains growing as sequence length increases.The multidirectional design assigns head groups different sliding directions, expanding the 3D receptive field while requiring only one attention pass.Source: §3.1 · p. 4

3.2 Text Encoder

  • Qwen2-VL-7B serves as the text encoder, selected for visual-language understanding and reasoning, with multilingual input support.Text features are extracted from its penultimate hidden layer, RMS-normalized, and transformed to reduce scale disparities with video features.Source: §3.2 · p. 4

3.3 Video Diffusion Transformer

  • MMDiT uses distinct fully connected structures and feed-forward networks for text and video modalities to enhance their alignment.The model concatenates text and video embeddings for self-attention and uses 3D RoPE; rectified-flow loss was selected over DDPM loss based on initial experiments.Source: §3.3 · pp. 4–5

3.4 3D Causal VAE

  • A 3D causal VAE compresses video across spatial and temporal dimensions, while spatial and temporal slicing reduces memory use during long, high-resolution decoding.Caching prior latent states supports causal processing; variable frame intervals during VAE training improve cross-frame encoding and decoding robustness, and a latent penalty reduces speckle artifacts.Source: §3.4 · p. 5

4 Model Training

EasyAnimate’s training pipeline curates video-text data, trains the diffusion transformer across token lengths and stages, then applies reward backpropagation post-training. The methods address data quality, uneven GPU workloads, and alignment with human preferences.

4.1 Data Curation

  • The pipeline splits videos into single-shot clips, filters them, and generates dense captions that are refined and summarized; caption-video similarity is computed to support alignment and prompt-following in training datasets.Motion filtering uses optical-flow scores and a camera-shake classifier, with filtering thresholds increased along with resolution; the authors curate approximately 34M video-text and 3M image-text pairs for joint image-video training.Source: §4.1 · p. 5

4.2 DiT Training

  • 120.91% more tokens are trained per iteration with Training with Token Length than with the naive method.Samples with similar sequence lengths are grouped, including videos with different resolutions and frame counts; the reported comparison uses batch size 256, 1024×1024 resolution, and 49 frames.Source: §4.2 · pp. 5–6, Table 3 · p. 6
  • DiT training progresses through VAE adaptation, pretraining, continued pretraining, and image-to-video finetuning at increasing resolutions.The stages use token lengths of 256 × 256 × 49, 512 × 512 × 49, and 1024×1024×49 for the specified pretraining and finetuning phases.Source: §4.2 · p. 6

4.3 Post Training with Reward Backpropagation

  • Reward backpropagation with LoRA fine-tunes the DiT using differentiable rewards to improve alignment with human preferences.For flow-matching sampling, EasyAnimate uses K=10 because optimizing only the last step was insufficiently fast and stable; it uses F=1 because multi-frame rewards consumed more memory and impaired dynamics and generalization.Source: §4.3 · p. 6

5 Experiment

EasyAnimate is reported to lead on multiple VBench metrics and in human preference across all evaluated categories. Ablations favor Qwen2-VL, six-directional window attention, combined reward models, and specific reward-backpropagation settings.

5.1 Evaluation

  • EasyAnimate achieves the best performance across multiple VBench metrics and the highest human preference across perceptual quality, text-video consistency, and physical-law adherence.Table 4 compares VBench results; Table 5 reports human-evaluation win rates.Source: §5.1 · p. 6, Table 4 · p. 7, Table 5 · p. 7

5.2 Ablation Study

  • Qwen2-VL significantly improves overall VBench performance over the CLIP–T5 dual encoder, whose text limits and nuanced-text understanding are described as suboptimal.CLIP limits text to 77 tokens.Source: Table 6 · p. 7, §5.2 · p. 7
  • Middle-layer window attention hurts performance least; the authors hypothesize it retains earlier global context and gains stability from later full-attention layers.Table 7 compares shallow, middle, and deep placement.Source: Table 7 · p. 7, §5.2 · p. 7
  • 352.3 FVD for 6-directional attention, versus 373.6 for 3-directional and 408.1 for 1-directional; smaller windows worsen FVD without notable speed gains.The authors describe the current window-size setting as a balance between speed and quality.Source: §5.2 · p. 7, Table 8 · p. 7
  • MPS and HPSv2.1 improve VBench composite, quality, and semantic scores, and their combination performs best; reward feedback also produces clearer, more textured visuals.Figure 6 compares outputs with and without reward feedback using the same prompt.Source: §5.2 · pp. 7–8, Table 9 · p. 8, Figure 6 · p. 8
  • K=1 reward backpropagation at only the final denoising step is unstable, with a sudden reward drop; its gradient norm may be much smaller than at K=10.Figure 7 shows the impact of backpropagation steps.Source: Figure 7 · p. 8, §5.2 · p. 8
  • F=1 is sufficient for convergence and generalization; multiple decoded frames can impair robot-motion dynamics, while F=17 can invite background artifacts from reward hacking.The authors speculate that frame-wise optimization directions may conflict with an image-based reward model.Source: Figure 8 · p. 8, §5.2 · p. 8

6 Limitations

EasyAnimate has color-accuracy and dynamic-degree limitations and currently supports videos up to five seconds long.

6 Limitations

  • Color accuracy and dynamic degree remain limited, and the current five-second maximum restricts use for longer-duration tasks.The paper gives mismatching requested object colors as an example and attributes the issues likely to dataset processing.Source: §6 · p. 8

7 Conclusion

EasyAnimate combines Hybrid Windows Attention, reward-model refinement, variable-length training, and MLLM text encoding in a transformer-based video-generation framework. The paper reports state-of-the-art performance on the video evaluation leaderboard.

7 Conclusion

  • EasyAnimate uses Hybrid Windows Attention to reduce long-sequence complexity while improving spatial and temporal dependency modeling, reward models for preference alignment, variable-resolution and frame-count training, and MLLM encoding for multilingual text understanding.The paper reports state-of-the-art video-evaluation leaderboard performance.Source: §7 · p. 8

Related papers

Loading 2405.18991v3…