Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation

Wenjing Wang; Huan Yang; Zixi Tuo; Huiguo He; Junchen Zhu; Jianlong Fu; Jiaying Liu

Source-linked AI summary

Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation

Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, Jiaying Liu

arXiv:2305.10874v4May 18, 2023cs.CV

TL;DR

Text-to-video generation is constrained by limited paired data and weak joint modeling of spatial and temporal information. The paper introduces Swap-CA within VideoFactory and releases HD-VG-130M with a refined subset, reporting superior spatial quality, temporal consistency, and text alignment. The authors note that limited resources prevent matching some closed-source industrial products.

Problem
Text-to-video generation lacks large-scale paired data and effective joint spatial-temporal modeling, while existing datasets may lack quality, scale, or accessibility.
Method
The paper combines swapped cross-attention, which alternates query roles between spatial and temporal features, with VideoFactory and the HD-VG-130M dataset.
Results
The method generates videos with superior spatial quality, temporal consistency, and alignment with text, while the refined dataset further improves performance.
Takeaways & Limitations
Joint spatial-temporal interaction and open, quality-filtered data provide the paper’s supported basis for high-quality open-domain text-to-video generation.
Takeaways & Limitations
Limited resources prevent the authors from matching the capabilities of some closed-source industrial products.

Abstract

from arXiv · show

With the explosive popularity of AI-generated content (AIGC), video generation has recently received a lot of attention. Generating videos guided by text instructions poses significant challenges, such as modeling the complex relationship between space and time, and the lack of large-scale text-video paired data. Existing text-video datasets suffer from limitations in both content quality and scale, or they are not open-source, rendering them inaccessible for study and use. For model design, previous approaches extend pretrained text-to-image generation models by adding temporal 1D convolution/attention modules for video generation. However, these approaches overlook the importance of jointly modeling space and time, inevitably leading to temporal distortions and misalignment between texts and videos. In this paper, we propose a novel approach that strengthens the interaction between spatial and temporal perceptions. In particular, we utilize a swapped cross-attention mechanism in 3D windows that alternates the "query" role between spatial and temporal blocks, enabling mutual reinforcement for each other. Moreover, to fully unlock model capabilities for high-quality video generation and promote the development of the field, we curate a large-scale and open-source video dataset called HD-VG-130M. This dataset comprises 130 million text-video pairs from the open-domain, ensuring high-definition, widescreen and watermark-free characters. A smaller-scale yet more meticulously cleaned subset further enhances the data quality, aiding models in achieving superior performance. Experimental quantitative and qualitative results demonstrate the superiority of our approach in terms of per-frame quality, temporal correlation, and text-video alignment, with clear margins.

1 Introduction

The paper addresses open-domain text-to-video generation by improving spatial-temporal interaction and providing large-scale, high-quality open data. Its VideoFactory framework combines Swap-CA with HD-VG-130M and a refined subset to improve generation quality and alignment.

Open-domain text-to-video generation is challenging because large-scale paired data are scarce and jointly modeling space and time is difficult.
Existing space-time separable approaches limit interaction between spatial and temporal features, causing temporal distortions and text-video discrepancies.
Swap-CA alternates the query role between spatial and temporal features, enabling bidirectional guidance and reciprocal information flow.
VideoFactory uses latent diffusion and a spatiotemporal U-Net for 3D noise prediction, integrating the proposed swapped cross-attention framework.
HD-VG-130M provides 130 million open-domain text-video pairs with high-definition, widescreen, and watermark-free videos.
A higher-quality subset applies text, motion, and aesthetic processing, while experiments report improved quantitative and qualitative video-generation performance.

2 Related Works

Prior text-to-video work ranges from limited-scene GANs to transformer and diffusion approaches, while available datasets often lack scale, quality, diversity, or accessibility. The paper responds with an open-source dataset whose labels and filtering target video-generation needs.

Early GAN-based text-to-video models combine convolutional architectures with recurrent networks but remain limited to simple scenes and motions.
Recent approaches extend text-to-video generation to open domains using large-scale transformers or diffusion models, but high-dimensional modeling remains difficult.
Existing datasets commonly lack sufficient scale or quality, or remain inaccessible to the research community.
HD-VG-130M uses premium captions rather than simplistic class labels or visually unsynchronized subtitles, and offers expansive scale for video generation.
Its HD-VG-40M subset applies stricter filtering for visual text, motion, and aesthetics to retain higher-quality training videos.

3 High-Definition Video Generation Dataset

The paper constructs HD-VG-130M, an open-domain dataset of 130 million high-definition, widescreen, watermark-free text-video pairs, then filters it into a higher-quality 40-million-sample subset. Processing targets caption relevance, motion, visual quality, and semantic coherence for text-to-video generation.

3.1 Data Collection, Processing and Annotation: HD-VG-130M contains 130M open-domain text-video pairs in high-definition, widescreen, and watermark-free formats.Videos are collected from YouTube, split into single-scene clips, and captioned with BLIP-2 using central keyframes.
3.1 Data Collection, Processing and Annotation: The dataset spans 15 video categories, mostly contains clips shorter than 20 seconds, and uses captions of roughly 10 words.The captions describe visual content and are intended to represent the corresponding single-scene clips.
3.2 Further Data Processing: Motion filtering removes insufficiently dynamic clips and image-like transformations using optical-flow statistics.The pipeline excludes videos with Oavg ≤ 0.2 and applies Oavg/Omd and Omd criteria to remove simplistic translation or scaling animations while retaining real-world camera motion.
3.2 Further Data Processing: Visual-text filtering removes channel names and subtitles while retaining ordinary scene text to preserve dataset diversity.CRAFT detects text near frame edges; the filtering removes 37.33% of videos, while 73.36% of retained videos still contain text.
3.2.4 Summary: The refined HD-VG-40M subset contains 40 million samples after filtering meaningless text, inadequate motion, and low-aesthetic content.Compared with HD-VILA-100M and HD-VG-130M, the subset is presented as having improved semantic coherence and suitability for text-to-video generation.
3.2.4 Summary: Detailed visual-content processing distinguishes this dataset from existing open-source alternatives and is associated with improved visual quality and text-video alignment versus Panda-70M-trained models.The comparison is reported in Tables 6–7.

4 High-Quality Text-to-Video Generation

The framework strengthens video generation by modeling spatial and temporal features through reciprocal cross-attention in 3D windows, within a latent-diffusion spatiotemporal U-Net. It combines this design with windowed computation, pretrained image-generation components, and super-resolution processing.

Spatiotemporal Connection: Swap-CA uses cross-attention between spatial and temporal features, alternating the query role across adjacent layers for bidirectional guidance.One feature supplies the query while the other supplies key/value; adjacent blocks reverse these roles to promote reciprocal information flow.
Spatiotemporal Connection: 3D window attention reduces computational costs by applying spatiotemporal cross-attention within non-overlapping local windows.The windows process features of size F × H × W using a window size Fw × Hw × Ww, avoiding global attention.
Model Architecture: The model uses a latent diffusion framework with an auto-encoder, CLIP text features, and a spatial-temporal separable U-Net for 3D noise prediction.Text features enter the U-Net through spatial cross-attention, while diffusion optimization occurs in a downsampled 3D latent space.
Model Architecture: Swap-CA is inserted at the ends of U-Net encoder and decoder blocks, while 1×1×1 convolution and temporal skip connections handle additional feature integration.This placement balances performance and efficiency and strengthens information flow among temporal modules separated by spatial resizing.
Super-Resolution: The framework adds super-resolution processing using a Real-ESRGAN hybrid degradation model to simulate degradation from diffusion and auto-encoding.The generated video is first produced at lower resolution and then enhanced through super-resolution training based on degraded original frames.

5 Experiments

Experiments evaluate the attention design, window size, dataset construction, and benchmark performance. Results favor reciprocal spatial-temporal interaction, HD-VG-130M data processing, and strong performance across validation and zero-shot datasets, while concept misinterpretation remains an observed failure mode.

Spatiotemporal Inter-Connection: Swap-CA improves performance more than the compared attention strategies, while Tune-A-Video is less effective in the open-domain setting.CogVideo and 3D spatial-temporal WSA outperform the baseline but provide less improvement than Swap-CA.
Spatiotemporal Inter-Connection: 8 × 3 × 6 achieves the best FVD and CLIPSIM scores with comparable efficiency among the evaluated attention window sizes.The spatial dimensions match the core U-Net feature resolution, while the temporal size broadens attention while reducing computation complexity.
Video Generation Dataset: Adding HD-VG-130M improves WebVid-10M validation FVD by 45.34 and enhances generated videos’ visual quality beyond watermark removal.The reported improvements include greater scenic beauty and richer detail.
Video Generation Dataset: Fine-tuning on HD-VG-40M enhances the FVD score to 418.40 and resolves static-scene and meaningless-text issues observed without the higher-quality subset.The resulting samples also show better contrast, clearer edges, and more vivid colors.
Video Generation Dataset: HD-VILA-100M training degrades FVD from 429.75 to 692.99 and CLIPSIM from 0.3082 to 0.2671 compared with the reported alternative setting.The authors associate this degradation with subtitles and the lack of scene detection and video captioning procedures.
Quantitative and Qualitative Comparison: On MSR-VTT, the zero-shot model achieves an average CLIPSIM of 0.3021, while WebVid-10M validation reports FVD 292.35 and CLIPSIM 0.3070.The paper reports these results as outperforming existing methods on the respective evaluations.
Quantitative and Qualitative Comparison: Qualitative samples are described as having high quality, clear motion, intricate details, and precise semantic alignment.The comparison includes Make-A-Video, Imagen Video, Video-LDM, and Gen-2.
Quantitative and Qualitative Comparison: The model can misinterpret concepts and produce unintended results for prompts whose meaning depends on a specific interpretation.The paper gives “A cat singing in a barbershop quartet” as an example.

6 Conclusion

The paper presents an open-domain framework and dataset for high-quality text-to-video generation, combining swapped cross-attention with 130 million open-domain pairs and a higher-quality subset. The authors report improved spatial quality, temporal consistency, and text alignment, while noting resource limits relative to some closed-source industrial systems.

Conclusion: The framework uses swapped cross-attention to make spatial and temporal information complement each other in high-definition, widescreen, watermark-free video generation.The conclusion reports improvements in spatial quality, temporal consistency, and alignment with text.
Conclusion: HD-VG-130M provides 130 million open-domain text-video pairs, complemented by a higher-quality subset intended to further promote performance.The dataset is presented as a contribution for maximizing the model’s potential and supporting field development.
Limitations and Future Work: The authors state that limited resources prevent matching the capabilities of some closed-source industrial products.They identify refining BLIP-2 captions and using more powerful text-to-image backbones as future directions.