Source-linked AI summary
SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration
Jianyi Wang, Zhijie Lin, Meng Wei, Yang Zhao, Ceyuan Yang, Fei Xiao, Chen Change Loy, Lu Jiang
TL;DR
Video restoration must recover temporally consistent detail from low-quality videos with unknown degradations, while existing diffusion methods remain limited in sampling efficiency and resolution flexibility. SeedVR introduces a shifted-window diffusion transformer with causal video encoding and scalable image/video training. It reports strong performance across synthetic, real-world, and AI-generated video benchmarks, while remaining over twice as fast as existing methods.
Problem
Diffusion-based video restoration remains costly and degrades when processing resolutions different from training, limiting long-duration, high-resolution restoration.
Method
SeedVR uses a shifted-window diffusion transformer with variable-sized boundary windows, a causal video autoencoder, mixed image/video training, and progressive training.
Results
SeedVR achieves superior performance on 4 of 6 benchmarks and is over twice as fast as existing diffusion-based restoration methods.
Takeaways & Limitations
SeedVR supports generic restoration across arbitrary video lengths and resolutions with strong perceptual quality and temporal detail consistency.
Takeaways & Limitations
Excessively strong generative ability can reduce output fidelity, so the final model does not enhance low-quality conditions this way.
Abstract
from arXiv · showhide
Video restoration poses non-trivial challenges in maintaining fidelity while recovering temporally consistent details from unknown degradations in the wild. Despite recent advances in diffusion-based restoration, these methods often face limitations in generation capability and sampling efficiency. In this work, we present SeedVR, a diffusion transformer designed to handle real-world video restoration with arbitrary length and resolution. The core design of SeedVR lies in the shifted window attention that facilitates effective restoration on long video sequences. SeedVR further supports variable-sized windows near the boundary of both spatial and temporal dimensions, overcoming the resolution constraints of traditional window attention. Equipped with contemporary practices, including causal video autoencoder, mixed image and video training, and progressive training, SeedVR achieves highly-competitive performance on both synthetic and real-world benchmarks, as well as AI-generated videos. Extensive experiments demonstrate SeedVR's superiority over existing methods for generic video restoration.
1. Introduction
Generic video restoration must recover high-quality videos from inputs with complex, often unknown degradations. SeedVR addresses diffusion-based restoration bottlenecks in arbitrary-resolution, long-video processing through shifted-window attention, efficient video encoding, mixed-resolution image/video training, and progressive training.
- Generic video restoration reconstructs high-quality outputs from low-quality videos affected by complex and often unknown real-world degradations.
- Diffusion-based restoration methods face high computational costs and performance degradation when processing resolutions unlike those used during training.
- 50% patch overlap can make diffusion-based inference slow; VEnhancer requires 387 seconds for 31 frames at 1344×768 with 50 sampling steps.
- SeedVR replaces full self-attention with large non-overlapping Swin windows in a diffusion transformer to address resolution constraints at lower computational cost.
- SeedVR combines a causal video autoencoder, mixed native-resolution image/video training, and multi-stage progressive training to reduce computational cost and accelerate convergence.
- SeedVR is at least 2× faster than existing diffusion-based video-restoration methods despite having 2.48B parameters.
2. Related Work
Related restoration work evolved from limited-receptive-field CNNs to transformer and diffusion approaches. Window attention reduces self-attention cost, while SeedVR investigates larger and variable-sized windows to support arbitrary-length, arbitrary-resolution restoration without tiled sampling.
- Attention Mechanism in Restoration: CNN-based restoration methods typically struggled to capture long-range pixel dependencies because of limited receptive fields.
- Attention Mechanism in Restoration: Transformer restoration methods introduced attention mechanisms, with window attention reducing the quadratic complexity of self-attention.
- SeedVR uses 64 × 64 windows in an 8× compressed latent space and variable-sized boundary windows to reduce resolution constraints and avoid tiled sampling.
- Diffusion-based restoration improves performance but remains limited by inherited diffusion-prior constraints and restricted generative capability for realistic textures.
3. Methodology
SeedVR combines shifted-window multimodal diffusion attention with a causal video autoencoder and scalable training procedures. The design supports arbitrary video lengths and resolutions while targeting efficient restoration and high-quality reconstruction.
- Shifted Window Based MM-DiT: SeedVR adapts the SD3-style architecture with a Swin-MMDiT block to handle video inputs with arbitrary lengths and resolutions.
- Shifted Window Based MM-DiT: Swin-MMDiT applies regular and half-window-shifted attention, allowing local windows to exchange information across successive transformer blocks.
- Shifted Window Based MM-DiT: Variable-sized windows at boundaries are flattened and processed without complex 3D masking, while 3D rotary positional embeddings model varying window sizes.
- Shifted Window Based MM-DiT: Video and text features use separate attention mechanisms whose keys and values are concatenated without increasing computational cost in practice.
- Causal Video VAE: The causal video autoencoder compresses space and time, uses 16 latent channels and temporal compression factor 4, and supports long videos by processing clips.
- Large-scale Training: SeedVR scales training through joint image/video data, varying resolutions, progressive resolution and duration growth, and noise-conditioned low-quality inputs.
- Large-scale Training: Excessively strong generative ability can reduce output fidelity, so SeedVR does not apply the corresponding enhancement to low-quality conditions.
4. Experiments
SeedVR is evaluated across synthetic, real-world, and AI-generated video benchmarks using perceptual, reference-based, and no-reference metrics. It achieves strong restoration quality and efficiency, while ablations show that its causal VAE and large spatiotemporal windows support reconstruction quality and practical training.
- Experimental Settings: SeedVR is compared on synthetic benchmarks, the real-world VideoLQ dataset, and the AIGC38 dataset using full-reference and no-reference metrics.Testing videos are processed to 720p while preserving their original length.
- Quantitative Comparisons: SeedVR achieves superior performance on 4 of 6 benchmarks and remains competitive on REDS30, where it obtains the best DOVER score.The authors note limitations on PSNR and SSIM because these metrics emphasize pixel-level fidelity and structural similarity, while SeedVR focuses on perceptual quality.
- Qualitative Comparisons: Qualitative comparisons show stronger degradation removal and texture generation on real-world and AIGC videos, including building structures, a panda’s nose, and a terracotta warrior’s face.Other approaches produce blurred details in the cited AIGC examples.
- Causal Video VAE: The causal video VAE reaches the lowest rFVD score, 69.5% lower than the second best, while achieving the best LPIPS and competitive PSNR and SSIM.These results indicate strong video reconstruction quality relative to the compared VAE models.
- Window Size for Attention: With 1 × 8 × 8 windows, training takes 455.49 sec/iter, 19.24 times longer than with 1 × 64 × 64 windows.The comparison uses models trained for 12.5k iterations under the same settings.
- Window Size for Attention: On YouHQ40, full attention and smaller spatial windows degrade as temporal length increases, whereas the 5 × 64 × 64 window improves performance by capturing long-range dependencies and supporting text-restoration alignment.These observations motivate SeedVR’s selected 5 × 64 × 64 attention window.
5. Conclusion
SeedVR is presented as a foundational diffusion-transformer architecture for high-quality video restoration across arbitrary resolutions and lengths. The authors report strong handling of synthetic and real-world degradations, improved visual realism and temporal detail consistency, and more than twice the speed of existing methods despite its larger parameter size.
- SeedVR is designed as a foundational architecture for high-quality video restoration with arbitrary resolutions and lengths.
- SeedVR handles both synthetic and real-world degradations while improving visual realism and detail consistency across frames.
- Over twice as fast as existing methods despite its larger parameter size, SeedVR combines restoration quality with higher inference speed.