Source-linked AI summary
FastVR: Efficient Streaming Video Restoration with One-Step Diffusion
Xiaoxu Chen, Qin Yang, Haoran Bai, Sibin Deng, Ying Chen
TL;DR
High-resolution video restoration with diffusion is constrained by costly VAE processing and full self-attention. FastVR uses a one-step streaming model with a lightweight VAE, chunk-wise causal attention, and trajectory-based training. It reports strong benchmark quality and temporal consistency while processing 1080p video at 11 FPS on one H20 GPU.
Problem
High-resolution video restoration is limited by costly VAE encoding and decoding and the quadratic cost of full self-attention in diffusion transformers.
Source: §1 · p. 2Method
FastVR is a one-step streaming restoration model combining a lightweight VAE and chunk-wise causal attention with continuous trajectory learning and velocity consistency regularization.
Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4Results
Across five benchmarks, FastVR achieves the highest MUSIQ and CLIP-IQA scores on all five datasets, ranks first or second in DOVER, and reaches 11 FPS on 1080p video with one H20 GPU.
Source: §1 · p. 3, §4.3 · p. 7Takeaways & Limitations
The results highlight the potential of jointly designing efficient inference and restoration-oriented training for practical high-resolution video restoration.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Diffusion-based video restoration recovers realistic details, but its practical deployment is limited by two efficiency bottlenecks: costly VAE encoding and decoding, and the quadratic cost of full self-attention in diffusion transformers (DiTs). This paper presents FastVR, a streaming video restoration framework built on a one-step diffusion model, which delivers strong restoration quality and temporal consistency while processing 1080p video at 11 FPS on a single H20 GPU. To improve inference efficiency, FastVR combines a lightweight VAE with chunk-wise causal attention, which substantially reduces the computational cost. During training, it further adopts velocity consistency regularization and continuous trajectory learning, which improve restoration quality. Extensive experiments show that FastVR is more efficient than the evaluated diffusion baselines while achieving state-of-the-art performance on synthetic and real-world benchmarks. We hope that this work supports further progress in the community.
1. Introduction
Video diffusion can recover realistic detail while modeling temporal dynamics, but its computational cost limits practical restoration. FastVR targets the VAE and attention costs with a one-step streaming framework and reports strong restoration quality and temporal consistency at 11 FPS on 1080p video.
1. Introduction
- Video diffusion priors can recover realistic textures while modeling motion and appearance jointly, but these benefits come at high computational cost.Image diffusion priors improve spatial detail but do not explicitly model temporal dynamics, which can disrupt consistency across frames.Source: §1 · p. 2
- After sampling is reduced to one step, quadratic global self-attention and reconstruction through a large causal video VAE can dominate inference; VAE decoding becomes dominant while dense attention remains inefficient at high resolution.SeedVR2 reports that its causal video VAE accounts for more than 95% of total runtime on a 100-frame 720p video.Source: §1 · p. 2
- FastVR combines a lightweight VAE and chunk-wise causal attention with continuous restoration-trajectory learning and velocity consistency regularization.The framework targets encoding, decoding, and attention costs; its training is reported to improve optimization stability and detail recovery.Source: §1 · pp. 2–3
- 11 FPS on 1080p video with a single H20 GPU, while maintaining strong restoration quality and temporal consistency.Experiments on synthetic and real-world benchmarks report competitive restoration quality and temporal consistency.Source: §1 · p. 3
2. Related Work
Prior work develops video-native diffusion restoration to preserve temporal consistency and improve detail fidelity, then accelerates inference through one-step methods. Even after sampling is compressed, VAE processing and attention remain important latency sources at high resolution.
2.1. Diffusion-based Video Restoration.
- Video-native diffusion models address temporal consistency alongside spatial fidelity, with SeedVR, STAR, and Vivid-VR introducing restoration-specific architectures and objectives.Earlier approaches added temporal modules to image backbones, but the underlying 2D prior limits robustness under severe spatiotemporal corruption.Source: §2.1 · p. 3
2.2. Video Diffusion Acceleration.
- One-step video restoration methods use distillation, adversarial post-training, or rectified-flow techniques, but high-resolution inference remains bottlenecked by VAE processing and DiT attention.FlashVSR combines one-step distillation, sparse causal attention, and a compact decoder in a streaming framework.Source: §2.2 · p. 3
3. Method
FastVR pairs one-step streaming inference with a lightweight VAE and chunk-wise causal attention. Its two-stage DiT training learns a latent restoration trajectory with velocity consistency, then refines decoded outputs under pixel-space supervision.
3. Method
- FastVR restores each temporal chunk and spatial tile in one DiT pass, using a lightweight VAE decoder conditioned on the LQ video.The model is trained in two stages with the frozen Wan VAE, while the lightweight VAE handles inference.Source: §3 · pp. 3–4, §3.1 · p. 4
3.1. Model Architecture
- Chunk-wise causal attention restricts each chunk to itself and its immediate predecessor, reducing temporal attention cost to O(Tk) from O(T^2) with KV caching.For fixed chunk size and spatial resolution, the KV cache remains O(k), independent of video length.Source: §3.1 · pp. 4–5
3.2. Two-Stage Latent–Pixel Training
- Stage I trains a continuous latent-space restoration trajectory from LQ to HQ states, with random timestep sampling supervising velocities across degradation levels.The trajectory runs from the HQ endpoint at t = 0 to the LQ endpoint at t = τ.Source: §3.2 · p. 5
- Self-distilled velocity consistency compares predictions at a reference latent and a state reached by the model's own update, addressing their potential trajectory mismatch.The model propagates a sampled state with an Euler update before evaluating velocity at the propagated state.Source: §3.2 · p. 5
- Stage II fine-tunes the one-step prediction using decoded pixel-space supervision because latent-space accuracy alone may not ensure faithful reconstruction.The Wan VAE decoder stays frozen; the decoded output is supervised with equally weighted pixel-wise ℓ1 and DISTS losses.Source: §3.2 · p. 6
4. Experiments
FastVR is evaluated on synthetic and real-world benchmarks, with comparisons covering restoration quality, visual detail, and inference efficiency. It shows strong no-reference quality and competitive visual detail, while retaining trade-offs against some baselines on synthetic full-reference metrics.
4.1. Implementation Details
- FastVR’s implementation uses filtered video training data, two-stage optimization, and tiled inference with chunk-wise causal attention.Training uses approximately 80,000 videos and 45-frame clips; inference uses three latent frames per chunk and overlapping 64 × 64 latent tiles.Source: §4.1 · p. 6
4.2. Evaluation and Metrics.
- Evaluation spans three synthetic datasets and two real-world datasets, using no-reference metrics for real-world videos and both full- and no-reference metrics for synthetic videos.Real-world evaluation includes NIQE, MANIQA, MUSIQ, CLIP-IQA, and DOVER; synthetic evaluation additionally reports PSNR, SSIM, and LPIPS.Source: §4.2 · p. 6, Table 1 · p. 7
4.3. Quantitative Results
- FastVR achieves the highest MUSIQ and CLIP-IQA scores on all five datasets, ranks first or second in DOVER, and leads all reported metrics on VideoLQ.On synthetic benchmarks, it improves LPIPS over both FlashVSR variants, while DOVE and SeedVR2 generally retain advantages in full-reference metrics.Source: §4.3 · p. 7, Table 1 · p. 7
4.4. Qualitative Results
- FastVR reconstructs clearer fine details and avoids some competing methods’ texture artifacts in real-world examples, while recovering fur and fabric detail in synthetic examples.The comparisons include clothing, landscapes, portraits, animals, text, feathers, illustrated patterns, fur, and knitted fabric.Source: §4.4 · p. 7, Figure 3 · p. 8, Figure 4 · p. 9
5. Conclusion
FastVR combines efficient streaming inference with restoration-oriented training. Experiments show strong no-reference quality and competitive visual detail, alongside favorable speed and memory use.
5. Conclusion
- FastVR combines a lightweight VAE, chunk-wise causal attention, and KV caching to reduce autoencoding and denoising overhead while bounding intermediate memory during long-video inference.Its two-stage training combines continuous latent-space trajectory learning and velocity consistency regularization with pixel-space refinement.Source: §5 · p. 9
- Experiments on synthetic and real-world benchmarks demonstrate strong no-reference quality and competitive visual detail, together with favorable inference speed and memory usage.The paper concludes that jointly designing efficient inference and restoration-oriented training has potential for practical high-resolution video restoration.Source: §5 · p. 9
- FastVR is presented as a one-step diffusion framework for efficient streaming video restoration.The conclusion highlights joint inference and training design as a direction for practical high-resolution restoration.Source: §5 · p. 9