PaperlayerView exact revision on arXiv

Source-linked AI summary

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution

Shangchen Zhou, Peiqing Yang, Jianyi Wang, Yihang Luo, Chen Change Loy

arXiv:2312.06640v1cs.CV

TL;DR

Real-world VSR requires realistic detail and temporal consistency despite unknown degradations and diffusion sampling randomness. Upscale-A-Video uses local temporal layers, global flow-guided latent propagation, and controllable prompts and noise, achieving state-of-the-art performance with visual realism and temporal consistency. Its scope remains bounded by difficulty maintaining global coherence across long videos.

  • Problem

    Real-world VSR must address complex unknown degradations while diffusion sampling makes temporal stability difficult, especially across lengthy sequences.

  • Method

    Upscale-A-Video combines temporal U-Net and VAE-Decoder layers with training-free flow-guided recurrent latent propagation in a text-guided latent diffusion framework.

  • Results

    Upscale-A-Video achieves state-of-the-art performance on existing benchmarks with visual realism and temporal consistency.

  • Takeaways & Limitations

    Text prompts and adjustable noise levels let the framework trade restoration fidelity against generated detail for more flexible real-world video enhancement.

  • Takeaways & Limitations

    Current methods remain constrained by short local input sequences, limiting their ability to ensure global temporal consistency in longer videos.

Abstract

from arXiv · show

Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

1. Introduction

Real-world video super-resolution must recover realistic detail from complex unknown degradations while preserving fidelity and temporal coherence. Upscale-A-Video addresses this with local and global temporal mechanisms, plus prompt and noise controls that balance restoration against generation.

  • Real-world VSR must handle unknown combinations of downsampling, noise, blur, flickering, and compression while preserving visual fidelity and temporal coherence.
  • Diffusion priors produce finer realistic details than CNN restorers but introduce temporal discontinuities through stochastic sampling.
  • Existing temporal-consistency methods remain limited because they process short local sequences and struggle to preserve coherence across longer videos.
  • Upscale-A-Video combines temporal U-Net and VAE-Decoder layers for local stability with training-free flow-guided recurrent latent propagation for long-range consistency.The VAE-Decoder reduces texture flickering, while latent propagation fuses information across short segments during inference.
  • Text prompts steer texture creation, while adjustable input noise controls the trade-off between restoration fidelity and generated detail.Lower noise favors restoration; higher noise encourages more refined details.
  • The resulting latent-diffusion approach achieves state-of-the-art benchmark performance with visual realism and temporal consistency.

2. Related Work

Earlier VSR methods often rely on predefined degradations or camera-specific paired data, limiting their generalizability to real-world videos. Diffusion-based alternatives improve restoration quality but leave temporal discontinuities in video as an under-explored challenge.

  • Most existing VSR methods assume predefined degradation processes, causing performance to deteriorate in real-world scenarios.
  • Camera-collected HR-LR pairs can work for similar mobile-camera videos, but their generalization to other devices remains uncertain.
  • Fine-tuning pretrained diffusion models with additional trainable layers has shown impressive performance, while temporal discontinuities remain an under-explored challenge for real-world VSR.

3. Methodology

Upscale-A-Video uses a local-global latent diffusion strategy to preserve temporal coherence in long-video super-resolution. Temporal layers stabilize short segments, while flow-guided latent propagation connects segments during selected denoising steps and a temporal VAE-Decoder reduces flicker.

  • Local-global temporal consistency: The framework processes videos as segments with a temporal U-Net for local consistency and recurrent latent propagation for global consistency.The global module operates during user-specified diffusion steps and propagates information across segments.
  • Latent diffusion foundation: The model is initialized from a pretrained text-guided SD ×4 Upscaler and adapts its image-based latent diffusion architecture to video.Its video design inflates 2D convolutions into 3D convolutions while retaining the pretrained upscaling capabilities.
  • Local temporal modeling: Temporal attention and 3D residual blocks are added to the U-Net while pretrained spatial layers remain fixed during temporal training.This concentrates optimization on learning local temporal consistency from video data.
  • Decoder consistency: A finetuned VAE-Decoder with temporal 3D residual blocks reduces flickering and improves low-level consistency when decoding latent sequences.Its newly added temporal layers are trained with L1, LPIPS perceptual, and temporal adversarial losses.
  • Global latent propagation: Flow-guided propagation validates optical flow with forward-backward consistency error and propagates only latents below a consistency threshold.Warped previous-frame latents are fused with current latents using weight β, set to 0.5 by default.
  • Inference control: Propagation need not occur at every diffusion step, allowing selected steps and timing to adapt to video jitter severity.Earlier integration is suggested for minor jitter, whereas later denoising steps are preferred for severe jitter such as AIGC videos.

4. Experiments

Experiments evaluate Upscale-A-Video on synthetic, real-world, and AI-generated videos using reference and nonreference metrics, qualitative comparisons, temporal profiles, and ablations. The method shows strong reconstruction, perceptual quality, artifact removal, and temporal consistency, while text prompts and noise control affect detail quality.

  • Evaluation Setup: The model is trained on WebVid10M and YouHQ, with PSNR, SSIM, LPIPS, and flow-warping error used for synthetic evaluation, and CLIP-IQA, MUSIQ, and DOVER for real-world and AIGC data.WebVid10M provides around 335K video-text pairs, while YouHQ contributes around 37K high-definition clips.
  • Quantitative Evaluation: Upscale-A-Video achieves the highest PSNR across four synthetic datasets, the lowest LPIPS on UDM10 and YouHQ40, and the highest CLIP-IQA and DOVER on real and AIGC videos.These results indicate strong reconstruction and perceptual quality across datasets from different sources.
  • Qualitative Evaluation: Qualitative comparisons show improved artifact removal and detail generation, including accurate wall structure, detailed koala fur, and recovery of the billboard text “EAT IN or TAKEAWAY”.The cited comparisons cover synthetic REDS30 and YouHQ40 videos as well as real-world VideoLQ videos.
  • Temporal Consistency: Upscale-A-Video achieves the best optical-flow error on UDM10 and second-best scores on REDS30, SPMCS, and YouHQ40, outperforming other diffusion and strong CNN-based methods.Temporal profiles also show a more seamless and smoother transition.
  • Ablation Study: Proper text prompts improve perceptual quality and faithful details, whereas low noise produces blurry details and excessive noise may cause oversharpening.Noise therefore controls the restoration-generation behavior observed during sampling.

5. Conclusion

The paper presents Upscale-A-Video for real-world video super-resolution, where diffusion models face temporal discontinuities from sampling randomness. Its local-global temporal strategy improves coherence, while text prompts and noise control support a fidelity-quality trade-off for texture generation.

  • Conclusion: Upscale-A-Video applies an image diffusion prior to real-world VSR while addressing temporal discontinuities caused by randomness during sampling.The paper characterizes real-world video diffusion applications as challenging and understudied.
  • Conclusion: A local-global temporal strategy within latent diffusion enhances temporal coherence across video content.The conclusion frames this strategy as the central mechanism for avoiding discontinuities.
  • Conclusion: Text prompts and noise-level control enable texture creation and a trade-off between restoration fidelity and generated quality.The paper presents these controls as supporting practical use in real-world scenarios.

A. Architecture

The architecture specification covers the pretrained Stable Diffusion ×4 Upscaler and the inserted temporal layers used by Upscale-A-Video. Training uses 320 × 320 video patches containing eight frames.

  • Architecture: Table 3 summarizes hyperparameters for the pretrained SD ×4 Upscaler, including its U-Net and VAE, together with the inserted temporal layers.The table follows the hyperparameter presentation style of Latent Diffusion.
  • Architecture: The model is trained on video patches of size 320 × 320 with 8 frames.This setting is stated in the architecture hyperparameter overview.

B.1. YouHQ Dataset

YouHQ is a large-scale high-definition video dataset collected from YouTube for training higher-quality VSR models. It contains roughly 37,000 clips spanning diverse scene categories.

  • YouHQ Dataset: YouHQ contains around 37,000 high-definition video clips collected from YouTube.The videos have a resolution of 1080 × 1920.
  • YouHQ Dataset: The dataset covers street view, landscape, animal, human face, static object, underwater, and nighttime scenes.This category diversity is illustrated by the dataset distribution.

C.1. Training Strategy for Watermark Removal

The training strategy uses staged U-Net training to remove watermarks introduced by WebVid10M, while patch-wise inference supports arbitrary video scales and lengths. Additional decoder conditioning and color correction address color consistency, and user studies compare the method with diffusion- and CNN-based baselines.

  • Training Strategy for Watermark Removal: A second 10k-iteration phase trains only on YouHQ to remove watermarks introduced during first-stage training on WebVid10M and YouHQ.The first stage runs for 70k iterations on both datasets; the second stage uses only YouHQ.
  • Inference Strategy: Patch-wise training and overlapping-patch inference allow processing videos with arbitrary scales and lengths without strict input-resolution requirements.Overlapping crops are processed separately and then combined to manage memory constraints.
  • Color Consistency: Input-conditioned VAE-Decoder finetuning and optional wavelet color correction improve consistency in low-frequency information such as color.The method uses decoder conditioning to address diffusion-related color shifts and adds training-free wavelet correction.
  • User Study: The user study evaluates real-world and AIGC videos against StableSR, SD ×4 Upscaler, and RealBasicVSR with 20 participants.Each volunteer viewed randomly selected video trials.
  • User Study: Human voters preferred Upscale-A-Video over other methods in the reported user study.

D.2. Ablation on Different Pretrained Priors

The ablation compares Stable Diffusion and SD ×4 Upscaler as pretrained priors. The SD ×4 Upscaler variant provides stronger restoration fidelity and temporal consistency, while the Stable Diffusion variant exhibits more color shift and artifacts.

  • Pretrained Prior Comparison: The Stable Diffusion-based variant suffers from more noticeable color shifts and unexpected artifacts, including distortions in a child’s face.Wavelet color correction is required to address the color shift.
  • Pretrained Prior Comparison: Even after wavelet color correction, the SD ×4 Upscaler-based model outperforms the Stable Diffusion variant in PSNR, SSIM, and LPIPS.
  • Pretrained Prior Comparison: The SD ×4 Upscaler-based model also demonstrates stronger generative restoration, including successful recovery of a wall in a visual example.
  • Pretrained Prior Comparison: The SD ×4 Upscaler prior shows clear advantages in restoration fidelity and temporal consistency on the YouHQ40 test set.The reported fidelity metrics are PSNR, SSIM, and LPIPS, while temporal consistency is measured by E∗ warp.

D.3. Ablation on Positions of Recurrent Latent Propagation Module

The ablation examines where to place recurrent latent propagation and compares text-guided versus empty-prompt results across synthetic, real-world, and AIGC videos. Propagation can be applied at selected diffusion steps, while proper prompts improve detail and realism.

  • Propagation Position: The recurrent latent propagation module can be inserted at selected diffusion steps rather than every inference step.The position study evaluates alternatives on the YouHQ40 test set.
  • Propagation Position: The temporal-profile comparison evaluates how different recurrent latent propagation positions affect video consistency.
  • Text Prompt Effect: Proper text prompts produce finer and more faithful details than empty prompts under classifier-free guidance.The improvement is reported for both real-world and AIGC videos.
  • Qualitative Evaluation: Qualitative comparisons cover synthetic, real-world, and AIGC videos against multiple state-of-the-art methods.The comparisons include Real-ESRGAN, SD ×4 Upscaler, ResShift, StableSR, DBVSR, and RealBasicVSR.
  • Qualitative Evaluation: A demo video provides additional comparisons on synthetic, real-world, and AIGC videos.
Loading 2312.06640v1…