PaperlayerView exact revision on arXiv

Source-linked AI summary

STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution

Rui Xie, Yinhong Liu, Penghao Zhou, Chen Zhao, Jun Zhou, Kai Zhang, Zhenyu Zhang, Jian Yang, Zhenheng Yang, Ying Tai

arXiv:2501.02976v1cs.CV

TL;DR

Real-world VSR must handle complex degradations while preserving spatial fidelity and temporal consistency, challenges that limit earlier GAN- and image-diffusion-based approaches. STAR leverages T2V diffusion priors with LIEM and Dynamic Frequency Loss, and reports superior spatial and temporal quality across synthetic and real-world datasets.

  • Problem

    Real-world VSR must restore clear, temporally consistent videos from low-resolution inputs with unknown degradations, while prior approaches face oversmoothing or insufficient temporal consistency.

  • Method

    STAR integrates T2V diffusion priors into real-world VSR, adding LIEM for local detail enhancement and Dynamic Frequency Loss for frequency-specific fidelity guidance.

  • Results

    STAR achieves superior spatial and temporal quality, including the highest DOVER clarity scores across all datasets compared with state-of-the-art methods.

  • Takeaways & Limitations

    Powerful T2V models can generate high-quality real-world VSR results in both spatial and temporal dimensions.

Abstract

from arXiv · show

Image diffusion models have been adapted for real-world video super-resolution to tackle over-smoothing issues in GAN-based methods. However, these models struggle to maintain temporal consistency, as they are trained on static images, limiting their ability to capture temporal dynamics effectively. Integrating text-to-video (T2V) models into video super-resolution for improved temporal modeling is straightforward. However, two key challenges remain: artifacts introduced by complex degradations in real-world scenarios, and compromised fidelity due to the strong generative capacity of powerful T2V models (\textit{e.g.}, CogVideoX-5B). To enhance the spatio-temporal quality of restored videos, we introduce\textbf{~\name} (\textbf{S}patial-\textbf{T}emporal \textbf{A}ugmentation with T2V models for \textbf{R}eal-world video super-resolution), a novel approach that leverages T2V models for real-world video super-resolution, achieving realistic spatial details and robust temporal consistency. Specifically, we introduce a Local Information Enhancement Module (LIEM) before the global attention block to enrich local details and mitigate degradation artifacts. Moreover, we propose a Dynamic Frequency (DF) Loss to reinforce fidelity, guiding the model to focus on different frequency components across diffusion steps. Extensive experiments demonstrate\textbf{~\name}~outperforms state-of-the-art methods on both synthetic and real-world datasets.

1. Introduction

Real-world VSR must recover clear spatial details and temporal consistency under unknown degradations, but existing approaches face oversmoothing or weak temporal modeling. STAR addresses these challenges with T2V priors, LIEM, and Dynamic Frequency Loss, and reports strong spatial and temporal quality.

  • Motivation: Real-world VSR reconstructs high-resolution videos from low-resolution inputs with unknown degradations while seeking clear details and temporal consistency.Unexpected noise, blur, and compression make joint spatial-temporal restoration difficult.
  • Limitations of prior methods: GAN-based methods improve details and motion consistency but often oversmooth videos because of limited generative capacity.Optical-flow maps help produce smooth motion across frames.
  • Limitations of prior methods: Image-diffusion VSR methods add temporal components, yet image-based training can still fail to ensure high temporal consistency.These models are primarily trained on static image data rather than videos.
  • STAR: STAR integrates powerful T2V diffusion priors into real-world VSR to improve spatial details and temporal consistency.The framework is presented as a spatio-temporal quality augmentation approach.
  • STAR: LIEM enriches local information before global self-attention, while DF Loss guides low- and high-frequency learning across diffusion steps.Together, these components target degradation artifacts and restoration fidelity.
  • Results: STAR achieves the highest clarity measured by DOVER scores across all datasets while maintaining robust temporal consistency.This is the paper’s reported comparison with state-of-the-art methods.

2. Related Work

Related VSR methods include recurrent and sliding-window approaches, while diffusion-based methods introduce generative priors for restoration. Text-to-image adaptations add temporal layers but often remain limited in temporal consistency, motivating T2V-based approaches.

  • Video Super-Resolution: Traditional VSR methods use recurrent processing or sliding windows to super-resolve video sequences.Both categories suffer from degradation mismatch in real-world applications.
  • Text-to-Video Diffusion Model: Large-scale T2V diffusion models include U-Net-based and DiT-based architectures.I2VGen-XL and CogVideoX are examples discussed in this context.
  • Diffusion Prior for Super-Resolution: Diffusion priors have been used for image and video super-resolution to balance fidelity and perceptual quality.Representative methods integrate restoration and generative modules or embed semantic information for guidance.
  • Diffusion Prior for Super-Resolution: Several VSR methods adapt text-to-image diffusion priors by adding temporal layers, but they often struggle with temporal consistency.These methods include Upscale-A-Video, MGLD-VSR, Inflating with Diffusion, and SATeCo.

3. Methodology

STAR integrates a text-to-video diffusion backbone with local-detail enhancement and frequency-aware fidelity optimization for real-world video super-resolution. Its pipeline combines VAE, text encoding, ControlNet, LIEM, and dynamic frequency constraints across diffusion steps.

  • Overview: STAR uses a VAE, text encoder, ControlNet, and T2V model with LIEM to restore HR videos from LR inputs.The VAE produces latent tensors, the text encoder supplies high-level embeddings, ControlNet guides the T2V model, and the model predicts diffusion velocity.
  • Dynamic Frequency Loss: DF Loss adaptively constrains low- and high-frequency components at different diffusion steps to improve fidelity alongside the v-prediction objective.The method emphasizes low-frequency structure earlier and high-frequency details later, reflecting the denoising process.
  • Optimization: The total objective combines the v-prediction loss Lv with the dynamic-frequency loss LDF using b(t) = 1 − t/tmax.The paper sets tmax to 999 for balancing the two loss terms.
  • Local Information Enhancement Module: LIEM is inserted before global attention to emphasize local information and address degradation artifacts before global feature aggregation.The module is motivated by the importance of local details under complex real-world degradations.
  • Local Information Enhancement Module: LIEM uses average and max pooling within a local attention block, enriching local features and producing clearer, more detailed results.The paper adopts the local attention block from CBAM; local processing is followed by global feature aggregation.
  • Dynamic Frequency Loss: DF Loss estimates the restored latent, decodes it to pixel space, applies DFT, and separates low- and high-frequency components with a predefined pass filter.The resulting frequency components are used to formulate the frequency-aware loss, whose weighting function is c(t) = (t/tmax)^α.

4. Experiments

STAR is evaluated on synthetic and real-world datasets using fidelity, perceptual quality, clarity, and temporal-consistency metrics. Experiments show that LIEM placement, DF-loss design, and larger T2V priors improve restoration quality, while STAR achieves strong quantitative and qualitative results.

  • Datasets and Implementation: STAR is evaluated on synthetic UDM10, REDS30, and OpenVid30 datasets and the real-world VideoLQ dataset.Synthetic datasets use generated LR-HR pairs, while VideoLQ contains real-world videos without ground truth.
  • Datasets and Implementation: The evaluation measures PSNR, SSIM, LPIPS, ILNIQE, DOVER, and E*_warp across fidelity, perceptual similarity, quality, clarity, and temporal consistency.Synthetic datasets use reference-based metrics, whereas VideoLQ uses non-reference metrics.
  • Comparisons: On VideoLQ, STAR achieves the best DOVER score and second-best ILNIQE and E*_warp scores.These results indicate strong spatial and temporal quality on the real-world dataset.
  • Comparisons: STAR produces realistic spatial details and the best degradation removal in qualitative comparisons, including text structure, human hands, horse shape, and fur.Human evaluators also preferred STAR’s visual results on synthetic and real-world datasets.
  • Ablation Study: Position (i) gives the best LIEM results, while inserting LIEM into both spatial and temporal blocks performs best overall.The study attributes poorer results at positions (ii) and (iii) to excessive influence on the model mapping when most weights are frozen.
  • Ablation Study: Separating frequency components and prioritizing low-frequency reconstruction early yields the best perceptual quality while maintaining high fidelity.The adopted configuration uses linear b(t) and α = 2 for c(t).
  • Scaling Up with Larger T2V Models: Replacing I2VGen-XL with CogVideoX improves SSIM from 0.6944 to 0.7400 and DOVER from 0.6609 to 0.7350.The evaluation uses a new 480×720 test set cropped from 10 OpenVid-1M videos because CogVideoX supports that input resolution.

5. Conclusion

The paper concludes that STAR uses T2V diffusion priors with LIEM and DF Loss to restore real-world videos with fewer artifacts, higher fidelity, and stronger temporal consistency. Experiments show superior spatial and temporal quality and support the use of powerful T2V models for real-world VSR.

  • Conclusion: STAR restores real-world videos with fewer artifacts, higher spatial fidelity, and stronger temporal consistency.The framework combines T2V diffusion priors with modules targeting degradation handling and fidelity.
  • Conclusion: LIEM improves degradation handling and fine-detail reconstruction, while DF Loss guides frequency-specific restoration across diffusion steps.These components are integrated into the T2V backbone to enhance fidelity.
  • Conclusion: Experiments show that STAR achieves superior performance in both spatial and temporal quality.The conclusion presents this result as evidence for applying T2V models to real-world VSR.

A. Perception-Distortion Trade-Off

STAR’s DF Loss exposes a perception–distortion trade-off through β: larger β favors fidelity, while smaller β favors perceptual quality.

  • A. Perception-Distortion Trade-Off: β controls the balance between fidelity and perceptual quality in STAR’s DF Loss.The weighting function b(t) adjusts the relative influence of the v-prediction and DF losses.
  • A. Perception-Distortion Trade-Off: Increasing β strengthens DF-loss weighting across diffusion steps and enhances fidelity.The authors associate larger β with greater fidelity and report improved PSNR and E*warp.
  • A. Perception-Distortion Trade-Off: Decreasing β reduces DF-loss influence, allowing v-prediction to produce more perceptual results.Lower β is associated with reduced LPIPS, indicating better perceptual quality.
  • A. Perception-Distortion Trade-Off: The β ablation illustrates that higher values favor fidelity, whereas lower values emphasize perceptual quality.Figure 11 visualizes the effect of β on the b(t) weighting function.

B.1. User Study

A user study compared STAR with diffusion- and GAN-based VSR methods on real-world and synthetic datasets using evaluations from 12 participants.

  • B.1. User Study: The user study evaluated STAR against Upscale-A-Video, MGLD-VSR, and RealViformer on VideoLQ and REDS30.VideoLQ provided real-world videos, while REDS30 provided synthetic videos.

B.2. Qualitative Comparisons

Qualitative comparisons show that STAR preserves richer details and greater realism on synthetic and real-world datasets, with further gains from larger T2V models.

  • B.2. Qualitative Comparisons: STAR preserves richer details and achieves greater realism in synthetic and real-world qualitative comparisons.These comparisons are presented in Figures 13 and 14.
  • B.2. Qualitative Comparisons: Scaling up the T2V model further improves restoration effects in synthetic and real-world results.The authors identify large, robust T2V models as strong base models for video super-resolution.

B.3. Video Demo

The supplementary demo video showcases STAR’s spatial and temporal advantages across synthetic, real-world, and AIGC videos, alongside visual comparison figures.

  • B.3. Video Demo: The demo video presents additional spatial and temporal results on synthetic, real-world, and AIGC videos.It is provided in the supplementary material as STAR-demo.mp4.
  • B.3. Video Demo: Human evaluators preferred STAR for both visual quality and temporal consistency.The user-study outcome is summarized in Figure 12.
  • B.3. Video Demo: Qualitative figures report more detailed and realistic synthetic results and clearer facial details with more accurate text structure on real-world data.Figures 13 and 14 provide the corresponding synthetic and real-world comparisons.
  • B.3. Video Demo: Larger T2V models enhance detail and realism in video super-resolution results.Figure 15 compares synthetic and real-world results obtained with larger T2V models.
Loading 2501.02976v1…