PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Super Resolution Based on Deep Learning: A Comprehensive Survey

Hongying Liu, Zhubo Ruan, Peng Zhao, Chao Dong, Fanhua Shang, Yuanyuan Liu, Linlin Yang, Radu Timofte

arXiv:2007.12928v3cs.CVeess.IV

TL;DR

Low-resolution videos require methods that exploit relationships among successive frames while avoiding temporal incoherence. The survey reviews 33 deep-learning VSR methods, organizes them by inter-frame information use, and compares their benchmark performance. IconVSR and EDVR show superior performance across three reported datasets, while deployment cost and perceptual video-quality evaluation remain open challenges.

  • Problem

    Frame-by-frame super-resolution can introduce artifacts and temporal incoherence, motivating methods that exploit relationships among successive video frames.

  • Method

    The survey reviews 33 deep-learning VSR methods, proposes a taxonomy based on inter-frame information use, details architectures, and compares benchmark performance.

  • Results

    IconVSR and EDVR show superior performance on the Vimeo-90K-T, Vid4, and REDS4 datasets.

  • Takeaways & Limitations

    The taxonomy and benchmark comparison organize recent deep-learning VSR techniques and support method selection for online, offline, motion, and lighting-change settings.

  • Takeaways & Limitations

    Conventional recurrent methods can struggle with very long-term dependence, while complex LSTM designs limit depth on hardware.

Abstract

from arXiv · show

In recent years, deep learning has made great progress in many fields such as image recognition, natural language processing, speech recognition and video super-resolution. In this survey, we comprehensively investigate 33 state-of-the-art video super-resolution (VSR) methods based on deep learning. It is well known that the leverage of information within video frames is important for video super-resolution. Thus we propose a taxonomy and classify the methods into six sub-categories according to the ways of utilizing inter-frame information. Moreover, the architectures and implementation details of all the methods are depicted in detail. Finally, we summarize and compare the performance of the representative VSR method on some benchmark datasets. We also discuss some challenges, which need to be further addressed by researchers in the community of VSR. To the best of our knowledge, this work is the first systematic review on VSR tasks, and it is expected to make a contribution to the development of recent studies in this area and potentially deepen our understanding to the VSR techniques based on deep learning.

1 Introduction

Video super-resolution extends image super-resolution to successive frames, using inter-frame relationships to improve reconstruction while addressing temporal incoherence. This survey organizes deep-learning VSR methods by inter-frame information use, reviews their architectures, compares benchmarks, and discusses challenges.

  • Video super-resolution processes successive frames to exploit inter-frame relationships, whereas image super-resolution typically processes one image at a time.
  • Frame-by-frame image super-resolution can introduce artifacts and unwanted temporal incoherence across video frames.
  • Traditional VSR methods use explicit motion, blur, and noise models, but these models remain inadequate for fitting varied video scenes.
  • Deep-learning VSR methods use LR and HR video sequences for inter-frame alignment, feature extraction and fusion, and high-resolution sequence reconstruction.
  • The survey reviews deep-learning VSR progress, proposes an inter-frame-information taxonomy, compares benchmark performance, and discusses applications, challenges, and future perspectives.

2 Background

VSR reconstructs high-resolution video from multiple degraded low-resolution frames by combining spatial information within frames and temporal relationships across frames. The background defines the degradation and reconstruction processes and introduces PSNR and SSIM as quality measures.

  • VSR restores high-resolution videos from multiple low-resolution frames and typically exploits inter-frame information beyond single-image super-resolution.
  • A high-resolution frame has spatial dimensions scaled by factor s, with example scale factors s=2, 4 or 8.
  • The temporal neighborhood for center frame ˆIi contains 2N+1 high-resolution frames, where N is the temporal radius.
  • The degradation function models factors including noise, motion blur, and downsampling, while D and B denote down-sampling and blur operations and Ei→j denotes motion-based warping.
  • VSR estimates ground-truth high-resolution frames from degraded sequences using model parameters θβ, with quality commonly evaluated by PSNR and SSIM.

3 Video Super-resolution Methods

This section organizes deep-learning VSR methods by how they use inter-frame information, distinguishing alignment strategies and several non-alignment architectures. It also identifies trends toward diversified methods, longer-range dependencies, learned motion estimation, and more practical evaluation settings.

  • Taxonomy: The taxonomy categorizes VSR methods according to their use of inter-frame information, primarily separating explicit alignment from no alignment.The survey notes that feature-extraction, fusion, and reconstruction modules may overlap across categories.
  • Method categories: The taxonomy includes MEMC, deformable convolution, 2D convolution, 3D convolution, recurrent convolutional, non-local, and other method groups.Examples include BasicVSR, EDVR, DUF, RISTN, PFNL, RBPN, and STARnet across the listed categories.
  • Observed trends: Since 2018, VSR methods have diversified, with many methods emerging without alignment while both alignment-based and non-alignment methods can obtain sound performance.Earlier methods from 2015–2017 mostly used frame alignment.
  • Observed trends: Later methods expand receptive fields from local to global by processing longer sequences, bidirectionally propagating features, or computing non-local correlations.BasicVSR propagates features forward and backward, while PFNL computes correlations across all possible pixels within and across frames.
  • Observed trends: MEMC methods increasingly use deep learning to estimate optical flow, while evaluations move toward videos with more complex motions and scene changes.The cited methods include FRVSR, STTN, SOFVSR, TecoGAN, and MuCAN; test videos progressed from Vid4 and UVGD to REDS.

4 Methods with Alignment

Methods with alignment explicitly align neighboring frames with the target before reconstruction. They mainly rely on motion estimation and compensation or deformable convolution.

  • 4 Methods with Alignment: Methods with alignment explicitly align neighboring frames with the target frame before subsequent reconstruction.The alignment uses extracted motion information.
  • 4 Methods with Alignment: Motion estimation and motion compensation are common techniques for aligning frames in VSR.These methods form the MEMC category.
  • 4 Methods with Alignment: Deformable convolution is another common technique used to align neighboring frames before reconstruction.The survey introduces alignment-based methods according to their primary alignment technique.

4.1 Motion Estimation and Compensation Methods

Motion estimation and compensation methods align neighboring video frames with a target frame before deep networks extract, fuse, and reconstruct super-resolved results. The surveyed methods use traditional or learned motion processing, including optical flow, spatial transformation, and recurrent designs.

  • Core MEMC pipeline: Motion estimation extracts inter-frame motion, while motion compensation warps neighboring frames to align them with the target frame.Optical flow is a predominant motion-estimation technique, and compensation can use bilinear interpolation or spatial transformers.
  • Taxonomy: MEMC methods are categorized as traditional or deep-learning-based according to whether motion estimation or compensation uses a deep neural network.Deep-DE, VSRnet, and RRCN are classified as traditional MEMC methods in this taxonomy.
  • Representative methods: Deep-DE generates multiple super-resolution drafts from adjusted TV-ℓ1 flow and MDP, then feeds them with a bicubic-interpolated target frame into a CNN.Its CNN uses three convolutional layers followed by a deconvolution layer for feature extraction, fusion, and super-resolution.
  • Representative methods: VSRnet extends SRCNN from one input frame to multiple motion-compensated frames and adds FSE and adaptive motion compensation.These mechanisms are used to accelerate training and reduce the impact of unreliable compensated frames.
  • Representative methods: RRCN is a bidirectional recurrent network that learns a residual image while super-resolving only the middle frame of multiple consecutive inputs.Its forward and backward outputs are summed, and the model uses GLG-TV for motion estimation and compensation.
  • Representative methods: VESPCN uses a CNN-based coarse-to-fine MCT module for motion estimation and compensation, followed by convolutional feature fusion and sub-pixel upsampling.The coarse and fine stages progressively estimate optical flow before the compensated frames enter the reconstruction network.

4.1.5 DRVSR

The surveyed aligned VSR methods combine motion estimation and compensation with reconstruction, using designs such as SPMC, spatial transformation, recurrent processing, and task-specific flow learning. Their architectures differ in how they align frames, exploit temporal information, and reconstruct high-resolution outputs.

  • DRVSR: DRVSR combines motion estimation, SPMC-based motion compensation, and fusion to jointly upsample and compensate neighboring frames using optical flow.Its SPMC layer contains a grid generator and sampler that transform LR coordinates into HR coordinates and sample the warped data.
  • RVSR: RVSR aligns neighboring frames with a spatial transformation layer and adaptively weights super-resolution branches operating at different temporal scales.The temporal modulation module produces a weighted sum of branch outputs, with each branch handling a different number of input frames.
  • FRVSR: FRVSR reuses the previously inferred HR estimate to super-resolve the subsequent frame, targeting temporal consistency while reducing computational cost.The previous HR frame is warped using upsampled optical flow, transformed back to an LR representation, and combined with the target frame.
  • STTN: STTN introduces a spatio-temporal transformer that estimates flow across multiple frames to address inaccurate pairwise optical-flow estimation under occlusion and luminance variation.Its pipeline includes spatiotemporal flow estimation, a spatio-temporal sampler, and a super-resolution module.
  • SOFVSR: SOFVSR reconstructs high-resolution optical flow with OFRnet, converts it to LR flow, warps neighboring frames, and feeds them to SRnet.The optical flow is estimated through a coarse-to-fine approach before alignment and super-resolution reconstruction.
  • TOFlow: TOFlow jointly trains optical-flow estimation and reconstruction so the flow network is tailored to tasks including video super-resolution, interpolation, and deblurring.For video super-resolution, SpyNet, an STN, and a four-layer image-processing network produce the final result.
  • MMCNN: MMCNN aligns consecutive frames, fuses details with multi-memory processing, reconstructs a residual image, and adds it to an upsampled LR target frame.Its five major modules include optical flow, feature extraction, multi-memory detail fusion, feature reconstruction, and sub-pixel upsampling.

4.1.12 MEMC-Net

MEMC-based VSR methods align neighboring frames with a target frame using motion estimation and compensation, then exploit those aligned inputs for reconstruction. The section covers adaptive warping, multi-stage refinement, temporal aggregation, recurrent propagation, and limitations under difficult motion or illumination.

  • MEMC-Net: MEMC-Net aligns neighboring frames using an adaptive warping layer driven by estimated optical flow and convolutional kernels.The flow and kernels are produced by separate motion-estimation and kernel-estimation networks.
  • RTVSR: RTVSR estimates motion with a fully convolutional motion convolutional kernel estimation network and warps neighboring frames with learned 1D kernels.
  • MultiBoot VSR: MultiBoot VSR uses two stages, feeding the first-stage super-resolved output, after downsampling and concatenation with the initial LR frame, into the second stage.FlowNet 2.0 computes optical flow for motion compensation before the first stage.
  • TecoGAN: TecoGAN combines recurrent inputs, optical-flow-based warping, and adversarial training with a spatio-temporal discriminator and Ping-Pong loss.The discriminator evaluates generated and ground-truth sequences using consecutive HR, upsampled LR, and warped HR frames.
  • MuCAN: MuCAN aggregates temporal and cross-scale correspondences before reconstruction, using multi-candidate matching and non-local attention to collect context.Its architecture includes TM-CAM, CN-CAM, and a reconstruction module.
  • BasicVSR: BasicVSR uses backward and forward recurrent propagation modules plus upsampling, with coupling propagation directly transferring backward outputs into forward propagation.
  • Limitations: MEMC alignment is common but can degrade sharply when videos contain dramatic lighting changes or large or complex motions.Under such conditions, optical-flow assumptions can fail, producing inaccurate alignment, artifacts, and blurring.

4.2 Deformable Convolution Methods

Deformable-convolution VSR methods adapt feature sampling to handle geometric variation and difficult motion, with EDVR combining deformable alignment and temporal-spatial attention. The survey also notes computational and convergence costs, while recurrent designs and training choices can affect practical performance.

  • Deformable convolution: Deformable convolution learns offsets rather than using a fixed geometric sampling structure, improving modeling of geometric transformations and complex motion or illumination.The survey describes this operation as expanding and reshaping the convolutional receptive field.
  • EDVR: EDVR is identified as the champion model in the NTIRE19 Challenge.
  • EDVR: EDVR combines PCD alignment for large motions with TSA fusion for effectively combining multiple frames.Its pipeline aligns frames, fuses them, refines features, and upsamples a residual image.
  • Other methods: DNLN couples deformable-convolution alignment with non-local attention and cascaded deformable convolutions for inter-frame feature processing.The cascaded design is described as making alignment more accurate.
  • Other methods: TDAN applies deformable convolution to target and neighboring frames to obtain offsets that warp neighboring frames into alignment.
  • Other methods: D3Dnet uses 3D deformable convolution and residual deformable 3D blocks for spatio-temporal feature modeling, motion compensation, and spatial information capture.
  • Other methods: VESR-Net uses a feature encoder, fusion module, and reconstruction module, with channel-attention residual blocks, PCD alignment, and separate non-local processing.
  • Limitations: Deformable convolution remains computationally expensive and has harsh convergence conditions, leaving room for further improvement.The survey also notes that MEMC performance depends on architecture, training data, preprocessing, hyperparameters, and iteration count.

5 Methods without Alignment

Methods without alignment avoid explicit neighboring-frame alignment and instead learn spatial or spatio-temporal correlations directly. The survey groups them by extraction strategy and describes representative 2D, 3D, and specialized architectures.

  • 2D convolution methods: Without alignment, networks directly process input frames for feature extraction, fusion, and super-resolution, learning inter-frame correlations internally.
  • 2D convolution methods: VSRResFeatGAN uses a convolutional residual-block generator and adversarial training, with a convolutional discriminator.
  • 2D convolution methods: FFCVSR combines local and context networks, using unaligned LR frames and the previous HR output to restore details and maintain temporal consistency.
  • 3D convolution methods: 3D convolution methods extract temporal correlations directly; DUF also performs dynamic filtering, upsampling, and residual-map estimation without motion estimation or compensation.
  • 3D convolution methods: FSTRN factorizes 3D kernels to reduce computation, while 3DSRNet detects scene boundaries and replaces mismatched frames with temporally closest same-scene frames.
  • Specialized architectures: DSMC targets large motion with deformable convolutions, U3D-RDN, multi-stage communicated upsampling, and dual-subnet training.
  • Limitations: 3D methods capture spatio-temporal correlations but generally have higher computational complexity, limiting real-time deployment; RCNN and LSTM designs also face training or depth constraints.

6 Performance Comparisons

The survey compares representative ×4 VSR methods across Vimeo-90K-T, Vid4, and REDS4, while noting that differing training and preprocessing choices limit fairness. IconVSR and EDVR perform strongly across datasets, with architecture complexity and parameter count varying substantially.

  • Evaluation setup: ×4 comparisons use PSNR and SSIM under bicubic or blur-downsampling degradation, but differing training data, preprocessing, and crops may make direct comparisons unfair.
  • Dataset results: 37.84 PSNR is reported for IconVSR on Vimeo-90K-T with BD degradation, compared with 37.61 for EDVR under BI degradation.
  • Dataset results: 28.04 PSNR is reported for IconVSR on Vid4 with BD degradation, followed by RSDN at 27.92 and EDVR at 27.85.
  • Dataset results: 31.67 PSNR is reported for IconVSR on REDS4 with BI degradation, followed by EDVR at 31.09 and MuCAN at 30.88.
  • Cross-dataset results: IconVSR and EDVR show superior performance across Vimeo-90K-T, Vid4, and REDS4.
  • Architecture and efficiency: EDVR uses 20.60 MB parameters and a more complex alignment-and-fusion architecture, whereas several other high-performing methods use fewer than 10.0 MB.
  • Model selection: For large motions, the survey recommends IconVSR, EDVR, DSMC, RBPN, and PFNL, while deformable-convolution methods are described as more robust to alignment errors.

7 Applications of Video Super-Resolution

Video super-resolution is applied across remote sensing, medical imaging, surveillance, faces, multiview and panorama video, thermal imaging, and 3D reconstruction. These applications use domain-specific data, architectures, or objectives to recover detail and support downstream tasks.

  • Application scope: VSR supports remote sensing, medical diagnosis, video decoding, and 3D reconstruction by enhancing video-frame resolution and visual quality.
  • Remote sensing: Remote-sensing methods combine spatial and spatiotemporal neighbors with nonlocal similarity, preserving edges and texture details in satellite videos.
  • Medical imaging: Medical-video approaches reconstruct high-resolution MRI, CT, and cardiac MRI data using multiple slices, iterative feedback, or bidirectional ConvLSTM.
  • Surveillance and faces: Surveillance and license-plate studies use deep CNNs or SRGAN-based enhancement, with surveillance experiments focused mainly on indoor datasets.
  • Specialized video formats: Face-video, multiview, panorama, and 3D-video methods adapt motion feedback, kernel regression, weighted losses, or client-side reconstruction to their capture settings.
  • Thermal imaging: Thermal-video super-resolution has been used to improve sequences for respiratory-rate estimation and evaluated with several deep neural networks.

8 Trends and Challenges

The survey identifies deployment efficiency, interpretability, larger scale factors, realistic degradation, scene changes, perceptual evaluation, and better inter-frame information use as open VSR challenges. These issues constrain transfer from benchmark performance to real-world video.

  • Deployment: Large parameter counts, computational and storage demands, and long training times make efficient real-world deployment difficult, especially on mobile devices.
  • Interpretability: Existing models lack theoretical interpretations of how convolutional networks recover low-resolution video sequences, motivating deeper interpretability studies.
  • Scaling factors: Most work targets ×2, ×3, and ×4 magnification, while ×8 and ×16 remain rarely explored and become more difficult as scale increases.
  • Scaling factors: Fixed-scale models limit generalization and portability, creating a need for VSR methods supporting arbitrary scale factors.
  • Degradation and supervision: Bicubic downsampling and Gaussian blur do not adequately represent complex, uncertain real-world video degradation, and synthetic paired data can therefore remain inaccurate.
  • Scene changes: Scene changes often require segmenting videos into unchanged-scene portions, increasing computation; 3DSRNet offers an initial boundary-detection and frame-replacement solution.
  • Evaluation: PSNR and SSIM may not reflect human-perceived video quality, motivating evaluation criteria aligned with perception.
  • Inter-frame information: Effective inter-frame information use remains difficult because 3D convolution and non-local modules are computationally costly and optical-flow accuracy is not guaranteed.

9 Conclusions

The survey reviews deep-learning approaches for video super-resolution, organizes them by how they leverage video-frame information, and compares representative methods on benchmarks. It also identifies eight open issues for future VSR research.

  • The survey classifies existing VSR algorithms into seven subcategories according to how they leverage information contained in video frames.
  • It describes representative methods and summarizes each method’s advantages and disadvantages.
  • The survey compares and analyzes representative methods on benchmark datasets and outlines wide applications of VSR algorithms.
  • It lists eight open issues intended to support further development of VSR algorithms.

Author Biography

The author biographies identify researchers affiliated with Xidian University, the Shenzhen Institute of Advanced Technology, the University of Wurzburg, and ETH Zurich, with interests spanning machine learning, image processing, and low-level vision.

  • Hongying Liu is a faculty member at Xidian University’s School of Artificial Intelligence and a senior IEEE member.Her degrees include a Ph.D. in Engineering from Waseda University.
  • Zhubo Ruan researches machine learning and video super-resolution after receiving an M.Sc. from Xidian University.
  • Peng Zhao is pursuing an M.Sc. at Xidian University and researches video super-resolution and medical image processing.
  • Chao Dong is an associate professor at the Shenzhen Institute of Advanced Technology whose interests include image and video super-resolution, denoising, and enhancement.His team won first place in NTIRE2018, PIRM2018, and NTIRE2019 super-resolution challenges.
  • Fanhua Shang and Yuanyuan Liu are professors at Xidian University with backgrounds in circuits and systems and prior research at the Chinese University of Hong Kong.
  • Linlin Yang is pursuing a Ph.D. at Xidian University and researches image processing.
  • Radu Timofte is a computer-vision professor at the University of Wurzburg and a lecturer and group leader at ETH Zurich.He received a 2022 Humboldt Professorship Award for Artificial Intelligence.
Loading 2007.12928v3…