PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Video Super-Resolution using HR Optical Flow Estimation

Longguang Wang, Yulan Guo, Li Liu, Zaiping Lin, Xinpu Deng, Wei An

arXiv:2001.02129v1cs.CV

TL;DR

Video SR needs accurate temporal dependency, but LR optical flows can hinder recovery of fine temporal details in HR outputs. SOF-VSR jointly super-resolves optical flows and images, using HR flows for motion compensation before SR. The method reports improved SR accuracy and consistency, state-of-the-art performance, and a PSNR/SSIM increase over bilinear flow upsampling.

  • Problem

    Existing deep video SR methods estimate optical flows in LR space, limiting the accuracy of temporal dependency needed to recover fine details.

  • Method

    SOF-VSR uses OFRnet to infer HR optical flows coarse-to-fine, applies motion compensation with those flows, and feeds compensated LR frames to SRnet.

  • Results

    SOF-VSR achieves state-of-the-art video SR performance, while optical flow SR raises PSNR/SSIM from 25.85/0.765 with bilinear upsampling to 26.00/0.772.

  • Takeaways & Limitations

    Super-resolving optical flows provides accurate temporal details and improves both SR accuracy and temporal consistency.

Abstract

from arXiv · show

Video super-resolution (SR) aims at generating a sequence of high-resolution (HR) frames with plausible and temporally consistent details from their low-resolution (LR) counterparts. The key challenge for video SR lies in the effective exploitation of temporal dependency between consecutive frames. Existing deep learning based methods commonly estimate optical flows between LR frames to provide temporal dependency. However, the resolution conflict between LR optical flows and HR outputs hinders the recovery of fine details. In this paper, we propose an end-to-end video SR network to super-resolve both optical flows and images. Optical flow SR from LR frames provides accurate temporal dependency and ultimately improves video SR performance. Specifically, we first propose an optical flow reconstruction network (OFRnet) to infer HR optical flows in a coarse-to-fine manner. Then, motion compensation is performed using HR optical flows to encode temporal dependency. Finally, compensated LR inputs are fed to a super-resolution network (SRnet) to generate SR results. Extensive experiments have been conducted to demonstrate the effectiveness of HR optical flows for SR performance improvement. Comparative results on the Vid4 and DAVIS-10 datasets show that our network achieves the state-of-the-art performance.

I. INTRODUCTION

Video SR relies on temporal dependency between consecutive LR frames, but LR-space optical flows limit temporal-detail recovery. SOF-VSR addresses this by super-resolving optical flows and images in one end-to-end network, with HR flows used for motion compensation before SR.

  • Motivation: Video SR exploits adjacent frames for additional information, making temporal dependency important for reconstructing HR videos.Traditional patch-based methods capture pixel-level dependency at high computational cost.
  • Limitations of Prior Work: Existing deep video SR methods estimate optical flows from LR frames for motion compensation, but their temporal dependency accuracy remains limited.These methods then map compensated LR frames directly to HR outputs.
  • Proposed Approach: SOF-VSR jointly super-resolves optical flows and images in an end-to-end network to recover temporal and spatial details.The unified design makes optical-flow SR contribute to image SR performance.
  • Proposed Approach: OFRnet infers HR optical flows from LR frames in a coarse-to-fine manner, after which HR flows align latent HR frames through motion compensation.A space-to-depth transformation bridges the resolution gap between HR optical flows and LR frames before SRnet generates the HR frame.
  • Results: SOF-VSR reports state-of-the-art performance against recent video SR methods on the Vid4 and DAVIS-10 datasets.The paper also reports a lightweight architecture and additional analyses compared with its conference version.

A. Single Image SR

Prior video SR methods progressively adopted temporal modeling, motion estimation, and end-to-end architectures, while the paper identifies LR-space dependency accuracy as a remaining limitation. SOF-VSR instead recovers temporal details through HR optical flows alongside spatial details.

  • Video SR Development: Early multi-image and video SR methods used patch similarities or neighborhood regression to exploit spatio-temporal information.These approaches targeted complex motion through patch-wise similarities or kernels.
  • Video SR Development: Deep video SR methods commonly estimate optical flow, compensate motion, and reconstruct an HR frame from compensated inputs.This two-step framework was later integrated into end-to-end CNN architectures.
  • Video SR Development: End-to-end methods incorporated motion estimation and compensation, while recurrent and adaptive architectures aggregated temporal contexts or prior HR estimates.Other approaches avoided explicit motion estimation but could struggle with large displacements and complicated motions.
  • SOF-VSR: SOF-VSR differs by super-resolving optical flows to recover temporal details while also restoring spatial details.The paper states that HR optical flows provide accurate temporal dependency and contribute to spatial-detail restoration.

III. METHODOLOGY

SOF-VSR uses an end-to-end pipeline that reconstructs HR optical flows from LR frames, applies coarse-to-fine scale-recurrent processing, and feeds motion-compensated information to SRnet.

  • Overview: SOF-VSR first uses OFRnet to infer HR optical flows from LR frame pairs, then applies motion compensation and SRnet to reconstruct the HR central frame.The network processes consecutive LR frames and is described as an end-to-end system integrating optical-flow reconstruction, compensation, and super-resolution.
  • OFRnet: At each coarse-to-fine level, the preceding flow estimate contributes to generating a residual optical flow for the next level.The first two levels estimate flow at different input scales, while the third level reconstructs HR optical flow from LR feature representations.
  • OFRnet: OFRnet estimates optical flow across multiple scales, using recurrent modules at the first two levels and an SR module at the third level to recover HR flow.The scale-recurrent design shares parameters across scales and is intended to remain lightweight while handling complex motion, including large displacements.
  • OFRnet: The first two levels use feature extraction, efficient residual blocks, and flow-estimation layers, while the HR level adds residual processing and subpixel resolution enhancement.Channel split, channel shuffle, and depth-wise convolution are used in the efficient residual blocks.
  • OFRnet: OFRnet is presented as a unified network that infers HR optical flows between latent HR images directly from LR inputs.The paper distinguishes this approach from methods that interpolate LR inputs or LR flows, because interpolation does not add SR information.

C. Motion Compensation Module

The motion-compensation module converts HR optical flows into LR-resolution flow cubes, uses them to warp neighboring LR frames, and produces multiple drafts for SRnet.

  • Space-to-depth transformation: Space-to-depth transformation folds HR optical flows into LR grids and places the grids in the channel dimension to form LR flow cubes.The resulting flow cube has the same spatial resolution as the LR frames, and flow magnitude is divided by the scale factor s.
  • Warping: Slices from each LR flow cube warp a neighboring LR frame through bilinear interpolation, producing multiple warped drafts.The warped drafts are concatenated into a draft representation for subsequent super-resolution.
  • Temporal dependency: Although compensation operates on LR frames, HR optical flows allow the compensated frames to encode accurate temporal dependency.This operation bridges the resolution mismatch between HR flow estimates and LR image inputs.

D. Super-Resolution Net (SRnet)

SRnet concatenates the central LR frame with motion-compensated drafts and reconstructs the HR central frame through feature extraction, residual processing, and subpixel upsampling.

  • Input construction: SOF-VSR concatenates all motion-compensated drafts with the central LR frame and feeds the resulting draft cube to SRnet.The draft cube includes neighboring-frame drafts together with the central frame before HR reconstruction.
  • SRnet architecture: SRnet extracts features with a 3 × 3 convolution, processes them through eight efficient residual blocks, and applies a subpixel layer for resolution enhancement.A final 3×3 convolutional layer generates the HR frame.

E. Loss Function

The paper trains SRnet and OFRnet with separate loss terms, including MSE for SR and intermediate supervision for OFRnet, then combines them in a joint objective.

  • SRnet is trained using mean square error (MSE) loss.
  • OFRnet uses intermediate supervision at each level of its pyramid.
  • The joint training loss is L = LSR + λ4LOFR, with λ4 empirically set to 0.01 to balance the two terms.

IV. EXPERIMENTS

The experiments cover datasets, ablation studies, state-of-the-art comparisons, and a face-recognition task to evaluate SOF-VSR for low- and high-level vision.

  • The experiments introduce datasets and implementation details before evaluating SOF-VSR through ablation studies on Vid4.
  • SOF-VSR is compared with state-of-the-art methods on the Vid4 and DAVIS-10 datasets.
  • A face-recognition task is used to further evaluate the network for high-level vision tasks.

A. Datasets

The study trains on diverse 1080P video clips, validates on four Derf clips, and evaluates on Vid4 and DAVIS-10 using frame-quality and temporal-consistency metrics.

  • Datasets: Training uses 145 1080P HD video clips from the CDVL Database covering diverse natural and urban scenes.
  • Datasets: Four Derf clips—Coastguard, Foreman, Garden, and Husky—are used for validation.
  • Datasets: Vid4 is used for fair comparison with state-of-the-art methods, while a 10-clip DAVIS subset provides further comparison.
  • Evaluation: Evaluation uses PSNR and SSIM for individual-frame accuracy, T-MOVIE for temporal consistency, and MOVIE for overall video quality.
  • Ablation study: The ablation removes OFRnet or replaces HR optical flows with LR optical flows to test motion compensation and flow resolution.
  • Ablation study: Removing OFRnet lowers PSNR/SSIM from 26.00/0.772 to 25.70/0.753 and raises T-MOVIE from 19.35 to 20.03.
  • Ablation study: Using LR optical flows yields PSNR/SSIM of 25.85/0.765, below the SOF-VSR result with HR optical flows.

3) Upsampled Flow vs. Super-resolved Flow:

Super-resolved optical flows outperform simple upsampling by recovering finer temporal details, improving flow and warping accuracy across optical-flow and video datasets.

  • Upsampled Flow vs. Super-resolved Flow: Bilinear interpolation produces little change, with PSNR/SSIM of 25.85/0.765 versus 25.83/0.766, whereas optical flow SR reaches 26.00/0.772.
  • Upsampled Flow vs. Super-resolved Flow: On Vid4, warping with super-resolved flows lowers RMSE from 3.46 to 3.26 and raises PSNR from 29.51 to 30.05.
  • Upsampled Flow vs. Super-resolved Flow: Visual comparisons show that upsampled flows have distorted, blurred edges and artifacts, while super-resolved flows recover clearer edges and finer details.
  • Upsampled Flow vs. Super-resolved Flow: Vid4 error maps show fewer erroneous pixels with super-resolved optical flows, indicating recovery of finer temporal details.
  • Upsampled Flow vs. Super-resolved Flow: Across Sintel, Middlebury, KITTI 2012, KITTI 2015, and Vid4, the results demonstrate finer temporal-detail recovery than upsampling.

4) SISR before Optical Flow Estimation.:

The study compares separate SISR before flow estimation with a compact, integrated design, finding that scale recurrence and efficient residual blocks reduce cost with comparable performance.

  • SISR before Optical Flow Estimation.: The SISR-before-flow variant first super-resolves LR frames separately before optical-flow estimation, providing an alternative route to HR optical flows.The paper evaluates this design against its integrated network.
  • Scale-Recurrent Architecture: The scale-recurrent OFRnet reduces overall model size from 1.33M to 1.00M while maintaining comparable performance to the scale-cascaded architecture.The authors attribute the reduction to avoiding redundant parameters across similar scale levels.
  • Efficient Residual Blocks: Replacing efficient residual blocks with vanilla blocks increases model size from 1.00M to 1.56M and FLOPs from 108.90G to 143.14G.The vanilla variant achieves slightly better performance, but the proposed blocks avoid an obvious performance drop while reducing complexity.
  • Efficiency: The compact SOF-VSR design is presented as more suitable for mobile computing devices because it reduces model size and computational complexity.This conclusion follows the reported lightweight architecture comparisons.
  • Experimental Comparison: The evaluation compares SOF-VSR with single-image and video SR methods under BI and BD degradation models on Vid4 and DAVIS-10.The comparison includes retraining SOF-VSR as SOF-VSR-BD for the BD setting.

1) Evaluation on the Vid4 Dataset:

Across benchmark and downstream evaluations, SOF-VSR improves accuracy and temporal consistency while recovering finer visual details and supporting face recognition.

  • Vid4 Quantitative Evaluation: SOF-VSR achieves the best BI performance for 2× and 3× SR on Vid4, and for 4× SR it outperforms TDVSR-L in PSNR, SSIM, and MOVIE with halved parameters and FLOPs.It also achieves comparable performance to the conference version with 1.0M rather than 1.5M parameters.
  • Vid4 Accuracy and Consistency: SOF-VSR and SOF-VSR-BD achieve better PSNR and T-MOVIE performance on Vid4 while remaining lightweight and compact.T-MOVIE measures consistency, whereas PSNR measures frame accuracy in the described evaluation.
  • Vid4 Qualitative Evaluation: SOF-VSR and SOF-VSR-BD recover finer details such as the word “MAREE” and building stripes in Vid4 visual comparisons.The temporal profiles show that “MAREE” is difficult to recognize in several competing results.
  • DAVIS-10 Quantitative Evaluation: On DAVIS-10, SOF-VSR achieves state-of-the-art PSNR and SSIM under BI degradation, while its T-MOVIE performance is comparable or better than competing methods.The network retains strong performance with 1.0M rather than 1.5M parameters.
  • DAVIS-10 Quantitative Evaluation: Under BD degradation on DAVIS-10, SOF-VSR-BD improves over SPMC by 1.26 in PSNR and 3.15 in T-MOVIE.The dataset includes fast-moving objects and complex motion patterns, especially large displacements.
  • Qualitative and Downstream Evaluation: SOF-VSR produces clearer details with fewer artifacts than competing methods in DAVIS-10 visual comparisons and yields the highest top-1 and top-5 face-recognition accuracy under both degradation models.It outperforms CARN by 1.2% in top-1 accuracy and 1.7% in top-5 accuracy.

V. CONCLUSION

The paper concludes that super-resolving optical flows improves video SR by providing more accurate temporal dependency. SOF-VSR combines HR-flow reconstruction, motion compensation, and SR inference in an end-to-end network.

  • Conclusion: OFRnet super-resolves optical flows from LR inputs to provide accurate temporal dependency for video SR.The conclusion identifies this as the first stage of the proposed network.
  • Conclusion: Motion compensation uses HR optical flows before SRnet infers the final super-resolved frames.These stages form the paper’s end-to-end video SR pipeline.
  • Conclusion: Experiments show that SOF-VSR improves both SR accuracy and temporal consistency while achieving state-of-the-art performance against existing video SR methods.The conclusion attributes the improvements to recovering accurate temporal details.
Loading 2001.02129v1…