PaperlayerView exact revision on arXiv

Source-linked AI summary

Frame-Recurrent Video Super-Resolution

Mehdi S. M. Sajjadi, Raviteja Vemulapalli, Matthew Brown

arXiv:1801.04590v4cs.CVcs.AIstat.ML

TL;DR

Existing sliding-window video super-resolution repeatedly processes LR frames and estimates outputs independently, limiting efficiency and temporal consistency. FRVSR recurrently uses the previous HR estimate to process the next frame, and the paper reports higher-quality, more efficient results that significantly outperform competing state-of-the-art approaches.

  • Problem

    Sliding-window methods repeatedly process input frames and estimate outputs independently, increasing computational cost and limiting temporal consistency.

  • Method

    FRVSR is an end-to-end trainable recurrent framework that warps and passes the previous HR estimate into the next super-resolution step.

  • Results

    FRVSR significantly outperforms competing state-of-the-art video super-resolution approaches qualitatively and quantitatively on a standard benchmark dataset.

  • Takeaways & Limitations

    Recurrent HR propagation provides higher-quality video super-resolution while remaining more efficient than existing sliding-window approaches.

  • Takeaways & Limitations

    Occlusions irreversibly destroy previously aggregated details in affected areas, limiting those regions to the performance of single-image super-resolution models.

Abstract

from arXiv · show

Recent advances in video super-resolution have shown that convolutional neural networks combined with motion compensation are able to merge information from multiple low-resolution (LR) frames to generate high-quality images. Current state-of-the-art methods process a batch of LR frames to generate a single high-resolution (HR) frame and run this scheme in a sliding window fashion over the entire video, effectively treating the problem as a large number of separate multi-frame super-resolution tasks. This approach has two main weaknesses: 1) Each input frame is processed and warped multiple times, increasing the computational cost, and 2) each output frame is estimated independently conditioned on the input frames, limiting the system's ability to produce temporally consistent results. In this work, we propose an end-to-end trainable frame-recurrent video super-resolution framework that uses the previously inferred HR estimate to super-resolve the subsequent frame. This naturally encourages temporally consistent results and reduces the computational cost by warping only one image in each step. Furthermore, due to its recurrent nature, the proposed method has the ability to assimilate a large number of previous frames without increased computational demands. Extensive evaluations and comparisons with previous methods validate the strengths of our approach and demonstrate that the proposed framework is able to significantly outperform the current state of the art.

1. Introduction

Video super-resolution can exploit temporal information across LR frames, but sliding-window methods repeatedly process inputs and estimate outputs independently. FRVSR addresses these issues with recurrent HR-frame propagation, reducing computation while promoting temporal consistency and achieving strong reported results.

  • 1. Introduction: Video super-resolution combines temporal information from multiple LR frames to reconstruct details missing from individual frames.Temporal relationships provide information beyond the spatial statistics available in single-image super-resolution.
  • 1. Introduction: Sliding-window methods repeatedly process each input frame and independently estimate outputs, increasing computation and causing temporally inconsistent flickering.These methods divide video super-resolution into many separate multiframe super-resolution subtasks.
  • 1. Introduction: FRVSR recurrently passes the previously estimated HR frame to the next iteration instead of estimating every frame independently.The recurrent HR estimate propagates information from past frames and helps recreate fine details.
  • 1. Introduction: Processing each input frame only once reduces computational cost, while recurrent propagation supports temporally consistent video results.The framework is end-to-end trainable and does not require pre-training stages.
  • 1.1. Our contributions: FRVSR significantly outperforms current state-of-the-art video super-resolution methods qualitatively and quantitatively on a standard benchmark dataset.The experiments also compare quality and efficiency across varied settings and baselines.

2. Video super-resolution

Video super-resolution reconstructs HR frames from consecutive LR frames, building on single-image and classical multi-frame approaches. Related deep methods use motion compensation, recurrent designs, or frame aggregation, while FRVSR combines flow-based warping with recurrent HR estimation.

  • 2. Video super-resolution: Given consecutive LR video frames, video super-resolution aims to generate HR estimates approximating the original HR frames.The LR frames are obtained by downsampling HR video frames by scale factor s.
  • 2.1. Related work: Single-image super-resolution methods include interpolation, example-based reconstruction, dictionary learning, and self-similarity approaches.These approaches address the ill-posed inverse problem of reconstructing HR images from LR inputs.
  • 2.1. Related work: Video and multi-frame methods combine multiple LR frames to recover details missing from individual frames, although classical formulations are computationally expensive.Deep learning methods use convolutional networks and alternative loss functions to advance reconstruction quality.
  • 2.1. Related work: Most existing deep video methods estimate one HR output from multiple LR inputs, using optical-flow warping, trainable motion compensation, or joint warping and HR mapping.Some methods process batches of up to seven LR frames or aggregate separately estimated HR frames with dynamic weights.
  • 2.1. Related work: Prior recurrent approaches use bidirectional processing, feature propagation, or previous output frames, with some video methods commonly using component-wise pre-training.FRVSR instead uses explicit motion compensation and an end-to-end trainable recurrent framework.
  • 2.1. Related work: FRVSR computes optical flow in LR space, upsamples it to HR space, warps the previous HR estimate, and feeds the result with the current LR frame to SRNet.Training applies a super-resolution loss through SRNet and FNet, plus an additional loss on the warped previous LR frame to aid FNet.

3. Method

FRVSR recurrently combines the current LR frame with a warped previous HR estimate, using optical flow and LR-space processing to generate each HR output. The framework is end-to-end trainable, propagates information through time, and processes videos frame by frame.

  • Framework: FNet estimates LR optical flow between consecutive inputs, which is upscaled and used to warp the previous HR estimate onto the current frame.The warped result is then mapped to LR space before super-resolution.
  • Framework: The space-to-depth transformation extracts shifted LR grids from the warped HR image and places them in the channel dimension.This mapping lets the computationally intensive processing remain in LR space.
  • Framework: SRNet concatenates the LR-mapped warped estimate with the current LR input and produces the final HR estimate.Both FNet and SRNet are fully convolutional; the illustrated 4x models operate in LR space.
  • Training: The training objective combines super-resolution loss with an auxiliary flow loss based on spatial mean squared error between warped LR inputs.The super-resolution loss backpropagates through both SRNet and FNet.
  • Efficiency: Each input frame is processed once, while recurrence propagates information across many frames without increasing computational demands.The fully convolutional networks can run frame by frame on videos of arbitrary size and length.
  • Framework: FRVSR passes the previous HR estimate into the next iteration instead of estimating each frame independently.This recurrent design propagates information from past frames through the HR estimate.
  • Training and inference: FRVSR is trained end to end through time on 10-frame clips, with the first previous HR estimate initialized as a black image.The first frame is independently upsampled, encouraging single-image upsampling during training.

4. Evaluation

The evaluation compares FRVSR with matched SISR and sliding-window VSR baselines under 4x upsampling, using YT10 and Vid4. FRVSR is reported to provide the strongest quality-efficiency results across the evaluated settings.

  • Baselines: The evaluation uses two baselines built from the same optical-flow and super-resolution networks for a fair comparison.SISR removes optical flow and prior information, while VSR uses a fixed sliding window of input frames.
  • Protocol: The experiments evaluate 4x upsampling on ten 3–5s 1080p YT10 clips and compare against state-of-the-art methods on Vid4.Video PSNR is computed on the brightness channel using mean squared error over all video pixels.
  • Efficiency: VSR is computationally more expensive than FRVSR because it runs FNet twice for each output frame.The SRNet computation is described as almost identical between the two models.
  • Blur robustness: FRVSR gives the best results for every tested blur size on YT10.The best FRVSR PSNR uses σ = 1.5.
  • Blur robustness: 1.00 dB and 0.39 dB higher PSNR are reported for FRVSR than the best SISR and VSR results, respectively.The comparison is FRVSR at σ = 1.5 against SISR at σ = 2.0 and VSR at σ = 1.5.

4.2. Blur size

FRVSR remains robust across blur sizes, outperforming SISR and VSR, while training-clip experiments show PSNR gains largely saturate beyond five frames.

  • Blur robustness: FRVSR significantly outperforms SISR and VSR across Gaussian blur sizes from σ = 0 to σ = 5.Smaller blur kernels can cause aliasing, whereas excessive blur removes high-frequency input information and makes fine-detail reconstruction harder.
  • Training clip length: 31.60, 32.01 and 32.10 average video PSNR result from training clip lengths of 2, 5 and 10 on YT10, respectively.The same recurrent model was trained with each clip length.
  • Training clip length: PSNR has already started to saturate with a clip length of 5, and going beyond 10 may not yield significant improvements.This indicates diminishing gains from substantially longer training clips under the tested settings.

4.4. Degraded inputs

Under noisy and compressed inputs, FRVSR consistently achieves higher PSNR than SISR and VSR, while temporal profiles show sharper, more temporally consistent results than either baseline.

  • Degraded inputs: FRVSR consistently outperforms SISR and VSR under varying Gaussian noise and JPEG compression levels.On YT10, the gains are 0.36–0.91 dB over SISR and 0.18–0.48 dB over VSR.
  • Degraded inputs: 0.36–0.91 dB and 0.18–0.48 dB are FRVSR’s reported YT10 gains over SISR and VSR, respectively.These comparisons cover the tested noise and JPEG-compression conditions.
  • Temporal consistency: FRVSR produces the most temporally consistent results while retaining finer details, whereas VSR is sharper than SISR but still flickers.Temporal-profile jitter and jagged lines indicate flickering artifacts.

4.6. Range of information flow

FRVSR propagates information across many frames without increasing computation, with performance stabilizing in normal operation and extending beyond 30 frames in a range test.

  • Comparison with fixed windows: Existing fixed-window approaches face a tradeoff: more input frames can improve quality but substantially increase computational cost.Their fixed input windows also limit the maximum range over which details can be propagated.
  • Frame range: FRVSR’s normal-mode PSNR improves as more frames are processed, then plateaus at 12 frames.The normal mode uses a black frame as the first previous HR estimate.
  • Frame range: With the first previous HR estimate replaced by ground-truth HR, FRVSR performs better than normal mode even after 50 frames.The red curve indicates propagation of high-frequency details across a large number of frames.
  • Information flow: FRVSR propagates information over more than 30 frames when initialized at different frames in the same Foliage clip.The performance gap between starts at frame 1 and frame 11 closes only toward the clip’s end.

4.7. Network size and computational efficiency

FRVSR achieves better results than VSR with smaller super-resolution networks and lower computation, while its tested Full HD 4x-upscaling runtimes vary with model size.

  • Network size: FRVSR achieves better results than VSR despite being faster as network size varies.The comparison uses different numbers of residual blocks and convolution filters in SRNet.
  • Network size: FRVSR with 5 residual blocks is both faster and better than VSR with 10 residual blocks.The figure reports this as an example of FRVSR’s quality-efficiency advantage with smaller networks.

4.8. Comparison with prior art

On Vid4, FRVSR achieves strong quantitative and visual results against prior video super-resolution methods, including with its smaller model. The larger FRVSR 10-128 model recovers the most accurate images and reaches the strongest reported comparison.

  • More than 0.5 dB separates FRVSR from the current state of the art, while FRVSR 3-64 beats all previous neural network-based methods by SSIM.FRVSR 3-64 remains close to the much larger VSR 10-128 model on Vid4.
  • FRVSR 3-64 recovers sharper, more detailed foliage than prior art and produces results similar to the much bigger VSR model.The visual comparison is reported on the Foliage sequence from Vid4.
  • FRVSR 10-128 recovers the most accurate image in the visual comparison.The model uses 10 residual blocks and 128 convolution filters.
  • FRVSR 3-64 produces better PSNR results than all prior art on Vid4, including Liu and Sun’s optimization-based method.The comparison uses average PSNR and SSIM on Vid4 at scaling factor s = 4.

5. Future work

The framework can propagate reconstructed details over many frames, but its single HR-estimate memory loses information when occlusions affect previously aggregated details. Future extensions include periodic HR-frame inputs, additional memory, and more advanced losses.

  • Occlusions can irreversibly destroy previously aggregated details because the HR estimate is the only channel passed to future iterations.In affected regions, the model can perform only as well as single-image super-resolution, unlike fixed-window methods that can use non-occluded frames.
  • An additional static or motion-compensated memory channel is a natural extension, although preliminary experiments did not improve overall performance.The proposed memory extension targets information that cannot be retained in the HR estimate alone.
  • Periodic insertion of the original HR frame every K-th frame could support video compression by storing only one in K HR frames.The remaining frames would be reconstructed by the model.
  • More advanced loss terms are a further direction because the recurrent architecture naturally encourages temporally consistent results.The paper identifies this combination as suitable for further research.

6. Conclusion

The paper concludes that FRVSR is a flexible, end-to-end trainable video super-resolution framework that produces higher quality results more efficiently than sliding-window approaches and outperforms competing methods across settings.

  • FRVSR generates higher-quality video super-resolution results more efficiently than existing sliding-window approaches.The conclusion describes the framework as flexible and end-to-end trainable.
  • Extensive experiments show that FRVSR outperforms competing baselines in various settings and state-of-the-art methods quantitatively and qualitatively on a standard benchmark.
Loading 1801.04590v4…