PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, Wenzhe Shi

arXiv:1611.05250v2cs.CV

TL;DR

Existing video SR methods either fail to exploit inter-frame redundancy in real time or incur substantial computational costs from upsampling and motion compensation. The paper develops spatio-temporal sub-pixel convolution networks and an end-to-end trainable spatial-transformer motion-compensation scheme, reporting improved accuracy, temporal consistency, and state-of-the-art efficiency.

  • Problem

    Existing HD video SR methods do not effectively exploit temporal correlations while operating in real time, and prior joint-frame methods can be extremely slow.

  • Method

    The paper combines sub-pixel convolution with early fusion, slow fusion, and 3D-convolution spatio-temporal models, plus multi-scale spatial-transformer motion compensation trained end-to-end.

  • Results

    The proposed algorithms surpass the current state of the art in video SR in both accuracy and efficiency, while explicit motion compensation improves PSNR and temporal consistency over spatio-temporal models alone.

  • Takeaways & Limitations

    Spatio-temporal processing can improve reconstruction accuracy and temporal consistency or reduce computational complexity relative to independent single-frame processing.

  • Takeaways & Limitations

    The data contain motion blur and disocclusions, which are not explicitly modeled but could potentially improve results if incorporated.

Abstract

from arXiv · show

Convolutional neural networks have enabled accurate image super-resolution in real-time. However, recent attempts to benefit from temporal correlations in video super-resolution have been limited to naive or inefficient architectures. In this paper, we introduce spatio-temporal sub-pixel convolution networks that effectively exploit temporal redundancies and improve reconstruction accuracy while maintaining real-time speed. Specifically, we discuss the use of early fusion, slow fusion and 3D convolutions for the joint processing of multiple consecutive video frames. We also propose a novel joint motion compensation and video super-resolution algorithm that is orders of magnitude more efficient than competing methods, relying on a fast multi-resolution spatial transformer module that is end-to-end trainable. These contributions provide both higher accuracy and temporally more consistent videos, which we confirm qualitatively and quantitatively. Relative to single-frame models, spatio-temporal networks can either reduce the computational cost by 30% whilst maintaining the same quality or provide a 0.2dB gain for a similar computational cost. Results on publicly available datasets demonstrate that the proposed algorithms surpass current state-of-the-art performance in both accuracy and efficiency.

1. Introduction

Video super-resolution must recover high-resolution data from ill-posed low-resolution inputs while exploiting temporal redundancy without sacrificing real-time efficiency. This paper combines spatio-temporal sub-pixel networks with jointly trainable spatial-transformer motion compensation to improve accuracy and temporal consistency efficiently.

  • Motivation: Video SR extends image SR by using temporal redundancy across frames, but motion compensation is often expensive or requires costly preprocessing.Existing approaches either process frames independently or use inefficient optical-flow and preprocessing pipelines.
  • Motivation: ESPCN reduces runtime by learning the mapping directly from low-resolution to high-resolution space instead of first applying bicubic upsampling.This efficiency enables real-time video SR when frames are processed independently, but that naive strategy misses inter-frame redundancies.
  • Contributions: The paper studies early fusion, slow fusion and 3D convolutions as alternative architectures for discovering spatio-temporal correlations.These architectures jointly process multiple consecutive frames rather than treating each frame independently.
  • Contributions: It proposes dense inter-frame motion compensation using a multi-scale spatial transformer network integrated with the SR model.Spatial transformers provide differentiable spatial mappings that can encode optical flow and be jointly trained with the SR objective.
  • Contributions: The combined approach is end-to-end trainable and targets efficient motion-compensated video SR with improved accuracy and temporal consistency.The paper reports this combination as a real-time approach based on sub-pixel convolution and spatio-temporal networks.

2. Methods

The methods combine sub-pixel convolution with spatio-temporal networks to process consecutive video frames efficiently, and add jointly trainable spatial-transformer motion compensation. The architectures include early fusion, slow fusion, and weight-shared 3D-convolution equivalents, with a multi-scale flow design for aligning frames.

  • 2.1. Single-image super-resolution: Sub-pixel convolution performs all convolutional processing in low-resolution space before rearranging outputs into the high-resolution image.The network learns the upscaling operation and avoids bicubic preprocessing, reducing computational cost.
  • 2.2. Spatio-temporal networks: Spatio-temporal networks replace a single low-resolution input with consecutive frames and extend convolutional filters with a temporal dimension.This work reconstructs one central output frame from each input block.
  • 2.2. Spatio-temporal networks: Early fusion collapses all temporal information in the first layer, whereas slow fusion merges smaller frame groups hierarchically through the network.Early fusion is a special case of slow fusion, and the paper illustrates early fusion with five input frames.
  • 2.2. Spatio-temporal networks: Weight sharing across the temporal dimension allows online reuse of prior activations and is equivalent to using 3D convolutions.In 3D convolutions, temporal information is merged through convolutions in both space and time.
  • 2.3. Spatial transformer motion compensation: The motion-compensation module estimates dense pixel-wise optical flow with bilinear interpolation and warps a target frame onto a reference frame.A multi-scale design first estimates coarse flow, then refines it using the warped image, coarse flow, and original images.
  • 2.3. Spatial transformer motion compensation: Spatial-transformer and super-resolution modules are differentiable, enabling joint end-to-end optimization of motion compensation and video super-resolution.The same motion parameters can model the outer frames relative to the central frame in a three-frame block.

3. Experiments and results

The experiments evaluate spatio-temporal video super-resolution under controlled computational complexity, including operation counts and steady-state reuse for slow fusion networks.

  • Experimental design: Experiments target ×3 and ×4 upscaling of uncompressed full HD video at 1080×1920 resolution.The study compares architectures while controlling network parameters so performance gains reflect design choices rather than increased capacity.
  • Experimental design: Comparable network sizes are enforced so accuracy improvements can be attributed to resource allocation choices rather than trivial capacity increases.The number of network parameters is treated as a fairness constraint across methods.
  • Complexity analysis: Per-frame computational complexity is approximated by counting floating-point operations for each network layer.The analysis introduces a layer-wise operation estimate for reconstructing a frame.
  • Complexity analysis: Slow fusion complexity is measured in steady-state operation, where reused layer outputs carry across consecutive frames.This accounts for feature reuse when assessing per-frame efficiency.

3.1. Experimental setup

The study trains and evaluates models on uncompressed full HD video data, using a fixed training protocol and architecture settings designed for controlled comparisons.

  • Data: The CDVL dataset contains 115 uncompressed full HD videos, with 100 used for training and 15 reserved for testing.Training extraction produces 3000 samples from 100 videos, with 5% used for validation.
  • Data: Training samples are single frame pairs for single-frame models and consecutive low-resolution frame blocks with a central high-resolution target for spatio-temporal models.This sample definition varies with the network architecture.
  • Training protocol: All models share the same training protocol and similar hyperparameters, including 3×3 filters, rectified linear hidden layers, and linear output activation.Biases use zero initialization and weights use orthogonal initialization.
  • Training protocol: Models are optimized with Adam at learning rate 10^-4 using spatial subsamples that begin with batch size 1 and grow to 128.The batch size doubles every 10 epochs until reaching its maximum.
  • Evaluation: Figure 4 compares CDVL ×3 super-resolution for single-frame models with early-fusion models using 3, 5, or 7 frames.The comparison is organized around SF and E3–E7 architectures.
  • Architecture control: Hidden-layer feature counts are adjusted according to temporal depth to maintain comparable network sizes across fusion architectures.The stated setting uses 24 features for temporal depth 1 and scales features by 24/D_l for deeper temporal layers.

3.2. Spatio-temporal video SR

Spatio-temporal architectures improve video SR accuracy over independent single-frame processing, while fusion strategy determines the trade-off between operations and reconstruction quality.

  • Input-frame analysis: Spatio-temporal correlations produce more accurate results than independently processing frames, with only a marginal complexity increase for early fusion.The comparison covers single-frame models and early fusion with 3, 5, and 7 input frames.
  • Input-frame analysis: E7 slightly underperforms because dependencies beyond 5 frames may be too complex to learn and may introduce noise.The passage presents this as a likely explanation rather than a demonstrated causal mechanism.
  • Early versus slow fusion: Early fusion achieves higher accuracy with a marginal 3% increase in operations relative to single-frame models.Slow fusion is faster because it uses fewer features in its initial layers.
  • Early versus slow fusion: A 9-layer slow-fusion network matches early-fusion accuracy while reducing complexity by about 30%.The passage contrasts this with a considerable accuracy decrease for the 7-layer slow-fusion network.
  • Weight sharing: Weight sharing reduces accuracy but also lowers per-frame operations through feature reuse.A 7-layer shared-weight slow-fusion network reduces operations by almost 30% with a minimal accuracy decrease relative to single-frame processing.
  • Weight sharing: Early and slow fusion consistently outperform shared-weight slow fusion in the reported performance–efficiency trade-off.The comparison also reports that 7-layer early fusion can outperform and operate faster than 9-layer shared-weight slow fusion.

3.3. Motion compensated video SR

The motion-compensated model jointly aligns consecutive frames and performs super-resolution, producing structurally improved reconstructions over spatio-temporal super-resolution without compensation.

  • Motion compensation: Figure 5 compares three-frame sections without and with motion compensation, showing less pronounced error maps after compensation.Its top panel displays the estimated flow map relating consecutive frames.
  • Results: Figure 6 reports structurally more accurate reconstructions when motion compensation and super-resolution are jointly applied than with spatio-temporal super-resolution alone.The comparison is between the motion-compensated result and the uncompensated spatio-temporal result.
  • Motion compensation: The motion-compensation module estimates flow maps for outer frames relative to the middle frame in three consecutive-frame inputs.The module is first trained independently before being combined with the super-resolution network.
  • Joint optimization: The E3-MC model is initialized from separately pretrained compensation and super-resolution networks before joint optimization.The full model is then optimized using the combined objective.
  • Results: E3-MC achieves PSNR improvements over E3 that are sometimes almost twice the improvement of E3 relative to SF on CDVL ×3 SR.The authors attribute this to adapting the super-resolution input to maximize temporal redundancy.

3.4. Comparison to state-of-the-art

On Vid4, VESPCN delivers stronger spatial and temporal reconstruction than the compared methods while retaining substantial efficiency advantages from sub-pixel convolution and efficient motion compensation.

  • Quality comparison: VESPCN surpasses the other compared methods in PSNR and SSIM, while its temporally coherent reconstruction significantly reduces the MOVIE index.The comparison includes SRCNN, ESPCN, VSRnet, and VESPCN variants on Vid4.
  • Evaluation protocol: The reported values may slightly deviate from original papers because the evaluation uses an own SSIM implementation and video PSNR instead of averaged per-frame PSNR.Borders and initial and final frames were also cropped for fair comparison against VSRnet 2.
  • Efficiency comparison: VSRnet’s reported comparison excludes motion-compensation operations, so its Table 4 complexity does not include that computational bottleneck.The table caption explicitly notes this exclusion.
  • Quality comparison: VESPCN produces visually accurate reconstructions in both spatial detail and temporal profiles over 25 consecutive frames.Figure 7 compares ×3 super-resolution results and temporal profiles sampled along a dashed image row.
  • Efficiency comparison: 20% fewer operations than ESPCN is achieved by VESPCN 5L-E3 while maintaining similar performance across all evaluated quality metrics.Sub-pixel convolution reduces operations relative to architectures that upscale low-resolution inputs before CNN processing.
  • Efficiency comparison: 10^3 slower than VESPCN 9L-E3-MC is the reported motion-compensation runtime of VSRnet relative to the proposed method.VESPCN motion compensation requires 3.6 GOps for ×3 and 2.0 GOps for ×4 upscaling, applied twice for each input frame requiring compensation.

4. Conclusion

The paper combines sub-pixel convolution with temporal fusion and jointly trainable spatial-transformer motion compensation for real-time video super-resolution. The resulting models improve accuracy and temporal consistency or reduce complexity, while motion-compensated approaches outperform spatio-temporal models alone and the current state of the art.

  • Conclusion: The proposed real-time models combine sub-pixel convolutions, temporal fusion strategies, and spatial-transformer motion compensation for video super-resolution.The study examines early fusion, slow fusion, and 3D convolutions for temporal processing.
  • Conclusion: Spatio-temporal models improve reconstruction accuracy and temporal consistency or reduce computational complexity relative to independent single-frame processing.
  • Conclusion: Motion-compensated approaches outperform spatio-temporal models alone in PSNR and temporal consistency and surpass current video SR state-of-the-art performance.
Loading 1611.05250v2…