PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Spatio-Temporal Residual Network for Video Super-Resolution

Sheng Li, Fengxiang He, Bo Du, Lefei Zhang, Yonghao Xu, Dacheng Tao

arXiv:1904.02870v1cs.CV

TL;DR

Video SR needs to use temporal information, but 3D convolutions can make networks too computationally expensive to deepen. FSTRN uses factorized spatio-temporal residual blocks and LR-to-HR cross-space residual learning; the authors report that it significantly outperforms current state-of-the-art SR methods.

  • Problem

    Frame-by-frame SR ignores temporal correlations and can yield inconsistent video, while 3D convolutions add computation that restricts network depth.

    Source: §1 · pp. 1–2

  • Method

    FSTRN combines factorized 3D spatio-temporal residual blocks with LR-space and cross-space residual learning linking LR inputs to HR outputs.

    Source: §1 · p. 2

  • Results

    The authors report that FSTRN significantly outperforms current state-of-the-art SR methods in benchmark evaluations.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The proposed design is reported to combine computational efficiency with improved video SR and a generalization guarantee.

    Source: §1 · p. 2, §6 · p. 8

    The theoretical bound assumes stated Lipschitz and spectral-norm bounds on network components and weights.

    Source: §4 · p. 5

Abstract

from arXiv · show

Recently, deep learning based video super-resolution (SR) methods have achieved promising performance. To simultaneously exploit the spatial and temporal information of videos, employing 3-dimensional (3D) convolutions is a natural approach. However, straight utilizing 3D convolutions may lead to an excessively high computational complexity which restricts the depth of video SR models and thus undermine the performance. In this paper, we present a novel fast spatio-temporal residual network (FSTRN) to adopt 3D convolutions for the video SR task in order to enhance the performance while maintaining a low computational load. Specifically, we propose a fast spatio-temporal residual block (FRB) that divide each 3D filter to the product of two 3D filters, which have considerably lower dimensions. Furthermore, we design a cross-space residual learning that directly links the low-resolution space and the high-resolution space, which can greatly relieve the computational burden on the feature fusion and up-scaling parts. Extensive evaluations and comparisons on benchmark datasets validate the strengths of the proposed approach and demonstrate that the proposed network significantly outperforms the current state-of-the-art methods.

1. Introduction

Video SR must use temporal information to avoid the inconsistencies of frame-by-frame methods, but 3D convolutions make deep networks computationally costly. FSTRN combines factorized spatio-temporal blocks with LR- and cross-space residual learning; the paper reports stronger benchmark performance and a generalization bound independent of network size.

1. Introduction

  • Frame-by-frame SR ignores temporal correlations and can produce inconsistent HR video with spurious flickering, while 3D convolutions increase computation enough to restrict network depth.Existing residual connections in HR space raise computational complexity; LR-space residuals burden feature fusion and upscaling.Source: §1 · pp. 1–2
  • FSTRN significantly outperforms current state-of-the-art SR methods in benchmark evaluations and has a generalization bound O(1/√n) with no explicit dependence on network size.The Dancing video ×4 comparison also reports the highest PSNR and SSIM, finest texture, and fewest artifacts.Source: §1 · p. 2, Figure 2 · p. 2
  • FRB factorizes each 3D filter into two lower-dimensional 3D filters, reducing computation while supporting deeper networks.The proposed block is compared with an EDSR residual block and a single C3D residual block in Figure 1.Source: Figure 1 · p. 1, §1 · p. 2
  • Global residual learning combines LR-space residual learning with cross-space residual learning to exploit similarities between LR inputs and HR outputs.CRL directly links LR and HR spaces, using LR video as an anchor for spatial information in the HR output.Source: §1 · pp. 2–3

2. Related work

Single-image SR work developed CNN mappings, residual learning, and efficient feature upscaling. Video SR methods extend image SR with temporal fusion or recurrent approaches, while 3D convolutions offer a way to learn spatio-temporal information jointly.

2.1. Single-image SR with CNNs

  • CNN-based single-image SR progressed from SRCNN's end-to-end LR-to-HR mapping to deeper residual architectures that improved SR performance.The related work also describes efficient sub-pixel and transposed-convolution layers for upscaling features at the network end.Source: §2.1 · p. 3
  • EDSR, SRResNet, SRDenseNet, and RDN use LR-space residual connections, which aid feature extraction but burden upscaling and feature fusion.The cited methods include [28], [26],, and [51].Source: §2.1 · p. 3

2.2. Video SR with CNNs

  • Video SR methods use sliding windows, motion compensation, recurrent networks, or 3D convolutions to handle temporal information and consistency.C3D can flexibly access varying temporal contexts, but prior C3D-based BRCN remained shallow; this work targets a deep end-to-end network with high computational efficiency.Source: §2.2 · p. 3

3. Fast spatio-temporal residual network

FSTRN combines low-resolution spatio-temporal feature extraction with residual learning across low- and high-resolution spaces. Its factorized residual block reduces the computational cost of 3D convolutions, and training focuses on center-frame reconstruction.

3.1. Network structure

  • FSTRN combines shallow LR feature extraction, FRBs, LR feature fusion and up-sampling, and global residual learning with LRL and CRL.The shallow features feed the FRBs; their outputs are fused and upsampled, with residual learning linking features and spaces.Source: §3.1 · pp. 3–4

3.2. Fast spatio-temporal residual blocks

  • FRB factorizes a k×k×k 3D filter into successive 1×k×k spatial and k×1×1 temporal filters, reducing computation.The block also uses a skip connection and PReLU.Source: §3.2 · p. 4

3.3. Global residual learning

  • Global residual learning combines LR-space residual learning with CRL, which adds a mapped LR video to the LSRNet output in HR space.The LR-to-HR mapping can use bilinear, nearest, bicubic, area, or deconvolution-based interpolation.Source: §3.3 · pp. 4–5

3.4. Network learning

  • Training uses an L1 objective approximated with the Charbonnier penalty, with the loss focused mainly on reconstructing the input sequence’s center frame.The network produces the same frames as the input, but the work focuses on center-frame reconstruction; ε is set to 1e−3.Source: §3.4 · p. 5

4. Theoretical analysis

The theoretical analysis bounds the complexity of FSTRN’s hypothesis space and its generalization error. The stated bound’s factors do not explicitly depend on network size.

Theoretical analysis

  • The analysis derives a covering bound for FSTRN’s hypothesis space and uses it to obtain an upper bound on generalization error.The paper describes the covering bound as constraining hypothesis-space complexity and the generalization bound as a theoretical guarantee.Source: §4 · p. 5
  • Theorem 2 states a generalization bound that holds with probability at least 1 − δ for any hypothesis Fθ.The paper says the result follows from Theorem 1.Source: §4 · p. 6
  • The bound’s factors do not explicitly depend on neural-network size, which the paper presents as supporting the result’s meaningfulness and FSTRN’s generalization ability.The analysis assumes bounded spectral norms for specified weight matrices and bounded differences from reference matrices.Source: §4 · pp. 5–6

5. Experiments

Experiments evaluate FSTRN’s efficiency, component behavior, and benchmark performance for ×4 video super-resolution. The results report lower FRB computation, improved convergence with the proposed components, and better benchmark metrics than competing methods.

5.1. Settings

  • The experiments use 25 training videos and evaluate ×4 SR on challenging test videos with PSNR and SSIM measured on the luminance channel.Augmentation and volume cropping yield 13,020 training volumes; inputs are smoothed with a Gaussian filter and downsampled bicubically.Source: §5.1 · p. 6

5.2. Study of FRB

  • For 64-channel feature maps and five 32 × 32 input frames, FRB reduces parameters and calculations by more than half versus C3DRB.Table 2 compares one FRB with one residual block built from a single C3D.Source: §5.2 · p. 7, Table 2 · p. 7

5.3. Ablation investigations

  • Adding FRBs and then LRL and CRL improves performance and convergence over the baseline; CRL interpolation choices converge to nearly the same performance except deconvolution.Figure 4 plots PSNR convergence for degradation models and CRL interpolation methods on a ×4 test video over 200 epochs.Source: Figure 4 · p. 6, §5.3 · p. 7

5.4. Comparisons with state-of-the-art

  • FSTRN surpasses state-of-the-art methods by 0.55 dB average PSNR and 0.2 average SSIM.The comparison uses ×4 test-video results; Table 1 covers PSNR and SSIM across the listed methods.Source: §5.4 · p. 7, Table 1 · p. 7
  • On the Flag video, FSTRN produces sharper outputs with smoother inter-frame transitions than the compared methods.The qualitative comparison presents original and super-resolved frames for SRCNN, RDN, BRCN, VESPCN, and FSTRN.Source: §5.4 · p. 7, Figure 5 · p. 8

6. Conclusion

The conclusion presents FSTRN as a video SR network combining efficient spatio-temporal feature extraction with LR- and cross-space residual learning. The paper reports theoretical generalization analysis and empirical gains over current state-of-the-art SR methods.

  • FSTRN is proposed for video super-resolution, with empirical results reported to outperform current state-of-the-art SR methods.Source: §6 · p. 8
  • FRB extracts spatio-temporal features simultaneously while maintaining computational efficiency.Source: §6 · p. 8
  • LR-space residuals support feature extraction, while cross-space residual learning exploits similarities between LR inputs and HR outputs.Source: §6 · p. 8

A. Proof

The appendix collects proofs omitted from the main text.

A.1. Preliminary

The preliminary material defines FSTRN’s hypothesis and loss framework and introduces complexity tools used in its analysis. The appendix then derives a covering bound for FSTRN and uses a Rademacher-complexity bound to obtain a generalization bound.

A.1. Preliminary

  • A tuned FSTRN is treated as a hypothesis mapping LR videos to HR videos, with loss, expected risk, empirical risk, and generalization error defined for its analysis.Source: §A.1 · p. 11

A.2. Covering bound of FSTRN

  • The covering-bound proof combines bounds for the FRB and network components to establish a covering bound for FSTRN.The proof invokes layer and FRB bounds before applying them to the network theorem.Source: §A.2 · pp. 12–13

A.3. Generalization Bound for FSTRN

  • The generalization-bound proof first upper-bounds Rademacher complexity, then applies that bound to obtain a generalization-error bound.Source: §A.3 · p. 13

B. Empirical Results

The appendix reports that FSTRN outperforms state-of-the-art methods in both qualitative and quantitative evaluations.

B.1. Quantitatively Results

On Vid4, FSTRN outperforms all other methods and shows excellent robustness across different scenarios, using PSNR and SSIM for evaluation.

  • On Vid4, FSTRN outperforms all other methods and shows excellent robustness across different scenarios, evaluated using PSNR and SSIM.Source: §B.1 · p. 13

B.2. Qualitatitve Results

The qualitative comparisons cover ×4 super-resolution results across four videos and compare FSTRN with six existing methods.

Loading 1904.02870v1…