PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Super-Resolution Transformer

Jiezhang Cao, Yawei Li, Kai Zhang, Luc Van Gool

arXiv:2106.06847v3cs.CV

TL;DR

VSR requires restoring high-resolution video while using spatial-temporal information, but standard Transformer layers neglect locality and cross-frame feature alignment. VSR-Transformer replaces them with spatial-temporal convolutional self-attention and bidirectional optical-flow-based feed-forward processing, with benchmark experiments demonstrating effectiveness.

  • Problem

    Standard Transformer blocks for VSR neglect data locality in fully connected self-attention and feature alignment in token-wise feed-forward processing.

  • Method

    VSR-Transformer uses spatial-temporal convolutional self-attention to exploit locality and a bidirectional optical-flow-based feed-forward layer to propagate and align features across frames.

  • Results

    Extensive experiments on several benchmark datasets demonstrate the effectiveness of VSR-Transformer, especially with limited numbers of frames.

  • Takeaways & Limitations

    The proposed design adapts Transformer sequence modeling to VSR by incorporating locality, spatial-temporal correlations, feature propagation, and alignment.

Abstract

from arXiv · show

Video super-resolution (VSR), with the aim to restore a high-resolution video from its corresponding low-resolution version, is a spatial-temporal sequence prediction problem. Recently, Transformer has been gaining popularity due to its parallel computing ability for sequence-to-sequence modeling. Thus, it seems to be straightforward to apply the vision Transformer to solve VSR. However, the typical block design of Transformer with a fully connected self-attention layer and a token-wise feed-forward layer does not fit well for VSR due to the following two reasons. First, the fully connected self-attention layer neglects to exploit the data locality because this layer relies on linear layers to compute attention maps. Second, the token-wise feed-forward layer lacks the feature alignment which is important for VSR since this layer independently processes each of the input token embeddings without any interaction among them. In this paper, we make the first attempt to adapt Transformer for VSR. Specifically, to tackle the first issue, we present a spatial-temporal convolutional self-attention layer with a theoretical understanding to exploit the locality information. For the second issue, we design a bidirectional optical flow-based feed-forward layer to discover the correlations across different video frames and also align features. Extensive experiments on several benchmark datasets demonstrate the effectiveness of our proposed method. The code will be available at https://github.com/caojiezhang/VSR-Transformer.

1 Introduction

VSR is a spatial-temporal sequence modeling task, but standard Transformer blocks inadequately capture locality, neighboring-frame correlations, and feature alignment. VSR-Transformer addresses these limitations with convolutional self-attention and optical-flow-based feed-forward processing.

  • VSR enhances low-resolution video to high-resolution video and can be formulated as sequence-to-sequence modeling.
  • Fully connected self-attention neglects local spatial information and does not clearly exploit correlations among neighboring video frames.Patch-based tokenization can divide local structures across tokens, while global attention is computed through fully connected layers.
  • Token-wise feed-forward processing independently handles embeddings, preventing feature propagation and alignment across video frames.
  • VSR-Transformer combines spatial-temporal convolutional self-attention with bidirectional optical-flow-based feed-forward processing.The first layer exploits locality; the second learns spatial-temporal information through feature propagation and alignment.
  • The paper reports extensive benchmark experiments demonstrating VSR-Transformer’s effectiveness, especially when using limited numbers of frames.

2 Related Work

Prior VSR research includes sliding-window alignment, deformable-convolution alignment, and recurrent approaches. These methods progressively address motion handling, feature alignment, temporal propagation, and recurrent stability.

  • Video super-resolution: Sliding-window VSR methods predict optical flow between low-resolution frames and align frames through spatial warping.
  • Video super-resolution: TDAN and EDVR use deformable convolutions for feature-level alignment, with EDVR extending this through multi-scale pyramid cascading.
  • Video super-resolution: RSDN and RRN adopt recurrent frameworks to exploit previous frames or stabilize recurrent training with residual mappings and skip connections.
  • Video super-resolution: BasicVSR uses bidirectional recurrence together with optical-flow-based feature alignment for VSR.

3 Preliminary and Problem Definition

The paper formulates VSR as learning a nonlinear mapping from extracted low-resolution video features to high-resolution frames. It motivates a Transformer design that addresses locality and cross-frame feature alignment.

  • The notation distinguishes sequence data or distributions, matrices, vectors, elements, softmax, ReLU, layer normalization, and the frame-index set [T].
  • Function distance measures how closely a learned function f matches a target function f* over inputs.
  • A k-pattern function depends only on a small pattern of consecutive input elements, providing a formal notion of data locality.
  • VSR extracts feature sequence X from low-resolution frames and learns a nonlinear mapping F that reconstructs high-resolution frames using spatial-temporal information.
  • The training objective minimizes a distance-based loss between generated high-resolution frames and ground-truth frames.The distance may use L1, L2, or Charbonnier loss.
  • A standard Transformer block combines self-attention and token-wise feed-forward layers, but its fully connected attention neglects locality and independent token processing produces misaligned features.
  • The proposed framework uses feature extraction, a Transformer encoder, and reconstruction to restore a high-resolution video.

4 Video Super-Resolution Transformer

VSR-Transformer adapts Transformer architecture to video super-resolution with spatial-temporal locality, feature alignment, and bidirectional frame interaction. Its encoder combines convolutional self-attention with optical-flow-based propagation while retaining parallel computing.

  • Overall architecture: VSR-Transformer extracts video features, maps them to continuous representations with a Transformer encoder, and reconstructs a high-resolution video.The architecture comprises a feature extractor, Transformer encoder, and reconstruction network.
  • Spatial-temporal convolutional self-attention: Theoretical analysis states that FCSA can have small initial gradients when k = Ω(log n), whereas STCSA can learn k-pattern functions with finite gradient-descent steps.The analysis links STCSA's learning behavior to its ability to capture frame locality.
  • Spatial-temporal convolutional self-attention: The STCSA layer uses convolutional projections and sliding local patches to capture spatial-temporal information before folding updated patches back into feature maps.Unfold extracts local patches, while fold recombines them into a feature map.
  • Bidirectional optical flow-based feed-forward: The BOFF layer estimates forward and backward optical flows, warps feature maps, propagates information in both directions, and fuses the resulting feature sequences.The flow-based feed-forward design models correlations across video frames and performs feature alignment.
  • Bidirectional optical flow-based feed-forward: Unlike BasicVSR, the proposed model recurrently estimates neither optical flows nor features, avoiding recursion and enabling parallel computing.The comparison specifically contrasts VSR-Transformer with BasicVSR's recurrent processing.

5 Experiments

Experiments evaluate VSR-Transformer on REDS, Vimeo-90K-T, and Vid4 using PSNR, SSIM, model size, and qualitative comparisons, including an optical-flow ablation.

  • Experimental setup: Experiments use REDS, Vimeo-90K-T, and Vid4, evaluating generated-image quality with PSNR and SSIM while also comparing model sizes.The compared methods include RCAN, VESPCN, SPMC, TOFlow, FRVSR, DUF, RBPN, EDVR, BasicVSR, and IconVSR.
  • Results on REDS: On REDS, VSR-Transformer recovers finer details and sharper edges than BasicVSR and IconVSR when trained on five frames.The reported qualitative examples include square patterns and horizontal and vertical strip patterns.
  • Results on Vimeo-90K-T and Vid4: On Vimeo-90K-T, VSR-Transformer achieves the highest PSNR and SSIM, while its Vid4 generalization is better than EDVR but worse than BasicVSR and IconVSR.The authors attribute part of the Vid4 comparison to different numbers of evaluated frames and possible distribution bias.
  • Results on Vimeo-90K-T: On Vimeo-90K-T, VSR-Transformer generates sharp, realistic high-resolution frames, whereas BasicVSR, IconVSR, and EDVR produce blurrier or messier textures.The qualitative comparison is for 4× VSR.
  • Ablation study: Removing optical flow reduces REDS performance to PSNR 30.37, while optical flow produces finer details and sharper edges.The ablation compares VSR-Transformer without optical flow against the version with optical flow.

6 Conclusion

The paper concludes that VSR-Transformer adapts Transformer architecture to video super-resolution by combining locality-aware attention with bidirectional feature propagation and alignment. Extensive benchmark experiments demonstrate the method’s effectiveness.

  • 6 Conclusion: VSR-Transformer uses spatial-temporal convolutional self-attention to leverage locality information instead of directly applying existing vision Transformer designs.The paper provides theoretical analysis supporting an advantage over fully connected self-attention.
  • 6 Conclusion: Its bidirectional optical flow-based feed-forward layer exploits correlations among frames while performing feature propagation and alignment.These operations are presented as the mechanism for using information across different video frames.
  • 6 Conclusion: Extensive experiments on several benchmark datasets demonstrate the effectiveness of VSR-Transformer.

Supplementary Materials: Video Super-Resolution Transformer

The supplementary materials provide notation, theoretical preliminaries, proof-supporting definitions, optimization background, and additional experimental information for the VSR-Transformer paper.

  • Supplementary organization: The supplement includes detailed proofs for all theorems and lemmas, additional experiment settings, network architectures, ablations, and qualitative results.These materials are organized into theory and experiment sections.
  • Preliminaries: Notation distinguishes sequence data, matrices, vectors, matrix elements, softmax, ReLU, layer normalization, index sets, indicators, and expectations.The notation defines σ1 as column-wise softmax, σ2 as ReLU, and ϕ as layer normalization.
  • Theoretical definitions: The supplement defines function distance between a model and target function, then introduces k-pattern functions to formalize locality.The k-pattern definition restricts a function’s dependence to a consecutive local input pattern.
  • Theoretical definitions: The locality definition permits comparison of whether convolutional or fully connected networks can learn local patterns under a specified loss tolerance.The stated condition is Lf,D(hu,W) ≤ ϵ.
  • Optimization background: The supplement states an online gradient-descent convergence theorem for sequences of convex functions under updates θs+1=θs−η∇fs(θs).

B Proofs of Theorem 1

The proof of Theorem 1 analyzes fully connected self-attention on local k-patterns using permutation arguments, orthogonality, and gradient calculations under stated assumptions.

  • Gradient analysis: The proof represents the target local pattern with χI and analyzes gradients with respect to network weights.The target is expressed using a product over a set I of consecutive bits.
  • Permutation argument: Permutation invariance is used by fixing a permutation that preserves an index while mapping local subsets to distinct subsets.The proof defines permutation actions on inputs and index sets and exploits uniform-distribution invariance.
  • Orthogonality argument: The proof uses an inner product and induced norm over the data distribution, with χI′ forming an orthonormal basis under Parseval’s equality.
  • Bound derivation: The argument sums contributions over neurons and selects a maximal permutation family to obtain the stated bound.

C Proofs of Theorem 2

The appendix proves properties of STCSA under random weight initialization, using pattern coverage, convexity, and gradient-descent arguments. The results establish probabilistic bounds for representing k-patterns and training the STCSA layer.

  • Theorem 2: If q > 2k+3 log(2k/δ), every k-pattern is covered with probability at least 1−δ over the random initialization.The proof uses the probability 2^-k that a random weight sign pattern matches a fixed z and applies a union bound.
  • Proof of Theorem 2: For each sign pattern z, the expected number of matching initialized weights is q2^-k, and concentration bounds lower-bound its realized count.Chernoff concentration provides the high-probability coverage needed by the theorem.
  • Pattern construction: The proof constructs u∗ parameters supported on selected pattern locations, with one bounded component and zero components for later locations.The construction uses ∥u∗(j∗)∥≤2k+1k/√q and sets ∥u∗(j)∥=0 for j > 1.
  • Representation argument: The STCSA representation follows from its definition, convolution properties, and a weight choice that matches the target local k-pattern.The proof explicitly invokes the m = 1 activation assumption and the existence of a suitable convolutional weight W.

D More Experiment Details and Network Architecture

The experiments use standardized preprocessing, augmentation, and architecture settings, then evaluate component contributions through REDS4 ablations. STCSA, BOFF with optical flow, and larger frame context are each associated with better restoration outcomes.

  • Experiment details: Training uses bicubic down-sampling, 64×64 LR crops, random flips and 90° rotations, fixed 448×256 resolution, and pre-trained SPyNet flow estimation.The batch size is 2 per GPU, residual-block channels are 64, and Transformer-block count equals the number of frames.
  • Network architecture: The architecture uses convolutional STCSA blocks, bidirectional BOFF propagation networks, feature fusion, a five-block feature extractor, and a 30-block reconstruction module.Each BOFF direction uses 30 residual blocks; convolutions use 3×3 kernels, stride 1, and padding 1.
  • Ablation study: Removing STCSA worsens quantitative performance on REDS4 and produces less detailed, less sharp high-resolution frames.The ablation is defined as VSR-Transformer without the spatial-temporal convolutional self-attention layer.
  • Ablation study: Removing BOFF also worsens performance, while retaining optical flow produces finer details and sharper edges in the reconstructed frames.The comparison replaces BOFF with a stack of Residual ReLU networks.
  • Ablation study: Training with only three frames degrades performance, whereas more frames help restore missing information from neighboring frames.The authors state that they plan to train VSR-Transformer with more frames in future work.

F.1 Results on REDS4

The paper presents qualitative 4× VSR comparisons on REDS4, Vimeo-90K-T, and Vid4. The supplied passages identify the datasets and zoom-in presentation but do not state a qualitative outcome.

  • REDS4: Figure 8 presents a qualitative comparison on the REDS4 dataset for 4× VSR.The figure instructs readers to zoom in for the best view.
  • Vimeo-90K-T: Figure 9 presents a qualitative comparison on Vimeo-90K-T for 4× VSR.The figure instructs readers to zoom in for the best view.
  • Vid4: Figure 10 presents a qualitative comparison on Vid4 for 4× VSR.The figure instructs readers to zoom in for the best view.

G More Results on Vid4

On Vid4, VSR-Transformer achieves comparable PSNR and SSIM to EDVR-L. BasicVSR and IconVSR perform worse in the small-frame setting, while their cited comparison uses seven frames.

  • Quantitative comparison: Comparable PSNR and SSIM to EDVR-L are achieved by VSR-Transformer on Vid4.Table 7 reports quantitative PSNR/SSIM comparisons for 4× VSR, with Y denoting evaluation on Y channels.
  • Quantitative comparison: BasicVSR and IconVSR perform much worse than EDVR and VSR-Transformer in the reported small-frame comparison.The passage notes that BasicVSR and IconVSR use seven frames in their training and testing setup, unlike the cited small-frame setting.
  • Interpretation: The results indicate that BasicVSR and IconVSR rely substantially on long-term sequence aggregation and generalize poorly when trained and tested with few frames.The passage presents this as an implication of the comparison and states that the results verify VSR-Transformer’s effectiveness and generalization ability.
Loading 2106.06847v3…