PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Trajectory-Aware Transformer for Video Super-Resolution

Chengxu Liu, Huan Yang, Jianlong Fu, Xueming Qian

arXiv:2204.04216v3eess.IVcs.CV

TL;DR

VSR methods often use only nearby frames, while recurrent approaches can have limited long-term modeling capability. TTVSR attends to visual tokens along pre-aligned trajectories and adds cross-scale tokenization; the authors report state-of-the-art results across four benchmarks, including gains on REDS4.

  • Problem

    Sliding-window VSR methods use limited adjacent frames, while recurrent methods can lack long-term modeling capability, constraining the use of temporal information across video sequences.

    Source: §1 · p. 1

  • Method

    TTVSR organizes video tokens into pre-aligned trajectories and computes attention among relevant tokens along each trajectory, with cross-scale tokenization to handle scale changes.

    Source: §1 · p. 2

  • Results

    On REDS4, TTVSR gains 0.70 dB PSNR over BasicVSR and 0.45 dB PSNR over IconVSR, and the authors report state-of-the-art performance across four widely used VSR benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TTVSR's trajectory-based attention reduces computational cost and enables long-range video modeling.

    Source: §1 · p. 2

    When rotation occurs, trajectories can be inaccurate and fail to transfer useful information, limiting performance.

    Source: §5 · p. 8

Abstract

from arXiv · show

Video super-resolution (VSR) aims to restore a sequence of high-resolution (HR) frames from their low-resolution (LR) counterparts. Although some progress has been made, there are grand challenges to effectively utilize temporal dependency in entire video sequences. Existing approaches usually align and aggregate video frames from limited adjacent frames (e.g., 5 or 7 frames), which prevents these approaches from satisfactory results. In this paper, we take one step further to enable effective spatio-temporal learning in videos. We propose a novel Trajectory-aware Transformer for Video Super-Resolution (TTVSR). In particular, we formulate video frames into several pre-aligned trajectories which consist of continuous visual tokens. For a query token, self-attention is only learned on relevant visual tokens along spatio-temporal trajectories. Compared with vanilla vision Transformers, such a design significantly reduces the computational cost and enables Transformers to model long-range features. We further propose a cross-scale feature tokenization module to overcome scale-changing problems that often occur in long-range videos. Experimental results demonstrate the superiority of the proposed TTVSR over state-of-the-art models, by extensive quantitative and qualitative evaluations in four widely-used video super-resolution benchmarks. Both code and pre-trained models can be downloaded at https://github.com/researchmm/TTVSR.

1. Introduction

VSR depends on temporal information, but sliding-window methods are costly to extend to distant frames and recurrent methods can lose long-term modeling capability. TTVSR organizes visual tokens into trajectories and attends along them, with cross-scale tokenization for scale changes; the authors report gains over BasicVSR and IconVSR on REDS4.

1. Introduction

  • The Figure 1 comparison highlights finer textures in TTVSR's recovered target frame from boxed regions tracked by trajectories.Source: Figure 1 · p. 1
  • Sliding-window VSR methods align adjacent frames, typically five or seven, while larger windows increase computational cost and limit access to distant frames.EDVR is cited as a sliding-window example using deformable convolutions.Source: §1 · p. 1
  • Recurrent methods convey information through hidden states, but vanishing gradients can limit their long-term modeling capability.The introduction cites IconVSR as a representative recurrent approach.Source: §1 · p. 1
  • TTVSR forms pre-aligned trajectories of visual tokens and computes self-attention only among relevant tokens on the same trajectory.The authors state that this design reduces computational cost and makes long-range video modeling practicable.Source: §1 · p. 2
  • A cross-scale feature tokenization module addresses scale changes that often occur in long-range videos.Source: §1 · p. 2
  • On REDS4, TTVSR gains 0.70 dB PSNR over BasicVSR and 0.45 dB PSNR over IconVSR.The paper also reports that TTVSR significantly outperforms existing state-of-the-art methods across four widely used VSR benchmarks.Source: §1 · p. 2

2. Related Work

Prior VSR work uses sliding windows or recurrent hidden states, while Transformer approaches to VSR have also been constrained by attention's computational cost. TTVSR is introduced as a trajectory-aware approach to long-term modeling that keeps attention computation within an acceptable range.

2.1. Video Super-Resolution

  • Sliding-window methods align and fuse adjacent-frame information, while recurrent methods pass information through hidden states but can lose long-term modeling capability due to vanishing gradients.The related-work discussion names EDVR and TDAN as deformable-convolution alignment methods and BasicVSR and IconVSR as recurrent methods.Source: §2.1 · p. 2

2.2. Vision Transformer

  • Vision Transformers model relationships among image patches, and TTSR uses a texture Transformer to search reference-image patches relevant to a low-resolution image.Source: §2.2 · p. 2
  • Transformer-based VSR methods such as VSR-Transformer and MuCAN use attention to align frames, but high video attention costs restrict them to narrow temporal windows.Source: §2.2 · p. 2
  • TTVSR introduces trajectory-aware attention to improve long-term VSR modeling while keeping attention computation within an acceptable range.Source: §2.2 · p. 2

3. Our Approach

TTVSR represents video frames as trajectories of aligned visual tokens and uses location maps to express and maintain those trajectories as matrices.

3.1. Trajectory-Aware Transformer

  • TTVSR represents video frames as trajectories of aligned visual tokens, with each trajectory a time-ordered coordinate sequence ending at its query token.The trajectory endpoint is associated with the query token's coordinate.Source: §3.1 · p. 3
  • Attention operates only along each spatio-temporal trajectory, selecting its most relevant token with hard attention and using soft attention to reduce the impact of irrelevant tokens.Restricting attention to trajectories avoids computation across the spatial dimension compared with vanilla vision Transformers.Source: §3.1 · pp. 3–4
  • Cross-scale tokenization expands receptive fields, pools features from different scales to a uniform size, and splits them into tokens so larger-scale textures can inform smaller-scale recovery.The module uses successive unfold and fold operations, pooling, and then unfolding to produce tokens.Source: §3.1 · p. 4

3.2. Location Maps for Trajectory Generation

  • Location maps replace costly trajectory calculations with matrix operations, updating per-frame coordinate maps through backward flow and parallel spatial sampling.The backward flow is obtained from a lightweight motion-estimation network.Source: §3.2 · pp. 4–5

3.3. TTVSR based on Location Maps

  • Representing trajectories as aligned location-map matrices provides an efficient way for TTVSR to use information from distant video frames and may motivate implementations for other video tasks.The location maps are described as an interchangeable formulation of the trajectories.Source: §3.3 · p. 5

3.4. Training Details

  • Training uses 50-frame sequences with bidirectional propagation; adjacent frames provide 1 × 1 tokens, while more distant frames are sampled at intervals for 4 × 4 tokens.The implementation also uses 4 × 4, 6 × 6, and 8 × 8 kernels for cross-scale feature tokenization.Source: §3.4 · p. 5

4. Experiments

TTVSR is evaluated on REDS4 and three additional VSR test sets, with PSNR and SSIM used for comparison. The experiments report stronger benchmark results, examine computational costs, and ablate trajectory generation, trajectory-aware attention, frame sampling, and cross-scale tokenization.

4.1. Datasets and Metrics

  • Evaluation uses 4× downsampling, with bicubic degradation on REDS4 and Gaussian-filter degradation on Vimeo-90K-T, Vid4, and UDM10.PSNR and SSIM are the evaluation metrics.Source: §4.1 · p. 5

4.2. Comparisons with State-of-the-art Methods

  • On REDS4, TTVSR achieves 32.12 dB PSNR and significantly outperforms IconVSR by 0.45 dB; the paper says this margin demonstrates its power in long-range modeling.Source: §4.2 · p. 6
  • 28.40 dB, 40.41 dB, and 37.92 dB PSNR on Vid4, UDM10, and Vimeo-90K-T, respectively, outperforming other state-of-the-art methods.TTVSR's reported gains over IconVSR are 0.36 dB on Vid4 and 0.38 dB on UDM10; improvement is greater on datasets with at least 30 frames than on the seven-frame Vimeo-90K-T sequences.Source: §4.2 · p. 6
  • Visual comparisons across four test sets show finer recovered textures, including more striped detail in stonework.The authors attribute the finer results to using textures from relevant tokens.Source: §4.2 · p. 6, Figure 4 · p. 7
  • TTVSR has higher performance than IconVSR with comparable parameters and FLOPs, and is lighter than MuCAN.The authors attribute its computational-cost reduction mainly to trajectories in attention calculation.Source: Table 3 · p. 6, §4.2 · p. 7

4.3. Ablation Study

  • Trajectory generation improves PSNR from 30.46 to 31.91, and adding trajectory-aware attention raises it to 31.99 on REDS4.The authors report that trajectory generation links relevant tokens, while attention integrates trajectory tokens for long-range modeling.Source: §4.3 · p. 7, Table 4 · p. 7
  • Performance increases with more sampled frames but gains diminish beyond 45; sampling every three frames, or 33 frames, is reported sufficient for the 100-frame sequences.The authors note that smaller intervals may add little information because adjacent frames are too similar.Source: §4.3 · p. 8, Table 5 · p. 8
  • Cross-scale feature tokenization improves PSNR from 31.99 to 32.12 and extracts finer textures, while oversized tokens such as scale 12 are undesirable.The model uses token sizes 4, 6, and 8 to address scale changes in long-range sequences.Source: §4.3 · p. 8, Figure 5 · p. 8

5. Limitations

Rotation can make TTVSR’s motion trajectories inaccurate, preventing useful information transfer and limiting performance. Other state-of-the-art methods also struggle with rotation, while TTVSR still reports greater gains.

5. Limitations

  • Rotation makes TTVSR’s motion trajectories inaccurate, preventing useful information from being transferred.The paper identifies this as a failure case.Source: §5 · p. 8, Figure 7 · p. 8
  • Inaccurate trajectories during rotation limit the performance of TTVSR.Source: §5 · p. 8
  • Other state-of-the-art methods also fail to obtain better performance on rotation, while TTVSR still achieves greater gains.The paper attributes the difficulty to modeling rotation and TTVSR’s gains to its long-range modeling ability.Source: §5 · p. 8

6. Conclusion

The conclusion presents TTVSR as a trajectory-aware Transformer for leveraging long-range frame dependencies in video super-resolution. It reports visual margins over state-of-the-art models and identifies broader low- and high-level vision tasks as future directions.

6. Conclusion

  • TTVSR shows clear visual margins over existing state-of-the-art models.Source: §6 · p. 8
  • TTVSR formulates video frames as pre-aligned visual-token trajectories and calculates attention along them.Source: §6 · p. 8
  • A location map records trajectories and can be updated online efficiently, supporting long-range video modeling with reduced computational costs.Source: §6 · p. 8
  • The authors identify evaluating low-level vision tasks and extending the approach to high-level vision tasks as future work.Source: §6 · p. 8

Supplementary Material

The supplementary material outlines algorithm details, architecture and experimental settings, complexity analysis, limitations, and further comparisons. Its algorithm details describe bidirectional propagation, cross-scale feature tokenization, and location-map updating.

Supplementary Material

  • The supplement covers algorithm details, architecture and experimental settings, complexity analysis, limitations, and additional comparisons.Source: Supplementary Material · p. 11

A. Algorithm Details

  • The TTVSR pseudocode shows forward propagation; the method also uses bidirectional propagation so frame features can propagate backward and forward.Source: A. Algorithm Details · p. 11
  • Cross-scale feature tokenization expands feature receptive fields, pools features from different scales to a common scale, then unfolds them into output tokens.This extracts larger-scale features without adding parameters and can process the whole temporal sequence in parallel.Source: A. Algorithm Details · p. 11
  • A lightweight motion-estimation network supplies backward flow for updating location maps, with adjacent-coordinate interpolation and spatial sampling.Average pooling makes the motion-estimation output match the location-map size.Source: A. Algorithm Details · pp. 11–12

B. Details of Architecture and Runtimes

The runtime analysis compares TTVSR’s components and reports that it is lighter and faster than attention-based methods, but slower than IconVSR. It attributes that runtime gap to attention’s many small matrix multiplications and notes the dependence on implementation and hardware.

B. Details of Architecture and Runtimes

  • Runtime measurements use one 180×320 LR frame, ×4 upsampling, and an NVIDIA Tesla V100 GPU.The architecture uses five residual blocks for feature extraction, 60 for reconstruction, and 64 feature channels.Source: B. Details of Architecture and Runtimes · p. 12
  • TTVSR is lighter and faster than attention-based methods, but slower than IconVSR.The authors attribute the IconVSR runtime gap to implementation and hardware dependence; attention calculations take more than half of TTVSR’s runtime.Source: B. Details of Architecture and Runtimes · p. 12
  • Compared with methods without attention, TTVSR achieves higher results with smaller parameter size and comparable FLOPs.The comparison is referenced to Table 3 in the main paper.Source: B. Details of Architecture and Runtimes · p. 12

C. More Experimental Settings

The experiments use different degradations and sequence settings for REDS and Vimeo-90K, with bidirectional propagation and a sequence-wide Charbonnier loss.

C. More Experimental Settings

  • REDS4 is evaluated with bicubic downsampling, while Vimeo-90K uses Gaussian filtering (σ = 1.6) followed by downsampling; Vid4, UDM10, and Vimeo-90K-T are test sets for Vimeo.The model uses bidirectional propagation with shared parameters, cascading the backward and forward features for each frame.Source: C. More Experimental Settings · p. 12
  • REDS inputs contain 50 frames; Vimeo-90K sequences are extended to 28 frames by flipping twice, and Vimeo-90K-T results average outputs from the same frame extension.Source: C. More Experimental Settings · p. 12
  • A Charbonnier penalty is applied between ground-truth and restored frames across the sequence; ε is 1 × 10−8, and experiments use eight NVIDIA Tesla V100 GPUs.The reported setup uses Python 3.9 and PyTorch 1.9.Source: C. More Experimental Settings · p. 12

D. Complexity Analysis

Trajectory-aware attention avoids attention calculations over the spatial token dimension, reducing computation and supporting use of distant video frames.

D. Complexity Analysis

  • The analysis compares the vanilla Transformer’s attention cost with trajectory-aware attention using token size C · D_h · D_w and a sequence of length T.Source: D. Complexity Analysis · p. 12
  • For each query token, the number of attention tokens falls from T · H/D_h · W/D_w to T because spatial attention is omitted.Source: D. Complexity Analysis · p. 12
  • Trajectory-aware attention reduces attention computation by a factor of (H/D_h · W/D_w), avoiding spatial-dimension computation and enabling use of distant frames.Source: D. Complexity Analysis · pp. 12–13

E. Limitation Analysis

TTVSR’s gains are limited on shorter sequences, while long-sequence training and inference have time and memory costs.

E. Limitation Analysis

  • Shorter sequences limit performance improvement; Vimeo-90K’s original seven-frame sequences are extended to 28 frames by flipping twice, but improvement remains limited.Source: E. Limitation Analysis · p. 13
  • Longer training sequences support long-range modeling but increase training time, which the authors plan to optimize.Source: E. Limitation Analysis · p. 13
  • Training and inference must store features for each moment, consuming GPU memory; the authors propose adaptive, selective feature storage as future work.Source: E. Limitation Analysis · p. 13

F. More Results

Additional experiments report strong perceptual-metric performance and provide visual comparisons across four video super-resolution benchmarks.

F. More Results

Related papers

Loading 2204.04216v3…