Source-linked AI summary
Recurrent Video Restoration Transformer with Guided Deformable Attention
Jingyun Liang, Yuchen Fan, Xiaoyu Xiang, Rakesh Ranjan, Eddy Ilg, Simon Green, Jiezhang Cao, Kai Zhang, Radu Timofte, Luc Van Gool
TL;DR
Video restoration must balance parallel methods’ temporal fusion against their model and memory costs and recurrent methods’ parameter sharing against limited long-range modeling and parallelizability. RVRT addresses this gap with locally parallel clip processing in a globally recurrent transformer, using guided deformable attention for clip-to-clip alignment. Experiments across super-resolution, deblurring, and denoising report state-of-the-art performance with balanced model size, memory usage, and runtime.
Problem
Existing video restoration methods trade off parallel temporal fusion and recurrent parameter efficiency against memory usage, long-range dependency modeling, and parallelizability.
Method
RVRT divides videos into clips, jointly updates frames within each clip, recurrently propagates clip features, and aligns clips with guided deformable attention.
Results
RVRT achieves state-of-the-art performance across video super-resolution, deblurring, and denoising on eight benchmark datasets with balanced model size, memory usage, and runtime.
Takeaways & Limitations
RVRT provides a globally recurrent video restoration model with locally parallel processing that combines advantages of parallel and recurrent methods.
Takeaways & Limitations
RVRT’s optical-flow pre-alignment complexity increases quadratically with clip length.
Abstract
from arXiv · showhide
Video restoration aims at restoring multiple high-quality frames from multiple low-quality frames. Existing video restoration methods generally fall into two extreme cases, i.e., they either restore all frames in parallel or restore the video frame by frame in a recurrent way, which would result in different merits and drawbacks. Typically, the former has the advantage of temporal information fusion. However, it suffers from large model size and intensive memory consumption; the latter has a relatively small model size as it shares parameters across frames; however, it lacks long-range dependency modeling ability and parallelizability. In this paper, we attempt to integrate the advantages of the two cases by proposing a recurrent video restoration transformer, namely RVRT. RVRT processes local neighboring frames in parallel within a globally recurrent framework which can achieve a good trade-off between model size, effectiveness, and efficiency. Specifically, RVRT divides the video into multiple clips and uses the previously inferred clip feature to estimate the subsequent clip feature. Within each clip, different frame features are jointly updated with implicit feature aggregation. Across different clips, the guided deformable attention is designed for clip-to-clip alignment, which predicts multiple relevant locations from the whole inferred clip and aggregates their features by the attention mechanism. Extensive experiments on video super-resolution, deblurring, and denoising show that the proposed RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory and runtime.
1 Introduction
RVRT combines locally parallel processing within video clips with global recurrence to balance the complementary strengths of parallel and recurrent restoration methods. It introduces guided deformable attention for one-stage clip-to-clip alignment and reports state-of-the-art results across three restoration tasks.
- Parallel methods fuse multiple frames effectively but often require large models and enormous memory for long videos.
- Recurrent methods reuse network blocks to save parameters, but sequential processing can cause long-range information loss and noise amplification.
- RVRT divides videos into fixed-length clips, processes neighboring frames jointly, and refines later clip features using previously inferred clip features.
- Guided deformable attention estimates multiple relevant locations across supporting-clip frames under optical-flow guidance and dynamically aggregates their features.
- GDA samples multiple relevant locations, supports arbitrary locations through bilinear interpolation, and uses dynamic aggregation weights rather than fixed weights.
- Experiments on eight benchmark datasets report state-of-the-art performance for video super-resolution, deblurring, and denoising with balanced model size, memory usage, and runtime.
2 Related Work
Related work spans parallel and recurrent video restoration, alignment methods, and transformer architectures. These approaches motivate RVRT’s combination of temporal processing, alignment, and transformer-based restoration.
- Parallel vs. recurrent methods: Parallel restoration methods estimate frames simultaneously and include sliding-window and transformer-based approaches.
- RVRT architecture: RVRT’s architecture is presented as shallow feature extraction, recurrent feature refinement, and high-quality frame reconstruction.
- Parallel vs. recurrent methods: Recurrent restoration methods propagate latent features sequentially from one frame to the next, accumulating information for later-frame restoration.
- Alignment in video restoration: Traditional video restoration commonly estimates optical flow between neighboring frames before warping images for alignment.
- Vision transformers: Transformers represent pixels or image patches as tokens and have achieved gains across vision and restoration tasks, including video restoration.
- Deformable attention: Some deformation-attention methods predict weights from query features or place supporting points uniformly, primarily for recognition tasks.
3 Methodology
RVRT combines global recurrent propagation across fixed-length clips with local parallel processing of neighboring frames. Its GDA module aligns successive clips by sampling multiple flow- and offset-guided locations and aggregating them through attention.
- Overall architecture: RVRT reconstructs high-quality video from a low-quality sequence, with scale factor s determining the output spatial resolution.The input and output video tensors have temporal length T, while the output dimensions are scaled to sH and sW.
- Overall architecture: The model extracts shallow features, repeatedly refines video features, and reconstructs high-quality frames.Shallow extraction uses convolutions and residual Swin Transformer blocks before recurrent refinement and frame reconstruction.
- Recurrent feature refinement: RVRT divides the video into N-frame clips, processes frames within each clip jointly in parallel, and propagates features across clips recurrently.This design processes local neighboring frames in parallel within a globally recurrent framework.
- Recurrent feature refinement: Modified residual Swin Transformer blocks use 3D N × h × w attention windows so each frame can attend to itself and other frames.The resulting implicit feature aggregation jointly updates different frames within a clip.
- Guided deformable attention: GDA aligns the previous clip toward the current clip by estimating multiple relevant locations and dynamically aggregating their sampled features with attention.It uses optical-flow-based pre-alignment, predicts offsets, samples features from the resulting locations, and computes attention-based aligned features.
- Guided deformable attention: For each target frame, GDA forms queries from the current clip and keys and values from NM bilinearly sampled locations in the previous clip.Projection matrices are applied before sampling, and an MLP with GELU-based channel interaction follows spatial aggregation.
4 Experiments
Experiments evaluate RVRT’s clip length, alignment, GDA components, and efficiency across video super-resolution, deblurring, and denoising. Results show improved restoration quality alongside reduced model size, testing memory, or runtime in several comparisons.
- Ablation Study: N = 2 mitigates noise amplification from a corrupted frame while using more neighboring frames than N = 1.After setting the 50-th frame to zero, N = 2 has a smaller performance drop and higher PSNR on all frames than N = 1, while affecting more neighboring frames.
- Ablation Study: GDA outperforms existing alignment methods for frame-to-frame alignment and improves further when aggregating features from the whole clip.The direct clip-level aggregation constitutes the proposed one-stage video-to-video alignment design.
- Ablation Study: Optical flow guidance increases PSNR by 1.11dB, while multilayer flow updates and channel interaction further improve GDA.The channel interaction compensates for attention’s spatial-only aggregation by modeling channel relationships in the MLP.
- Ablation Study: Increasing deformable groups initially raises PSNR before saturation; doubling attention heads slightly improves results but increases computation.Too many heads hurt performance because the head dimension may become too small.
- Video Super-Resolution: RVRT achieves the best performance on REDS4 and Vid4 for both video super-resolution degradations, improving PSNR over BasicVSR++ by 0.2∼0.5dB.RVRT also outperforms VRT on REDS4 and Vid4 by up to 0.36dB and restores sharper textures and details visually.
- Video Super-Resolution: Compared with parallel methods, RVRT uses at least 50% fewer model parameters and testing memory and reduces runtime by at least 25%.Compared with BasicVSR++, RVRT improves PSNR by 0.26dB.
5 Conclusion
RVRT combines globally recurrent processing with locally parallel frame updates, using guided deformable attention for clip-to-clip alignment. Experiments covered video super-resolution, deblurring, and denoising.
- 5 Conclusion: RVRT is a globally recurrent model with locally parallel designs that combines advantages of parallel and recurrent video restoration methods.It extracts neighboring-frame features jointly within clips and accumulates information across clips and layers.
- 5 Conclusion: Guided deformable attention aligns video clips by aggregating information from multiple neighboring locations under optical-flow guidance.The mechanism performs one-stage clip-to-clip alignment with adaptive attention.
- 5 Conclusion: Extensive experiments evaluated RVRT on video super-resolution, video deblurring, and video denoising.The supplied comparison captions cover video deblurring on DVD and GoPro and video denoising on DAVIS and Set8.
6 Limitations and Societal Impacts
RVRT has a computational limitation because optical-flow pre-alignment complexity grows quadratically with clip length, and its use can raise privacy and medical-diagnosis risks.
- 6 Limitations and Societal Impacts: Optical-flow pre-alignment has complexity that increases quadratically with clip length.The paper suggests directly predicting all optical flows as one possible solution.
- 6 Limitations and Societal Impacts: Restoring blurry videos may create privacy concerns and lead to misjudgments if RVRT is used for medical diagnosis.The paper suggests limiting use for sensitive or critical videos.