PaperlayerView exact revision on arXiv

Source-linked AI summary

A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift

Dasong Li, Xiaoyu Shi, Yi Zhang, Ka Chun Cheung, Simon See, Xiaogang Wang, Hongwei Qin, Hongsheng Li

arXiv:2206.10810v2eess.IVcs.CV

TL;DR

Video restoration must recover clear frames from noisy or blurred videos, yet existing methods can rely on computationally costly alignment and attention architectures. This paper uses grouped spatial-temporal shifts to aggregate frame information, and reports better state-of-the-art performance with fewer FLOPs across deblurring and denoising.

  • Problem

    Video restoration needs inter-frame information, but existing methods often use computationally costly optical flow, deformable convolution, or cross-frame self-attention.

    Source: §1 · p. 1

  • Method

    Grouped spatial-temporal shifts implicitly model temporal correspondences, with 2D convolution blocks fusing shifted features for multi-frame aggregation.

    Source: §1 · pp. 1–2

  • Results

    The framework outperforms state-of-the-art methods with fewer FLOPs on both video deblurring and video denoising.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The framework offers a lower-computation alternative for video deblurring and denoising while maintaining state-of-the-art performance.

    Source: §1 · p. 2, §5 · p. 8

    The network is not designed for video super-resolution, where optical flow could greatly help, and may not perform well on that task.

    Source: §1 · p. 9

Abstract

from arXiv · show

Video restoration, which aims to restore clear frames from degraded videos, has numerous important applications. The key to video restoration depends on utilizing inter-frame information. However, existing deep learning methods often rely on complicated network architectures, such as optical flow estimation, deformable convolution, and cross-frame self-attention layers, resulting in high computational costs. In this study, we propose a simple yet effective framework for video restoration. Our approach is based on grouped spatial-temporal shift, which is a lightweight and straightforward technique that can implicitly capture inter-frame correspondences for multi-frame aggregation. By introducing grouped spatial shift, we attain expansive effective receptive fields. Combined with basic 2D convolution, this simple framework can effectively aggregate inter-frame information. Extensive experiments demonstrate that our framework outperforms the previous state-of-the-art method, while using less than a quarter of its computational cost, on both video deblurring and video denoising tasks. These results indicate the potential for our approach to significantly reduce computational overhead while maintaining high-quality results. Code is avaliable at https://github.com/dasongli1/Shift-Net.

1. Introduction

Video restoration must use information across degraded frames, but existing alignment and long-range modeling approaches can be computationally costly or struggle with difficult motion and image conditions. The paper proposes grouped spatial-temporal shifts for efficient multi-frame aggregation and reports better performance with fewer FLOPs on video deblurring and denoising.

1. Introduction

  • Handheld video can contain sensor noise and severe blur from camera shake or object movement, motivating video restoration.Source: §1 · p. 1
  • Existing approaches use optical flow, deformable convolution, or self-attention for temporal modeling, but can require complex or costly architectures and struggle with large displacements, noise, or blur.The passage also notes that very large kernels do not necessarily improve performance.Source: §1 · p. 1
  • Grouped spatial-temporal shift blocks provide large effective receptive fields for temporal correspondence, while basic 2D U-Nets handle frame-wise feature encoding and restoration.The shifts operate separately across temporal and spatial dimensions, and 2D convolution blocks fuse the shifted features.Source: §1 · pp. 1–2
  • Unlike TSM [37], the method combines alternative temporal shifts with grouped spatial shifts and fusion convolutions, yielding receptive fields such as 23 × 23.The passage attributes TSM's difficulty in multi-frame aggregation to channel misalignment across three frames.Source: §1 · p. 2
  • The framework surpasses state-of-the-art methods with fewer FLOPs on both video deblurring and video denoising.The authors present this as evidence of generalization across the two tasks.Source: §1 · p. 2

2. Related Work

Prior video-restoration work addresses temporal alignment and long-term information aggregation through explicit alignment, fixed windows, recurrent models, and shift operations. These approaches face trade-offs in complexity, robustness, or receptive-field size, motivating the paper's extension of shifts to obtain large receptive fields with small convolutions.

2. Related Work

  • Explicit alignment with optical flow or deformable convolution can increase complexity and may fail with large displacement, noise, or blurry regions; small-kernel implicit convolution can have narrow receptive fields.Other approaches include dynamic filters and optical-flow-guided deformable alignment.Source: §2 · p. 2
  • Sliding-window methods aggregate information only within a fixed set of adjacent frames, while recurrent methods can accumulate faulty predictions and misalignment over time.Source: §2 · p. 2
  • Shift operations have been used in efficient convolution alternatives and vision models, while VRT [35] shifts two-frame clips for temporal self-attention but remains computationally costly.The paper extends shift operations to obtain a large receptive field with small kernel convolutions.Source: §2 · pp. 2–3

3. Method

Group Shift-Net combines grouped spatial-temporal shifts with 2D U-Net-like processing to aggregate information across video frames. Its staged design uses frame-wise feature extraction, multi-frame fusion, and final restoration.

3.1. Overview of Group Shift-Net

  • Group Shift-Net extracts frame features, fuses neighboring-frame features with grouped spatial-temporal shifts, then restores each output frame.The fusion stage stacks shift blocks to aggregate long-term information.Source: §3.1 · p. 3

3.2. Frame-wise Processing

  • Stacked slim 2D U-Nets perform feature extraction and final restoration, with residual blocks and adjustable network size.Average pooling and bilinear upsampling adjust feature resolutions.Source: §3.2 · pp. 3–4

3.3. Grouped Spatial-temporal Shift

  • Alternating FTS and BTS blocks fuse adjacent frames, while grouped spatial shifts apply 25 candidate offsets to propagated feature slices.The two temporal-shift block types retain half the channels for current-frame appearance and shift the other half for inter-frame aggregation.Source: §3.3 · pp. 4–5, Figure 4 · p. 4

3.4. How Grouped Spatial Shift Help Restoration?

  • Qualitative examples show more recovered detail on GoPro and improved reconstruction of leaves and a moving tire on DVD.Source: Figure 5 · p. 5, Figure 7 · p. 6
  • Local attribution maps show that shifted features aligned with inter-frame motion contribute more to restoration, with expansive effective receptive fields.Red-dot saturation represents each area's contribution to restoring the marked patch.Source: Figure 6 · p. 5, §3.4 · p. 6

4. Experiments

Experiments evaluate Group Shift-Net on video deblurring and denoising, then ablate its shift design and compare alternatives. The reported results favor the proposed method across both restoration tasks, with ablations identifying useful shift settings.

4. Experiments

  • Experiments cover GoPro and DVD for deblurring and DAVIS for denoising, with small and base models offered for different computational requirements.The reported inference setup processes 100 frames for Ours-s and 50 frames for Ours and Ours+ due to memory limits.Source: §4 · p. 6

4.1. Video Deblurring Results

  • Ours+ outperforms VRT by 1.07 dB PSNR on GoPro and 0.42 dB PSNR on DVD, using only 21% of its FLOPs.The visual comparison also shows sharper letters, building structures, and moving-leg boundaries in difficult deblurring cases.Source: §4.1 · p. 7

4.2. Video Denoising Results

  • The method achieves the best performance at five noise levels on DAVIS with less computational cost, and its small model beats prior lightweight models.In Figure 8, it reconstructs more texture and text detail than competing methods.Source: §4.2 · p. 7, Table 3 · p. 7, Figure 8 · p. 7

4.3. Ablation Study

  • Removing grouped spatial shift drops PSNR by 0.35 dB, while replacing alternative temporal shift with bidirectional shift drops it by 0.33 dB.The compared models use the same training settings as the base model; the bidirectional variant aggregates 3/4 of the reference-frame channels and 1/8 from each neighboring frame.Source: §4.3 · pp. 7–8, Table 4 · p. 8
  • Spatial shifts outperform larger-kernel fusion by about 0.3 dB PSNR, while larger kernels do not improve performance as their receptive fields grow.The authors suggest that extremely large kernels may be difficult to optimize.Source: §4.3 · p. 8, Table 5 · p. 8
  • Grouped shifts {0, ±5, ±9} perform best among the tested offsets; 25 and 49 shifts perform similarly and about 0.15 dB PSNR better than 9 shifts.Performance degrades when shift length exceeds the fusion kernel size, which the authors attribute to shifted features no longer being filtered seamlessly.Source: §4.3 · p. 8, Table 6 · p. 8
  • The method has similar temporal consistency to VRT and better performance with less computation than optical-flow, DCN, and cross-frame self-attention alternatives.The consistency test uses adjacent-output mean absolute error on DAVIS sequences with 12 noise seeds; the replacement comparisons include both fusion-layer and whole-shift-block variants.Source: §4.3 · p. 8, Table 7 · p. 8, Table 8 · p. 8
  • Shift length s=5 gives the best performance for both deblurring and denoising and across all tested resolutions.Resolution tests use downsampled, original, and upsampled DAVIS.Source: §4.3 · p. 8, Table 9 · p. 8

5. Conclusion

The paper proposes a simple video-restoration framework based on spatial-temporal shift blocks. It reports better performance than state-of-the-art methods with less computation on video deblurring and denoising.

  • Grouped spatial-temporal shift provides a simple video-restoration framework that outperforms state-of-the-art methods with less computation on deblurring and denoising.Source: §5 · p. 8

1. Optical Flow Analysis in Video Restoration

The analysis finds that optical flow affects restoration tasks differently: it helps super-resolution but contributes little to deblurring and denoising. The paper relates this limitation to inaccurate flow in blurry or noisy frames and identifies super-resolution as outside its network's intended scope.

  • Optical flow improves video super-resolution by 0.56 dB PSNR but does not greatly improve video deblurring or denoising.The analysis compares BasicVSR++ with and without flow at matched computational cost.Source: §1 · p. 9, Table 10 · p. 9
  • Optical flow estimates are inaccurate in noisy or blurry frames, including task-oriented flow trained on GoPro or DAVIS.Source: §1 · p. 9
  • Because optical flow contributes little to deblurring and denoising, the authors design grouped spatial-temporal shift for implicit temporal correspondence modeling when flow is inaccurate.Source: §1 · p. 9
  • The network is not designed for super-resolution, where optical flow estimation could help, and may not perform well on that task.Source: §1 · p. 9

2. More Results

Qualitative videos show stable, temporally consistent restoration without flicker, with more texture and detail than VRT in deblurring and denoising. On Set8 denoising, the method achieves performance approximately equal to concurrent RVRT.

  • The restored videos are described as free of flicker and temporally consistent and stable.Source: §2 · p. 9
  • Compared with VRT, the method restores more textures and details in both video deblurring and video denoising.Source: §2 · p. 9
  • On Set8 video denoising, the method achieves approximately the same performance as concurrent RVRT.Source: §2 · p. 9

3. Further Analysis of Grouped Spatial Shift

The analysis relates the contribution of shifted feature groups to optical-flow direction. It also notes that optical-flow estimates on blurry or noisy images can be inaccurate.

3. Further Analysis of Grouped Spatial Shift

  • LAM contribution weights rank shifted features, whose top-ranked vectors are averaged into pseudo optical flow for local grids.The analysis also compares pseudo flows from ranks 5–8, 9–12, and 13–16 with flow estimated from clean frames.Source: §3 · p. 9
  • Shifted feature groups usually contribute more when their shift direction is similar to the ground-truth optical flow.The analysis calculates x- and y-axis correlations between the flow and pseudo flow.Source: §3 · p. 9
  • Optical-flow estimates on blurry and noisy images are inaccurate, and task-oriented flow is usually smooth and inaccurate.Source: Figure 9 · p. 10

4. Motion magnitudes

The section evaluates deblurring performance across motion magnitudes on GoPro. The base model exceeds VRT in the largest-motion group.

4. Motion magnitudes

  • 33.94 dB in the 10% largest magnitudes surpasses VRT (32.45 dB) by +1.49 dB.Source: §4 · p. 10
  • GoPro frames are categorized according to motion magnitude for the deblurring comparison.Source: §4 · p. 10
  • Motion magnitude is computed by averaging SPyNet optical flows between each clean frame and its two adjacent ground-truth frames.Source: §4 · p. 10

5. Network Architecture

The architecture uses three-stage, multi-scale U-Nets for frame-wise processing and restoration, with grouped spatial-temporal shift blocks for multi-frame fusion. Model variants differ in their U-Net count, frame-wise channel width, and fusion-channel count.

5. Network Architecture

  • The three-stage design uses a three-scale U-Net backbone with MPRNet-like encoding, average pooling, and 2D bilinear upsampling.Source: §5 · p. 10
  • Channel Attention Blocks extract frame-wise features and process skip connections, while GSTS blocks aggregate and communicate multi-frame features.Source: §5 · pp. 10–11
  • Each GSTS block combines grouped spatial-temporal shifts with a lightweight FusionConv layer for fusing shifted features.FusionConv uses point-wise and depth-wise convolutions, LayerNorm, channel attention, and gated layers.Source: §5 · p. 11
  • Ours-s stacks three 14-channel slim U-Nets with 64 fusion channels; Ours and Ours+ stack five 24-channel slim U-Nets with 80 fusion channels.Source: §5 · p. 11

Related papers

Loading 2206.10810v2…