Source-linked AI summary
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration
Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, Luc Van Gool
TL;DR
Image restoration requires recovering missing content while efficiently modelling dependencies across global, regional, and local image ranges. The paper introduces anchored stripe self-attention and integrates it with window attention and channel-enhanced convolution in GRL. GRL achieves state-of-the-art performance across various restoration tasks, including reported PSNR gains over Restormer on GoPro and HIDE deblurring datasets.
Problem
Image restoration needs efficient long-range dependency modelling beyond regional windows while explicitly representing global, regional, and local features.
Method
GRL combines anchored stripe self-attention, window self-attention, and channel-attention-enhanced convolution to model image hierarchies across three ranges.
Results
GRL achieves state-of-the-art performance across various image restoration tasks, including 1.01 dB and 0.43 dB PSNR improvements over Restormer on GoPro and HIDE, respectively.
Takeaways & Limitations
The architecture provides efficient and explicit hierarchical modelling capacity for image restoration across global, regional, and local ranges.
Takeaways & Limitations
The anchored-attention rationale relies on interpreting similarity propagation through anchors despite the paper’s attention similarity being based on dot products rather than a metric distance.
Abstract
from arXiv · showhide
The aim of this paper is to propose a mechanism to efficiently and explicitly model image hierarchies in the global, regional, and local range for image restoration. To achieve that, we start by analyzing two important properties of natural images including cross-scale similarity and anisotropic image features. Inspired by that, we propose the anchored stripe self-attention which achieves a good balance between the space and time complexity of self-attention and the modelling capacity beyond the regional range. Then we propose a new network architecture dubbed GRL to explicitly model image hierarchies in the Global, Regional, and Local range via anchored stripe self-attention, window self-attention, and channel attention enhanced convolution. Finally, the proposed network is applied to 7 image restoration types, covering both real and synthetic settings. The proposed method sets the new state-of-the-art for several of those. Code will be available at https://github.com/ofsoundof/GRL-Image-Restoration.git.
1. Introduction
Image restoration must recover missing content while modeling natural-image features across local, regional, and global ranges. GRL addresses this hierarchy with anchored stripe self-attention, window self-attention, and channel-attention-enhanced convolution, and evaluates the architecture across seven restoration tasks.
- Motivation: Image restoration is an ill-posed inverse problem because degradation removes important image content.The target output is a high-quality image reconstructed from inputs degraded by blur, sub-sampling, noise, or JPEG compression.
- Image hierarchies: Natural images contain local, regional, and global feature ranges that require different modelling mechanisms.CNNs suit local edges and colors, window attention suits regional consistency and similarity, while global features include symmetry, repeated patterns, and structural similarity.
- Research gap: Existing convolutional and window-attention networks struggle to explicitly capture long-range dependencies efficiently, especially as image resolution increases.Global modelling is often achieved through repeated feature propagation, while pairwise comparisons and similarity searches become costly for high-resolution images.
- Research questions: The paper asks how to efficiently model global features, explicitly represent all three ranges, and improve restoration consistently across tasks.These questions organize the paper’s proposed mechanism, network architecture, and experiments.
- Proposed approach: Anchored stripe self-attention uses lower-dimensional anchors and horizontal or vertical stripes to reduce self-attention complexity while retaining a global image view.Cross-scale similarity motivates anchors, anisotropic features motivate stripes, and shifted stripe modes balance complexity with modelling capacity beyond the regional range.
- Proposed approach: GRL explicitly models global, regional, and local dependencies through parallel anchored stripe attention, window attention, and channel-attention-enhanced convolution.The architecture is applied to seven tasks spanning real, synthetic, and data-synthesis-based restoration settings.
2. Related Works
Prior image-restoration methods model local patterns with convolutions and broader dependencies with non-local operations or self-attention. Regional window attention improves efficiency but limits the modelling range, motivating methods that extend self-attention beyond regional neighborhoods.
- Convolution for local range modelling: Convolution is well matched to detecting repetitive local patterns because it slides learnable kernels across an image.Stacking convolutional layers expands the processing depth while retaining local-pattern modelling.
- Non-local and global priors: Non-local methods and self-attention extend image restoration beyond local features by modelling repetitive or global image dependencies.These approaches have been applied to tasks including denoising and super-resolution, while global self-attention remains computationally expensive.
- Non-local and global priors: Global self-attention has quadratic computational complexity in the number of tokens, motivating research into more efficient variants.The attention map contributes an O(N^2) space term, making high-resolution images especially challenging.
- Regional self-attention: Regional self-attention, including shifted-window attention, improves efficiency but restricts interactions to a regional range.Recent work suggests that involving a wider range of pixels can improve image restoration, motivating modelling beyond regional windows.
3. Motivation
Global self-attention explicitly models long-range dependencies but becomes impractical for high-resolution images because its pairwise comparisons scale quadratically. The paper motivates anchored stripe self-attention by exploiting cross-scale similarity and anisotropic features to reduce complexity while extending modelling beyond regional windows.
- Self-attention for dependency modelling: Self-attention compares each token with all others and computes outputs as similarity-weighted sums, enabling explicit long-range dependency modelling.Its attention map captures pairwise pixel similarity.
- Self-attention for dependency modelling: Global self-attention has O(N^2d) time and O(N^2) space complexity, making high-resolution feature maps computationally burdensome.Window attention reduces the participating tokens but limits modelling to a regional range.
- Motivation I: cross-scale similarity: Cross-scale similarity motivates using lower-dimensional anchors to approximate query-key attention without directly forming the full attention map.Natural-image structures can remain similar across resolutions, supporting anchors as an intermediate representation.
- Motivation I: cross-scale similarity: Anchored self-attention factorizes attention through maps Me and Md, distilling values into an intermediate feature before expanding them back to query positions.The formulation is Y = Me · (Md · V), with anchors mediating both stages.
- Motivation II: anisotropic image features: Anchored self-attention removes the quadratic N^2 term, but its remaining dependence on image tokens N can still be unaffordable at large resolutions.This motivates further restricting attention to anisotropic stripes.
- Motivation II: anisotropic image features: Anisotropic natural-image features motivate horizontal, vertical, and shifted stripe attention as a trade-off between global-range capacity and computational control.The mechanism combines stripe attention with anchors to extend efficient attention beyond regional ranges.
- Discussion: The approximate attention map Me · Md preserves the major structure of the exact map M, with a reported Pearson correlation coefficient of 0.9505.The comparison supports the plausibility of using anchors as an intermediate for attention approximation.
4. Modelling Image Hierarchies
GRL explicitly models global, regional, and local image dependencies within one transformer layer and network architecture. Its components combine anchored stripe attention, window attention, and channel-attention-enhanced convolution.
- GRL incorporates global-, regional-, and local-range image modelling capacities in one network architecture.
- The network takes a degraded low-quality image as input and estimates a recovered high-quality image after feature extraction, representation learning, and reconstruction.
- The transformer layer processes feature maps through parallel self-attention and channel-attention-enhanced convolution branches.The convolution branch captures local structures, while self-attention includes window and anchored stripe attention.
- Window attention captures regional dependencies, while the convolution branch captures local structures.
- Anchored stripe self-attention uses lower-dimensional anchors to reduce long-range attention complexity to O(NMd) time and O(NM) space.Anchor projection summarizes the feature map before the two-stage attention operation.
5. Experimental Results
GRL is evaluated across real, synthetic, and data-synthesis-based restoration settings, spanning deblurring, denoising, super-resolution, artifact removal, and demosaicking. It reports strong results across these tasks, including state-of-the-art performance and favorable efficiency comparisons.
- Real image restoration: GRL improves single-image motion deblurring PSNR over Restormer by 1.01 dB on GoPro and 0.43 dB on HIDE.These improvements are achieved under a smaller parameter budget.
- Real image restoration: 40.20 PSNR establishes GRL-B as state of the art on RealBlur-R, while GRL-B uses 24% fewer parameters than Restormer.
- Synthetic image restoration: On Gaussian denoising, GRL-T reduces model complexity to 0.76% of [5] and 2.7% of DRUNet without sacrificing network accuracy.GRL-B also exceeds Restormer on Urban100 by 0.44 dB PSNR for color images at noise level 50.
- Synthetic image restoration: GRL-T outperforms convolutional and self-attention-based lightweight super-resolution networks, while GRL-B sets the state of the art for accurate image super-resolution.Compared with EDT, GRL-T gains 0.44 dB on Urban100 and 0.22 dB on Manga109 for ×4 SR.
- Synthetic image restoration: GRL-S outperforms prior JPEG artifact-removal methods across datasets and quality factors, with much smaller complexity than FBCNN, and GRL also significantly outperforms prior demosaicking methods.The reported FBCNN and GRL-S parameter counts are 71.92M and 3.12M, respectively.
- Real-world image restoration: In real-world ×4 super-resolution, GRL removes more artifacts from low-resolution images than the compared methods in visual evaluation.No ground-truth images are available, so only visual comparisons are reported.
- Ablation study: Dot-product similarity remains competitive with metric-based comparison, and average pooling followed by linear projection is selected for anchor projections.The anchor-projection choice considers both accuracy and parameter budget.
6. Conclusion
The paper concludes that GRL provides efficient, explicit hierarchical modelling for image restoration through anchored stripe self-attention. Across diverse restoration tasks, the architecture achieves state-of-the-art performance.
- GRL combines efficient anchored stripe self-attention with global, regional, and local hierarchical modelling for image restoration.
- The proposed architecture achieves state-of-the-art performance across various image restoration tasks.