Source-linked AI summary
Accurate Image Restoration with Attention Retractable Transformer
Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, Xin Yuan
TL;DR
Image-restoration Transformers reduce convolutional limitations but often restrict attention to dense local regions, producing limited receptive fields. ART alternates dense and sparse attention modules to combine local and global interactions, and it achieves state-of-the-art results across super-resolution, denoising, and JPEG artifact reduction benchmarks.
Problem
Existing efficient image-restoration Transformers mainly use dense attention over local regions, restricting receptive fields and limiting long-range interactions.
Method
ART alternates dense attention blocks based on local windows with sparse attention blocks based on sparse grids.
Results
ART achieves state-of-the-art performance across image super-resolution, denoising, and JPEG compression artifact reduction tasks.
Takeaways & Limitations
Combining dense and sparse attention provides simultaneous local and global receptive fields for image restoration.
Abstract
from arXiv · showhide
Recently, Transformer-based image restoration networks have achieved promising improvements over convolutional neural networks due to parameter-independent global interactions. To lower computational cost, existing works generally limit self-attention computation within non-overlapping windows. However, each group of tokens are always from a dense area of the image. This is considered as a dense attention strategy since the interactions of tokens are restrained in dense regions. Obviously, this strategy could result in restricted receptive fields. To address this issue, we propose Attention Retractable Transformer (ART) for image restoration, which presents both dense and sparse attention modules in the network. The sparse attention module allows tokens from sparse areas to interact and thus provides a wider receptive field. Furthermore, the alternating application of dense and sparse attention modules greatly enhances representation ability of Transformer while providing retractable attention on the input image.We conduct extensive experiments on image super-resolution, denoising, and JPEG compression artifact reduction tasks. Experimental results validate that our proposed ART outperforms state-of-the-art methods on various benchmark datasets both quantitatively and visually. We also provide code and models at https://github.com/gladzhang/ART.
1 INTRODUCTION
Image restoration seeks to recover high-quality images from degraded inputs, but CNNs and windowed Transformer methods have limited long-range or receptive-field coverage. ART addresses this by alternating dense local and sparse global attention.
- Motivation: Image restoration recovers high-quality images from low-quality inputs across applications including super-resolution and denoising.It is described as an ill-posed inverse problem because many original images may correspond to one input.
- Motivation: CNNs have limited ability to model long-range dependencies because convolutional receptive-field scaling is parameter-dependent and interactions are content-independent.This limitation motivated introducing self-attention into vision systems.
- Problem: Windowed Transformer methods group tokens from dense image regions, reducing computation but causing restricted receptive fields.SwinIR uses shifted non-overlapping windows, while IPT splits features into patches treated as tokens.
- Approach: ART combines dense attention blocks using local windows with sparse attention blocks using sparse grids.The two block types are designed to provide retractable attention over the input feature.
- Contribution: Alternating dense and sparse attention enables simultaneous local and global receptive fields in the Transformer.The authors evaluate ART on super-resolution, denoising, and JPEG compression artifact reduction and report state-of-the-art performance.
2 RELATED WORK
Prior image-restoration research developed CNN-based methods and later explored Transformer architectures for content-dependent global receptive fields. The cited work spans restoration models and efficient vision Transformers.
- Image Restoration: CNN-based image-restoration methods achieved superior performance over conventional restoration approaches.Examples include SRCNN for super-resolution and DnCNN for image denoising.
- Image Restoration: SRCNN introduced CNNs to image super-resolution, while DnCNN became a representative image-denoising method.These methods are presented as foundational examples in CNN-based restoration research.
- Vision Transformer: Vision Transformers use content-dependent global receptive fields and have been introduced into computer vision systems.ViT projected large image patches into token sequences, while MaxViT and CrossFormer explored efficient or multi-axis attention designs.
3 PROPOSED METHOD
ART combines dense local attention with sparse long-range attention in a residual Transformer architecture for image restoration. Its alternating attention blocks are designed to capture both local and global receptive fields while keeping attention computation more efficient than full attention.
- Overall Architecture: ART extracts shallow features, processes them through residual groups of Transformer attention blocks, and reconstructs the high-quality image from the combined feature map.The architecture applies convolutions before and after deep feature extraction, then uses element-wise addition to form the final feature map.
- Restoration Tasks: ART supports super-resolution, denoising, and compression-artifact reduction with task-specific restoration and loss choices.Super-resolution uses upsampling and L1 loss, while denoising and JPEG artifact reduction use direct reconstruction and Charbonnier loss.
- Design Motivation: The proposed retractable attention strategy addresses the restricted receptive fields of windowed Transformers while retaining lower-cost attention modules than full-image self-attention.Existing methods use shorter token sequences from dense image areas, whereas ART also models sparse-area interactions; D-MSA and S-MSA have lower computational cost because W^2 ≪ hw and hI < hw.
- Attention Blocks: Dense attention groups neighboring tokens within fixed non-overlapping windows, restricting interactions to local image regions.The dense module divides tokens into groups of W×W tokens and applies self-attention within each group.
- Attention Blocks: Sparse attention groups tokens from spaced positions with interval size I, enabling interactions across sparse image regions.The sparse strategy complements dense windows by connecting tokens that are not confined to one local neighborhood.
- Attention Blocks: ART alternates dense attention blocks and sparse attention blocks, placing DAB before SAB and adding long-distance residual connections across block pairs.The resulting Attention Retractable Transformer provides interactions for both local dense tokens and global sparse tokens.
4 EXPERIMENTAL RESULTS
ART is evaluated across image super-resolution, denoising, and JPEG compression artifact reduction, with ablations examining attention design, interval size, and model variants. The experiments report strong quantitative and visual performance while balancing accuracy against computational cost.
- Ablation Study: ART alternates dense and sparse attention blocks because using either block alone lacks local or global receptive fields.The alternating DAB-SAB structure achieves higher performance in the ablation study.
- Ablation Study: Smaller sparse-attention intervals improve performance, so the experiments select task-specific intervals to balance accuracy and complexity.The selected interval sizes are (4, 4, 4, 4, 4, 4) for SR, (16, 16, 12, 12, 8, 8) for denoising, and (18, 18, 13, 13, 7, 7) for JPEG CAR.
- Ablation Study: ART-S has model size comparable to SwinIR but achieves better performance in the reported comparison.ART-S uses an MLP ratio of 2 and interval size 8, whereas ART uses an MLP ratio of 4.
- Image Super-Resolution: ART achieves the best PSNR/SSIM performance on all five image super-resolution benchmark datasets and improves over SwinIR across scale factors.Self-ensemble further improves ART results.
- Image Denoising: ART achieves the highest color-image-denoising performance on three datasets except Kodak24 and improves over Restormer by up to 0.25dB on Urban100.Visual comparisons report preserved textures and high-frequency components alongside reduced noise corruption.
- JPEG Compression Artifact Reduction: ART reports the best performance among compared methods for JPEG compression artifact reduction, with further gains from self-ensemble.The evaluation uses PSNR and SSIM at compression quality factors 40, 30, and 10.
REPRODUCIBILITY STATEMENT
The paper supports reproducibility by providing implementation details, experiment settings, source code, and pre-trained models for verifying the reported results.
- The paper documents its model architecture, dense and sparse attention modules, implementation details, and experiment settings.
- Source code and pre-trained models are provided so readers can reproduce the training and testing process.
- The pre-trained models enable verification of the validity of the corresponding results.