Source-linked AI summary
SwinIR: Image Restoration Using Swin Transformer
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, Radu Timofte
TL;DR
Image restoration seeks to recover high-quality images from degraded inputs, but convolutional and patch-based Transformer approaches have limitations in content adaptation, long-range modeling, and border handling. SwinIR uses residual Swin Transformer blocks for restoration and outperforms competing methods across super-resolution, denoising, and JPEG artifact reduction tasks, often with fewer parameters.
Problem
Image restoration lacks methods that flexibly model content-dependent interactions and long-range dependencies without patch-border artifacts or extra overlap computation.
Method
SwinIR combines shallow feature extraction, residual Swin Transformer blocks for deep feature extraction, and high-quality reconstruction with residual connections.
Results
SwinIR outperforms competing methods across super-resolution, denoising, and JPEG artifact reduction, including a maximum 0.26dB PSNR gain on Manga109 at scale factor 4.
Takeaways & Limitations
The results demonstrate SwinIR’s effectiveness and generalizability across three representative image restoration tasks.
Abstract
from arXiv · showhide
Image restoration is a long-standing low-level vision problem that aims to restore high-quality images from low-quality images (e.g., downscaled, noisy and compressed images). While state-of-the-art image restoration methods are based on convolutional neural networks, few attempts have been made with Transformers which show impressive performance on high-level vision tasks. In this paper, we propose a strong baseline model SwinIR for image restoration based on the Swin Transformer. SwinIR consists of three parts: shallow feature extraction, deep feature extraction and high-quality image reconstruction. In particular, the deep feature extraction module is composed of several residual Swin Transformer blocks (RSTB), each of which has several Swin Transformer layers together with a residual connection. We conduct experiments on three representative tasks: image super-resolution (including classical, lightweight and real-world image super-resolution), image denoising (including grayscale and color image denoising) and JPEG compression artifact reduction. Experimental results demonstrate that SwinIR outperforms state-of-the-art methods on different tasks by $\textbf{up to 0.14$\sim$0.45dB}$, while the total number of parameters can be reduced by $\textbf{up to 67%}$.
1. Introduction
Image restoration reconstructs clean high-quality images from degraded inputs, but CNNs are limited by content-independent filtering and weak long-range dependency modeling. SwinIR addresses these issues with a Swin Transformer-based architecture combining shallow feature extraction, deep feature extraction, and high-quality image reconstruction.
- Motivation: Image restoration targets high-quality reconstruction from degraded images in tasks including super-resolution, denoising, and JPEG artifact reduction.CNNs have become the primary workhorse for these tasks.
- Limitations of CNNs: CNN restoration methods use content-independent convolution kernels and struggle to model long-range dependencies effectively.Applying the same kernel across image regions may be suboptimal because interactions do not depend on image content.
- SwinIR: SwinIR comprises shallow feature extraction, deep feature extraction, and high-quality image reconstruction modules based on Swin Transformer.The shallow feature is directly transmitted to reconstruction to preserve low-frequency information.
- Contributions: Compared with CNN-based restoration models, SwinIR provides content-based attention interactions, shifted-window long-range dependency modeling, and better performance with fewer parameters.The attention weights can be interpreted as spatially varying convolution.
2. Related Work
Learning-based, especially CNN-based, methods became popular for image restoration because of their impressive performance, while Transformers were increasingly applied to vision and restoration by modeling global interactions. However, the IPT restoration model relied on over 115.5M parameters.
- CNN-based restoration: CNN-based methods became popular for image restoration because they learn low-quality-to-high-quality mappings from large-scale paired datasets.The passage contrasts these methods with generally model-based traditional approaches and cites SRCNN, DnCNN, and ARCNN as pioneering examples.
- Transformer-based vision: Transformers gained popularity in computer vision and were applied to classification, detection, segmentation, crowd counting, and image restoration.In vision problems, Transformers explore global interactions between different image regions to attend to important regions.
- Transformer-based restoration: Over 115.5M parameters were required by IPT, a standard-Transformer backbone for various image restoration problems.IPT was introduced for multiple restoration tasks after Transformers demonstrated impressive performance in computer vision.
3. Method
SwinIR uses shared shallow and deep feature extraction modules followed by task-specific high-quality image reconstruction. Its deep extractor stacks residual Swin Transformer blocks, while reconstruction aggregates low- and high-frequency features using task-appropriate upsampling or convolution.
- Architecture: SwinIR comprises shallow feature extraction, deep feature extraction, and high-quality image reconstruction, with shared extraction modules but task-specific reconstruction modules.A 3 × 3 convolution extracts shallow features, while the deep extractor contains K residual Swin Transformer blocks and a final 3 × 3 convolution.
- Image reconstruction: Shallow features carry low frequencies, deep features recover lost high frequencies, and a long skip connection transfers low-frequency information directly to reconstruction.For super-resolution, reconstruction uses a sub-pixel convolution layer to upsample; denoising and JPEG artifact reduction use a single convolution layer without upsampling.
- Loss function: Super-resolution training uses L1 pixel loss, with pixel, GAN, and perceptual losses combined for real-world super-resolution, while denoising and JPEG reduction use Charbonnier loss.Classical and lightweight super-resolution use naive L1 pixel loss, whereas real-world super-resolution adds GAN and perceptual losses to improve visual quality.
- Residual Swin Transformer block: RSTBs apply L Swin Transformer layers, followed by convolution and a residual connection that aggregates features across blocks.The convolution enhances translational equivariance, while residual connections provide identity-based paths for aggregating different feature levels.
- Swin Transformer layer: Swin Transformer layers use local multi-head self-attention with alternating regular and shifted windows to establish cross-window connections.Each layer also includes LayerNorm, an MLP with GELU nonlinearity, and residual connections.
4. Experiments
Experiments show that SwinIR improves image restoration across classical and lightweight super-resolution, JPEG artifact reduction, and denoising, while ablations confirm that larger architectural hyperparameters generally improve PSNR. The model also produces sharper, more natural restored details than competing approaches.
- Ablation study: PSNR increases with channel number, RSTB number, and STL number in the ablation study, although increasing channel number causes quadratic parameter growth.The ablation trains on DIV2K for classical image SR (×2) and tests on Manga109.
- Ablation study: Using three 3 × 3 convolution layers reduces parameters but slightly decreases performance compared with the RSTB design.The comparison concerns extracting local neighbouring information.
- Real-world image SR: A larger SwinIR model trained on much larger datasets handles more complex corruptions and achieves better performance on real-world images than the current model.Details are provided on the project page.
- JPEG compression artifact reduction: SwinIR achieves average PSNR gains of at least 0.11dB and 0.07dB on the two JPEG artifact-reduction testing datasets.Evaluation uses Classic5 and LIVE1 with JPEG quality factors 10, 20, 30, and 40.
- Image denoising: SwinIR outperforms all compared denoising methods, surpassing DRUNet by up to 0.3dB on Urban100.The comparison covers grayscale and color denoising at noise levels 15, 25, and 50.
5. Conclusion
SwinIR is a Swin Transformer-based image restoration model with shallow feature extraction, deep feature extraction, and high-resolution reconstruction modules. Experiments across three restoration tasks and six settings demonstrate its state-of-the-art performance, effectiveness, and generalizability.
- SwinIR comprises shallow feature extraction, deep feature extraction, and high-resolution reconstruction modules.
- Residual Swin Transformer blocks form the deep feature extractor, combining Swin Transformer layers, a convolution layer, and a residual connection.
- State-of-the-art performance is reported across six settings spanning classic, lightweight, and real-world image super-resolution, grayscale and color denoising, and JPEG artifact reduction.
- The results demonstrate SwinIR's effectiveness and generalizability across representative image restoration tasks.
- Future work will extend SwinIR to image deblurring and deraining.