Source-linked AI summary
Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration
Marcos V. Conde, Ui-Jin Choi, Maxime Burchi, Radu Timofte
TL;DR
Compressed images are efficient to transmit and store but lose information and develop artifacts, creating a need for effective restoration. The paper introduces Swin2SR, a SwinV2-based improvement to SwinIR, and evaluates it across three restoration tasks. Swin2SR improves convergence and performance, achieves competitive AIM 2022 results, and is ranked among the challenge’s top five solutions.
Problem
Compression introduces artifacts and information loss, while existing restoration methods face limitations including content-independent convolutional kernels and training challenges for large vision transformers.
Method
Swin2SR replaces SwinIR’s transformer blocks with SwinV2 components and uses auxiliary low-resolution and high-frequency losses for compressed-input super-resolution.
Results
Swin2SR achieves state-of-the-art performance across JPEG artifact removal, classical and lightweight super-resolution, and compressed-image super-resolution, with a top-five AIM 2022 ranking.
Takeaways & Limitations
Swin2SR advances super-resolution of compressed inputs while supporting faster training and convergence, larger capacity, and higher resolution.
Takeaways & Limitations
Qualitative results show a clear blur effect and difficulty recovering fine details and sharpness; comparisons follow the SwinIR experimental setup.
Abstract
from arXiv · showhide
Compression plays an important role on the efficient transmission and storage of images and videos through band-limited systems such as streaming services, virtual reality or videogames. However, compression unavoidably leads to artifacts and the loss of the original information, which may severely degrade the visual quality. For these reasons, quality enhancement of compressed images has become a popular research topic. While most state-of-the-art image restoration methods are based on convolutional neural networks, other transformers-based methods such as SwinIR, show impressive performance on these tasks. In this paper, we explore the novel Swin Transformer V2, to improve SwinIR for image super-resolution, and in particular, the compressed input scenario. Using this method we can tackle the major issues in training transformer vision models, such as training instability, resolution gaps between pre-training and fine-tuning, and hunger on data. We conduct experiments on three representative tasks: JPEG compression artifacts removal, image super-resolution (classical and lightweight), and compressed image super-resolution. Experimental results demonstrate that our method, Swin2SR, can improve the training convergence and performance of SwinIR, and is a top-5 solution at the "AIM 2022 Challenge on Super-Resolution of Compressed Image and Video".
1 Introduction
Compressed images support efficient transmission and storage but suffer artifacts and information loss, motivating restoration methods. Swin2SR applies SwinV2 to improve SwinIR for compressed-image restoration and evaluates it across multiple tasks.
- Compression enables efficient image and video transmission and storage but introduces artifacts and information loss that degrade visual quality.
- CNN-based restoration methods face content-independent kernels and limited receptive fields, motivating architectures that model broader image interactions.
- Transformers use self-attention to capture global context, while Swin Transformer combines local attention with shifted windows for long-range dependency modeling.
- Swin2SR uses SwinV2 to improve SwinIR and address training instability, training duration, and resolution gaps between pre-training and fine-tuning.
- The paper reports extensive comparisons and competitive results in the AIM 2022 Challenge on Super-Resolution of Compressed Image and Video.
2 Related Work
Related work spans CNN-based restoration and Transformer-based vision models. Swin Transformer reduces attention cost through shifted windows, while SwinV2 modifies normalization, attention, and positional-bias mechanisms for scaling.
- CNNs represent leading approaches for many low-level image-restoration tasks, including denoising, deblurring, super-resolution, and compression-artifact removal.
- Transformers model complex global image interactions through attention and have been applied to classification, detection, segmentation, and perceptual quality assessment.
- Swin Transformer uses shifted-window self-attention and progressive downsampling, reducing attention complexity for large images.
- For image size h × w with fixed M × M patches, window attention replaces quadratic image-wide complexity with linear complexity in image area.
- SwinV2 replaces pre-normalization, dot-product attention, and parameterized positional bias with post-normalization, scaled cosine attention, and log-spaced continuous bias.
- SwinV2 attention applies softmax to scaled cosine similarities plus positional bias before multiplying by the value matrix.
3 Our Method
Swin2SR adapts SwinIR with SwinV2-based residual transformer blocks and restoration-oriented training components. Its experiments use established image-super-resolution protocols and add losses targeting low-resolution consistency and high-frequency detail.
- Swin2SR updates SwinIR’s residual transformer blocks with SwinV2 layers and attention to increase model capacity and resolution.
- Post-normalization reduces deeper-layer feature variance and improves numerical training stability, while scaled cosine attention reduces dominance by particular heads.
- Experiments evaluate JPEG artifact removal, classical and lightweight super-resolution, and compressed-image super-resolution, mainly using DIV2K with PSNR and SSIM on the Y channel.
- The model contains shallow feature extraction, deep Residual SwinV2 Transformer Blocks, and high-quality image reconstruction modules.
- Compressed-input super-resolution experiments explore additional loss functions intended to improve performance and robustness.
- The auxiliary loss minimizes L1 distance between downsampled prediction and reference, enforcing consistency at lower resolution.
- The high-frequency loss compares blurred-detail residuals between prediction and reference to improve sharpness and output quality.
4 Experimental Results
Experiments evaluate Swin2SR across JPEG artifact removal, classical and lightweight super-resolution, real-world images, and compressed-image super-resolution. The results emphasize flexible single-model operation, faster convergence, competitive quality, and strong challenge performance, while fine-detail recovery remains limited.
- 4.1 JPEG Compression Artifacts Removal: Swin2SR trains a single model across four JPEG quality factors, unlike comparison methods commonly trained separately for each quality.The authors also compare against SwinIR using half the training iterations and JPEG compression augmentation.
- 4.1 JPEG Compression Artifacts Removal: Swin2SR restores compressed images and produces high-quality results for grayscale or color JPEG artifact removal.The paper reports qualitative restoration results and describes the approach as more general and flexible across compression qualities.
- 4.2 Classical Image Super-Resolution: 12M parameters make Swin2SR competitive with CNN models using 15.4∼44.3M parameters, while matching SwinIR performance with less training.For classical SR, Swin2SR reaches essentially the same performance as SwinIR after 400k iterations from scratch, compared with SwinIR’s 500k iterations and ×4 fine-tuning.
- 4.2 Classical Image Super-Resolution: A dynamic upsampling layer lets one Swin2SR-D model perform ×2, ×3, and ×4 super-resolution on-line, whereas SwinIR trains one model per scale.The reported implementation supports ×2 and ×4 super-resolution indistinctly within a single model.
- 4.2 Classical Image Super-Resolution: Swin2SR-s matches SwinIR (small) results while training almost 33% fewer iterations and outperforming several lightweight SR methods with similar parameters and MACs.The comparison reports PSNR and SSIM alongside parameter counts and multiply-accumulate operations measured on a 1280 × 720 image.
- 4.4 Compressed Image Super-Resolution: For ×4 super-resolution of JPEG-compressed images at quality 10, Swin2SR ranks among the top challenge solutions, with 1.41s average testing time on one A100 GPU.The paper evaluates both an end-to-end solution and a two-stage artifact-removal-plus-upscaling solution using DIV2K and Flickr2K.
- 4.4 Compressed Image Super-Resolution: The compressed-input results retain structural information and high-frequency content but show a clear blur effect, indicating difficulty recovering fine details and sharpness.The same high-frequency-detail issue is reported for SwinIR and other state-of-the-art methods.
5 Conclusion
The paper proposes Swin2SR, a SwinV2 Transformer-based model for super-resolution and restoration of compressed images. It reports state-of-the-art performance across three restoration tasks and a top-five ranking in the AIM 2022 challenge.
- Swin2SR is a SwinV2 Transformer-based model for super-resolution and restoration of compressed images.
- The method achieves state-of-the-art performance for JPEG artifact removal, classical and lightweight super-resolution, and compressed image super-resolution.
- Swin2SR ranks among the top five in the AIM 2022 Challenge on Super-Resolution of Compressed Image and Video.