Source-linked AI summary
SwinFIR: Revisiting the SwinIR with Fast Fourier Convolution and Improved Training for Image Super-Resolution
Dafeng Zhang, Feiyu Huang, Shizhuo Liu, Xiaobing Wang, Zhezhu Jin
TL;DR
SwinIR’s local receptive fields limit global information extraction for image super-resolution; SwinFIR replaces deep-feature convolutions with spatial-frequency blocks based on Fast Fourier Convolution. It achieves state-of-the-art performance across benchmarks, exceeding SwinIR by 0.30–0.80 dB on Manga109.
Problem
SwinIR’s local receptive fields limit extraction of global image information, which is important for image super-resolution.
Source: §1 · p. 1, §3.1 · p. 3Method
SwinFIR replaces SwinIR’s deep-feature convolution with a spatial-frequency block that uses Fast Fourier Convolution for global features and a residual module for local features.
Source: §1 · p. 2Results
Across all benchmarks, SwinFIR achieves state-of-the-art performance, exceeding SwinIR by 0.30–0.80 dB on Manga109.
Source: §1 · p. 2Takeaways & Limitations
Feature ensemble improves super-resolution performance without increasing training or testing time.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Transformer-based methods have achieved impressive image restoration performance due to their capacities to model long-range dependency compared to CNN-based methods. However, advances like SwinIR adopts the window-based and local attention strategy to balance the performance and computational overhead, which restricts employing large receptive fields to capture global information and establish long dependencies in the early layers. To further improve the efficiency of capturing global information, in this work, we propose SwinFIR to extend SwinIR by replacing Fast Fourier Convolution (FFC) components, which have the image-wide receptive field. We also revisit other advanced techniques, i.e, data augmentation, pre-training, and feature ensemble to improve the effect of image reconstruction. And our feature ensemble method enables the performance of the model to be considerably enhanced without increasing the training and testing time. We applied our algorithm on multiple popular large-scale benchmarks and achieved state-of-the-art performance comparing to the existing methods. For example, our SwinFIR achieves the PSNR of 32.83 dB on Manga109 dataset, which is 0.8 dB higher than the state-of-the-art SwinIR method.
1. Introduction
SwinFIR extends SwinIR with a spatial-frequency block for global and local feature extraction, and combines it with data augmentation and feature ensemble. The paper reports state-of-the-art results across benchmarks, including gains over SwinIR and EDT on Manga109 and Urban100.
1. Introduction
- SwinIR’s shifted-window self-attention expands its receptive field, but the limited shift cannot effectively capture global information in early layers.The introduction motivates global information as essential to super-resolution because it can activate more pixels and benefit image reconstruction.Source: §1 · p. 1
- SwinFIR replaces deep-feature-extraction convolutions with a Spatial Frequency Block that uses Fast Fourier Convolution for global features and a residual spatial branch for local features.The authors present the block as a way to use global information for image super-resolution while enhancing local feature expression.Source: §1 · p. 2
- SwinFIR reports state-of-the-art performance on all benchmarks, with gains of 0.30–0.80 dB over SwinIR on Manga109 and 0.24–0.44 dB over EDT on Urban100.Figure 1 compares SwinFIR with SwinIR and EDT at ×2, ×3, and ×4 scales and identifies SwinFIR as best at each scale.Source: Figure 1 · p. 1, §1 · p. 2
- The authors report that channel shuffle and Mixup augmentation can considerably improve super-resolution performance, challenging the assumption that pixel-domain augmentation harms the task.They motivate augmentation as a way to inhibit overfitting and improve generalization.Source: §1 · p. 2
- Feature ensemble combines multiple trained models into a more comprehensive model and improves performance without increasing training or testing time.The authors describe it as a zero-cost post-processing strategy inspired by self-ensemble.Source: §1 · p. 2
2. Related Works
Related work traces the development of CNN- and Transformer-based super-resolution, motivating SwinFIR’s use of large-receptive-field operations in early network layers.
2. Related Works
- CNN-based SR progressed from SRCNN [11] to EDSR [29], RCAN [45], and HAN [33], but CNN receptive-field limitations constrain remote modeling.RCAN models interdependencies between feature channels, while HAN extends attention across layers, channels, and locations.Source: §2 · p. 2
- IPT [3], HAT [4], and SwinIR [28] apply Transformer approaches to low-level vision and image restoration, but leave Transformer potential underexploited.HAT combines channel attention, self-attention, and overlapping cross-attention.Source: §2 · p. 2
- Inspired by LaMa [36], SwinFIR introduces SFB based on FFC to apply large-receptive-field operations in early layers and exploit long dependencies.The passage says these operations use more input pixels for better performance.Source: §2 · p. 2
3. Methodology
SwinFIR extends SwinIR with a global-context Spatial Frequency Block, while revisiting loss, augmentation, and feature-ensemble strategies to improve super-resolution with little or no added computation. Its feature ensemble combines selected models without lengthening training or testing.
3. Methodology
- The methodology revisits low- or no-extra-cost strategies, including loss and augmentation, and examines more training data, larger Swin Transformer windows, and pre-training.The paper describes loss and data augmentation as training- and testing-free methods; it does not discuss the other examined strategies further here.Source: §3 · p. 3
3.1. Model Design
- SwinFIR replaces SwinIR’s local 3×3 convolution with a Spatial Frequency Block that fuses spatial features and Fast Fourier Convolution’s global features.The SFB splits features into spatial and frequency domains, applies convolution and 2-D FFT/inverse FFT processing, then concatenates and convolves their outputs; the spatial branch uses a residual module for classical SR and a hourglass residual module for lightweight SR.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
3.2. Loss Function
- SwinFIR uses Charbonnier loss [23], which the authors report performs better than other loss functions.The loss optimizes the model against high-resolution targets during training.Source: §3.2 · p. 4
3.3. Data Augmentation
- Pixel-domain augmentation explores RGB channel shuffle, Mixup, Blend, CutMix, and CutMixup alongside flips and rotations; all except CutMix and CutMixup improve performance.The authors attribute the exceptions to their disruption of visual continuity.Source: §3.3 · p. 5
3.4. Feature Ensemble
- Feature ensemble combines validation-selected models using a weighted average, steadily improving performance without lengthening training or testing.Unlike multi-model and self-ensemble inference, it combines model features rather than inference results or transformed outputs; the authors say it can apply to low- and high-level tasks.Source: §3.4 · p. 5
4. Experiments · 4.1. Datasets
SwinFIR is pre-trained on ImageNet 2012, then fine-tuned for Classical and Lightweight SR on cropped DF2K data, with benchmark validation and training augmentation.
4.1. Datasets
- SwinFIR is pre-trained on ImageNet 2012, following EDT [26].Source: §4.1 · p. 5
- For Classical and Lightweight SR fine-tuning, 384×384 sub-images are cropped from DF2K, combining DIV2K [29] and Flicker2K.Source: §4.1 · p. 5
- Validation uses Set5, Set14, BSD100, Urban100, and Manga109 for Classical and Lightweight SR.Source: §4.1 · p. 5
- Training augmentation includes horizontal and vertical flips, rotation, channel shuffle, and Mixup to improve robustness.Source: §4.1 · p. 5
4.2. Implementation Details
SwinFIR replaces SwinIR’s RSTB convolution with a Spatial Frequency Block (SFB) based on Fast Fourier Convolution (FFC). The implementation also specifies configurations for classical, lightweight, and stereo super-resolution, plus optimization and training schedules.
4.2. Implementation Details
- SwinFIR replaces the convolution layer in SwinIR’s RSTB with an FFC-based Spatial Frequency Block (SFB).The authors describe FFC as an efficient global feature extractor.Source: §4.2 · p. 5
- For classical image SR, SwinFIR uses SwinIR’s configuration with a window size of 12 and patch size of 60.The authors report selecting the larger window and patch sizes after investigating their effect on SR performance.Source: §4.2 · p. 5
- For lightweight image SR, SwinFIR uses 4 RSTBs and 60 channels, with 5 STL in the second and third RSTBs.The authors state that using 5 STL in those RSTBs accelerates training and inference.Source: §4.2 · p. 5
- For stereo SR, SwinFIRSSR replaces HAT’s 3×3 convolution with an FFC-based SFB and uses SCAM to fuse left- and right-view features.The extension follows NAFSSR [6] and HAT [4]; the architecture is shown in Figure 6.Source: §4.2 · p. 5, Figure 6 · p. 6
- Training uses Adam with β1 = 0.9, β2 = 0.99, zero weight decay, and Charbonnier loss; pre-training runs 1,000,000 iterations before fine-tuning at 1e-5.The pre-training learning rate starts at 2e-4 and is halved at 500,000, 800,000, 900,000, and 950,000 iterations.Source: §4.2 · pp. 5–6
4.3. Comparison to state-of-the-arts methods
SwinFIR achieves strong results across classical, lightweight, and stereo image super-resolution comparisons. In classical SR, FFC supports broader use of input pixels and recovery of high-frequency detail.
4.3.1 Classical Image Super-Resolution
- SwinFIR achieves the best classical SR performance at ×2, ×3, and ×4; on ×4 Urban100 and Manga109, its PSNR is 28.12 dB and 32.83 dB, versus SwinIR’s 27.45 dB and 32.03 dB, respectively.SwinFIR is trained on ImageNet and fine-tuned on DF2K; even without pre-training, it achieves better or comparable performance to pre-trained EDT, with a 0.13 dB advantage on ×4 Manga109.Source: §4.3.1 · pp. 6–7, Table 1 · p. 7
- SwinFIR restores clearer high-frequency detail, especially in images with periodic transformations, while its LAM results show broader input use than SwinIR and EDT.The LAM attribution map indicates that almost all input LR pixels contribute to reconstruction; the passages attribute this wider pixel use to FFC’s long dependencies and note that current CNN and Transformer methods struggle on challenging samples.Source: §4.3.1 · pp. 7–8
4.3.3 Stereo Image Super-Resolution
- SwinFIRSSR surpasses the compared single-image and stereo SR methods; on ×2 stereo SR, it exceeds parameter-comparable NAFSSR-L by 0.19 dB, 0.20 dB, 0.64 dB, and 0.46 dB on KITTI2012, KITTI2015, Middlebury, and Flickr1024, respectively.All models are trained on 800 Flickr1024 and 60 Middlebury images; SwinFIRSSR’s reconstructed images are described as clearer and richer in details and textures.Source: §4.3.3 · pp. 8–9
4.3.2 Lightweight Image Super-Resolution
- At comparable parameter counts, lightweight SwinFIR achieves 31.50 dB PSNR on ×4 Manga109, 0.58 dB above SwinIR and 0.15 dB above EDT-T.It achieves the best results across all reported metrics and better or comparable performance to EDT with a smaller window size; visual results are described as sharper with more high-frequency detail.Source: §4.3.2 · p. 8, Table 2 · p. 9
4.4. Ablation Study
The ablations show gains from selected data augmentations, feature ensemble, and applying SwinFIR components across model architectures and low-level restoration tasks. They also motivate SFB’s local-feature role and reject augmentation schemes that disrupt visual continuity.
4.4.2 Impact of Data Augmentations
- The reported augmentations improved PSNR from 32.78 dB to 32.93 dB, a 0.15 dB gain over flip and rotation, as shown in Figure 2.Augmentations that insert new pixels sometimes improve super-resolution performance. CutMix and CutMixup can disrupt visual continuity and reduce useful information.Source: §4.4.2 · pp. 10–11
4.4.1 Impact of RFB
- Replacing the RSTB convolution with the original FB does not yield the expected SR improvement, whereas SFB adds a residual module to extract input-image local features.LAM indicates a higher contribution from local information; HSTL performs better than SFB on Manga but uses 2.6 times as many parameters (37.20M versus 14.03M), so it is rejected.Source: §4.4.1 · p. 9, Table 4 · p. 11
4.4.3 Robustness
- 0.16 dB higher PSNR: HAT FIR reaches 32.64 dB versus HAT’s 32.48 dB on Manga109 (×4).Adding feature ensemble to HAT FIR improves performance further.Source: Table 5 · p. 11, §4.4.3 · p. 11
- 0.08 dB higher PSNR on DIV2K: applying the method to ABPN [14] improves performance without adding inference time.SwinFIR also improves results on grayscale and color image denoising and JPEG artifact reduction benchmarks.Source: Table 6 · p. 11, §4.4.3 · p. 11
4.4.4 Impact of Feature Ensemble
- Feature ensemble improves super-resolution performance while requiring only one inference, unlike self-ensemble’s longer inference.It ensembles parameters of trained models rather than their output results.Source: §4.4.4 · pp. 11–12
5. Conclusion
SwinFIR extends SwinIR with an FFC-based global feature extractor to capture long dependencies, alongside strategies for improving super-resolution. The paper reports that feature ensemble improves performance without increasing training or testing time, and that experiments on popular benchmarks show SwinFIR surpassing current models.
5. Conclusion
- SwinFIR uses Fast Fourier Convolution as an efficient global feature extractor, giving it a wider receptive field than SwinIR and enabling it to use long dependencies.The authors attribute its use of more pixels for better performance to FFC-based long-dependency modeling.Source: §5 · p. 12
- The paper also revisits data augmentation, loss function, pre-training, and feature ensemble as strategies for improving super-resolution performance.Source: §5 · p. 12
- Feature ensemble steadily improves performance without lengthening training or testing time.Source: §5 · p. 12
- Experiments on popular benchmarks show that SwinFIR surpasses current models and that the proposed methods can significantly improve image-restoration performance.Source: §5 · p. 12