Source-linked AI summary
HiT-SR: Hierarchical Transformer for Efficient Image Super-Resolution
Xiang Zhang, Yulun Zhang, Fisher Yu
TL;DR
Transformer-based SR methods often use fixed small windows because larger windows make self-attention computationally costly, limiting multi-scale feature aggregation. HiT-SR uses expanding hierarchical windows and spatial-channel correlation; its variants report improved performance and efficiency over their original models.
Problem
Fixed small windows limit transformer-based SR to a single scale, while window self-attention has quadratic complexity in window size, making larger receptive fields costly.
Source: §1 · p. 2Method
HiT-SR replaces fixed small windows with expanding hierarchical windows and uses spatial-channel correlation to aggregate their features with linear complexity to window sizes.
Source: §1 · p. 2Results
HiT-SIR, HiT-SNG, and HiT-SRF achieve better performance with fewer parameters and FLOPs, and ∼7× faster speeds than their original models.
Source: §1 · p. 3Takeaways & Limitations
The strategy converts SwinIR-Light, SwinIR-NG, and SRFormer-Light into hierarchical SR transformers while retaining efficient design.
Source: §1 · pp. 2–3
Abstract
from arXiv · showhide
Transformers have exhibited promising performance in computer vision tasks including image super-resolution (SR). However, popular transformer-based SR methods often employ window self-attention with quadratic computational complexity to window sizes, resulting in fixed small windows with limited receptive fields. In this paper, we present a general strategy to convert transformer-based SR networks to hierarchical transformers (HiT-SR), boosting SR performance with multi-scale features while maintaining an efficient design. Specifically, we first replace the commonly used fixed small windows with expanding hierarchical windows to aggregate features at different scales and establish long-range dependencies. Considering the intensive computation required for large windows, we further design a spatial-channel correlation method with linear complexity to window sizes, efficiently gathering spatial and channel information from hierarchical windows. Extensive experiments verify the effectiveness and efficiency of our HiT-SR, and our improved versions of SwinIR-Light, SwinIR-NG, and SRFormer-Light yield state-of-the-art SR results with fewer parameters, FLOPs, and faster speeds ($\sim7\times$).
1 Introduction
Transformer-based SR methods face a trade-off: fixed small windows limit multi-scale information, while larger windows incur quadratic costs. HiT-SR combines expanding hierarchical windows with spatial-channel correlation to use larger receptive fields efficiently.
1 Introduction
- Fixed small windows limit transformer-based SR to a single scale, while quadratic window-attention costs make larger receptive fields expensive.Prior approaches reduce channels to support larger windows, but retain quadratic complexity and trade off spatial and channel information.Source: §1 · p. 2
- HiT-SR replaces fixed small windows with expanding hierarchical windows to aggregate multi-scale features and establish long-range dependencies.The approach converts popular transformer-based SR networks into hierarchical transformers.Source: §1 · pp. 2–3
- Spatial-channel correlation combines spatial and channel information and aggregates hierarchical features with linear complexity to window sizes.Its design includes dual feature extraction and spatial and channel self-correlation, and avoids softmax and time-consuming window shifting.Source: §1 · p. 3
- HiT-SIR, HiT-SNG, and HiT-SRF achieve better performance with fewer parameters and FLOPs, and approximately 7× faster speeds than their original models.The figure compares SwinIR-Light, SwinIR-NG, and SRFormer-Light with their corresponding HiT-SR versions.Source: Figure 1 · p. 2, §1 · p. 3
2 Related Work
Prior SR approaches include efficient CNN designs and transformer methods that establish long-range dependencies. The related work identifies the computational cost of large-window attention as a barrier to hierarchical features, which HiT-SR addresses with spatial-channel correlation.
Transformer-based SR
- Transformer-based SR methods use approaches such as N-Gram, omni self-attention, and permuted self-attention, while large-window attention makes hierarchical features difficult to use.The cited methods are SwinIR, N-Gram, OSA, and PSA.Source: §2 · p. 3
Efficient SR
- Efficient CNN approaches use compact blocks, information distillation, lattice filters, and pruning, but often struggle to utilize long-range dependencies.The passage names CARN, IMDN, LatticeNet, and pruning methods.Source: §2 · p. 3
Hierarchical Feature
- ELAN's group-wise multi-scale self-attention remains quadratic in window size and trades off channel information through channel splitting; HiT-SR's SCC uses linear complexity while preserving spatial and channel information.This contrast motivates the use of SCC to efficiently exploit hierarchical features.Source: §2 · pp. 3–4
3 Method
HiT-SR converts transformer-based SR networks into hierarchical transformers by combining expanding windows with spatial-channel correlation. This design gathers multi-scale and long-range information while supporting larger windows with linear computational complexity in window size.
3.1 Hierarchical Transformer
- HiT-SR replaces fixed small-window attention with hierarchical windows and spatial-channel correlation to aggregate multi-scale features efficiently.The correlation method is designed to support window scaling with linear computational complexity in window size.Source: §3.1 · pp. 4–5
3.2 Block-Level Design: Hierarchical Windows
- Expanding windows progress from small windows in early layers to larger windows in later layers, building receptive fields without shifting fixed windows.This uses cascaded transformer layers to gather local features first and then information from longer-range dependencies.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
3.3 Layer-Level Design: Spatial-Channel Correlation
- Dual feature extraction combines spatial features from a convolutional branch with channel features from a linear branch through element-wise multiplication.The spatial branch uses an hourglass structure with three convolutional layers and a reduced hidden dimension for efficiency.Source: §3.3 · p. 6
- Spatial self-correlation aggregates spatial information using projected values, avoids softmax, and has linear computational complexity in window size.Its projected values are bounded by the fixed base window size, allowing larger windows to be processed efficiently.Source: §3.3 · p. 7
- Channel self-correlation uses a single head to enable full channel interaction, while spatial self-correlation uses multiple heads across channel subspaces.The single-head strategy avoids limiting channel aggregation to a subset of other channels.Source: §3.3 · pp. 7–8
4 Experiments and Analysis
HiT-SR improves efficient transformer-based super-resolution through hierarchical windows and spatial-channel correlation. Experiments report stronger benchmark performance, faster convergence, broader information aggregation, and favorable ablation results.
4.1 Experimental Settings
- Models use DIV2K training and five benchmark datasets, with bicubic degradation and PSNR/SSIM evaluation on the Y channel; complexity is measured at a 720 × 1280 output size.The improved models use the same settings, and Table 3 reports complexity at ×2 on an A100 GPU.Source: §4.1 · pp. 8–9, Table 2 · p. 9, Table 3 · p. 10
4.2 Benchmarking
- HiT-SRF sets state-of-the-art results across upscaling factors; HiT-SIR matches SRFormer-Light with fewer parameters and FLOPs, while HiT-SIR/SNG/SRF gain 0.26/0.23/0.22 dB PSNR on Urban100 (×2).HiT-SR models achieve ∼7× inference speed-up; HiT-SRF is 7.8× faster than SRFormer-Light.Source: Table 3 · p. 10, §4.2 · p. 10
- On Urban100 (×2), HiT-SIR gains 0.26 dB PSNR over its original model; on Urban100 and Manga109 (×2), it converges faster than SwinIR-Light and achieves similar SR performance in around half the iterations under the same training settings.In challenging scenes, HiT-SIR and HiT-SRF produce finer details and sharper textures, and HiT-SR methods better retain overall structure.Source: §4.2 · pp. 10–11, Figure 5 · p. 10
4.3 Method Analysis
- LAM comparisons show HiT-SIR expands informative areas beyond SwinIR-Light’s local regions, supporting wider information aggregation through hierarchical features.The paper identifies similar structures and repetitive patterns as examples of information gathered from wider regions.Source: §4.3 · p. 12, Figure 7 · p. 12
- SCC has linear complexity in window size, whereas W-SA rises sharply and PSA remains quadratic, supporting efficient scaling to larger windows.The comparison uses 60 channels and 6 heads for each method.Source: §4.3 · p. 12, Figure 8 · p. 12
4.4 Ablation Study
- Expanding hierarchical windows outperform fixed-small, fixed-large, and shrinking-window variants, combining multi-scale features with a small-to-large progression.The stated rationale is that small windows first use relevant information and improve feature representations before larger windows establish long-range dependencies.Source: §4.4 · pp. 12–13, Table 4 · p. 13
- DFE’s two-branch fusion of spatial and channel information improves performance over linear and convolutional projection alternatives.The cited alternatives are limited to channel information or, in the case of efficiency-oriented convolution, insufficient channel information.Source: §4.4 · pp. 13–14, Table 4 · p. 13
- The QV setting performs similarly to QKV with lower computation; removing softmax and combining S-SC with C-SC improve the reported results.C-SC alone performs better than S-SC, while the combined spatial and channel information gives the best overall performance.Source: Table 4 · p. 13, §4.4 · p. 14
5 Conclusion
HiT-SR converts transformer-based SR networks into hierarchical transformers using expanding windows and spatial-channel correlation. The paper reports state-of-the-art results for its improved models while maintaining efficient feature aggregation.
5 Conclusion
- HiT-SR applies expanding hierarchical windows to leverage multi-scale features and establish long-range dependencies in transformer blocks.The paper presents this as a general strategy for converting popular transformer-based SR methods.Source: §5 · p. 14
- Its spatial-channel correlation method has linear complexity in window size, enabling efficient aggregation of hierarchical features.The conclusion contrasts this design with the quadratic complexity of self-attention methods.Source: §5 · p. 14
- HiT-SIR, HiT-SNG, and HiT-SRF are reported to set new state-of-the-art results for efficient image SR.Source: §5 · p. 14