Source-linked AI summary
Efficient Long-Range Attention Network for Image Super-resolution
Xindong Zhang, Hui Zeng, Shi Guo, Lei Zhang
TL;DR
Transformer-based SR offers effective self-attention but incurs high computational cost and includes operations that may be redundant for SR. ELAN combines efficient local extraction, multi-scale group-wise attention, and shared attention in a simple ELAB cascade. It achieves competitive or state-of-the-art SR performance with substantially lower complexity, including about ×4.5 acceleration in an ablation, while some competing latency comparisons are unavailable.
Problem
Self-attention improves feature modeling for SR, but its expensive computation and redundant operations limit efficient long-range attention.
Method
ELAN stacks ELABs that combine two shift-conv operations, group-wise multi-scale self-attention, accelerated attention, and shared attention.
Results
ELAN achieves highly competitive SR performance against previous state-of-the-art models in lightweight and performance-oriented settings with less complexity.
Takeaways & Limitations
ELAN provides a simple and efficient way to capture local structure and long-range dependency for single-image super-resolution.
Takeaways & Limitations
Latency and FLOPs cannot be reported for competing methods whose source codes are unavailable.
Abstract
from arXiv · showhide
Recently, transformer-based methods have demonstrated impressive results in various vision tasks, including image super-resolution (SR), by exploiting the self-attention (SA) for feature extraction. However, the computation of SA in most existing transformer based models is very expensive, while some employed operations may be redundant for the SR task. This limits the range of SA computation and consequently the SR performance. In this work, we propose an efficient long-range attention network (ELAN) for image SR. Specifically, we first employ shift convolution (shift-conv) to effectively extract the image local structural information while maintaining the same level of complexity as 1x1 convolution, then propose a group-wise multi-scale self-attention (GMSA) module, which calculates SA on non-overlapped groups of features using different window sizes to exploit the long-range image dependency. A highly efficient long-range attention block (ELAB) is then built by simply cascading two shift-conv with a GMSA module, which is further accelerated by using a shared attention mechanism. Without bells and whistles, our ELAN follows a fairly simple design by sequentially cascading the ELABs. Extensive experiments demonstrate that ELAN obtains even better results against the transformer-based SR models but with significantly less complexity. The source code can be found at https://github.com/xindongzhang/ELAN.
1 Introduction
Transformer-based SR can model image dependencies with self-attention, but its quadratic cost and redundant components constrain efficient long-range attention. ELAN addresses this with shift-conv, GMSA, shared attention, and a simple ELAB-based architecture.
- CNN-based SR methods often require deep, complicated topologies to enlarge receptive fields, consuming substantial computational resources.
- Self-attention models image dependencies effectively, but its quadratic complexity with input feature size limits use in low-level vision.
- ELAN uses two shift-conv operations to extract local structure with a larger receptive field and 1 × 1-convolution-level arithmetic complexity.
- GMSA divides features into groups with different window sizes, expanding long-range attention more flexibly than fixed windows while controlling computation.
- ELAB combines shift-conv and GMSA with shared attention, and ELAN stacks these blocks to form a simple SR network.
- ELAN reports new state-of-the-art image-SR results with significantly less complexity than existing vision-transformer SR methods.
2 Related work
SR research progressed from increasingly deep CNNs and attention mechanisms toward transformer-based models that capture broader dependencies. However, transformer-derived SR designs can retain unnecessary complexity and small-window attention limitations.
- CNN-based SR evolved from shallow convolutional mappings toward deep residual, dense, and attention-enhanced architectures.
- Many CNN-based methods enlarge receptive fields through deeper and more complicated networks or feature-correlation mechanisms.
- Transformer self-attention models dependencies across data and have been applied to both high-level and low-level vision tasks.
- IPT computes local features and self-attention on non-overlapped patches, which may lose information useful for reproducing image details.
- SwinIR adapts shifted-window transformers to image restoration, but its structure is largely inherited from high-level vision and is described as redundant for SR.
3 Methodology
ELAN uses a compact SR pipeline that extracts shallow and deep features, reconstructs the HR image, and builds ELABs from shift-conv and GMSA. Shared attention and shifted windows further reduce attention cost while extending feature interactions.
- 3.1 Overall Pipeline of ELAN: The ELAN pipeline performs shallow feature extraction, cascaded ELAB-based deep feature extraction, and HR reconstruction with a global shortcut.
- 3.2 Efficient Long-range Attention Block (ELAB): Each ELAB combines residual local-feature extraction with GMSA to progressively increase the receptive field of attention.
- 3.2 Efficient Long-range Attention Block (ELAB): Shift-conv splits features into groups, shifts selected groups spatially, and applies one 1 × 1 convolution to enlarge local receptive fields efficiently.
- 3.2 Efficient Long-range Attention Block (ELAB): Window-based attention costs 2M^2HWC, so GMSA uses groups with different window sizes to model long-range dependencies without directly enlarging one window.
- 3.2 Efficient Long-range Attention Block (ELAB): ASA removes layer normalization and other attention overheads to reduce computation and memory cost without sacrificing SR performance.
- 3.2 Efficient Long-range Attention Block (ELAB): Shared attention reuses scores from one SA module across adjacent modules, avoiding repeated reshape operations and 1 × 1 convolutions.
- 3.2 Efficient Long-range Attention Block (ELAB): Shifted windows circularly shift features before GMSA and invert the shift afterward, introducing connections across neighboring local windows.
4 Experiments
Experiments evaluate ELAN on light-weight and classic SR benchmarks using PSNR/SSIM, latency, parameters, and FLOPs. ELAN generally matches or exceeds competing methods while reducing computational cost, and ablations show how its components trade efficiency against reconstruction quality.
- Experimental setup: Experiments use DIV2K training data and five benchmark datasets, reporting PSNR/SSIM on the Y channel plus latency, parameters, and FLOPs.Latency is measured during inference on a single NVIDIA 2080Ti GPU; parameters and FLOPs are reference efficiency measures.
- Light-weight SR comparison: ELAN-light achieves the best or second-best PSNR/SSIM across five datasets and three scales, while running about ×4.5 faster than SwinIR-light.It also uses fewer parameters and FLOPs than SwinIR-light.
- Light-weight SR comparison: ELAN-light restores clearer and sharper structures than competing light-weight models in qualitative ×4 comparisons, where CNN models often produce blurry or distorted edges.The reported comparisons cover six example images in Figures 3 and 4.
- Classic SR comparison: ELAN achieves the best results on almost all classic-SR benchmarks and upscaling factors, with better PSNR/SSIM than SwinIR in almost all settings and more than ×2 faster inference.The comparison also reports fewer parameters and FLOPs than SwinIR.
- Classic SR comparison: ELAN restores more accurate and sharper edges than representative classic SR models, which commonly yield blurry or inaccurate edges and textures.The qualitative examples include tiger fur and skirt edges in Figures 5 and 6.
- Ablation studies: Ablations show that shifted windows reduce latency from 247ms to 177ms with nearly unchanged performance, while ASA reduces it from 177ms to 66ms without PSNR/SSIM loss.GMSA then improves PSNR by 0.21dB on Urban100 and 0.19dB on Manga109 for only a 9ms latency increase.
- Ablation studies: Shared attention further accelerates ELAN-light with little performance drop, yielding about ×4.5 acceleration while maintaining comparable performance to SwinIR-light.Increasing shared-attention depth reduces parameters, FLOPs, and latency but costs performance; an appropriate depth provides a trade-off.
5 Conclusion
ELAN uses a simple cascade of efficient long-range attention blocks to capture local structure and long-range dependency for single-image super-resolution. Experiments report competitive performance with lower cost than prior transformer-based SR methods.
- ELAN sequentially cascades ELABs, each combining two shift-conv layers with GMSA to expand self-attention's receptive field.Shared attention and accelerated self-attention calculation improve efficiency.
- ELAN captures local structure and long-range dependency efficiently through shift-conv, GMSA, and shared attention.
- ELAN achieves highly competitive performance against previous state-of-the-art SR models in both lightweight and performance-oriented settings.
- ELAN is more economical than previous transformer-based SR methods, although its self-attention remains computation- and memory-intensive compared with lightweight CNNs.The authors identify more efficient self-attention implementations or approximations as future work.