Source-linked AI summary
Dual Aggregation Transformer for Image Super-Resolution
Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, Fisher Yu
TL;DR
Image SR requires recovering HR images from LR inputs, but CNN locality limits global dependency modeling and existing attention methods emphasize separate dimensions. DAT combines spatial and channel aggregation across and within Transformer blocks using alternating attention, AIM, and SGFN. The paper reports superior SR results, including improved visual quality, while retaining lower complexity and model size.
Problem
CNN locality hinders global dependency modeling in image SR, while spatial and channel attention are typically explored separately.
Method
DAT alternates spatial-window and channel-wise self-attention across blocks and uses AIM plus SGFN for intra-block spatial-channel aggregation.
Results
DAT achieves superior visual SR results against recent methods and is reported to outperform state-of-the-art methods with lower complexity and model size.
Takeaways & Limitations
DAT provides a Transformer SR design that aggregates spatial and channel information through complementary inter-block and intra-block mechanisms.
Takeaways & Limitations
SwinIR and CAT-A suffer from blurring artifacts in the discussed comparison.
Abstract
from arXiv · showhide
Transformer has recently gained considerable popularity in low-level vision tasks, including image super-resolution (SR). These networks utilize self-attention along different dimensions, spatial or channel, and achieve impressive performance. This inspires us to combine the two dimensions in Transformer for a more powerful representation capability. Based on the above idea, we propose a novel Transformer model, Dual Aggregation Transformer (DAT), for image SR. Our DAT aggregates features across spatial and channel dimensions, in the inter-block and intra-block dual manner. Specifically, we alternately apply spatial and channel self-attention in consecutive Transformer blocks. The alternate strategy enables DAT to capture the global context and realize inter-block feature aggregation. Furthermore, we propose the adaptive interaction module (AIM) and the spatial-gate feed-forward network (SGFN) to achieve intra-block feature aggregation. AIM complements two self-attention mechanisms from corresponding dimensions. Meanwhile, SGFN introduces additional non-linear spatial information in the feed-forward network. Extensive experiments show that our DAT surpasses current methods. Code and models are obtainable at https://github.com/zhengchen1999/DAT.
1. Introduction
Image SR must recover an HR image from an LR input despite multiple possible solutions, while CNN locality limits global dependency modeling. DAT addresses this by aggregating spatial and channel information across and within Transformer blocks, with experiments reporting stronger results and lower complexity and model size.
- Motivation: Image SR recovers a high-resolution image from a low-resolution counterpart, but its ill-posedness permits multiple solutions.
- Motivation: CNNs use local convolutions, hindering global dependencies and restricting image-SR performance.
- Dual Aggregation Transformer: DAT alternates spatial-window and channel-wise self-attention in successive blocks to capture spatial and channel context across dimensions.
- Dual Aggregation Transformer: AIM couples parallel convolution and self-attention branches through spatial- and channel-interaction operations for intra-block aggregation.
- Dual Aggregation Transformer: SGFN inserts a spatial-gate module between feed-forward layers, adding nonlinear spatial information and relieving channel redundancy.
- Results: Experiments report that DAT outperforms state-of-the-art SR methods while retaining lower complexity and model size.
2. Related Work
Prior image-SR research includes deep CNNs and spatial- or channel-attention mechanisms, while Transformer work develops efficient attention for vision. Feature-aggregation studies motivate combining spatial and channel modeling, which CNN methods still struggle to perform globally in both dimensions.
- Image Super-Resolution: Deep CNN methods established strong image-SR performance, including SR-CNN and RCAN's residual-in-residual architecture.
- Image Super-Resolution: Image-SR methods use attention mechanisms along spatial or channel dimensions to improve modeling ability.
- Image Super-Resolution: Most CNN-based methods still struggle to model global dependencies in both spatial and channel dimensions.
- Vision Transformer: Vision Transformer research improves efficiency and effectiveness through designs such as local windows, shifted interactions, and dual self-attention.
- Feature Aggregation: Feature-aggregation research combines spatial and channel attention across CNN and Transformer architectures to enhance feature expressions.
3. Method
DAT is an image super-resolution Transformer that aggregates spatial and channel features across successive blocks and within each block. Its architecture combines alternating spatial/channel attention with AIM and SGFN to enrich feature representations.
- Architecture: DAT processes LR images through shallow extraction, stacked residual groups of DATBs, and pixel-shuffle reconstruction into HR outputs.Convolution produces shallow features, while reconstruction aggregates features before and after upsampling.
- Inter-block Aggregation: Each DATB pair alternates a spatial Transformer block and a channel Transformer block to aggregate complementary spatial and channel context across blocks.DSTB uses spatial-window attention, whereas DCTB uses channel-wise attention.
- Self-attention: Spatial-window self-attention projects input features into query, key, and value matrices, partitions them into windows, and recombines attended outputs.Shifted windows are used by default to capture additional spatial information.
- Self-attention: Channel-wise self-attention reshapes projected features across spatial positions, divides channels into heads, and concatenates the resulting channel-attended features.A learnable temperature parameter adjusts inner products before softmax.
- Intra-block Aggregation: AIM adaptively fuses parallel convolution and self-attention branches through spatial- and channel-interaction operations.These interactions re-weight features from the complementary dimension, producing adaptive spatial and channel self-attention.
- Intra-block Aggregation: SGFN inserts a spatial-gate module between FFN projections, using depth-wise convolution and element-wise multiplication to add nonlinear spatial information and reduce channel redundancy.The depth-wise convolution is intended to maintain computational efficiency.
4. Experiments
Experiments evaluate DAT through implementation details, ablations, benchmark comparisons, visual results, and complexity analyses. The results show benefits from alternating spatial and channel attention and from the proposed intra-block modules, while maintaining competitive efficiency.
- Experimental Settings: DAT-S and DAT use different complexity settings, with DAT enlarging the SGFN channel expansion factor and spatial-attention window.DAT-S uses 6 residual groups with 3 DSTB–DCTB pairs per group; DAT uses expansion factor 4 and an 8×32 window, compared with 2 and 8×16 for DAT-S.
- Alternate Strategy: Alternating channel-wise and spatial-window self-attention produces sharper textures and edges than using either attention strategy alone.Feature maps before upsampling show improved texture and edge sharpness for the alternating strategy.
- Adaptive Interaction Module: AIM raises PSNR from 33.41 to 33.52 dB after parallel depth-wise convolution is added to self-attention.The convolution branch first gives a 0.07 dB gain over the 33.34 dB baseline; complete bidirectional AIM performs best among the interaction variants.
- Spatial-Gate Feed-Forward Network: SGFN reduces parameters to 2.18M and FLOPs to 32.25G relative to regular FFN while improving performance.Removing depth-wise convolution highlights the role of spatial information, while removing channel splitting increases model size and complexity and slightly lowers PSNR.
- Quantitative Results: DAT+ outperforms compared methods across all benchmark datasets at ×2, ×3, and ×4 scales, while DAT is generally stronger than prior methods.On Manga109 at ×2, DAT improves over SwinIR and CAT-A by 0.41 dB and 0.23 dB, respectively; DAT trails CAT-A only in Urban100 ×4 PSNR.
- Visual Results: On challenging ×4 examples, DAT preserves more structures and finer details while mitigating blurring artifacts, distortions, and inaccurate texture restoration.The visual comparison highlights clearer textures and correct structures in img_015, with similar observations in img_047 and img_049.
- Model Size Analyses: DAT provides comparable or better ×4 performance than CAT-A with lower complexity and model size, while DAT-S uses fewer FLOPs and parameters than SwinIR.Complexity comparisons report PSNR on Urban100 and Manga109 together with FLOPs and parameter counts.
5. Conclusion
The paper concludes that DAT aggregates spatial and channel features through alternating attention across blocks and AIM and SGFN within blocks. Extensive experiments indicate that DAT outperforms previous image SR methods.
- Conclusion: DAT alternates spatial-window and channel-wise self-attention across successive Transformer blocks to model global dependencies and aggregate features between dimensions.This inter-block strategy captures spatial and channel context.
- Conclusion: AIM and SGFN enhance individual blocks by complementing self-attention and feed-forward processing with cross-dimensional interaction and nonlinear spatial information.Together, the modules provide intra-block feature aggregation.
- Conclusion: Extensive experiments indicate that DAT outperforms previous image SR methods.