Source-linked AI summary
CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution
Xin Liu, Jie Liu, Jie Tang, Gangshan Wu
TL;DR
Transformer-based super-resolution methods capture long-range dependencies but commonly restrict attention to content-agnostic local regions to control quadratic complexity. CATANet instead aggregates long-range content-similar tokens and applies intra- and inter-group attention, achieving superior lightweight-SR results, including a reported 0.33 dB maximum PSNR improvement and nearly double inference speed over SPIN.
Problem
Transformer-based super-resolution attention has quadratic spatial complexity, while local windows and stripes limit content-aware long-range information interaction.
Method
CATANet aggregates long-range content-similar tokens with shared token centers and combines intra-group self-attention with inter-group cross-attention.
Results
0.33 dB maximum PSNR improvement and nearly double inference speed over SPIN are reported for CATANet.
Takeaways & Limitations
CATANet achieves superior performance while maintaining high inference efficiency in lightweight image super-resolution.
Abstract
from arXiv · showhide
Transformer-based methods have demonstrated impressive performance in low-level visual tasks such as Image Super-Resolution (SR). However, its computational complexity grows quadratically with the spatial resolution. A series of works attempt to alleviate this problem by dividing Low-Resolution images into local windows, axial stripes, or dilated windows. SR typically leverages the redundancy of images for reconstruction, and this redundancy appears not only in local regions but also in long-range regions. However, these methods limit attention computation to content-agnostic local regions, limiting directly the ability of attention to capture long-range dependency. To address these issues, we propose a lightweight Content-Aware Token Aggregation Network (CATANet). Specifically, we propose an efficient Content-Aware Token Aggregation module for aggregating long-range content-similar tokens, which shares token centers across all image tokens and updates them only during the training phase. Then we utilize intra-group self-attention to enable long-range information interaction. Moreover, we design an inter-group cross-attention to further enhance global information interaction. The experimental results show that, compared with the state-of-the-art cluster-based method SPIN, our method achieves superior performance, with a maximum PSNR improvement of 0.33dB and nearly double the inference speed.
1. Introduction
Transformer-based super-resolution captures long-range dependencies but often restricts attention to content-agnostic local regions for efficiency. CATANet addresses this with content-aware token aggregation and complementary intra- and inter-group attention, reporting stronger performance and efficiency than SPIN.
- SISR reconstructs high-resolution images from low-resolution inputs and supports applications including medical imaging, digital photography, and streaming media.
- Transformers capture long-range dependencies, but local windows and axial stripes limit access to long-range similar tokens and may introduce irrelevant information.
- SPIN uses clustered token centers as attention proxies, but sparse centers provide only a coarse approximation for long-range information propagation.
- CATANet combines content-aware token aggregation, intra-group self-attention, and inter-group cross-attention in a lightweight Token-Aggregation Block.The aggregation module shares token centers across image tokens and updates them only during training.
- 0.33 dB maximum PSNR improvement and nearly double inference speed over SPIN are reported alongside computational complexity comparable to local-region methods.
2. Related Work
Image super-resolution research spans increasingly capable CNN and transformer architectures, while lightweight methods target resource-constrained deployment. Token-clustering approaches aggregate similar tokens for long-range attention but incur substantial clustering costs.
- CNN-based super-resolution methods use residual, U-shaped, or other efficient structures, while transformers improve long-range modeling but often require many parameters.
- Lightweight super-resolution methods address the need to deploy networks on resource-constrained devices.
- Clustering methods such as BOAT and SPIN gather similar tokens and use clusters or aggregated tokens to support attention over long ranges.
- Token-clustering methods show promise but their clustering process has significant computational cost, hindering lightweight super-resolution applications.
3. Method
CATANet’s Token-Aggregation Block combines content-aware grouping with intra-group and inter-group attention to capture fine-grained long-range dependencies efficiently. The network further integrates local-region attention, convolutional refinement, residual groups, and image reconstruction.
- Network Architecture: CATANet extracts shallow features, processes them through sequential residual groups, and reconstructs the super-resolved image with global residual information.Each residual group contains TAB, LRSA, and a 3 × 3 convolution; reconstruction combines the learned residual with an upsampled LR image.
- Local Feature Refinement: LRSA enhances local dependencies and a 3 × 3 convolution refines local features while implicitly learning positional embeddings after TAB processing.The architecture also applies shared projection weights across overlapping local patches and uses residual shortcuts and normalization around transformer-style modules.
- Token-Aggregation Block: TAB combines Content-Aware Token Aggregation, Intra-Group Self-Attention, Inter-Group Cross-Attention, and a 1×1 convolution for efficient fine-grained long-range interaction.Its computational complexity is described as similar to local-region attention.
- Content-Aware Token Aggregation: CATA shares token centers across image tokens, initializes them by average pooling, and updates them with EMA during training to form global content-aware groups.The token centers are not updated during inference, avoiding their impact on inference speed.
- Content-Aware Token Aggregation: Image tokens are grouped by content similarity over long ranges, then divided into fixed-size subgroups to improve parallelism.Subgrouping addresses unequal group sizes; CATANet’s inference speed is approximately doubled when sub-grouping is used.
- Intra-Group Self-Attention: Intra-group attention projects subgroups into Q, K, and V, while each subgroup attends to consecutive subgroup keys and values to preserve adjacent content relationships.Attending to adjacent subgroups enhances performance without adding computational overhead, and PushBack restores tokens to their original feature-map positions.
- Inter-Group Cross-Attention: Inter-group cross-attention lets each subgroup interact with the shared token centers, using M ≪ N to maintain low computational cost.The resulting cross-attention output is fused with intra-group features through convolution.
4. Experiments
CATANet is evaluated against lightweight SR methods through benchmark comparisons, ablations, visualizations, and efficiency analyses. Results show strong reconstruction quality, component benefits, larger receptive fields, and faster inference.
- State-of-the-art comparison: CATANet outperforms compared lightweight SR methods across all benchmark datasets and scaling factors, with 15K fewer parameters than SPIN.The comparison covers CNN- and transformer-based models at ×2, ×3, and ×4.
- State-of-the-art comparison: 0.33 dB PSNR improvement over SPIN is achieved without self-ensemble, while the self-ensemble variant reaches a maximum 0.60 dB gain.The reported metrics are PSNR improvements against SPIN.
- Visualization analysis: TAB produces a larger receptive field and captures more useful long-range token information than the model without TAB, RCAN, and SwinIR-light.LAM visualizations compare the effective receptive fields of these variants and methods.
- Model size and running time analyses: CATANet-L/M/S achieve higher PSNR than other lightweight methods at each evaluated model size; CATANet-M exceeds DITN-Tiny by 0.59dB with fewer than 60K parameters.CATANet-S also exceeds SeemoRe-T by a maximum PSNR improvement of 0.28 dB.
5. Conclusion
CATANet aggregates content-similar tokens into content-aware regions and combines intra-group and inter-group attention for long-range interaction. Experiments report superior lightweight SR performance while maintaining high inference efficiency.
- Conclusion: CATANet uses token centers to aggregate content-similar tokens into content-aware regions.The Token-Aggregation Block is the network’s core component.
- Conclusion: Intra-Group Self-Attention enables fine-grained interaction within content-aware regions, while Inter-Group Cross-Attention enhances global information interaction.The two attention mechanisms operate at complementary interaction levels.
- Conclusion: CATANet achieves superior lightweight image super-resolution performance while maintaining high inference efficiency across benchmark datasets.The conclusion summarizes the reported experimental outcome.