Source-linked AI summary
Omni Aggregation Networks for Lightweight Image Super-Resolution
Hang Wang, Xuanhong Chen, Bingbing Ni, Yutian Liu, Jinfan Liu
TL;DR
Lightweight ViT-based super-resolution models are limited by uni-dimensional spatial or channel interactions and homogeneous aggregation that can constrain their effective receptive fields. Omni-SR addresses these issues with Omni Self-Attention and multi-scale aggregation, achieving state-of-the-art lightweight performance, including 26.95dB on Urban100 ×4 with 792K parameters.
Problem
Lightweight ViT-based super-resolution models are limited by uni-dimensional self-attention and homogeneous aggregation, which restrict comprehensive spatial-channel interactions and effective receptive fields.
Method
Omni-SR combines Omni Self-Attention for simultaneous spatial-channel interaction with an omni-scale aggregation scheme for local, meso-, and global-scale feature aggregation.
Results
Omni-SR achieves state-of-the-art lightweight super-resolution performance, including 26.95dB on Urban100 ×4 and 31.50dB on Manga109 ×4.
Takeaways & Limitations
Omni-SR provides a lightweight framework that combines omni-axis interaction with progressively hierarchical contextual aggregation for image super-resolution.
Abstract
from arXiv · showhide
While lightweight ViT framework has made tremendous progress in image super-resolution, its uni-dimensional self-attention modeling, as well as homogeneous aggregation scheme, limit its effective receptive field (ERF) to include more comprehensive interactions from both spatial and channel dimensions. To tackle these drawbacks, this work proposes two enhanced components under a new Omni-SR architecture. First, an Omni Self-Attention (OSA) block is proposed based on dense interaction principle, which can simultaneously model pixel-interaction from both spatial and channel dimensions, mining the potential correlations across omni-axis (i.e., spatial and channel). Coupling with mainstream window partitioning strategies, OSA can achieve superior performance with compelling computational budgets. Second, a multi-scale interaction scheme is proposed to mitigate sub-optimal ERF (i.e., premature saturation) in shallow models, which facilitates local propagation and meso-/global-scale interactions, rendering an omni-scale aggregation building block. Extensive experiments demonstrate that Omni-SR achieves record-high performance on lightweight super-resolution benchmarks (e.g., 26.95 dB@Urban100 $\times 4$ with only 792K parameters). Our code is available at \url{https://github.com/Francis0625/Omni-SR}.
1. Introduction
Lightweight ViT-based super-resolution models face limitations from uni-dimensional attention and homogeneous aggregation, motivating Omni-SR’s omni-axis and multi-scale design. The framework reports state-of-the-art lightweight performance, including 26.95dB on Urban100 ×4 and 31.50dB on Manga109 ×4.
- Motivation: Lightweight ViT-based super-resolution models remain difficult to improve despite progress from larger ViT-based frameworks.The paper focuses on boosting restoration performance under a typically sub-1M-parameter budget.
- Challenges: Uni-dimensional spatial-only or channel-only aggregation ignores explicit information from the other dimension.The paper identifies this incomplete interaction as a central limitation of lightweight self-attention models.
- Challenges: Homogeneous operators can saturate interaction range prematurely, producing a suboptimal effective receptive field in shallow lightweight models.Lightweight models cannot stack enough layers to compensate for this limitation.
- Contributions: Omni Self-Attention simultaneously exploits spatial and channel information through cascaded cross-covariance computations.OSA is designed as a plug-in component for mainstream self-attention variants and provides higher-order receptive-field information.
- Results: 26.95dB on Urban100 ×4 and 31.50dB on Manga109 ×4 are reported for Omni-SR at the lightweight model scale.The framework also shows faster convergence, a smoother loss landscape, and better robustness than existing ViT-based super-resolution frameworks.
2. Related Works
Related work covers CNN-based and lightweight transformer approaches to image super-resolution, including attention mechanisms and convolution-transformer combinations. The supplied passages emphasize resource-efficient architectural design but provide limited detailed discussion of the related-work methods.
- Image Super-resolution: CNN-based super-resolution methods use residual, recursive, and local-global learning designs to improve reconstruction quality or reduce parameters.The passage introduces SRCNN and mentions skip connections, DRCN, and DRRN as representative approaches.
- Image Super-resolution: Lightweight super-resolution architectures explore compact designs for obtaining better reconstruction quality under limited computing resources.The related-work passage situates lightweight architectural design as a response to resource constraints.
- Lightweight Vision Transformer: Lightweight vision transformers combine convolutional and transformer components to learn local and global representations.LVT and MobileViT are cited as examples of this design direction.
OSAG OSAG ... OSAG
The Omni-SR architecture combines shallow feature extraction, deep feature extraction, and image reconstruction, with OSAG serving as the named aggregation component. The supplied passages identify OSAG in the overall structure but do not fully describe its internal stages.
- Overall Structure: Omni-SR comprises shallow feature extraction, deep feature extraction, and image reconstruction.The supplied architecture label also identifies Omni-Scale Aggregation Group within the framework.
- Overall Structure: The framework uses ViT architecture to pursue lightweight and accurate image super-resolution.The supplied passage does not provide further implementation details for OSAG.
3. Methodology
Omni-SR combines Omni Self-Attention, which models spatial and channel interactions, with omni-scale aggregation spanning local, meso, and global levels. These components form a lightweight architecture that progressively enlarges receptive-field interactions while reconstructing high-resolution images from shallow and deep features.
- Omni Self-Attention: Spatial attention and channel attention provide complementary capabilities, motivating their compact integration for stronger representation.Spatial attention does not transmit information between channels, whereas covariance-based channel attention enables channel re-weighting and information transmission.
- Omni Self-Attention: OSA jointly models spatial and channel context through sequential matrix operations, rotation, and spatial- and channel-wise attention.Window strategies reduce resource overhead, while the channel-wise stage models channel relations before inverse rotation produces the final aggregation.
- Omni-Scale Aggregation Group: OSAG progressively aggregates local, meso, and global information to enlarge the receptive field with low computational complexity.Its local stage uses a channel-attention-enhanced inverted bottleneck, while Meso-OSA and Global-OSA handle meso- and global-scale interactions.
- Omni-Scale Aggregation Group: Meso-OSA partitions features into non-overlapping P × P blocks, whereas Global-OSA divides them into a uniform G × G grid with adaptive lattice size.The two OSA variants differ only in their window partition strategies.
- Network Architecture: Omni-SR extracts shallow features, cascades K OSAGs and a convolution for deep features, then reconstructs the HR image from fused shallow and deep features.A 3 × 3 convolution performs shallow extraction, and PixelShuffle is used for upsampling in reconstruction.
- Network Architecture: Each OSAG contains a local convolution block, meso-OSA block, global-OSA block, and ESA block arranged with convolutional aggregation and residual feature fusion.The residual path combines the OSAG output with its input before the subsequent convolution and ESA processing.
4. Experiments
Experiments show that Omni-SR delivers strong lightweight super-resolution performance across standard benchmarks, while its omni-axis attention and multi-scale aggregation improve reconstruction, efficiency, and optimization behavior.
- Quantitative comparison: Omni-SR surpasses existing lightweight methods with comparable model sizes across five benchmark datasets, including transformer baselines SwinIR and ESRT.The comparison uses PSNR/SSIM on Set5, Set14, B100, Urban100, and Manga109.
- Efficiency and scalability: 28% lower computational complexity than SwinIR is achieved at similar parameters, with 36G versus 50G FLOPs at 1280×720.The reported comparison is Omni-SR versus SwinIR under similar parameter budgets.
- Visual comparison: Omni-SR reconstructs finer details and avoids blurred edges or artifacts more effectively than other lightweight methods in ×4 visual comparisons.The reported example highlights recovery of detailed wall texture that other methods fail to restore.
- Efficiency and scalability: Increasing OSAG number from K=1 produces stable performance improvements, while Omni-SR achieves the best PSNR-versus-parameter results across settings.The experiments examine the trade-off between model size and performance on Urban100 ×4.
- Ablation study: Removing channel interaction reduces performance by 0.13 dB, while combining local, meso, and global interactions yields the best OSAG ablation result.The ablations support both the contribution of channel-wise interaction and the value of multi-scale aggregation.
5. Conclusion
Omni-SR is a lightweight image super-resolution framework combining omni-axis self-attention with omni-scale aggregation. Experiments and analyses validate its super-resolution performance.
- Omni-SR is a lightweight framework for image super-resolution.
- Omni Self-Attention simultaneously models spatial and channel interactions to mine correlations across omni-axis dimensions.
- Omni-scale aggregation enlarges receptive fields with low computational complexity through progressively hierarchical contextual encoding.
- Extensive benchmark experiments and analytical studies validate Omni-SR’s prominent super-resolution performance.