PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-scale Attention Network for Single Image Super-Resolution

Yan Wang, Yusen Li, Gang Wang, Xiaoguang Liu

arXiv:2209.14145v3eess.IVcs.CV

TL;DR

Single-image super-resolution must recover high-frequency information from an ill-posed low-resolution input, while ConvNets have lagged transformer-based models in this setting. The paper proposes MAN, combining multi-scale large kernel attention and gated spatial attention units, and reports performance that keeps pace with transformer-based methods while offering efficiency trade-offs across model complexities.

  • Problem

    Super-resolution must reconstruct missing high-frequency information from an ill-posed low-resolution image, and ConvNets have not matched transformers in this low-level task.

  • Method

    MAN combines multi-scale large kernel attention for varied-range correlations with gated spatial attention units that incorporate spatial attention while reducing parameters and calculations.

  • Results

    MAN achieves better performance than previous state-of-the-art ConvNets and keeps pace with transformer-based methods in a more efficient manner.

  • Takeaways & Limitations

    Stacking different numbers of MLKA and GSAU produces MAN variants that trade off model complexity and performance across lightweight and performance-oriented super-resolution tasks.

Abstract

from arXiv · show

ConvNets can compete with transformers in high-level tasks by exploiting larger receptive fields. To unleash the potential of ConvNet in super-resolution, we propose a multi-scale attention network (MAN), by coupling classical multi-scale mechanism with emerging large kernel attention. In particular, we proposed multi-scale large kernel attention (MLKA) and gated spatial attention unit (GSAU). Through our MLKA, we modify large kernel attention with multi-scale and gate schemes to obtain the abundant attention map at various granularity levels, thereby aggregating global and local information and avoiding potential blocking artifacts. In GSAU, we integrate gate mechanism and spatial attention to remove the unnecessary linear layer and aggregate informative spatial context. To confirm the effectiveness of our designs, we evaluate MAN with multiple complexities by simply stacking different numbers of MLKA and GSAU. Experimental results illustrate that our MAN can perform on par with SwinIR and achieve varied trade-offs between state-of-the-art performance and computations.

1. Introduction

Single-image super-resolution must recover missing high-frequency information from an ill-posed low-resolution input. MAN addresses the limitations of increasingly costly network designs by combining multi-scale large-kernel attention with gated spatial processing to balance reconstruction quality and complexity.

  • Motivation: Super-resolution reconstructs missing high-frequency information, but one low-resolution image can correspond to many plausible high-resolution images.This makes finding correct low-resolution/high-resolution pixel correlations difficult.
  • Reported trade-off: MANs achieve higher PSNR with fewer parameters and computations on Manga109 at ×2 super-resolution scale.Figure 1 presents the performance-complexity trade-off.
  • Existing approaches: Increasing model capacity through larger datasets, pre-training, or training strategies improves general capability but increases training and data-collection costs.Examples include IPT, HAT, LSDIR, and RCAN-it.
  • Existing approaches: Deeper and wider networks, complex topologies, and attention mechanisms expand receptive fields and capture more information, but can suffer from over-training, high training costs, or limited improvement.Prior designs include multi-scale and non-local attention mechanisms.
  • ConvNets versus transformers: Transformer-based models have strong representation ability, while pure ConvNets such as VAN can be competitive in high-level tasks but still trail transformers in super-resolution.The paper therefore reassesses VAN’s large kernel attention for SR.
  • Proposed approach: MAN combines multi-scale large kernel attention with gated mechanisms to model correlations at multiple ranges while reducing computation and mitigating dilation-related blocking artifacts.Its modules are stacked into models with different complexity levels.

2. Related Work

Related super-resolution work spans performance-oriented and lightweight models, with researchers improving reconstruction through depth, width, attention, and efficient feature processing. Attention mechanisms increasingly target channel, spatial, and long-range correlations, while lightweight methods prioritize reduced model size and computation.

  • Super-resolution model families: Single-image super-resolution models are commonly grouped into classical performance-oriented methods and lightweight methods according to model complexity.This distinction reflects the trade-off between reconstruction quality and deployment constraints.
  • Classical super-resolution: Classical SR methods improve reconstruction by increasing depth and width, residual learning, channel attention, and residual-in-residual structures.Representative models include VDSR, EDSR, and RCAN.
  • Lightweight super-resolution: Lightweight SR methods reduce model size for mobile deployment, although recursive module reuse can lower parameters without reducing computation substantially.Later methods use channel splits and attention to exploit hierarchical features.
  • Attention mechanisms: Attention modules select informative regions or features while suppressing irrelevant information in SR feature maps.Prior work includes channel, second-order channel, spatial, and spatial-channel attention.
  • Long-range attention: Self-attention and varied spatial attention model long-range or coarse-to-fine image relationships for information aggregation and recovery quality.Examples include IPT, SwinIR, DAT, and GRL.

3. Methodology

MAN combines multi-scale large kernel attention and gated spatial attention within stacked multi-scale attention blocks, then reconstructs super-resolved images through residual feature processing. Its modules target broader context, local continuity, reduced computation, and flexible complexity.

  • Network Architecture: MAN extracts shallow features, processes them through cascading multi-scale attention blocks, and reconstructs high-quality images from residual high-frequency features.The reconstruction module uses a 3×3 convolution and pixel-shuffle layer.
  • Multi-scale Attention Block: Each multi-scale attention block combines multi-scale large kernel attention with a gated spatial attention unit in a MetaFormer-style structure.Layer normalization and learnable scaling factors are used around the module operations.
  • Multi-scale Large Kernel Attention: MLKA splits features into groups and applies LKA variants with different kernel and dilation settings to capture information from local to global ranges.The three described variants are 3-5-1, 5-7-1, and 7-9-1, corresponding to progressively larger receptive fields.
  • Gated Aggregation: A spatial gate adaptively combines each LKA output to reduce blocking artifacts while retaining local texture and long-range responses.Larger receptive fields react more to long-range dependence, whereas smaller ones retain local texture.
  • Complexity Analysis: Grouped multi-scale processing controls computational cost while flexibly using varied kernels to capture local and global information.The complexity analysis compares MLKA with LKA and window-based self-attention using theoretical FLOPs.
  • Gated Spatial Attention Unit: GSAU combines spatial attention with gated linear processing, using depth-wise convolution and element-wise multiplication to remove the nonlinear layer and capture local continuity.The design targets lower parameter and calculation costs than a conventional wide-channel MLP.
  • Large Kernel Attention Tail: The tail module uses a 7-9-1 LKA wrapped by two 1 × 1 convolutions to establish longer-range connections for reconstruction features.This replaces the vanilla convolution commonly used at the end of deep SR backbones.

4. Experiments

Experiments show that MAN’s components improve restoration quality with modest overhead, while its variants achieve strong accuracy–complexity trade-offs across classical, tiny, lightweight, and transformer comparisons.

  • Ablation Studies: MAN-tiny and MAN-light improve Urban100 PSNR by 0.25 dB and 0.29 dB, respectively, with negligible increases in parameters and calculations.Removing LKAT or the multi-scale mechanism reduces PSNR by 0.09 dB, while GSAU reduces 15K parameters and 3.6G calculations.
  • Ablation Studies: The transformer-style MAB raises Set5 PSNR from 32.15 dB to 32.33 dB over the RCAN-style block.The comparison indicates a better performance–computation balance for the transformer-style structure.
  • Ablation Studies: MLKA outperforms the evaluated LKA variants throughout training and produces higher detail intensity and more activated pixels in Urban100 visualizations.Using all three decomposition types trades off parameters, computations, and performance.
  • Ablation Studies: GSAU achieves performance comparable to CFF while using 73% of its parameters and calculations.The study compares GSAU with MLP, Simple Gate, and CFF as alternative feed-forward designs.
  • Comparisons with classical SR models: MAN exceeds other convolutional methods by up to 0.69 dB, 0.77 dB, and 0.81 dB PSNR at ×2, ×3, and ×4 upscaling, respectively.At ×2, MAN improves Manga109 PSNR from 39.92 dB to 40.02 dB over SwinIR; at ×4, it is slightly behind SwinIR.
  • Visual comparisons: On Urban100 ×4 qualitative examples, MAN reconstructs a clear fence structure and restores building windows that competing models do not.The tiny and light MAN variants also restore texture more clearly than other methods in the shown Urban100 examples.
  • Comparisons with tiny/light SR models: MAN-tiny uses fewer than 150K parameters while matching EDSR-baseline restoration quality, and MAN-light uses 2% of EDSR’s parameters and computations.MAN-tiny/light outperform the compared tiny/lightweight methods, while MAN-light leads IMDN and SwinIR-light/ELAN-light by 0.66 dB/0.23 dB on Urban100 ×4.
  • Comparisons with SR Transformers: MAN achieves similar quality to EDT-B with 75% of its parameters and 52% of its FLOPs, and performs on par with or better than similarly sized transformer methods.HAT and DAT are larger models that perform better than both EDT and MAN.

5. Conclusion

MAN is a CNN-based super-resolution network designed for multiple complexity levels, combining transformer-style blocks with MLKA and GSAU to model long-range and local information efficiently. Experiments report better performance than previous state-of-the-art ConvNets while remaining competitive with transformer-based methods.

  • MAN targets super-resolution across multiple complexity levels using transformer-style blocks for improved representation modeling.
  • MLKA combines large-kernel decomposition with multi-scale mechanisms to establish flexible long-range correlations among image regions.
  • GSAU integrates gating and spatial attention to activate local information while reducing model complexity.
  • MAN achieves better performance than previous state-of-the-art ConvNets and keeps pace with transformer-based methods more efficiently.
Loading 2209.14145v3…