PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Image Super-Resolution Using Pixel Attention

Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, Chao Dong

arXiv:2010.01073v1eess.IVcs.CV

TL;DR

The paper addresses the need for lightweight image super-resolution networks under practical computation constraints. It introduces pixel attention with SC-PA and U-PA blocks, yielding PAN, which achieves comparable performance to lightweight baselines with 272K parameters. The authors report that PA is especially useful for small networks, while its effectiveness decreases as network scale increases.

  • Problem

    Deep-learning-based super-resolution methods face computation costs that restrict real-world applications, motivating lightweight networks with fewer parameters.

  • Method

    The paper introduces pixel attention and integrates it into SC-PA for the main branch and U-PA for the reconstruction branch.

  • Results

    PAN achieves comparable performance with state-of-the-art lightweight networks while using only 272K parameters.

  • Takeaways & Limitations

    PA provides a lightweight attention component demonstrated effective in lightweight super-resolution networks.

  • Takeaways & Limitations

    When PAN is expanded beyond 50 blocks, training becomes difficult without additional techniques such as dense connections, which increase computation.

Abstract

from arXiv · show

This work aims at designing a lightweight convolutional neural network for image super resolution (SR). With simplicity bare in mind, we construct a pretty concise and effective network with a newly proposed pixel attention scheme. Pixel attention (PA) is similar as channel attention and spatial attention in formulation. The difference is that PA produces 3D attention maps instead of a 1D attention vector or a 2D map. This attention scheme introduces fewer additional parameters but generates better SR results. On the basis of PA, we propose two building blocks for the main branch and the reconstruction branch, respectively. The first one - SC-PA block has the same structure as the Self-Calibrated convolution but with our PA layer. This block is much more efficient than conventional residual/dense blocks, for its twobranch architecture and attention scheme. While the second one - UPA block combines the nearest-neighbor upsampling, convolution and PA layers. It improves the final reconstruction quality with little parameter cost. Our final model- PAN could achieve similar performance as the lightweight networks - SRResNet and CARN, but with only 272K parameters (17.92% of SRResNet and 17.09% of CARN). The effectiveness of each proposed component is also validated by ablation study. The code is available at https://github.com/zhaohengyuan1/PAN.

1 Introduction

The paper targets efficient image super-resolution by introducing pixel attention and a concise PAN architecture with specialized main and reconstruction blocks. PAN seeks a favorable reconstruction-performance/model-size trade-off while using 272K parameters.

  • Motivation: Image super-resolution predicts a high-resolution image from a low-resolution observation, but rising computation costs restrict real-world applications.The motivation includes real-time zooming and interactive editing.
  • Results: 272K parameters make PAN the fewest-parameter challenge entry, while its reconstruction performance and model size form a better trade-off than previous models.The paper reports 161K fewer parameters than the first-ranked entry and 415K fewer than the second-ranked entry.
  • Pixel Attention: Pixel attention (PA) is the paper’s main contribution, designed as an effective attention scheme for lightweight super-resolution networks.PA is motivated by channel and spatial attention schemes.
  • Network Blocks: SC-PA integrates pixel attention with Self-Calibrated convolution in the main branch as a simple, efficient building block.Its two convolution portions separately support higher-level feature manipulation and preservation of original information.
  • Network Blocks: U-PA applies pixel attention in the reconstruction branch and combines nearest-neighbor upsampling with convolution layers.The block is motivated by reducing redundancy and parameter cost in reconstruction modules.

2 Related Work

Related work has improved super-resolution through efficient convolutional designs, attention mechanisms, and varied reconstruction strategies. The paper distinguishes PA by targeting lower-complexity 3D attention and places U-PA attention in the reconstruction stage.

  • Efficient Super-Resolution: Deep super-resolution networks improve reconstruction results, but their parameter counts and computational costs limit practical applications.
  • Efficient Super-Resolution: Efficiency-oriented methods include direct low-resolution processing, group convolution, depth-wise separable convolution, and Self-Calibrated convolution.CARN-M and IMDN are cited as efficient super-resolution examples.
  • Attention Mechanisms: Attention methods improve deep-model performance using channel, spatial, or combined 3D attention representations, including SE-Net, BAM, CBAM, ECA, RCAN, and SAN.
  • Attention Mechanisms: PA differs from more complex attention modules by generating 3D attention features with a 1 × 1 convolution layer at lower computation complexity.
  • Reconstruction Methods: Although reconstruction modules commonly use upsampling and convolution, few researchers have investigated attention mechanisms in the reconstruction stage.The paper addresses this gap with its U-PA block for better reconstruction.

3 Proposed Method

PAN combines pixel attention with lightweight feature-mapping and reconstruction blocks. Its PA mechanism produces per-pixel, per-channel attention, while SC-PA and U-PA simplify the main and reconstruction branches.

  • 3.1 Network Architecture: PAN comprises feature extraction, stacked SC-PA nonlinear mapping, and U-PA reconstruction modules.The reconstruction module also includes a convolution layer and a global bilinear-interpolation connection.
  • 3.2 Pixel Attention Scheme: Pixel attention generates a 3D C × H × W attention matrix, unlike channel attention's 1D vector and spatial attention's 2D map.A 1×1 convolution and sigmoid produce attention maps that are multiplied with the input features.
  • 3.3 SC-PA Block: SC-PA splits features into two branches, applying pixel attention and two 3 × 3 convolutions in one branch while preserving information in the other.The branches are concatenated, passed through a 1 × 1 convolution, and combined with a shortcut to produce the output.
  • 3.4 U-PA Block: U-PA places pixel attention between convolution layers and uses nearest-neighbor upsampling in the reconstruction stage.This design targets lower parameter cost than conventional reconstruction structures based on deconvolution or pixel shuffle.
  • 3.5 Discussion: PA is especially useful for small networks, while its effectiveness decreases as network scale increases.The paper attributes this trend to PA improving convolution expression capacity, which matters more in lightweight networks.

4 Experiments

The experiments compare PAN with state-of-the-art single-image super-resolution methods and validate its components through ablation studies. Evaluation uses five benchmark datasets and PSNR and SSIM on the Y channel.

  • Experimental Setup: PAN is compared with state-of-the-art single-image super-resolution algorithms, alongside ablation studies for its proposed components.The comparison covers five commonly used benchmark datasets.
  • Datasets and Metrics: DIV2K and Flickr2K provide the training data, with low-resolution images generated by bicubic downsampling of high-resolution images.Testing uses Set5, Set14, B100, Urban100, and Manga109.
  • Datasets and Metrics: PSNR and SSIM on the Y channel are used as the evaluation metrics.These metrics are reported on the five standard benchmark datasets.
  • Training Details: Training uses L1 loss with Adam, random rotations and horizontal flips, and cosine annealing from 1e−3 to 1e−7.The cosine period is 250k iterations, with 256×256 high-resolution patches and minibatches of 32.

4.3 Comparison with SRResNet and CARN

PAN is evaluated against SRResNet and CARN for ×2, ×3, and ×4 super-resolution. It outperforms CARN across all five datasets and upscaling factors while using substantially fewer parameters.

  • Comparison with SRResNet and CARN: PAN outperforms CARN on all five benchmark datasets for upscaling factors ×2, ×3, and ×4.The comparison is based on Table 1 results for SRResNet, CARN, and PAN.
  • Comparison with SRResNet and CARN: For ×4, PAN has 17.92% of SRResNet's parameters and 17.09% of CARN's parameters.PAN obtains higher PSNR than SRResNet on Manga109 and B100 but lower results on the other three datasets.

4.4 Ablation Study

The ablation compares residual blocks using channel, spatial, or pixel attention for ×4 super-resolution. Pixel attention yields the strongest reported PSNR improvement among these attention variants.

  • Comparison of different attention schemes.: 0.09dB: RB-PA improves PSNR over basic RB, exceeding the 0.03dB gain from RB-CA.RB-SA is slightly worse than RB; results are mean PSNR over 328 images from five datasets.
  • Comparison of different attention schemes.: RB, RB-CA, RB-SA, and RB-PA compare basic residual, channel-attention, spatial-attention, and pixel-attention variants.The attention module is inserted after the second convolution in each residual block.

The effectiveness of Self-Calibrated (SC) block.

The study evaluates whether Self-Calibrated structure improves a pixel-attention residual block. SC-PA provides a larger PSNR gain than RB-PA with a small parameter increase over RB.

  • The effectiveness of Self-Calibrated (SC) block.: 0.12dB: SC-PA outperforms RB-PA in mean PSNR for ×4 super-resolution.Both RB-PA and SC-PA improve PSNR over the basic RB baseline.
  • The effectiveness of Self-Calibrated (SC) block.: 410 additional parameters: SC-PA requires only 1/30 of RB-PA's additional parameters compared with RB.The comparison uses five datasets and records results after 5 × 10^5 iterations.

The effectiveness of PA.

The ablations test pixel attention in both the Self-Calibrated and upsampling blocks, and across networks of different sizes. PA improves the lightweight PAN setting but can reduce performance in larger networks.

  • The effectiveness of PA.: 0.06dB: adding PA to both the SC-PA and U-PA blocks improves the reported result over removing PA.The measurement is PSNR for ×4 super-resolution on five benchmark datasets.
  • The effectiveness of PA.: 0.11dB: PA increases PSNR on PAN-SC-U, whereas RCAN-PA drops 0.06dB relative to RCAN.The experiments record mean PSNR after 1 × 10^6 iterations.
  • The effectiveness of PA.: PA is reported as more effective in lightweight models than in the larger SRResNet and RCAN networks tested.SRResNet and RCAN contain 1,518K and 15,592K parameters, respectively.

4.5 Comparison with State-of-the-art Methods

PAN is compared with lightweight and larger single-image super-resolution methods across five benchmark datasets and ×2, ×3, and ×4 scaling. It achieves competitive reconstruction quality with substantially fewer parameters than key lightweight baselines, while visual examples show fewer artifacts in selected details.

  • 4.5 Comparison with State-of-the-art Methods: Less than 300K parameters: PAN outperforms most state-of-the-art methods across the reported benchmark comparisons.Table 5 reports PSNR, SSIM, and parameter counts for ×2, ×3, and ×4 upscaling.
  • 4.5 Comparison with State-of-the-art Methods: About six times: CARN has approximately 1,592K parameters versus PAN's fewer than 300K, with similar reported performance.PAN also exceeds IMDN in PSNR on Set14, B100, and Urban100.

5 Conclusions

The paper proposes PAN, a lightweight CNN for image super resolution built around pixel attention and two specialized blocks. The framework targets improved SR performance with little parameter cost and achieves comparable performance to state-of-the-art lightweight networks.

  • PAN uses pixel attention (PA), a low-parameter scheme designed to improve reconstruction results.
  • The SC-PA and U-PA blocks apply PA to the main and reconstruction branches, respectively.
  • PAN improves SR performance with little parameter cost and achieves comparable performance to state-of-the-art lightweight networks.
Loading 2010.01073v1…