PaperlayerView exact revision on arXiv

Source-linked AI summary

A Dynamic Residual Self-Attention Network for Lightweight Single Image Super-Resolution

Karam Park, Jae Woong Soh, Nam Ik Cho

arXiv:2112.04488v1eess.IVcs.CV

TL;DR

Deep SISR networks can exceed practical parameter and computation budgets, while fixed lightweight structures may not adapt to diverse image statistics. DRSAN dynamically adjusts residual connections and adds residual self-attention; the authors report better or comparable lightweight SR performance with more than 20% fewer parameters while maintaining reconstruction quality.

  • Problem

    Deep SISR models impose substantial parameter and computation costs, while fixed lightweight structures may not fit diverse input statistics and can require exhaustive experiments to design.

    Source: §I · pp. 1–2

  • Method

    DRSAN adapts residual connections to input image characteristics through DRA and uses RSA to produce a 3D attention map in cooperation with residual structures without additional parameters.

    Source: §I · p. 2

  • Results

    More than 20% fewer parameters while maintaining reconstruction quality, with performance better than or comparable to state-of-the-art lightweight SR networks.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that DRSAN achieves better or comparable performance to state-of-the-art models with complex configurations on well-known benchmark datasets.

    Source: §V · p. 10

Abstract

from arXiv · show

Deep learning methods have shown outstanding performance in many applications, including single-image super-resolution (SISR). With residual connection architecture, deeply stacked convolutional neural networks provide a substantial performance boost for SISR, but their huge parameters and computational loads are impractical for real-world applications. Thus, designing lightweight models with acceptable performance is one of the major tasks in current SISR research. The objective of lightweight network design is to balance a computational load and reconstruction performance. Most of the previous methods have manually designed complex and predefined fixed structures, which generally required a large number of experiments and lacked flexibility in the diversity of input image statistics. In this paper, we propose a dynamic residual self-attention network (DRSAN) for lightweight SISR, while focusing on the automated design of residual connections between building blocks. The proposed DRSAN has dynamic residual connections based on dynamic residual attention (DRA), which adaptively changes its structure according to input statistics. Specifically, we propose a dynamic residual module that explicitly models the DRA by finding the interrelation between residual paths and input image statistics, as well as assigning proper weights to each residual path. We also propose a residual self-attention (RSA) module to further boost the performance, which produces 3-dimensional attention maps without additional parameters by cooperating with residual structures. The proposed dynamic scheme, exploiting the combination of DRA and RSA, shows an efficient trade-off between computational complexity and network performance. Experimental results show that the DRSAN performs better than or comparable to existing state-of-the-art lightweight models for SISR.

I. INTRODUCTION

Deep SISR models improve reconstruction but impose parameter and computation costs, while fixed lightweight structures may not suit diverse input statistics. DRSAN adapts residual connections to image characteristics and combines DRA with RSA.

I. INTRODUCTION

  • Lightweight SISR seeks to balance reconstruction performance against parameter and computation costs, since deep networks can exceed practical resource limits.The passage gives EDSR as an example, reporting 40.73 M parameters and 9384.7 G operations, compared with 57 K parameters and 52.7 G operations for SRCNN.Source: §I · p. 1
  • Prior lightweight methods often use fixed operations despite diverse input statistics, and finding effective structures can require exhaustive experiments.Different inputs may benefit from different operations, making a single fixed structure potentially suboptimal for some inputs.Source: §I · p. 2
  • DRSAN uses dynamic residual attention to adapt residual connections to input image characteristics and residual self-attention to recalibrate features with a 3D attention map without additional parameters.RSA works jointly with residual connections.Source: §I · p. 2
  • The paper extends preliminary dynamic residual connection work with RSA and a different architecture that uses fewer parameters and obtains better results.The authors also analyze DRA and RSA behavior to support understanding of the attention mechanisms for SR.Source: §I · p. 2
  • More than 20% fewer parameters while maintaining reconstruction quality and achieving state-of-the-art lightweight SR performance.The authors also report that the proposed network performs better than or comparably to state-of-the-art lightweight SR networks.Source: §I · p. 2

II. RELATED WORKS

The related work traces SISR from CNN-based reconstruction to efficient lightweight networks and attention mechanisms. It identifies fixed residual paths and attention that omits skip-connection features as gaps addressed by DRSAN.

A. CNNs for SISR

  • CNN-based SISR developed from SRCNN's pre-upsampling approach toward pixel-shuffle post-upsampling, which processes features in LR space before upscaling.Residual learning and post-upsampling have been combined in deeper SISR networks.Source: A. CNNs for SISR · p. 2

B. Lightweight SISR

  • Many lightweight SR networks reduce costs through recursive structures, cascading, neural architecture search, or adaptive residual weights, but most are manually designed with fixed operations.DRSAN instead learns correlations between image statistics and residual paths, changing connections according to the input.Source: B. Lightweight SISR · pp. 2–3
  • DRSAN comprises convolution layers, an upsampler, and DRAG building blocks containing residual and attention modules.Each DRAG also includes concatenation and a 1 × 1 convolution; its residual block cascades convolutions and PReLUs.Source: Figure 1 · p. 3

C. Attention Mechanisms for SISR

  • Prior attention methods include channel, spatial, and pixel attention, but commonly focus on residual-block features rather than skip connections.DRSAN applies DRA to skip connections and uses RSA to generate a 3D attention map in cooperation with residual structure, without additional modules or complex computation.Source: C. Attention Mechanisms for SISR · p. 3

III. PROPOSED METHOD

DRSAN extracts shallow features, processes them through stacked DRAGs, and reconstructs an HR image. Its mapping blocks use residual and attention modules to adaptively combine features.

A. Network Structure

  • The network has feature extraction, nonlinear mapping with multiple DRAGs, and reconstruction modules.The reconstruction module uses an upsampler and convolution.Source: A. Network Structure · p. 3

1) Feature Extraction Module:

  • A 3 × 3 convolution extracts shallow features from the LR input, then sends them to the nonlinear mapping and reconstruction modules.The resulting feature is x0.Source: 1) Feature Extraction Module · pp. 3–4

2) Nonlinear Mapping Module:

  • The nonlinear mapping module stacks DRAGs to extract information for HR reconstruction; each DRAG contains residual blocks, a DRM, and DRSA.The DRM controls residual connections so the DRAG can combine residual-block features with appropriate weights for varying image properties; DRSA uses DRA and RSA.Source: 2) Nonlinear Mapping Module · p. 4

3) Reconstruction Module:

The reconstruction module upsamples deep features and adds the shallow feature before pixel shuffle. Within each DRAG, the DRM controls residual connections by learning how input statistics relate to residual paths.

3) Reconstruction Module:

  • The reconstruction module adds shallow feature x0 to deep feature xK before a convolution and pixel-shuffle upsampling operation.The final convolution produces the reconstructed SR image.Source: 3) Reconstruction Module · p. 4

B. Dynamic Residual Attention Group (DRAG)

1) Dynamic Residual Module (DRM):

  • The DRM learns correlations between input image statistics and residual paths to combine low- and high-level features using different residual-path combinations.It uses two convolutional layers and does not use sigmoid activation, allowing residual coefficients to take a wider range of values.Source: 1) Dynamic Residual Module (DRM) · p. 4

2) Dynamic Residual Self-Attention (DRSA):

DRSA combines dynamically weighted residual features and residual self-attention. Its RSA coefficient generates a 3D attention map that scales features across channels and spatial positions.

2) Dynamic Residual Self-Attention (DRSA):

  • DRM-selected parameters weight residual paths so each residual feature can combine features from preceding blocks, adapting the residual structure to the input image.Source: 2) Dynamic Residual Self-Attention… · p. 4
  • The residual self-attention coefficient applies sigmoid activation to the residual-block output, producing a 3D map that scales features channel-wise and spatial-wise.RSA recalibrates features to help the network focus on more informative features.Source: 2) Dynamic Residual Self-Attention… · p. 5
  • After residual features are determined, the DRAG output xk is formed by channel-wise concatenation followed by a 1×1 convolution.Source: 2) Dynamic Residual Self-Attention… · p. 5

A. Dataset

The evaluation uses four SR benchmark datasets and reports PSNR and SSIM on test-image Y channels. The implementation details specify the training setup and model variants.

A. Dataset

  • PSNR and SSIM are evaluated on the Y channel using Set5, Set14, B100, and Urban100 test images.Training uses 800 DIV2K high-resolution images, 48×48 RGB patches, and flip and rotation augmentation.Source: A. Dataset · p. 5

B. Training Details

  • Training uses ADAM with L1 loss, batch size 16, and an initial learning rate of 2 × 10^-4, decayed by 0.85 every 2 × 10^5 iterations.Source: B. Training Details · p. 5

C. Implementation Details

  • Five DRSAN variants vary channel width and depth, with 32- and 48-channel models using four and three residual blocks per DRAG, respectively.The variants use 4, 8, or 10 DRAGs; ESPCN is the upsampler, and results are evaluated on an NVIDIA TITAN XP.Source: C. Implementation Details · p. 5

D. Model Analysis

The analysis compares activation patterns across attention schemes and examines how DRSA scales residual features. It reports that 3D attention stays active across blocks, while its weighting varies with feature type and block location.

1) Activation Analysis:

  • 3D attention maps remain active across residual blocks, whereas some SA maps are nearly inactive because SA shares one spatial map across channels.The first row’s fourth and sixth SA maps are cited as examples; Fig. 4 shows activation maps for SA, PA, and DRSA.Source: 1) Activation Analysis · p. 7, Figure 4 · p. 7
  • PA and DRSA assign higher weights to textures except in the last residual block, where edge regions receive higher weights before upsampling.The reported shared tendency applies to PA and DRSA, not SA.Source: 1) Activation Analysis · p. 7
  • DRSA activations cluster near 0.5, while PA activations range near zero, so DRSA scales features more strongly and differentiates their importance.The activation histograms use Set5 (×2) images and show values from each residual block.Source: 1) Activation Analysis · p. 7, Figure 5 · p. 7

2) Behavior Analysis:

The comparisons report that DRSAN performs better than existing methods in similar parameter scopes and can approach heavyweight models with substantially fewer parameters. The behavior analysis also links DRA variation to image structure and reconstruction changes, especially around edges.

2) Behavior Analysis:

  • DRA values vary across patch groups with rectangular textures, strong edges, or few edges and textures; variation is relatively small in the low-detail group.The DRA distributions come from the first DRAG of an ablation model, with patches from benchmark images at ×2.Source: 2) Behavior Analysis · pp. 7–8, Figure 6 · p. 8
  • Changing DRA values changes SR outputs, with differences concentrated near edges and stripe patterns best reconstructed using DRA from similar patches.The comparisons use smooth, horizontal-edge, and diagonal-edge patches; Fig. 7 visualizes the reconstructions and differences on the Y channel.Source: 2) Behavior Analysis · p. 8, Figure 7 · p. 8

E. Comparison with state-of-the-art methods

  • DRSAN models outperform existing lightweight methods in the same parameter scope; DRSAN-32m is comparable to AWSRN-M and A2F-M despite using fewer parameters and operations.The comparison is summarized in Table II; the text also suggests favoring depth over channels for similar network complexity.Source: E. Comparison with state-of-the-art… · p. 8
  • DRSAN-32m requires at least ten times fewer parameters than heavyweight SOTA SR methods while achieving similar performance (0.1 ∼0.2 dB) on BSD100 and a 0.4 ∼0.8 dB difference on structurally complex Urban100.Source: E. Comparison with state-of-the-art… · p. 8
  • Compared with heavy SR methods, the authors’ methods require three to nine times less memory and inference time while sacrificing 0.1 to 0.2 dB of performance, suggesting an efficient trade-off.Source: E. Comparison with state-of-the-art… · p. 9
  • On BSDS100 and Urban100 stripe patterns, DRSAN reconstructs more detail and reduces blurring artifacts relative to other methods.The qualitative comparison is shown in Fig. 8.Source: E. Comparison with state-of-the-art… · p. 9, Figure 8 · p. 9

F. Ablation Study

The ablations examine residual-connection choices and DRM activation functions. They report a parameter-free RSA gain, support for recalibrating residual features, and better performance without sigmoid activation in this design.

1) Dynamic Residual Connection:

  • RSA improves Urban100 (×2) performance by 0.12 dB without increasing parameters, and the reported comparisons support its joint use with residual connections.Table V compares the residual-connection configurations.Source: 1) Dynamic Residual Connection · pp. 9–10, Table V · p. 10
  • Connecting all preceding residual features does not improve performance and can degrade it, while the reported results indicate that DRM recalibrates residual-feature weights.The comparison includes the original SRResNet connection, all possible residual connections, and dynamic residual connections.Source: 1) Dynamic Residual Connection · p. 10, Table V · p. 10
  • DRM without an activation function performs better than with tanh, while sigmoid performs worse in this design; the authors conjecture that limiting scaling to 0–1 can hamper feature-combination diversity.The passage notes that sigmoid often helps stable convergence and performance generally, but reports a different outcome for this scheme; Table VI compares sigmoid and tanh.Source: 1) Dynamic Residual Connection · p. 10, Table VI · p. 10

2) Residual Self-Attention:

The attention ablation finds DRSA gives the largest reported PSNR gain with a small parameter increase. The conclusion presents RSA as a residual-cooperating mechanism and reports DRSAN performs better than or comparably to complex SOTA models.

2) Residual Self-Attention:

  • The ablation compares attention schemes on Set5, Set14, BSDS100, and Urban100 (×2), using a two-residual-group DRSAN baseline without DRSA.The tested alternatives are CA, SA, and PA.Source: 2) Residual Self-Attention · p. 10
  • DRSA boosts PSNR by 0.12 dB with 2 K additional parameters, compared with PA’s 0.10 dB gain and 13 K increase.CA and SA improve PSNR by 0.07 dB and 0.04 dB, respectively, with parameter increases of 1 K to 3 K.Source: 2) Residual Self-Attention · p. 10

V. CONCLUSION

  • The conclusion reports that DRSAN performs better than or comparably to SOTA models with complex configurations on well-known benchmark datasets.It also states that the design schemes and attention mechanisms are applicable to other residual networks without complex network design.Source: §V · p. 10
  • RSA boosts network performance without additional modules by cooperating with the residual structure.Source: §V · p. 10

Related papers

Loading 2112.04488v1…