PaperlayerView exact revision on arXiv

Source-linked AI summary

Channel-wise and Spatial Feature Modulation Network for Single Image Super-Resolution

Yanting Hu, Jie Li, Yuanfei Huang, Xinbo Gao

arXiv:1809.11130v1cs.CV

TL;DR

SISR must recover HR images from LR inputs while distinguishing feature information with unequal reconstruction contributions and retaining long-term information across deep layers. CSFM uses stacked FMM modules with CSAR attention blocks, gated fusion, and dense connections to modulate and preserve features, achieving higher PSNR with fewer parameters than EDSR and RDN in the reported comparison.

  • Problem

    SISR features contain information with different reconstruction contributions, while long-term information from preceding layers can weaken or disappear as networks deepen.

  • Method

    CSFM stacks FMM modules containing chained CSAR blocks and a GF node within a densely connected structure to modulate multi-level features and preserve hierarchical information.

  • Results

    72% and 47% fewer parameters than EDSR and RDN respectively, while achieving higher PSNR on Set5 with 4× upscaling.

  • Takeaways & Limitations

    CSFM provides a better reported tradeoff between performance and model size than the compared networks.

Abstract

from arXiv · show

The performance of single image super-resolution has achieved significant improvement by utilizing deep convolutional neural networks (CNNs). The features in deep CNN contain different types of information which make different contributions to image reconstruction. However, most CNN-based models lack discriminative ability for different types of information and deal with them equally, which results in the representational capacity of the models being limited. On the other hand, as the depth of neural networks grows, the long-term information coming from preceding layers is easy to be weaken or lost in late layers, which is adverse to super-resolving image. To capture more informative features and maintain long-term information for image super-resolution, we propose a channel-wise and spatial feature modulation (CSFM) network in which a sequence of feature-modulation memory (FMM) modules is cascaded with a densely connected structure to transform low-resolution features to high informative features. In each FMM module, we construct a set of channel-wise and spatial attention residual (CSAR) blocks and stack them in a chain structure to dynamically modulate multi-level features in a global-and-local manner. This feature modulation strategy enables the high contribution information to be enhanced and the redundant information to be suppressed. Meanwhile, for long-term information persistence, a gated fusion (GF) node is attached at the end of the FMM module to adaptively fuse hierarchical features and distill more effective information via the dense skip connections and the gating mechanism. Extensive quantitative and qualitative evaluations on benchmark datasets illustrate the superiority of our proposed method over the state-of-the-art methods.

I. INTRODUCTION

Single-image super-resolution must reconstruct an HR image from an LR image despite information distributed across feature channels, spatial locations, and network layers. CSFM addresses this by modulating informative features while preserving hierarchical information through attention, memory, gating, and dense connections.

  • SISR reconstructs a high-resolution image from a single low-resolution counterpart and is an ill-posed inverse problem.
  • Deep SR features contain information with different reconstruction difficulties and contributions, especially across frequency, texture, channels, spaces, and layers.
  • CSFM adaptively attends to multi-level, channel-wise, and spatial feature responses while exploiting hierarchical features to maintain persistent memory.
  • FMM modules stack CSAR blocks to modulate multi-level features and use a GF node to fuse short-term and long-term states.
  • The CSFM network stacks FMM modules with dense connections to enhance discriminative learning, exploit multi-level information, and maximize information flow.

II. RELATED WORK

Related SISR research progressed through deeper architectures, residual and dense connections, attention mechanisms, and post-upscaling designs. CSFM combines feature modulation with channel-wise and spatial attention to emphasize important information in SR features.

  • CNN-based SISR methods improved performance through residual learning, recursive learning, symmetric skip connections, and cascading memory blocks.
  • Post-upscaling methods extract features from LR images and increase spatial resolution near the network end using deconvolution or sub-pixel convolution.
  • CSFM applies sub-pixel convolution at the network end and uses feature modulation to enhance discriminative ability for different information types.
  • Attention mechanisms are used across tasks including image generation, captioning, classification, and restoration.
  • CSFM combines channel-wise and spatial attention within residual blocks to modulate representations globally and locally.

C. Skip Connections

CSFM combines residual paths with densely connected paths to improve information flow, feature re-exploitation, and preservation of important information across a deep SR network.

  • Residual blocks improve information flow and ease training in deep SR networks through skip connections.
  • CSFM stacks attention-based residual blocks within modules and connects modules densely to re-exploit features and preserve important information.
  • The network uses IFENet for low-resolution feature extraction, FTNet for feature transformation, and UpNet for the final reconstruction pipeline.
  • Channel attention summarizes each feature map globally, then selectively emphasizes informative channels and suppresses redundant ones.The attention weights rescale input features through channel-wise multiplication.
  • The channel-attention process adaptively modulates features according to input statistics and increases channel-wise feature discriminability.

2) The SA Unit:

Spatial attention complements channel attention by modeling differences across local positions, emphasizing regions associated with difficult high-frequency reconstruction.

  • Spatial attention targets local regions because edges and textures generally contain more high-frequency information than smooth areas.
  • The SA unit generates a spatial mask from the input feature maps using a two-layer neural network followed by a sigmoid function.The mask has one channel and spatial dimensions H × W.
  • A first convolution produces per-channel attentive maps, and a second 1 × 1 convolution combines them into one attentive map.
  • The sigmoid normalizes spatial attention values to [0, 1], which are multiplied element-wise with spatial positions in each feature map.
  • Spatial modulation operates locally and can interact with global channel-wise modulation to enhance network representational power.

3) Integration of CA and SA into the Residual Block:

The CSAR block embeds channel-wise and spatial attention within a residual block, modulating residual features while retaining the residual pathway’s information-flow benefits.

  • The CSAR block combines channel-wise and spatial attention units with a residual block to improve image SR representations.
  • Its output adds attention-modulated residual features to the block input: H_O = H_I + Φ(R(H_I)).Q denotes the CSAR function, R the residual branch, and Φ the combined attention model.
  • The residual branch uses two stacked convolutional layers with ReLU activation to produce residual features.
  • Channel-wise and spatially weighted residual features are concatenated and fused by a 1 × 1 convolution with learned adaptive weights.
  • Modulating only residual features preserves identical mapping, while stacking CSAR blocks applies both attentions to multi-level features.

B. The FMM Module

The FMM module chains CSAR blocks for multi-level feature modulation and uses gated fusion with dense connections to preserve and adaptively combine long-term information.

  • An FMM module contains a CSAR blockchain and a gated fusion node.
  • The CSAR blockchain stacks multiple CSAR blocks sequentially to perform channel-wise and spatial modulation at multiple levels.
  • The GF node concatenates outputs from preceding FMM modules and the current blockchain, then adaptively fuses them through a convolutional layer.
  • The GF convolution learns adaptive weights that control output information from the fused hierarchical features.
  • Short-term skip connections within the blockchain support information flow across layers and modules while the blocks capture important features.
  • Dense connections among FMM modules alleviate long-term information loss and contribute to multi-level information fusion for image SR.

C. Network Architecture

CSFM transforms low-resolution features through stacked FMM modules, densely connected feature transformation, and upscaling. Its design combines global residual learning with gated hierarchical fusion to preserve informative long-term features.

  • Feature Transformation: The FTNet stacks FMM modules to transform initial features into high-informative features for the UpNet.Each FMM module is represented by G_m, and the transformed features are subsequently upsampled for HR reconstruction.
  • Global Residual and Upscaling: The FTNet adds the initial extracted features through a global residual branch before the UpNet reconstructs the HR image.The UpNet uses a sub-pixel convolutional layer followed by a convolutional layer.
  • Training Objective: The CSFM network is trained by minimizing L1 distance between super-resolved images and corresponding ground-truth images.The objective is defined over LR-HR training patch pairs.
  • Dense Connectivity: Dense connections between FMM modules support multi-level information use and maintain information flow across the network.The network is designed to improve discriminative learning for different information types.

IV. EXPERIMENTS AND ANALYSIS

The experiments analyze component contributions and compare CSFM with state-of-the-art methods using ablation studies and benchmark evaluations.

  • Experimental Scope: The evaluation includes implementation details, ablations of CSAR and GF components, parameter studies, and comparisons with state-of-the-art methods.The analyses cover different component designs and numbers of CSAR blocks and FMM modules.

A. Datasets and Metrics

CSFM is evaluated on five standard super-resolution datasets using PSNR and SSIM under common luminance-channel evaluation protocols. The experiments also examine visual quality, information persistence, and architectural depth.

  • Datasets: The study uses Set5, Set14, BSD100, Urban100, and Manga109, with 800 DIV2K images for training.The datasets cover natural scenes, detailed urban images, and Japanese comics with fine structures.
  • Metrics: PSNR and SSIM evaluate luminance-channel quality after boundary pixels are removed, with higher values indicating better reconstruction.These metrics are computed under the standard SISR protocol described by the authors.
  • Visual Evaluation: Visual comparisons test 4× upscaling on Urban100 image “img015” across BR, CSAR, and CSAR-plus-GF networks.The reported comparison focuses on the contribution of attentive modulation and long-term information maintenance.
  • Implementation Details: The implementation uses 64-filter convolutions, mostly 3 × 3 kernels, with reduction ratio r=16 and increase ratio γ=2 in CSAR attention units.The CA and SA units, along with GF nodes, include 1 × 1 convolutions.
  • Ablation Design: The ablation studies compare attention-block variants, GF usage, feature-weight statistics, and different numbers of FMM modules and CSAR blocks.All comparison models use 3 × 10^5 mini-batch updates.

1) The CSAR Block:

Ablations show that combining channel-wise and spatial attention improves CSFM over residual blocks with no attention or only one attention type. GF nodes also preserve useful long-term information for reconstruction.

  • Comparative Evaluation: Table II reports average PSNR and SSIM for scale factors of 2×, 3×, and 4× when comparing state-of-the-art SR methods.Bold and underlined entries identify the best and second-best performances, respectively.
  • GF Node: GF nodes receive short-term and long-term feature maps, with larger filter-weight norms indicating greater contribution to fusion.The authors normalize these norms to [0,1] and average them by feature type.
  • GF Node: Long-term features from preceding FMM modules make non-negligible contributions, especially in late modules, at both 2× and 4× scales.The authors therefore associate GF-based information persistence with improved SR performance.

3) The Number of FMM Modules and the Number of CSAR Blocks in each FMM Module:

The study varies the numbers of FMM modules and CSAR blocks, finding that deeper configurations improve PSNR while appropriately allocating skip connections reduces redundancy and computational cost. The selected CSFM configuration is M = 8 and B = 16.

  • 3) The Number of FMM Modules and the Number of CSAR Blocks in each FMM Module:: Increasing either the number of FMM modules M or CSAR blocks B improves PSNR on BSD100 at 2× upscaling.The experiment represents configurations as MmBb, where m is the module count and b is the block count per module.
  • 3) The Number of FMM Modules and the Number of CSAR Blocks in each FMM Module:: Deeper models remain advantageous, although properly allocating skip connections preserves accuracy while reducing redundancy and computational cost.The comparison includes M4B8 versus M2B16 and M4B16 versus M8B8.
  • Visual evaluation: At 4× upscaling, CSFM reconstructs clearer stripes, more recognizable characters, correctly colored balcony lines, and sharper grids than competing methods.These visual evaluations cover Urban100 and Manga109 examples.
  • 3) The Number of FMM Modules and the Number of CSAR Blocks in each FMM Module:: The authors adopt M = 8 and B = 16 for subsequent comparison experiments to exploit long-term skip connections while controlling computational cost.

D. Comparisons with the State-of-the-arts

CSFM is evaluated against state-of-the-art SISR methods using quantitative, visual, and parameter-efficiency comparisons. It achieves higher benchmark performance, reconstructs challenging details more faithfully, and offers a favorable performance–model-size tradeoff.

  • Experimental setup: The evaluation uses public trained models for several baselines and rebuilds VDSR in PyTorch because its trained model was unavailable.
  • Quantitative evaluation: CSFM outperforms the compared methods in PSNR and SSIM across five benchmark datasets and scale factors of 2×, 3×, and 4×.It also achieves the highest performance with fewer skip connections than MemNet and RDN.
  • Quantitative evaluation: On Urban100, CSFM improves over the prior state of the art by 0.19dB, 0.18dB, and 0.14dB at 2×, 3×, and 4×, respectively.
  • Quantitative evaluation: On Manga109, CSFM exceeds EDSR by 0.21dB, 0.32dB, and 0.29dB in PSNR at 2×, 3×, and 4×, respectively.
  • Visual evaluation: Visual comparisons show that CSFM more clearly reconstructs stripe patterns, grid structures, texture regions, and characters than prior methods.The reported competing outputs contain severe distortions and noticeable artifacts.
  • Model-size efficiency: CSFM achieves higher PSNR than EDSR and RDN with 72% and 47% fewer parameters, respectively, on Set5 at 4× upscaling.The comparison reports a better tradeoff between performance and model size.
Loading 1809.11130v1…