PaperlayerView exact revision on arXiv

Source-linked AI summary

Attentional Feature Fusion

Yimian Dai, Fabian Gieseke, Stefan Oehmcke, Yiquan Wu, Kobus Barnard

arXiv:2009.14082v2cs.CV

TL;DR

Simple addition and concatenation leave feature fusion across layers and branches insufficiently addressed, especially when fused features differ in scale and semantics. The paper proposes AFF and MS-CAM, with iAFF refining initial integration, and reports stronger performance with smaller parameter budgets, including a 0.3% ImageNet advantage using 60% of the parameters in one comparison.

  • Problem

    Feature fusion is widespread but commonly relies on simple operations, while existing attention methods do not provide a unified solution for inconsistent scales, semantics, and initial integration.

  • Method

    AFF unifies attentional feature fusion across same-layer, short-skip, long-skip, and internal integration scenarios using MS-CAM, while iAFF adds another attention stage for initial integration.

  • Results

    The proposed AFF/iAFF networks outperform state-of-the-art networks under smaller parameter budgets; on ImageNet, iAFF-ResNet-50 exceeds Gather-Excite-θ+-ResNet-101 by 0.3% with 60% parameters.

  • Takeaways & Limitations

    Feature fusion quality, rather than simply increasing network depth, is presented as an important design target for improving CNN performance.

  • Takeaways & Limitations

    Iterative fusion can increase optimization difficulty, and iAFF-ResNet performance degraded when network depth increased from b = 3 to b = 4.

Abstract

from arXiv · show

Feature fusion, the combination of features from different layers or branches, is an omnipresent part of modern network architectures. It is often implemented via simple operations, such as summation or concatenation, but this might not be the best choice. In this work, we propose a uniform and general scheme, namely attentional feature fusion, which is applicable for most common scenarios, including feature fusion induced by short and long skip connections as well as within Inception layers. To better fuse features of inconsistent semantics and scales, we propose a multi-scale channel attention module, which addresses issues that arise when fusing features given at different scales. We also demonstrate that the initial integration of feature maps can become a bottleneck and that this issue can be alleviated by adding another level of attention, which we refer to as iterative attentional feature fusion. With fewer layers or parameters, our models outperform state-of-the-art networks on both CIFAR-100 and ImageNet datasets, which suggests that more sophisticated attention mechanisms for feature fusion hold great potential to consistently yield better results compared to their direct counterparts. Our codes and trained models are available online.

1. Introduction

The paper treats feature fusion as a central but under-addressed source of CNN representation quality. It proposes a unified attentional framework with multi-scale context and iterative integration to handle inconsistent feature semantics and scales.

  • Motivation: Feature fusion is pervasive across Inception modules, residual networks, FPNs, and U-Net, but is commonly implemented with simple fusion operations.These architectures fuse features across branches, short skips, or layers to address object-scale variation, deep-network training, or semantic-resolution differences.
  • Limitations: Existing attention-based methods mainly address same-layer feature selection, leaving cross-layer fusion and a generalized treatment of scale inconsistency unresolved.The paper identifies limited scenarios and the lack of a consistent approach as central shortcomings.
  • Limitations: Initial feature integration can bottleneck attention-based fusion because addition may combine features with inconsistent scales and semantics before fusion weights are generated.The paper describes this initial integration as an involuntary but influential stage in methods such as SKNet.
  • Contribution: The attentional feature fusion framework extends attention-based fusion to same-layer, short-skip, long-skip, and internal integration scenarios.AFF replaces existing fusion operators, while iAFF iteratively refines the initial integration used to generate fusion weights.
  • Contribution: MS-CAM aggregates channel context from multiple receptive-field scales to address scale variation and inconsistent feature semantics during fusion.Its multi-scale context can emphasize globally distributed large objects and locally distributed small objects.

2. Related Work

Prior work addresses scale variation, attention, and skip connections through complementary mechanisms, but the paper positions MS-CAM and AFF as a unified alternative for channel-aware feature fusion.

  • Scale variation: Multi-scale CNN features and image pyramids address object-scale variation by combining information across resolutions or layers.The related work describes feature pyramids as approximations of image pyramids for obtaining semantic features with spatial detail.
  • Attention mechanisms: Early attention mechanisms primarily modeled global dependencies, while later approaches began incorporating scale-aware context.SENet, self-attention, and related methods are presented as examples of global or increasingly scale-aware attention.
  • Channel attention: MS-CAM differs from prior multi-scale context approaches by addressing scale in channel attention with point-wise convolution inside the attention module.The paper states that multi-scale channel attention had not previously been discussed to its knowledge.
  • Skip connections: Short skip connections support gradient flow, whereas long skips combine fine-detail low-level features with coarse high-level semantic features.These connections motivate feature fusion across different depths and resolutions.
  • Skip connections: Compared with Highway Networks, AFF generates element-wise fusion weights from both input features and explicitly addresses initial feature integration.Highway Networks use a scalar weight generated from one input feature, while MS-CAM produces weights matching the feature-map size.

3. Multi-scale Channel Attention

The paper develops MS-CAM by combining global and local channel contexts, producing spatially resolved attention weights that refine feature maps while remaining lightweight.

  • 3.1. Revisiting Channel Attention in SENet: SENet compresses each H × W feature map into a scalar through global average pooling, which can favor globally distributed objects over small objects.The paper identifies this coarse global descriptor as a motivation for multi-scale channel contexts.
  • 3.2. Aggregating Local and Global Contexts: MS-CAM varies spatial pooling scale and adds local channel context to global channel context inside the attention module.This design targets scale variation while keeping the module lightweight.
  • 3.2. Aggregating Local and Global Contexts: Point-wise convolutions aggregate local channel interactions at each spatial position through a bottleneck structure.The bottleneck reduces parameters while preserving local context.
  • 3.2. Aggregating Local and Global Contexts: The local context has the same shape as the input feature, allowing MS-CAM to preserve and highlight subtle low-level details.The refined feature is obtained from global and local context through attention weights and element-wise feature modulation.

4. Attentional Feature Fusion

AFF unifies feature fusion across same-layer, short-skip, and long-skip scenarios by using attention to combine inputs of different scales. iAFF further addresses the quality of the initial feature integration with a second attention stage.

  • Unified fusion scenarios: AFF treats same-layer fusion, short skip connections, and long skip connections within one attention-based formulation.The scenarios correspond to InceptionNet, ResNet, and feature pyramids, respectively.
  • Multi-scale attention: AFF uses a multi-scale channel attention module to address feature inconsistency caused by different receptive fields, semantics, and scales.The formulation assumes Y has the larger receptive field by default.
  • Attentional Feature Fusion: The fused feature Z is a weighted combination of X and Y, with complementary weights M(X ⊎Y) and 1−M(X ⊎Y) between 0 and 1.This enables soft selection or weighted averaging between the two feature maps.
  • Design rationale: The unified formulation abstracts implementation differences among fusion strategies and supports replacing addition or concatenation in existing network modules.The paper applies these replacements to InceptionNet, ResNet blocks, and FPNs; Table 1 summarizes the resulting strategy differences.
  • Iterative fusion: iAFF adds another attention module for initial integration, reformulating X ⊎Y as an attention-weighted combination before final fusion.This two-stage design targets the initial integration issue associated with fully context-aware fusion.

5. Experiments

Experiments evaluate AFF and iAFF across multiple feature-fusion scenarios and datasets, showing consistent gains from multi-scale, attention-based integration. Iterative fusion improves performance further in most cases but can make optimization harder as networks deepen.

  • Experimental setup: AFF experiments cover InceptionNet, ResNet, and FPN scenarios on CIFAR-100, ImageNet, and StopSign.These represent same-layer, short-skip, and long-skip feature fusion settings.
  • Multi-scale context aggregation: Global + Local contextual aggregation outperforms single-scale alternatives across all tested settings.The compared modules use the same parameter count and differ only in contextual aggregation scale.
  • Feature integration strategy: Attention-based nonlinear fusion consistently outperforms addition and concatenation across the evaluated scenarios.The comparison controls the parameter budget while varying feature-integration strategies.
  • Feature integration strategy: The fully context-aware selective strategy is slightly but consistently better than other non-iterative alternatives.The results support using context-aware feature selection for multiple feature integration.
  • Iterative fusion: iAFF significantly outperforms the other integration strategies in most cases, although deeper iAFF-ResNet can degrade when optimization becomes harder.Performance did not improve when the ResBlock number increased from b = 3 to b = 4.
  • Comparison with state-of-the-art networks: AFF- and iAFF-integrated networks outperform comparison networks across all evaluated fusion scenarios.The reported comparisons include SKNet, SENet, and GAU-FPN against AFF-integrated InceptionNet, ResNet, and FPN.

6. Conclusion

The paper generalizes attention as dynamic feature fusion across same-layer, short-skip, long-skip, and internal integration scenarios. MS-CAM and iterative fusion address scale or semantic inconsistency and initial-integration bottlenecks, while experiments show strong performance with fewer layers or parameters.

  • AFF generalizes attention-based feature fusion across same-layer, short-skip, long-skip, and internal integration scenarios.
  • MS-CAM combines local channel contexts with global channel statistics to address semantic and scale inconsistencies among input features.
  • Iterative attentional feature fusion adds another attention level to alleviate the initial integration bottleneck.
  • Experiments on CIFAR-100 and ImageNet show that the models outperform state-of-the-art networks with fewer layers or parameters.

Implementation Details

The experiments use MXNet and GluonCV implementations with architectures trained from scratch for fairness. Dataset-specific optimization schedules and evaluation details are provided, including mIoU for semantic segmentation and selective use of mixup.

  • All architectures are implemented with MXNet and GluonCV, and experimental models are trained from scratch because most cannot use pretrained weights.
  • CIFAR-100 experiments use weight decay of 1e-4 and learning-rate reductions by 0.1 at epochs 300 and 350.
  • ImageNet experiments use label smoothing and cosine annealing without weight decay.
  • The StopSign semantic-segmentation experiment uses cross entropy and mean intersection over union (mIoU), addressing substantial scale variation.
  • Mixup is used for the networks in Tables 5 and 6 but not for the remaining experiments or ablations.

Local and Global Fusion Strategies

The ablation compares fixed, learned, and dynamically inferred fusion of local and global channel contexts. Iterative AFF performs best on CIFAR-100, while Recursive AFF performs worst, plausibly because nested Sigmoid functions complicate optimization; AFF and Concat-AFF are similar.

  • Half-AFF, AFF, and iterative AFF add local and global contexts with equal constant weights of 0.5.
  • Concat-AFF concatenates local and global contexts, then uses point-wise convolution with weights learned during training and fixed afterward.
  • Recursive AFF dynamically allocates fusion weights for local and global contexts during inference using MS-CAM.
  • Iterative AFF achieves the best CIFAR-100 performance, whereas Recursive AFF is among the worst-performing modules.
  • AFF and Concat-AFF perform very similarly, so the authors choose the squeeze-and-excitation form for simplicity.

Analysis of FLOPs

AFF adds point-wise-convolution FLOPs, but the reported overhead is marginal relative to host-network convolutions. Replacing every residual block can reach about 7.7% overhead, while the authors summarize typical added cost as around 3% to 5%.

  • 4.88% additional FLOPs are reported when replacing ResNet-50 with AFF-ResNet-50, increasing cost from 4.1 to 4.3 GFlops.
  • AFF point-wise convolutions contribute marginal FLOPs compared with the convolutional layers in the host network.
  • Around 7.7% is the maximum additional FLOPs percentage when AFF replaces every ResBlock from beginning to end.
  • The authors conclude that AFF typically adds around 3% to 5% FLOPs.
Loading 2009.14082v2…