PaperlayerView exact revision on arXiv

Source-linked AI summary

A^2-FPN: Attention Aggregation based Feature Pyramid Network for Instance Segmentation

Miao Hu, Yali Li, Lu Fang, Shengjin Wang

arXiv:2105.03186v1cs.CV

TL;DR

FPN builds semantically strong pyramids, but feature extraction and fusion have limitations that hinder discriminative multi-scale representations. A^2-FPN uses attention-guided aggregation to address them, with reported mask AP gains across instance-segmentation frameworks and backbones. The method assumes feature stages differ in semantic richness and intensity and contain diverse semantic entities.

  • Problem

    FPN’s feature extraction and fusion have intrinsic defects that limit further aggregation of discriminative features for multi-scale instance recognition.

    Source: §1 · pp. 1–2

  • Method

    A^2-FPN gathers global context across feature levels and uses content-aware sampling and channel-wise reweighting to fuse adjacent features.

    Source: §1 · p. 2

  • Results

    A^2-FPN consistently improves mask AP across frameworks; Mask R-CNN gains are 2.1% with ResNet-50 and 1.6% with ResNet-101, with 2.0% and 1.4% gains over Cascade Mask R-CNN and HTC baselines.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The paper concludes that pyramid construction depends on feature-aggregation node operations as well as cross-scale connections.

    Source: §1 · p. 2

    The context-collection assumption is that feature stages differ in semantic richness and intensity and comprise diverse semantic entities.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Learning pyramidal feature representations is crucial for recognizing object instances at different scales. Feature Pyramid Network (FPN) is the classic architecture to build a feature pyramid with high-level semantics throughout. However, intrinsic defects in feature extraction and fusion inhibit FPN from further aggregating more discriminative features. In this work, we propose Attention Aggregation based Feature Pyramid Network (A^2-FPN), to improve multi-scale feature learning through attention-guided feature aggregation. In feature extraction, it extracts discriminative features by collecting-distributing multi-level global context features, and mitigates the semantic information loss due to drastically reduced channels. In feature fusion, it aggregates complementary information from adjacent features to generate location-wise reassembly kernels for content-aware sampling, and employs channel-wise reweighting to enhance the semantic consistency before element-wise addition. A^2-FPN shows consistent gains on different instance segmentation frameworks. By replacing FPN with A^2-FPN in Mask R-CNN, our model boosts the performance by 2.1% and 1.6% mask AP when using ResNet-50 and ResNet-101 as backbone, respectively. Moreover, A^2-FPN achieves an improvement of 2.0% and 1.4% mask AP when integrated into the strong baselines such as Cascade Mask R-CNN and Hybrid Task Cascade.

1. Introduction

Instance segmentation must recognize objects across scales, and FPN’s feature extraction and fusion leave information loss, content-agnostic sampling, and semantic inconsistency unresolved. A^2-FPN addresses these issues with attention-guided feature aggregation and improves mask AP across the reported frameworks and backbones.

1. Introduction

  • FPN supports multi-scale recognition by fusing adjacent features through lateral connections and a top-down pathway, while PAFPN adds a bottom-up pathway to improve localization.The need for multi-scale representations follows from recognizing instances across varying scales and locations.Source: §1 · p. 1
  • FPN construction loses high-level information through drastic channel reduction, uses content-agnostic sampling, and adds adjacent features despite their semantic gap.These defects arise in feature extraction and the two feature-fusion steps.Source: §1 · pp. 1–2
  • The method collects global context across the feature hierarchy for distribution to each level, then uses location-wise reassembly kernels and channel-wise reweighting during fusion.The reweighting enhances semantic consistency before element-wise addition.Source: §1 · p. 2
  • A^2-FPN improves mask AP over FPN-based Mask R-CNN by 2.1% with ResNet-50 and 1.6% with ResNet-101, and exceeds Cascade Mask R-CNN and HTC baselines by 2.0% and 1.4%.The reported comparisons are on the MS COCO dataset.Source: §1 · p. 2
  • The paper argues that feature-aggregation node operations, not only cross-scale connections, matter to pyramid construction and reports consistent gains across frameworks and backbones.The evaluation is described as comprehensive experiments on COCO.Source: §1 · p. 2

2. Related Work

Related work covers instance-segmentation approaches, feature-pyramid architectures for multi-scale tasks, and attention mechanisms for contextual feature modeling.

2. Related Work

  • Detection-based approaches produce masks from proposals or boxes, while segmentation-based approaches group pixels; examples include Mask R-CNN and metric-learning methods [9] [29] [10].Other detection-based examples include PANet [26] and HTC [5].Source: §2 · p. 2
  • Feature-pyramid work ranges from FPN’s lateral and top-down pathways [22] to bottom-up augmentation [26], repeated bidirectional fusion [35], and searched structures [12].This paper instead explores the node operations used to aggregate features.Source: §2 · p. 2
  • Attention research includes scaled dot-product self-attention [36], non-local operations [38], graph reasoning [7] [42], and channel attention [16], while multi-level global context is rarely explored in detectors.DANet [11] and GCNet [4] combine self-attention and channel attention for contextual dependencies.Source: §2 · p. 2

3. Methodology

A^2-FPN builds its pyramid progressively with multi-level context extraction followed by attention-guided top-down and bottom-up fusion. Its modules collect, reason over, and redistribute global context, then use content-aware sampling and channel reweighting to fuse adjacent levels.

3. Methodology

  • A^2-FPN progressively extracts and fuses pyramid features through MGC, GACARAFE, and GACAP.MGC produces context-rich features; GACARAFE augments the top-down path, and GACAP augments the bottom-up path.Source: Figure 2 · p. 3, §3 · p. 3

3.1. Multi-level Global Context

  • MGC collects context features from each level, reasons over relations within and across levels, and distributes the resulting context to each level.The Context Collector produces multiple context features per backbone level, while scaled cosine-similarity attention is used for distribution.Source: §3.1 · pp. 3–4

3.2. Global Attention CARAFE

  • GACARAFE aggregates complementary adjacent-level information to predict location-wise reassembly kernels for content-aware upsampling and applies channel attention before fusion.The channel attention reweights adjacent features before their element-wise addition.Source: §3.2 · p. 4

3.3. Global Attention Content-Aware Pooling

  • GACAP extends content-aware reassembly to bottom-up pooling, combining adjacent features and predicting content-aware downsampling kernels.It upsamples the top-down feature, concatenates it with the bottom-up feature, then downsamples the combined feature.Source: §3.3 · p. 5

4. Experiments

Experiments on MS COCO evaluate A2-FPN across instance segmentation and object detection frameworks, backbones, and ablations. The reported results show gains over FPN-based baselines, while component and complexity analyses examine the modules and added computation.

4.1. Datasets and Evaluation Metrics

  • Experiments use MS COCO, training on train2017 and reporting val2017 for ablations and test-dev for comparison, with mask and box AP metrics.The stated default training uses 8 images across 4 GPUs for 12 epochs; images are resized to a maximum of 1333 × 800 pixels.Source: §4.1 · p. 5, §4.2 · p. 5, §4.3 · p. 5

4.3. Benchmarking Results

  • Replacing FPN in Mask R-CNN yields 36.6% mask AP with ResNet-50, 2.1% higher than the counterpart, and a 1.6% gain with ResNet-101.The benchmark table reports instance-segmentation mask AP and object-detection box AP on COCO test-dev.Source: §4.3 · p. 6, Table 1 · p. 6
  • A2-FPN with Cascade Mask R-CNN gains 2.0% and 1.7% for ResNet-50 and ResNet-101, while HTC gains 1.4% with ResNet-50.HTC with R-101-A2-FPN and X-101-A2-FPN reaches 40.8% and 42.1% mask AP, respectively.Source: §4.3 · pp. 6–7, Table 1 · p. 6
  • For object detection, A2-FPN improves Mask R-CNN box performance by 2.6% and 1.8% with ResNet-50 and ResNet-101, and reaches 50.4% box AP with a 2x schedule and multi-scale training.The passage also reports gains for Cascade Mask R-CNN and HTC across several backbones.Source: Table 1 · p. 6, §4.3 · p. 7

4.4. Ablation Study

  • On COCO val2017, MGC adds 1.0% to both mask and box AP; GACARAFE adds 1.3% mask and 1.8% box AP, while GACAP adds 0.6% mask and 0.7% box AP.Combining all three reaches 40.0% box AP and 36.2% mask AP, gains of 2.4% and 1.8%; improvements are reported across object scales.Source: §4.4 · p. 7, Table 2 · p. 7
  • Removing GCNs reduces MGC and A2-FPN performance by 0.2% and 0.3%, while GACARAFE exceeds CARAFE by 0.5% box AP and 0.3% mask AP.MGC performance varies by at most 0.1% across context-feature counts; 2·sigmoid improves GACARAFE by 0.5% box AP and 0.2% mask AP over sigmoid.Source: §4.4 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 8

4.5. Complexity Analysis

  • A2-FPN adds computation, but A2-FPN-Lite reduces it greatly at a small performance sacrifice and improves over PAFPN with acceptable complexity.The authors attribute most of the improvement to attention-guided feature aggregation.Source: §4.5 · p. 8, Table 6 · p. 8

5. Conclusion

A^2-FPN improves multi-scale feature learning through attention-guided aggregation and consistently improves baseline performance across backbones and instance segmentation frameworks.

  • A^2-FPN consistently and substantially improves baseline performance across various backbones and instance segmentation frameworks.Source: §5 · p. 8
  • A^2-FPN extracts discriminative features by collecting and distributing multi-level global context features.Source: §5 · p. 8
  • It fuses adjacent feature levels through content-aware sampling and channel-wise reweighting before element-wise addition.Source: §5 · p. 8

Related papers

Loading 2105.03186v1…