Source-linked AI summary
AugFPN: Improving Multi-scale Feature Learning for Object Detection
Chaoxu Guo, Bin Fan, Qian Zhang, Shiming Xiang, Chunhong Pan
TL;DR
FPN’s feature-pyramid design does not fully exploit multi-scale features because of semantic gaps, highest-level information loss, and heuristic RoI assignment. AugFPN addresses these issues with three specialized components and consistently improves FPN-based detectors, including Faster R-CNN, RetinaNet, and FCOS. The gains come with slower reported training and inference than FPN in the runtime comparison.
Problem
FPN contains design defects across feature preparation, fusion, and RoI assignment that prevent multi-scale features from being fully exploited.
Method
AugFPN integrates Consistent Supervision, Residual Feature Augmentation, and Soft RoI Selection to address semantic gaps, highest-level information loss, and RoI feature selection.
Results
AugFPN consistently improves FPN-based detectors, including Faster R-CNN, RetinaNet, and FCOS, across the reported backbones and detector settings.
Takeaways & Limitations
AugFPN provides a general feature-pyramid replacement that improves baseline performance on MS COCO across two-stage and one-stage detectors.
Takeaways & Limitations
AugFPN runs slower than FPN in the reported Faster R-CNN runtime comparison, at 11.1 fps versus 13.4 fps for inference.
Abstract
from arXiv · showhide
Current state-of-the-art detectors typically exploit feature pyramid to detect objects at different scales. Among them, FPN is one of the representative works that build a feature pyramid by multi-scale features summation. However, the design defects behind prevent the multi-scale features from being fully exploited. In this paper, we begin by first analyzing the design defects of feature pyramid in FPN, and then introduce a new feature pyramid architecture named AugFPN to address these problems. Specifically, AugFPN consists of three components: Consistent Supervision, Residual Feature Augmentation, and Soft RoI Selection. AugFPN narrows the semantic gaps between features of different scales before feature fusion through Consistent Supervision. In feature fusion, ratio-invariant context information is extracted by Residual Feature Augmentation to reduce the information loss of feature map at the highest pyramid level. Finally, Soft RoI Selection is employed to learn a better RoI feature adaptively after feature fusion. By replacing FPN with AugFPN in Faster R-CNN, our models achieve 2.3 and 1.6 points higher Average Precision (AP) when using ResNet50 and MobileNet-v2 as backbone respectively. Furthermore, AugFPN improves RetinaNet by 1.6 points AP and FCOS by 0.9 points AP when using ResNet50 as backbone. Codes will be made available.
1. Introduction
FPN improves object detection through multi-scale feature pyramids, but design defects across feature preparation, fusion, and RoI assignment prevent full exploitation of multi-scale features. AugFPN addresses these defects with three components and improves FPN-based detectors across backbones and detector types.
- FPN design defects: FPN propagates semantically strong high-level features to lower levels, but its pyramid has intrinsic flaws before fusion, during fusion, and after fusion.These stages correspond to semantic alignment, highest-level feature preservation, and RoI assignment.
- FPN design defects: Semantic gaps between independently projected feature levels make direct fusion inconsistent and can degrade multi-scale representations.Each level uses an independent 1 × 1 convolution before feature fusion.
- FPN design defects: Highest-level feature maps lose information after channel reduction, while global pooling can discard spatial relations and details when multiple objects appear.The paper identifies this as a feature-fusion defect and motivates retaining richer spatial context.
- FPN design defects: Heuristic RoI assignment uses one pyramid level based on proposal scale, ignoring potentially useful features from other levels.PANet’s max fusion can also discard smaller responses and adds independent fully connected layers.
- AugFPN: AugFPN combines Consistent Supervision, Residual Feature Augmentation, and Soft RoI Selection to address the three FPN defects respectively.The components align semantics, add diverse spatial context residually, and adaptively exploit RoI features across pyramid levels.
- Results: 2.3 AP and 1.6 AP are the reported Faster R-CNN improvements over FPN with ResNet50 and MobileNet-V2 backbones respectively.AugFPN also improves RetinaNet by 1.6 AP and FCOS by 0.9 AP with ResNet50.
2. Related Work
Related work spans two-stage and one-stage detectors, deep supervision, context exploitation, and RoI assignment. AugFPN differs from prior RoI fusion by learning data-dependent weights across levels without extra fully connected adaptation layers.
- Deep Object Detectors: Two-stage detectors generate and refine region proposals, whereas one-stage detectors are more efficient yet less accurate.The related-work discussion places FPN, RetinaNet, and other detectors within these two paradigms.
- Deep Supervision: Deep supervision trains or constrains intermediate layers with additional signals, while AugFPN applies instance-level supervision across pyramid features.Prior methods attach classifiers, pixel-level losses, or detection heads to intermediate representations.
- Context Exploitation: Prior context methods use atrous convolution or pyramid pooling, whereas AugFPN uses ratio-invariant adaptive pooling to generate diverse spatial context.This context is used in Residual Feature Augmentation.
- Strategy of RoI Assignment: FPN assigns each RoI to one scale heuristically, while PANet pools all levels and fuses them with a max operation after fully connected adaptation.AugFPN instead generates adaptive weights from the data and absorbs features from all levels.
- Strategy of RoI Assignment: AugFPN requires fewer parameters than PANet because it uses no extra fully connected layers to adapt RoI features.Its data-dependent weighting is designed to exploit features at different levels more fully.
3. Methodology
AugFPN addresses three FPN design defects—semantic gaps, highest-level information loss, and heuristic RoI assignment—through three corresponding components. These components supervise intermediate features, augment highest-level context, and adaptively fuse RoI features across pyramid levels.
- Consistent Supervision: FPN directly fuses different-scale features despite their large semantic gaps, which can degrade multi-scale representation.AugFPN introduces Consistent Supervision to narrow these gaps before fusion by applying the same supervision signals to multi-scale features.
- Consistent Supervision: Consistent Supervision maps each proposal to all pyramid levels, extracts RoI features from intermediate features, and applies auxiliary losses during training.The auxiliary branches are discarded during testing, so this component adds no inference parameters or computation.
- Residual Feature Augmentation: FPN’s highest-level feature loses information after channel reduction and contains only single-scale context, motivating Residual Feature Augmentation.The method adds diverse spatial context through a residual branch before the augmented feature is fused with lower pyramid levels.
- Residual Feature Augmentation: Ratio-invariant adaptive pooling accounts for image ratio and produces multiple context features, while Adaptive Spatial Fusion combines them instead of simple summation.This differs from PSP’s fixed-size pooling and addresses the reported inferiority of simple summation in ablation experiments.
- Soft RoI Selection: FPN heuristically assigns each RoI to one pyramid level, whereas Soft RoI Selection pools all levels and adaptively combines them with spatial weights.Its Adaptive Spatial Fusion module uses two convolution layers and fewer parameters than PANet’s extra fully connected layers, and is learned by back-propagation.
4. Experiments
Experiments on MS COCO show that AugFPN consistently improves FPN-based detectors across backbones, detector paradigms, and tasks, while each component addresses a distinct feature-pyramid defect. The gains come with increased training time and reduced inference speed.
- Main Results: 1.6 AP and 0.9 AP improvements are reported for RetinaNet and FCOS, respectively, when replacing FPN with AugFPN using ResNet50.RetinaNet gains 1.6 AP with ResNet50, while FCOS rises from 37.0 to 37.9 AP.
- Main Results: 2.0 AP detection and 1.9 AP instance-segmentation gains are obtained for Mask R-CNN with ResNet50.With ResNet101, the corresponding improvements are 1.5 AP for both detection and instance segmentation.
- Ablation Study: Consistent Supervision, Residual Feature Augmentation, and Soft RoI Selection improve the baseline by 0.9 AP, 1.0 AP, and 0.8 AP, respectively.Residual Feature Augmentation raises detection performance from 36.3 to 37.3 AP; Soft RoI Selection contributes most through APm (+1.0 AP) and APl (+0.9 AP).
- Ablation Study: Combining all three components achieves 38.7 AP, a 2.4 AP improvement over the baseline, indicating complementary effects.Consistent Supervision plus Soft RoI Selection alone yields a 1.7 AP improvement.
- Ablation Study: All-level supervision outperforms single-level supervision and the baseline, with λ = 0.25 giving 0.5 and 0.9 AP improvements, respectively.Single-level supervision alone improves the baseline by 0.7 AP at λ = 1.0.
- Ablation Study: Soft RoI Selection averages feature contributions by pyramid level for RoIs originally assigned to P2–P5, and features from all levels contribute together.The analysis uses RoIs from COCO val2017 and averages ASF-generated weights across positions and RoIs.
5. Conclusion
The paper identifies that FPN does not fully exploit multi-scale features and proposes AugFPN to address these issues with three components.
- AugFPN targets underexploited multi-scale features in FPN through three integrated components.The components are Consistent Supervision, Residual Feature Augmentation, and Soft RoI Selection.
- Consistent Supervision, Residual Feature Augmentation, and Soft RoI Selection address distinct feature-pyramid design problems.
- AugFPN improves the baseline method by a large margin on the challenging MS COCO dataset.