Source-linked AI summary
AFPN: Asymptotic Feature Pyramid Network for Object Detection
Guoyu Yang, Jie Lei, Zhikuan Zhu, Siyu Cheng, Zunlei Feng, Ronghua Liang
TL;DR
Conventional feature pyramid propagation can lose or degrade information between non-adjacent levels, limiting multi-scale feature fusion. AFPN progressively fuses adjacent-to-higher-level features and applies adaptive spatial weighting, with competitive results across two-stage and one-stage detectors.
Problem
Conventional feature pyramid propagation can lose or degrade semantic and detailed information during interaction across non-adjacent levels.
Method
AFPN progressively fuses low-level, higher-level, and top-level features, using adaptive spatial fusion to mitigate contradictory information across levels.
Results
AFPN achieves competitive performance across two-stage and one-stage detectors, including 1.6% and 2.6% improvements for ResNet-50 and ResNet-101 Faster R-CNN.
Takeaways & Limitations
The reported experiments support AFPN as a feature pyramid applicable to multiple detector frameworks and competitive with existing pyramid methods.
Abstract
from arXiv · showhide
Multi-scale features are of great importance in encoding objects with scale variance in object detection tasks. A common strategy for multi-scale feature extraction is adopting the classic top-down and bottom-up feature pyramid networks. However, these approaches suffer from the loss or degradation of feature information, impairing the fusion effect of non-adjacent levels. This paper proposes an asymptotic feature pyramid network (AFPN) to support direct interaction at non-adjacent levels. AFPN is initiated by fusing two adjacent low-level features and asymptotically incorporates higher-level features into the fusion process. In this way, the larger semantic gap between non-adjacent levels can be avoided. Given the potential for multi-object information conflicts to arise during feature fusion at each spatial location, adaptive spatial fusion operation is further utilized to mitigate these inconsistencies. We incorporate the proposed AFPN into both two-stage and one-stage object detection frameworks and evaluate with the MS-COCO 2017 validation and test datasets. Experimental evaluation shows that our method achieves more competitive results than other state-of-the-art feature pyramid networks. The code is available at \href{https://github.com/gyyang23/AFPN}{https://github.com/gyyang23/AFPN}.
I. INTRODUCTION
Feature pyramids address scale variation, but conventional propagation can degrade information between hierarchical levels. AFPN uses asymptotic fusion and adaptive spatial fusion, achieving competitive results across detector frameworks.
- Motivation: Feature pyramid architectures mitigate information loss from detecting objects at different scales compared with single-scale feature extraction.FPN is widely used in both one-stage and two-stage detectors.
- Limitations of Existing Pyramids: High-level information can degrade while propagating downward, whereas bottom-up pathways can degrade low-level detail during interaction.These complementary losses impair fusion between non-adjacent feature levels.
- Proposed Approach: AFPN asymptotically fuses two low-level features before progressively incorporating higher-level and top-level features.This staged interaction avoids the large semantic gap between non-adjacent levels while exchanging semantic and detailed information.
- Results: 1.6% and 2.6% improvements are reported for ResNet-50 and ResNet-101 Faster R-CNN, respectively, compared with FPN-based Faster R-CNN.AFPN is also reported to have the lowest FLOPs among compared feature pyramid networks.
- Contributions: AFPN incorporates adaptive spatial fusion to reduce conflicting information from different feature levels at each spatial location.The paper evaluates the method on MS COCO 2017 validation and test datasets and extends it to YOLOv5.
II. RELATED WORK
Single-scale features perform poorly when object or scene sizes vary. Feature pyramids combine multiple scales, while subsequent fusion modules further refine pyramid representations.
- Related Work: Traditional single-scale feature extraction can produce poor detection for objects or scenes with different sizes or scales.Feature pyramids were constructed to overcome this limitation.
- Related Work: Feature pyramids incorporate features at multiple scales, and feature fusion modules augment or refine these representations.These approaches target improved detector performance.
A. Feature Pyramids
Existing pyramids use fixed directional or searched connections, while fusion modules refine their representations. AFPN instead combines levels asymptotically and uses adaptive spatial fusion to manage conflicting information.
- Feature Pyramids: FPN transfers high-level features downward, while PAFPN adds a bottom-up path to provide high-level features with low-level detail.NASFPN searches automatically for feature connections rather than using a fixed architecture.
- Feature Fusion Modules: Feature fusion modules augment pre-existing fixed-topology pyramids, including operators for upsampling, spatial weighting, and channel refinement.ASFF assigns weights to features at different levels to address contradictory information.
- AFPN: AFPN fuses two low-level features first, then adds higher-level and top-level features through successive stages.The architecture is designed to avoid directly fusing levels separated by a large semantic gap.
- Adaptive Spatial Fusion: Adaptive spatial fusion assigns different weights across feature levels to mitigate contradictory information from multiple objects at the same location.The illustrated operation uses three levels but can be adapted to other numbers of levels.
III. ASYMPTOTIC FEATURE PYRAMID NETWORK
AFPN receives multi-level backbone features and fuses them progressively, beginning with low-level inputs before incorporating deeper levels. Its detector-specific outputs preserve the required multi-scale feature hierarchy.
- A. Extracting Multi-level Features: Faster R-CNN supplies AFPN with backbone features {C2, C3, C4, C5} at different scales.The low-level features C2 and C3 enter first, followed by C4 and then C5.
- A. Extracting Multi-level Features: Faster R-CNN AFPN produces {P2, P3, P4, P5, P6} with feature strides of {4, 8, 16, 32, 64} pixels.P6 is generated from P5 using stride-2 and stride-1 convolutions.
- A. Extracting Multi-level Features: YOLO uses {C3, C4, C5} as inputs and generates {P3, P4, P5}.This differs from the five-level feature output used in the Faster R-CNN experiments.
B. Asymptotic Architecture
AFPN asymptotically integrates features from adjacent to increasingly higher levels during backbone extraction, reducing the semantic gap between non-adjacent features.
- B. Asymptotic Architecture: AFPN initially fuses low-level features, then progressively incorporates deep and top-level features during bottom-up extraction.The architecture integrates low-level, high-level, and top-level features in sequence.
- B. Asymptotic Architecture: The asymptotic design addresses the larger semantic gap between non-adjacent hierarchical features by avoiding their immediate direct fusion.Adjacent hierarchical features have a smaller semantic gap than non-adjacent features.
- B. Asymptotic Architecture: AFPN uses 1×1 convolution and bilinear interpolation for dimensional alignment and upsampling before fusion.Downsampling uses convolution kernels and strides selected according to the required rate.
- B. Asymptotic Architecture: The architecture uses three feature levels in YOLO, so it does not require eight-times upsampling or downsampling.
C. Adaptive spatial fusion
Adaptive spatial fusion assigns location-dependent weights to multi-level features, emphasizing useful levels and reducing contradictory information during AFPN fusion.
- C. Adaptive spatial fusion: AFPN uses ASFF to assign varying spatial weights to features from different levels during multi-level fusion.The fusion combines features from three levels at each spatial position.
- C. Adaptive spatial fusion: The fused feature at each position is obtained as a linear combination of the aligned multi-level feature vectors.
- C. Adaptive spatial fusion: The spatial weights correspond to the three feature levels and are constrained to sum to one.
- C. Adaptive spatial fusion: AFPN uses a stage-specific number of adaptive spatial fusion modules because the number of fused features differs across stages.
A. Experiment Setup
Experiments evaluate AFPN on MS COCO 2017 with standard detection metrics, controlled training settings, and comparisons across feature pyramids and fusion operations.
- A. Experiment Setup: Evaluation uses MS COCO 2017 train2017, val2017, and test-dev splits with AP, AP50, AP75, APS, APM, and APL metrics.The splits contain 118k training, 5k validation, and 20k testing images, although test-dev labels are unavailable.
- A. Experiment Setup: Comparative experiments use similar input resolutions because model performance depends heavily on image size.
- A. Experiment Setup: Table III compares two-stage detectors, while Tables IV and V cover AFPN’s contribution to YOLOv5 and fusion-operation ablations.
- A. Experiment Setup: With ResNet-101, AFPN improves AP by 2.6% over FPN on MS COCO test-dev.
C. Results on Different Detectors
AFPN improves detection performance across two-stage and one-stage detectors, with especially clear gains for large objects and high-precision positioning, but not small-target detection.
- C. Results on Different Detectors: AFPN significantly enhances performance in both two-stage and one-stage detector frameworks.
- C. Results on Different Detectors: In two-stage detectors, AFPN particularly improves large-object detection but does not improve small-target detection according to APS results.
- C. Results on Different Detectors: AFPN is inferior to FPN in AP50 but superior in AP75, making it more suitable for high-precision positioning scenarios.
- C. Results on Different Detectors: For YOLOv5-n and YOLOv5-s, AFPN improves APL by 3.4% and 2.6%, respectively.
D. Learnable Parameters and Computational Cost
AFPN reduces feature dimensions to optimize the model while retaining improved detection performance and low computational cost.
- AFPN reduces feature dimensions to the original 1/8 in two-stage detectors and 1/4 in one-stage detectors.The reduction is motivated by AFPN's enhanced representation ability from increased network depth.
- AFPN achieves improved performance on YOLOv5 while using fewer parameters.
- AFPN has the lowest GFLOPs among the methods reported in the referenced table.The paper attributes this computational advantage mainly to reducing feature dimensions.
E. Ablation Studies
Ablation experiments compare adaptive spatial fusion with alternative fusion operations, while broader experiments report strong performance across detection frameworks.
- Element-wise concatenation achieves performance comparable to adaptive spatial fusion in the Faster R-CNN ablation study.The comparison uses ResNet-50 as the backbone.
- The ablation study replaces adaptive spatial fusion with element-wise sum and element-wise concatenation to investigate its efficacy.The experiments are conducted within the Faster R-CNN framework using ResNet-50.
- Extensive experiments demonstrate AFPN's superior performance over baseline methods across various detection frameworks.