Source-linked AI summary
Learning Spatial Fusion for Single-Shot Object Detection
Songtao Liu, Di Huang, Yunhong Wang
TL;DR
Single-shot detectors using feature pyramids are limited by inconsistency across feature scales, while image pyramids increase inference time. ASFF learns spatially adaptive fusion after resizing features to a common shape, and with YOLOv3 achieves a state-of-the-art speed-accuracy trade-off on COCO with marginal overhead.
Problem
Feature pyramids address scale variation in object detection but suffer from cross-scale inconsistency, while image-pyramid solutions sharply increase inference time.
Method
ASFF resizes features from other levels to a common shape and learns spatial fusion weights that filter contradictory information before combination.
Results
42.4% mAP at 45 FPS and 43.9% mAP at 29 FPS are reported for an enhanced YOLOv3 with ASFF.
Takeaways & Limitations
ASFF improves strong single-shot detector baselines with tiny inference overhead and achieves a state-of-the-art speed-accuracy trade-off among single-shot detectors.
Takeaways & Limitations
Feature-pyramid methods remain constrained by cross-scale inconsistency, while scale-specific alternatives may suffer from missing higher-resolution maps for small instances.
Abstract
from arXiv · showhide
Pyramidal feature representation is the common practice to address the challenge of scale variation in object detection. However, the inconsistency across different feature scales is a primary limitation for the single-shot detectors based on feature pyramid. In this work, we propose a novel and data driven strategy for pyramidal feature fusion, referred to as adaptively spatial feature fusion (ASFF). It learns the way to spatially filter conflictive information to suppress the inconsistency, thus improving the scale-invariance of features, and introduces nearly free inference overhead. With the ASFF strategy and a solid baseline of YOLOv3, we achieve the best speed-accuracy trade-off on the MS COCO dataset, reporting 38.1% AP at 60 FPS, 42.4% AP at 45 FPS and 43.9% AP at 29 FPS. The code is available at https://github.com/ruinmessi/ASFF
1. Introduction
Object detection must handle objects across widely varying scales, but feature pyramids introduce cross-scale inconsistency that limits single-shot detectors. ASFF learns spatially adaptive fusion to address this issue while preserving real-time performance.
- Object detection remains challenging because images contain multiple objects across a wide range of scales.
- Image pyramids can improve scale-invariant prediction but sharply increase inference time, limiting their use in real-world applications.
- Feature pyramids improve scale handling, but heuristic level assignments create conflicting information across scales in single-shot detectors.Positive regions at one level are treated as background at other levels, interfering with gradient computation.
- ASFF learns to spatially filter and fuse features from different levels, retaining useful information while suppressing contradictory information.Features are resized to a common resolution before adaptive fusion at each spatial location.
- 42.4% mAP at 45 FPS and 43.9% mAP at 29 FPS are reported after applying ASFF to an enhanced YOLOv3 baseline.The enhanced YOLOv3 baseline reaches 38.8% mAP at 50 FPS before ASFF.
2. Related Work
Prior feature-pyramid methods improve multi-scale representations but still suffer from cross-scale inconsistency. ASFF instead learns spatial fusion weights after resizing features from other levels to a common shape.
- FPN and related methods construct multi-scale representations through top-down, lateral, or additional cross-scale connections.
- Feature-pyramid methods still suffer from cross-scale inconsistency, limiting further performance gains.Ignore regions can increase false positives, while scale-specific branches may lose benefits from higher-resolution maps for small instances.
- ASFF learns connections among different feature maps rather than relying only on fixed multi-level fusion structures.
- ASFF resizes features from other levels to the target level's shape and then adaptively fuses them using learned spatial weights.
- Element-wise gating has succeeded in semantic segmentation, but object detection requires heuristic-guided feature selection across levels.
3. Method
ASFF strengthens a YOLOv3-based single-shot detector by resizing feature maps to a common resolution and learning spatially adaptive fusion weights. Its fusion addresses gradient inconsistency across pyramid levels while retaining computationally efficient detection.
- Strong YOLOv3 baseline: 38.8% mAP at 50 FPS is achieved by the enhanced YOLOv3 baseline on COCO 2017 val before applying ASFF.The baseline combines training tricks, an anchor-free branch, anchor guiding, and an additional IoU loss without heavy inference cost.
- Adaptive spatial feature fusion: ASFF resizes features from other pyramid levels to the target level’s resolution before adaptively fusing them at each spatial location.Upsampling uses channel compression followed by interpolation; downsampling uses strided convolution, with max pooling for a 1/4 scale ratio.
- Adaptive spatial feature fusion: The network learns spatial importance weights for the three level-specific features and uses them to produce each output feature map.The weights can be shared across channels and are learned through standard back-propagation.
- Consistency property: ASFF harmonizes inconsistent gradients by learning fusion coefficients that can reduce conflicting contributions from other pyramid levels while preserving background supervision.The analysis contrasts ASFF with ignoring corresponding regions, whose relaxation tends to cause inferior predictions and false positives.
- Training and inference: The detector jointly optimizes network parameters and fusion parameters using the YOLOv3 objective augmented with IoU regression loss.Inference retains YOLOv3-style classification and box regression followed by class-wise NMS at a threshold of 0.6.
4. Experiments
Experiments show that ASFF improves YOLOv3’s detection accuracy while adding little inference cost, and qualitatively filters conflicting cross-level features. Comparisons and visualizations support its effectiveness across object sizes and detector settings.
- Experimental setup: 115k COCO train-2017 images, 5k val-2017 images, and 20k test-dev images support training, ablation, and main evaluation.Main results report COCO AP on test-dev, while ablations and sensitivity studies use val-2017.
- Baseline: 38.8% AP is achieved by the final baseline after adding training and regression components, establishing the reference detector.The reported baseline uses BoF, guided anchoring, and an additional IoU loss.
- ASFF ablation: 40.6% box AP is obtained with ASFF versus 38.8% for the baseline, while inference increases by only 2 ms to 46 FPS.Most gains come from APS and APM, each increasing by 2.9% relative to corresponding reference scores.
- Fusion comparison: Element-wise sum and concatenation improve APS and APM but sharply reduce APL, unlike ASFF’s reported improvement across the evaluated scales.The comparison indicates that cross-level inconsistency negatively affects training and limits pyramidal feature representations.
- Visual analysis: Qualitative maps show ASFF filtering resized features at spatial locations where another level provides the relevant object representation.For same-sized objects, level-specific fusion suppresses conflicting features; for mixed sizes, a tennis racket predicted at level 1 draws central features mainly from resized level 2.
- Transfer to RetinaNet: ASFF consistently increases RetinaNet accuracy with both ResNet-50 and ResNet-101 backbones.The paper also evaluates ASFF beyond YOLOv3 by adding it to FPN features from P3 to P5.
- State-of-the-art comparison: YOLOv3 with enhanced ASFF* reaches the performance level of state-of-the-art single-shot detectors while retaining YOLOv3’s efficiency.ASFF* integrates DropBlock and RFB, uses 1.5× longer training, and improves performance more significantly at lower input resolutions.
5. Conclusion
The work identifies cross-scale inconsistency as a primary limitation of feature-pyramid single-shot detectors and proposes ASFF to filter it during training. ASFF improves strong baselines with tiny inference overhead and achieves a state-of-the-art speed-accuracy trade-off among single-shot detectors.
- ASFF learns adaptive spatial fusion weights during training to filter inconsistency across feature scales.
- ASFF significantly improves strong single-shot detector baselines while adding tiny inference overhead.