Source-linked AI summary
Feature Selective Anchor-Free Module for Single-Shot Object Detection
Chenchen Zhu, Yihui He, Marios Savvides
TL;DR
Anchor-based single-shot detectors rely on heuristic feature selection and overlap-based sampling, which can assign instances to unsuitable pyramid levels. FSAF trains multi-level anchor-free branches with online, content-based feature selection and can run alongside anchor-based branches. On COCO, it improves strong baselines with minimal inference cost, and the best model reaches 44.6% mAP.
Problem
Anchor-based detection is limited by heuristic feature selection and overlap-based anchor sampling, which can prevent instances from being trained at their optimal feature levels.
Method
FSAF attaches anchor-free branches to feature-pyramid levels and dynamically assigns each instance to a suitable level based on instance content during training.
Results
44.6% mAP is achieved on COCO with the final multi-scale detector, while joint FSAF improves RetinaNet by 1.8% AP with only 6ms additional inference latency.
Takeaways & Limitations
FSAF significantly improves strong single-shot detection baselines with tiny inference overhead and outperforms recent state-of-the-art single-shot detectors.
Abstract
from arXiv · showhide
We motivate and present feature selective anchor-free (FSAF) module, a simple and effective building block for single-shot object detectors. It can be plugged into single-shot detectors with feature pyramid structure. The FSAF module addresses two limitations brought up by the conventional anchor-based detection: 1) heuristic-guided feature selection; 2) overlap-based anchor sampling. The general concept of the FSAF module is online feature selection applied to the training of multi-level anchor-free branches. Specifically, an anchor-free branch is attached to each level of the feature pyramid, allowing box encoding and decoding in the anchor-free manner at an arbitrary level. During training, we dynamically assign each instance to the most suitable feature level. At the time of inference, the FSAF module can work jointly with anchor-based branches by outputting predictions in parallel. We instantiate this concept with simple implementations of anchor-free branches and online feature selection strategy. Experimental results on the COCO detection track show that our FSAF module performs better than anchor-based counterparts while being faster. When working jointly with anchor-based branches, the FSAF module robustly improves the baseline RetinaNet by a large margin under various settings, while introducing nearly free inference overhead. And the resulting best model can achieve a state-of-the-art 44.6% mAP, outperforming all existing single-shot detectors on COCO.
1. Introduction
Object detectors use feature pyramids and anchor boxes to handle scale variation, but anchor-based assignment can select non-optimal feature levels. FSAF addresses this by letting instances choose feature levels online through anchor-free branches and achieves strong COCO results with low overhead.
- Object detection supports downstream applications including instance segmentation, facial analysis, autonomous driving, and video analysis.
- Feature pyramids generate predictions at multiple scales, while anchor boxes discretize possible instance boxes using predefined locations, scales, and aspect ratios.
- Anchor-based detection uses heuristic feature selection and overlap-based anchor sampling, so an instance’s selected feature level may not be optimal.
- FSAF attaches an anchor-free branch to each pyramid level and dynamically assigns each instance to the most suitable level based on instance content.
- FSAF can operate independently or jointly with anchor-based branches during inference and is designed to remain agnostic to the backbone network.
- 44.6% mAP is achieved by the final detector with multi-scale testing, outperforming existing single-shot detectors on COCO.
2. Related Work
Single-shot detectors commonly use predefined anchors across feature pyramids, while anchor-free methods directly predict boxes or represent them through geometric points. The paper positions FSAF as a feature-selective anchor-free module that can integrate with conventional detectors.
- SSD, FPN, DSSD, RetinaNet, and DetNet use feature pyramids or multi-level towers, but retain predefined anchor boxes for encoding and decoding instances.
- FSAF is presented as a module that plugs into single-shot detectors with feature pyramids and assigns instances to pyramid levels through feature selection.
- Prior anchor-free approaches include direct box prediction, IoU-based regression, anchor-free region proposals, corner-pair detection, and integration of anchor-based and anchor-free methods.
3. Feature Selective Anchor-Free Module
FSAF adds anchor-free branches to each feature-pyramid level and dynamically assigns instances to the level with the lowest combined classification and regression loss. This replaces heuristic, anchor-constrained feature selection with online selection based on instance content.
- 3.1. Network Architecture: FSAF attaches an anchor-free classification and regression branch to every feature-pyramid level, alongside RetinaNet’s anchor-based branches.The module adds two convolutional layers per pyramid level while preserving a fully convolutional architecture.
- 3.2. Ground-truth and Loss: During training, each instance is represented in anchor-free form at an arbitrary pyramid level rather than being constrained by predefined anchors.The instance’s projected box and effective or ignored regions provide supervision for classification and regression maps.
- 3.2. Ground-truth and Loss: Classification uses focal loss, while box regression uses IoU loss over effective regions; locations outside those regions are ignored.The regression targets encode distances from each pixel to the projected box boundaries, normalized by S = 4.0.
- 3.3. Online Feature Selection: Unlike heuristic selection based purely on box size, FSAF chooses the feature level from the instance’s current losses and content.At inference, all levels produce predictions, so explicit feature selection is unnecessary.
- 3.3. Online Feature Selection: Online feature selection forwards each instance through all anchor-free levels and selects the level minimizing the sum of averaged focal and IoU losses.The selected level receives the supervision signal for that instance during training.
4. Experiments
Experiments on COCO ablations and test-dev comparisons show that online feature selection improves detection, especially for challenging instances, while adding little inference cost.
- Experimental setup: COCO experiments use trainval35k for training, minival for ablations, and test-dev for state-of-the-art comparisons.Ablations use 5k minival images; test-dev labels are private and require the evaluation server.
- Ablation studies: Online feature selection raises AP50 by 2.5%, APS by 1.5%, and APL by 2.2% when anchor-free and anchor-based branches work jointly.The joint model learns instances that are hard to model with anchor-based branches alone.
- Ablation studies: Online feature selection improves anchor-free branches by 1.2% AP over heuristic feature selection.The improvement overcomes the anchor-free branches' parameter disadvantage.
- Ablation studies: Online selection usually assigns larger instances to upper pyramid levels and smaller instances to lower levels, but also captures exceptions to anchor-based assignments.These exceptions allow FSAF to use better features for challenging objects.
- Efficiency and robustness: On ResNeXt-101, FSAF alone outperforms anchor-based counterparts by 1.2% AP and runs 68ms faster.When combined with anchor-based branches, it improves RetinaNet by 1.8% AP with only 6ms additional inference latency.
- Comparison to state of the art: The best detector reaches 44.6% AP with ResNeXt-101-64x4d, surpassing previous state-of-the-art single-shot detectors.The comparison is reported on COCO test-dev; the final model uses RetinaNet with FSAF.
5. Conclusion
The paper identifies heuristic feature selection as a primary limitation of anchor-based feature-pyramid detectors and addresses it with online selection in anchor-free branches.
- Conclusion: FSAF applies online feature selection to train anchor-free branches in a feature pyramid.It improves strong baselines with tiny inference overhead and outperforms recent state-of-the-art single-shot detectors.