Source-linked AI summary
M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network
Qijie Zhao, Tao Sheng, Yongtao Wang, Zhi Tang, Ying Chen, Ling Cai, Haibin Ling
TL;DR
Existing feature pyramids inherit classification-oriented backbone structure and may provide insufficiently representative, single-level features for scale-varying objects. The paper proposes MLFPN, which fuses multi-level features and aggregates decoder features across scales, then integrates it into SSD as M2Det. On MS-COCO, M2Det reports 41.0 AP at 11.8 FPS with single-scale inference and 44.2 AP with multi-scale inference.
Problem
Existing feature pyramids can be limited because they rely on classification backbones and mainly provide single-level information for each detection scale.
Method
MLFPN fuses backbone features, processes them through alternating Thinned U-shape and Feature Fusion Modules, and aggregates equal-scale decoder layers into a detection pyramid.
Results
41.0 AP at 11.8 FPS with single-scale inference and 44.2 AP with multi-scale inference are reported for M2Det on MS-COCO.
Takeaways & Limitations
MLFPN supplies deeper, multi-level multi-scale features, and M2Det reports a new state-of-the-art result among one-stage detectors.
Abstract
from arXiv · showhide
Feature pyramids are widely exploited by both the state-of-the-art one-stage object detectors (e.g., DSSD, RetinaNet, RefineDet) and the two-stage object detectors (e.g., Mask R-CNN, DetNet) to alleviate the problem arising from scale variation across object instances. Although these object detectors with feature pyramids achieve encouraging results, they have some limitations due to that they only simply construct the feature pyramid according to the inherent multi-scale, pyramidal architecture of the backbones which are actually designed for object classification task. Newly, in this work, we present a method called Multi-Level Feature Pyramid Network (MLFPN) to construct more effective feature pyramids for detecting objects of different scales. First, we fuse multi-level features (i.e. multiple layers) extracted by backbone as the base feature. Second, we feed the base feature into a block of alternating joint Thinned U-shape Modules and Feature Fusion Modules and exploit the decoder layers of each u-shape module as the features for detecting objects. Finally, we gather up the decoder layers with equivalent scales (sizes) to develop a feature pyramid for object detection, in which every feature map consists of the layers (features) from multiple levels. To evaluate the effectiveness of the proposed MLFPN, we design and train a powerful end-to-end one-stage object detector we call M2Det by integrating it into the architecture of SSD, which gets better detection performance than state-of-the-art one-stage detectors. Specifically, on MS-COCO benchmark, M2Det achieves AP of 41.0 at speed of 11.8 FPS with single-scale inference strategy and AP of 44.2 with multi-scale inference strategy, which is the new state-of-the-art results among one-stage detectors. The code will be made available on \url{https://github.com/qijiezhao/M2Det.
Introduction
Existing feature pyramids can be limited by relying on backbone layers designed for classification and single-level information. MLFPN addresses this by constructing deeper, multi-level multi-scale features, and M2Det reaches strong MS-COCO results.
- Existing detectors often construct pyramids directly from classification backbones, making their feature maps insufficiently representative for detection.
- Single-level pyramid features combine poorly with scale and appearance variation because shallow and deep layers provide complementary localization and semantic information.
- MLFPN fuses backbone features, alternates Thinned U-shape and Feature Fusion Modules, and aggregates equivalent-scale decoder layers into the detection pyramid.
- Each MLFPN pyramid map combines multiple levels and uses decoder layers deeper than the backbone, producing more representative detection features.
- 41.0 AP at 11.8 FPS with single-scale inference and 44.2 AP with multi-scale inference establish M2Det's reported MS-COCO results among one-stage detectors.
Related Work
Feature-pyramid detection offers a lower-cost alternative to image pyramids while retaining multi-scale processing, and it has been incorporated into several detector architectures.
- Image pyramids produce semantically representative multi-scale features by running predictions on differently resized copies of the input image.
- Image-pyramid methods can improve recognition accuracy and localization precision but impose time and memory costs that hinder real-time use.
- Feature pyramids extracted from inherent network layers use a single-scale image and require substantially less additional memory and computation.
- Feature-pyramid construction modules can be revised and integrated into deep neural-network detectors for training and testing.
- MS-CNN, SSD, and FPN exemplify detectors that use multi-resolution or multi-scale network features for object detection.
Proposed Method
M2Det’s MLFPN combines backbone features across levels, generates multi-level multi-scale features with alternating TUMs and FFMs, and aggregates equivalent scales into a feature pyramid. The resulting features support dense bounding-box localization and category scoring.
- M2Det uses MLFPN features to produce dense bounding boxes and category scores, with six pyramidal features and six anchors per pixel at three ratios.The detector follows SSD for default-box scale ranges and applies separate convolutional layers for localization and classification.
- MLFPN uses FFMv1 to fuse shallow and deep backbone features into a semantically enriched base feature.
- Alternating TUMs and FFMv2s generate multi-level multi-scale features, with each TUM producing feature maps at different scales.FFMv2 fuses the base feature with the largest output feature map from the preceding TUM before the next TUM processes it.
- TUM decoder outputs form multi-scale features within each level, while stacked TUMs provide shallow-, medium-, and deep-level features.The TUM encoder uses stride-2 convolutions, and its decoder references outputs from those encoder layers.
- SFAM concatenates features with equivalent scales across channels and applies channel-wise attention to aggregate them adaptively.The resulting pyramid contains features from multiple depth levels at each scale.
- The default MLFPN configuration uses eight TUMs, while input sizes follow SSD, RefineDet, and RetinaNet at 320, 512, and 800.Each TUM has five striding convolutions and five upsampling operations, with 256 channels allocated to each scale of its features.
Experiments
Experiments evaluate M2Det on MS-COCO through state-of-the-art comparisons, ablations, configuration studies, and speed measurements. The results show strong accuracy, efficiency, and consistent gains from the proposed components and configurations.
- Experimental setup: M2Det is evaluated on MS-COCO bounding-box detection using trainval35k for training, test-dev for comparisons, and minival for ablations.The experiments include implementation details, comparisons with state-of-the-art approaches, ablations, and different MLFPN settings.
- Comparison with state-of-the-art: 147M parameters are used by M2Det800-VGG, compared with 201M for CornerNet with Hourglass and 205M for Mask R-CNN with ResNeXt-101-32x8d-FPN.The comparison is presented to show that M2Det’s improvement is not entirely attributable to greater depth or parameter count.
- Ablation study: AP improves from 25.8 to 27.5 with Deconv layers, then reaches 30.8 after replacing the U-shape module with s-TUM in the ablation sequence.Stacking eight s-TUMs improves performance by 3.1 over the preceding operation, and replacing TUM by s-TUM produces the best result in that sequence.
- Ablation study: 32.7 AP is obtained when the base feature is fed into each TUM, providing necessary localization information from shallow features.The preceding ablation identifies a limitation from using the base feature only once: reducing channels lowers abstraction, while increasing them raises parameter count.
- Ablation study: All evaluation metrics improve when SFAM is included, with greater accuracy for small, medium, and large boxes.The ablation compares the architecture with SFAM against the architecture without SFAM.
- Ablation study: AP rises from 33.2 to 34.1 when VGG-16 is replaced with ResNet-101, and the improvement remains consistent across other AP metrics.This result is reported as the backbone-feature ablation outcome.
- Variants of MLFPN: More TUMs improve detection accuracy, while more internal channels also consistently benefit results; with similar parameter increases, more TUMs provide larger improvement than more channels.These variants fix VGG-16 and 320×320 input size and evaluate configurations on COCO minival.
Discussion
The authors attribute M2Det’s accuracy gains mainly to MLFPN, whose deeper decoder features and multi-level aggregation better address scale and appearance variation.
- Discussion: MLFPN’s accuracy improvement is mainly attributed to deeper decoder features that are more representative for object detection than backbone layers.The decoder layers are deeper than backbone layers, while existing detectors use backbone or minimally deepened extra layers.
- Discussion: Each feature map aggregates decoder layers from multiple levels, enabling multi-level features to detect objects at every scale.This design is intended to handle appearance-complexity variation across object instances.
- Discussion: The activation visualization uses scale and level dimensions to examine whether MLFPN learns features for different object scales and large appearance variation.The example contains persons, cars, and a traffic light with differing and similar object sizes.
- Discussion: The discussion concludes that multi-level features help handle scale and appearance-complexity variation and are necessary for similarly sized objects.The conclusion specifically connects multi-level features with objects that share similar sizes.
Conclusion
The paper proposes MLFPN to construct effective multi-level, multi-scale feature pyramids and builds M2Det by integrating it into SSD. On MS-COCO, M2Det reports state-of-the-art one-stage detection results.
- Conclusion: MLFPN fuses backbone features into a base feature, extracts multi-level multi-scale decoder features through TUMs and FFMs, and aggregates equal-scale features with SFAM.The resulting feature pyramid is used for object detection.
- Conclusion: M2Det integrates MLFPN into SSD as an end-to-end one-stage object detector.The detector produces the paper’s reported benchmark results.
- Conclusion: AP 41.0 at 11.8 FPS with single-scale inference and AP 44.2 with multi-scale inference are reported on MS-COCO.The paper describes these as new state-of-the-art results among one-stage detectors.