Source-linked AI summary
FSSD: Feature Fusion Single Shot Multibox Detector
Zuoxin Li, Lu Yang, Fuqiang Zhou
TL;DR
Scale variation remains a critical challenge in object detection, and conventional SSD struggles to combine detailed and semantic features across levels. FSSD adds lightweight feature fusion that concatenates transformed multi-scale features and generates pyramid features through down-sampling, improving accuracy and efficiency over conventional SSD and other detectors.
Problem
Scale variation remains a critical object-detection challenge, while conventional SSD does not fully combine local detailed features with global semantic features across levels.
Method
FSSD adds a lightweight feature-fusion module that transforms and concatenates features from different layers, then uses down-sampling blocks to generate pyramid features for detection.
Results
FSSD substantially improves conventional SSD and outperforms several state-of-the-art detectors in accuracy and efficiency, especially for small objects.
Takeaways & Limitations
Multi-level feature concatenation and subsequent pyramid generation can fully utilize features and improve object detection without additional bells and whistles.
Takeaways & Limitations
The authors identify stronger backbone networks as future work to improve FSSD performance on MSCOCO.
Abstract
from arXiv · showhide
SSD (Single Shot Multibox Detector) is one of the best object detection algorithms with both high accuracy and fast speed. However, SSD's feature pyramid detection method makes it hard to fuse the features from different scales. In this paper, we proposed FSSD (Feature Fusion Single Shot Multibox Detector), an enhanced SSD with a novel and lightweight feature fusion module which can improve the performance significantly over SSD with just a little speed drop. In the feature fusion module, features from different layers with different scales are concatenated together, followed by some down-sampling blocks to generate new feature pyramid, which will be fed to multibox detectors to predict the final detection results. On the Pascal VOC 2007 test, our network can achieve 82.7 mAP (mean average precision) at the speed of 65.8 FPS (frame per second) with the input size 300$\times$300 using a single Nvidia 1080Ti GPU. In addition, our result on COCO is also better than the conventional SSD with a large margin. Our FSSD outperforms a lot of state-of-the-art object detection algorithms in both aspects of accuracy and speed. Code is available at https://github.com/lzx1413/CAFFE_SSD/tree/fssd.
1. Introduction
FSSD addresses SSD’s difficulty combining multi-scale features by concatenating transformed features from different layers and generating a new detection pyramid. The resulting lightweight module improves accuracy over conventional SSD with only a slight speed cost.
- Motivation: Scale variation remains a critical object-detection challenge, while independently processing image scales is inefficient and single-scale features limit receptive-field coverage.These limitations motivate better use of multi-level ConvNet features.
- Motivation: Top-down fusion improves detection but requires multiple layer-by-layer merging processes and is not efficient when many layers must be combined.The paper contrasts this design with direct fusion of multiple feature levels.
- Method: FSSD projects and concatenates features from different layers and scales, normalizes them with Batch Normalization, then applies down-sampling blocks to generate a new feature pyramid.The generated pyramid is fed to multibox detectors for final predictions.
- Contributions: FSSD introduces a novel lightweight way to combine feature maps from different levels and generate a feature pyramid for fuller feature utilization.The paper identifies this framework and its influencing factors through analysis and experiments.
2. Related Work
Related detectors use region-based or one-stage designs, while feature-fusion methods combine information across ConvNet layers to improve object detection. SSD builds a progressively smaller feature pyramid, and later methods modify or enrich this design.
- Object detectors: Region-based detectors use region proposals or pooled ConvNet features, whereas one-stage detectors directly perform localization and classification for efficient inference.Examples include R-CNN-family methods, YOLO, and SSD.
- SSD: SSD predicts class scores and location offsets from multiple convolutional layers, assigning anchor sizes according to each layer’s receptive-field size.Non-maximum suppression post-processes the final detections.
- SSD extensions: DSSD adds deconvolution layers and additional large-scale context but slows inference because of increased model complexity.This illustrates an accuracy–speed trade-off in SSD extensions.
- SSD extensions: RSSD uses rainbow concatenation through pooling and concatenation to exploit relationships between layers in the feature pyramid with little speed loss.DSOD instead focuses on training from scratch and parameter efficiency through a DenseNet architecture.
- Feature fusion: Feature-fusion approaches concatenate multiple layer features, use skip connections, or apply top-down structures to combine low- and high-level information.Examples include HyperNet, ION, FCN, U-Net, SharpMask, and FPN.
3. Method
FSSD augments SSD with a feature-fusion module that combines multi-level features and regenerates a detection pyramid. The method uses concatenation, down-sampling, and selected training configurations to retain efficiency while improving feature utilization.
- Feature Fusion Module: FSSD concatenates transformed feature maps from different layers, then applies down-sampling blocks to generate pyramid features for multibox detection.The fusion framework focuses on selecting layers, transforming and merging features, and generating the new pyramid.
- Feature Fusion Module: The fusion design selects conv3_3, conv4_3, fc7, and conv7_2 as candidate source layers, while excluding feature maps smaller than 10×10.The selected feature maps are spatially aligned to the conv4_3 size before fusion.
- Feature Fusion Module: Concatenation is preferred over element-wise summation because it avoids requiring equal channel dimensions and offers greater fusion flexibility.The source feature maps are first projected with 1×1 convolutions before concatenation.
- Pyramid Feature Generation: The selected pyramid generator uses several simple convolution-plus-ReLU blocks after fusion rather than directly detecting from the fusion feature map.Among three tested structures, this design performs slightly better than the alternatives.
- Training: FSSD can be initialized from either a pretrained SSD or VGG16 model, with SSD initialization converging faster but VGG16 initialization achieving slightly better final results.The default training procedure uses the pretrained SSD model.
- Training: FSSD retains SSD’s training objective, bounding-box encoding, matching, hard-negative mining, and data-augmentation strategies.The comparison uses the same training objective and associated strategies as SSD.
4. Experiments
The experiments evaluate FSSD against conventional SSD on PASCAL VOC and MS COCO using a VGG16-based setup. Detection performance is measured with mAP, with VOC correctness defined by an IoU threshold above 0.5.
- Datasets and Setup: Experiments compare FSSD and conventional SSD using VGG16 on PASCAL VOC 2007, VOC 2012, and MS COCO datasets.The VGG16 backbone is preprocessed like SSD.
- Evaluation: VOC detections are counted as correct when bounding-box IoU with ground truth exceeds 0.5, and mAP is used to evaluate detection performance.MS COCO results are submitted for evaluation through the dataset’s evaluation process.
4.1. Ablation Study on PASCAL VOC2007
The ablations examine source-layer selection, feature normalization, fusion operation, and pyramid-extractor structure on VOC2007. Concatenation and the selected extractor design provide the strongest reported choices among the tested alternatives.
- Fusion Layer Selection: 78.8% mAP results when conv3_3 is removed, compared with 78.6% mAP when conv3_3, conv4_3, fc7, and conv7_2 are fused.The conv3_3 down-sampled feature map provides no benefit in this comparison, while conv7_2 is retained.
- Fusion Operation: 78.6% mAP results with concatenation, versus 76.3% mAP with element-wise summation, a difference of 2.3 points.The experiment directly compares the two feature-fusion operations.
- Feature Normalization: 0.7% mAP improvement results from adding Batch Normalization after concatenation to rescale feature maps.The normalization addresses differing value ranges among feature maps.
- Pyramid Feature Extractor Design: The pyramid structure that excludes direct prediction from the fusion feature map performs slightly better than the other two tested structures.The comparison includes simple blocks and bottleneck blocks for generating lower-resolution maps.
4.2. Results on PASCAL VOC
On PASCAL VOC, FSSD improves over conventional SSD across VOC2007 and VOC2012 while retaining competitive speed, with results varying by training data and input size.
- 78.8% mAP on VOC2007 test improves 1.1 points over conventional SSD300 for FSSD300.
- 82.7% mAP for FSSD300 trained with COCO exceeds SSD300 by 1.5% and DSOD by 1%.
- 80.9% mAP for FSSD512 improves SSD512 from 79.8%, while DSSD512 is more accurate but slower.
- 82.0% mAP on VOC2012 for FSSD300 exceeds conventional SSD's 79.3% by 2.7 points.
- 84.2% mAP for FSSD512 exceeds conventional SSD512's 82.2% by 2 points and leads one-stage detectors on the VOC2012 leaderboard.
4.3. Results on MS COCO
On MS COCO test-dev, FSSD improves substantially over conventional SSD, although it remains below DSOD and DSSD in overall performance.
- COCO training uses 115K train images and evaluates FSSD on the 20K test-dev set.
- 27.1% test-dev performance for FSSD300 exceeds SSD300*'s 25.1% by 2 points.
- 31.8% for FSSD512 outperforms conventional SSD's 28.8% by 3 points.
- FSSD512 is slightly below DSSD513 overall, but its mAP on small objects is higher than DSSD513's.
4.4. Speed
FSSD preserves high inference speed while improving accuracy, though its added layers increase inference time relative to SSD.
- 65.8 FPS is achieved by FSSD with 300 × 300 input on a single Nvidia 1080Ti GPU.
- FSSD consumes about 25% extra inference time because it adds layers to SSD.
- FSSD remains much faster than DSSD and DSOD while achieving competitive accuracy against most object detectors.
4.5. Performance improvement from SSD
FSSD improves SSD's detection behavior by reducing repeated detections of object parts and helping detect whole objects in challenging cases.
- FSSD reduces repeated detections of multiple parts from one object and merging multiple objects into one.
- In the dog example, SSD detects the head as a separate dog, whereas FSSD detects the whole dog at once.
5. Conclusion and future work
FSSD combines features across layers and generates pyramid feature maps through strided convolutions, improving conventional SSD and other detectors in accuracy and efficiency. Future work includes stronger backbones and applying the fusion module to Mask R-CNN.
- FSSD enhances SSD with a lightweight feature fusion module that concatenates features from different layers before generating pyramid feature maps.Several convolutional layers with stride 2 generate the pyramid features used for detection.
- FSSD improves conventional SSD and outperforms several state-of-the-art object detectors in accuracy and efficiency without additional bells and whistles.
- Future work proposes using stronger backbones, including ResNet and DenseNet, to improve MSCOCO performance.
- Replacing the FPN in Mask R-CNN with FSSD's feature fusion module is identified as another future research direction.