Source-linked AI summary
A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection
Zhaowei Cai, Quanfu Fan, Rogerio S. Feris, Nuno Vasconcelos
TL;DR
Multi-scale CNNs address the difficulty of detecting objects whose sizes span multiple scales, particularly small objects in datasets such as KITTI and Caltech. MS-CNN uses shared proposal and detection subnetworks with scale-specific output layers and feature upsampling, achieving high detection rates at speeds of up to 15 fps. Its scope includes datasets where object scales span multiple octaves, motivating multi-scale training and design choices.
Problem
CNN-based detectors have limited effectiveness across widely varying object scales, while input upsampling for small objects increases memory and computation costs.
Method
MS-CNN jointly trains proposal and detection subnetworks, performs detection at multiple output layers matched to scale ranges, and uses deconvolutional feature upsampling as an alternative to input upsampling.
Results
Up to 15 fps is achieved while the detector reports high detection rates on KITTI and Caltech; proposal recall reaches about 98% with 100 proposals in the reported comparison.
Takeaways & Limitations
A single feedforward pass can detect objects across scales, while feature upsampling provides memory and computation savings relative to input upsampling.
Takeaways & Limitations
The approach is motivated and evaluated for datasets such as Caltech and KITTI where object scales can span multiple octaves, including many small objects in KITTI.
Abstract
from arXiv · showhide
A unified deep neural network, denoted the multi-scale CNN (MS-CNN), is proposed for fast multi-scale object detection. The MS-CNN consists of a proposal sub-network and a detection sub-network. In the proposal sub-network, detection is performed at multiple output layers, so that receptive fields match objects of different scales. These complementary scale-specific detectors are combined to produce a strong multi-scale object detector. The unified network is learned end-to-end, by optimizing a multi-task loss. Feature upsampling by deconvolution is also explored, as an alternative to input upsampling, to reduce the memory and computation costs. State-of-the-art object detection performance, at up to 15 fps, is reported on datasets, such as KITTI and Caltech, containing a substantial number of small objects.
1 Introduction
MS-CNN addresses multi-scale detection by combining scale-specific detectors across network layers in one end-to-end CNN, while feature upsampling reduces the costs of handling small objects. The approach reports strong proposal recall and fast detection on challenging benchmarks.
- Motivation: Deep CNN detectors made progress on multiclass detection, but detecting objects across multiple scales remained difficult, especially for small objects.Input upsampling used by prior methods increases memory and computation costs.
- MS-CNN: MS-CNN performs detection at multiple output layers whose receptive fields target different object-scale ranges.Lower layers match small objects, while higher layers suit large objects.
- MS-CNN: The proposal and detection sub-networks are learned end-to-end and share computations within a unified architecture.The proposal sub-network combines complementary scale-specific detectors.
- Results: Over 95% recall is achieved with only 100 proposals on KITTI, whose objects vary substantially in scale.This result concerns the combined multi-scale proposal detector.
- Feature upsampling: Feature upsampling through deconvolution replaces input upsampling, enlarging feature-map responses for small objects while reducing memory and computation costs.Without image upsampling, MS-CNN reaches 10 fps on KITTI and 15 fps on Caltech.
2 Related Work
Related work improved detector speed through efficient feature extraction, cascade learning, shared CNN computation, and intermediate-layer supervision. These approaches established important components but left multiclass or scale-consistency challenges for subsequent detectors.
- Classical detectors: Real-time sliding-window detectors improved speed through fast feature extraction and cascade learning.Examples include integral-image Haar features, aggregate feature channels, soft cascades, and complexity-aware cascade formulations.
- Classical detectors: Detector cascades remained difficult to adapt to multiclass detection.This is identified as the main current limitation of the cascade architecture.
- CNN detectors: R-CNN combined object proposals with CNN classification, but proposal generation and repeated CNN evaluation limited speed.Spatial pyramid pooling reduced repeated computation by allowing CNN features to be computed once per image.
- Deep supervision: Intermediate-layer losses had been used in deep networks as a form of supervision and regularization.Prior work applied multiple weighted classification losses or deeply supervised objectives at intermediate layers.
3 Multi-scale Object Proposal Network
The MS-CNN proposal network addresses multi-scale detection by applying scale-specific detectors at multiple network layers, using different receptive fields while sharing a CNN trunk. It trains these branches with scale-selected samples and a multi-task classification and bounding-box regression loss.
- Multi-scale Detection: Multi-scale detection balances accuracy and efficiency by either rescaling inputs repeatedly or applying multiple classifiers to one image.Repeated input scaling is costly, while single-image multi-classifier strategies can produce weak detectors.
- Multi-scale Detection: The MS-CNN uses several feature-map resolutions and intermediate-layer templates to cover different object scales with variable receptive fields.This extends a few-scale strategy to deep CNNs while using a single input scale.
- Network Architecture: The proposal network has a shared CNN trunk with detection branches from different layers, whose outputs are combined as final proposal detections.The branches consist of single detection layers, with a buffer convolution on the branch after conv4-3.
- Training: The unified network optimizes a weighted multi-task loss combining classification and bounding-box regression across detection branches.Only the scale-selected subset contributes to each branch’s loss, and positive samples receive the bounding-box loss.
- Sampling: Hard-negative mining is used to address the severe imbalance between object and non-object samples, with mixture sampling performing similarly to bootstrapping.The final negative set is constrained to |S−| = γ|S+|.
- Training: Each detection branch is trained only on examples from its corresponding object-scale range.For example, the smallest-scale samples train the det-8 detector.
4 Object Detection Network
The detection sub-network strengthens proposal-based detection through ROI-pooled object and context features, shared computation, and jointly optimized classification and box regression. Feature upsampling improves the resolution available for small objects without the extra costs of input upsampling.
- The proposal network's sliding windows do not cover objects well, so a detection network is added to increase accuracy.
- The detection sub-network applies ROI pooling, then fully connected and output layers for classification and bounding box regression.Its outputs are class probability and bounding box.
- Detection training extends the multi-task loss with classification and smoothed L1 bounding box regression losses, jointly optimizing shared parameters W and detection parameters Wd.
- A 32×32 object maps to only 4×4 on conv4-3 and 2×2 on conv5-3, limiting the information available for 7×7 ROI pooling.
- Feature upsampling with a deconvolution layer increases feature-map resolution instead of input resolution, avoiding extra memory and computation costs.The authors report significantly improved detection performance, especially for small objects.
- Object and context regions are stacked after ROI pooling, with the context region 1.5 times larger than the object region.
5 Experimental Evaluation
The experiments evaluate MS-CNN proposal generation and object detection on KITTI and Caltech, emphasizing scale robustness, proposal quality, accuracy, and speed. Results show strong recall and state-of-the-art detection performance with reduced reliance on input upsampling.
- Experimental Setup: The evaluation uses KITTI and Caltech because they contain many small objects, with KITTI covering cars, pedestrians, and cyclists.KITTI provides 7,481 training/validation images and 7,518 test images; experiments use standard moderate evaluation where applicable.
- Proposal Evaluation: Detection accuracy varies by object scale across layers, while combining the scale-specific detectors achieves high recall across object sizes.Each detection layer performs best for objects matching its receptive-field scale.
- Proposal Evaluation: 98% proposal recall is achieved with 100 proposals, compared with approximately 2,000 for 3DOP and 10,000 for EdgeBoxes.The comparison is reported on the KITTI validation set using proposal recall curves.
- Object Detection Evaluation: Input upsampling by 1.5∼2 times significantly improves detection, while larger factors provide little additional gain and increase computational and memory requirements.The paper notes that a factor of 3.5 was required by a prior method, whereas larger factors make detectors exponentially slower.
- Object Detection Evaluation: The MS-CNN sets a new KITTI record for pedestrian and cyclist detection, with gains of 6 and 7 points over 3DOP on the moderate tasks.On original KITTI images, detection reaches 10 fps; Caltech speed is approximately 8 fps on 960×720 images and 15 fps on original images.
6 Conclusions
The paper proposes MS-CNN for fast multi-scale object detection by detecting at intermediate layers whose receptive fields match different object scales. Feature approximation through deconvolution reduces memory and computation, and the detector reaches speeds up to 15 fps.
- Conclusion: MS-CNN detects objects at multiple intermediate network layers whose receptive fields match different object scales.A single input image is processed feedforward through the network to detect across scales.
- Conclusion: Feature approximation is explored as an alternative to input upsampling and yields significant memory and computation savings.The approach uses CNN feature maps rather than enlarging the input image.
- Conclusion: The MS-CNN detector achieves high detection rates at speeds of up to 15 fps.