Source-linked AI summary
A MultiPath Network for Object Detection
Sergey Zagoruyko, Adam Lerer, Tsung-Yi Lin, Pedro O. Pinheiro, Sam Gross, Soumith Chintala, Piotr Dollár
TL;DR
COCO challenges object detection with broad scale variation, non-iconic cluttered images, occlusion, and demanding localization metrics. The paper modifies Fast R-CNN with MultiPath features, foveal context, and integral localization loss, coupling it with DeepMask proposals. The combined system improves over the Fast R-CNN with Selective Search baseline by 66% overall and nearly 4× on small objects, placing second in both 2015 COCO tracks.
Problem
COCO introduces broad object scales, frequent clutter and occlusion, and evaluation that rewards more precise localization.
Method
The paper modifies Fast R-CNN with skip connections, foveal regions, and an integral loss, then couples the classifier with DeepMask proposals and mask prediction.
Results
66% improvement over Fast R-CNN with Selective Search was achieved overall, with nearly 4× higher AP on small objects; the system placed second in both COCO 2015 tracks.
Takeaways & Limitations
MultiPath provides a COCO baseline system combining multi-scale features, contextual views, precise localization, and DeepMask proposals.
Takeaways & Limitations
Small-object detection remains challenging despite the reported improvements.
Abstract
from arXiv · showhide
The recent COCO object detection dataset presents several new challenges for object detection. In particular, it contains objects at a broad range of scales, less prototypical images, and requires more precise localization. To address these challenges, we test three modifications to the standard Fast R-CNN object detector: (1) skip connections that give the detector access to features at multiple network layers, (2) a foveal structure to exploit object context at multiple object resolutions, and (3) an integral loss function and corresponding network adjustment that improve localization. The result of these modifications is that information can flow along multiple paths in our network, including through features from multiple network layers and from multiple object views. We refer to our modified classifier as a "MultiPath" network. We couple our MultiPath network with DeepMask object proposals, which are well suited for localization and small objects, and adapt our pipeline to predict segmentation masks in addition to bounding boxes. The combined system improves results over the baseline Fast R-CNN detector with Selective Search by 66% overall and by 4x on small objects. It placed second in both the COCO 2015 detection and segmentation challenges.
1 Introduction
COCO introduces challenges involving scale, clutter, occlusion, context, and precise localization. The paper adapts Fast R-CNN with MultiPath modifications and DeepMask proposals, achieving strong detection and segmentation results.
- Motivation: COCO contains 300,000 images with segmented instances in 80 categories and introduces broad scale variation, clutter, occlusion, and localization demands.The dataset averages 7 object instances per image and includes a high percentage of small objects.
- Motivation: The paper adapts Fast R-CNN through experiments targeting small-object detection, contextual recognition, and improved localization.COCO drives the evaluation of these modifications.
- Approach: MultiPath combines skip connections, foveal regions, and an integral loss to use multi-layer features, multiple contexts, and higher-precision localization.These modifications create multiple information paths through the classifier.
- Results: 33.5 AP was achieved for COCO detection with an ensemble of 6 models, versus 19.3 AP for Fast R-CNN with Selective Search, a 66% improvement.The combined system uses DeepMask proposals.
- Results: The system placed second in both the 2015 COCO bounding-box and segmentation tracks.Only the deeper ResNet classifier outperformed it.
2 Related Work
Related work establishes region proposals and CNN classifiers as the dominant detection paradigm while motivating contextual, skip-connected, and CNN-based proposal approaches.
- Detection paradigm: Modern detectors commonly generate candidate regions with proposal algorithms and classify each region with a CNN.This region-proposal paradigm follows the work on Region CNNs.
- Context: Context-based methods use surrounding regions, whole-image features, or differently sized pooling regions to improve region classification.These approaches provide precedents for contextual processing around detected objects.
- Skip connections: Skip architectures aggregate features from multiple convolutional layers and had shown improved results in pedestrian detection and semantic segmentation.Concurrent work also revisited skip connections for general object detection.
- Object proposals: CNN-based object proposals produced substantial gains over earlier proposals based on grouping cues, edges, and superpixels.This work uses DeepMask segmentation proposals, including top-down refinement in an early VGG-A-based variant.
- Classifiers: The paper uses VGG-A for DeepMask and VGG-D for MultiPath while noting concurrent progress from deeper Residual Networks.Classifier architecture is presented as an important component of detector accuracy.
Methods
The MultiPath detector extends Fast R-CNN with multi-context heads, multi-resolution skip connections, and an integral localization loss. Its loss aggregates predictions across overlap thresholds to favor tighter localization.
- Base architecture: The baseline applies a pretrained VGG-D network and RoI-pooling to proposals before classification and bounding-box regression.MultiPath retains this Fast R-CNN framework.
- Foveal structure: Four foveal heads observe context regions at 1×, 1.5×, 2×, and 4× the proposal-box size.Their outputs are concatenated for classification and bounding-box regression.
- Skip connections: Small objects lose spatial information after 16× feature downsampling, motivating higher-resolution features from earlier layers.COCO objects below 32×32 and 16×16 pixels may reduce to 2×2 or 1×1 features at conv5.
- Skip connections: Skip connections concatenate normalized conv3, conv4, and conv5 RoI-pooled features, giving classifiers access to multiple resolutions.Connections are sparsified so earlier layers feed selected foveal heads.
- Integral loss: The integral loss addresses the limitation of a single IoU threshold by training performance across multiple thresholds.COCO averages AP between IoU 50 and 95, unlike metrics based only on AP50.
- Integral loss: The model approximates the integral with du = 5 and outputs multiple predictions p_u whose softmax probabilities are averaged at inference.It uses n = 6 thresholds u ∈ {50,55,...,75}; higher-overlap proposals are encouraged to receive higher scores.
4 Experiments
Experiments evaluate MultiPath against Fast R-CNN on COCO, analyzing its network modifications, proposal choices, localization, and efficiency. The modifications improve AP, while DeepMask proposals provide stronger accuracy with fewer proposals.
- Training and Testing Setup: Experiments use Fast R-CNN as the baseline, DeepMask proposals, and COCO images to analyze the MultiPath detector.The setup reports AP and AP50 on COCO validation images and examines the three core modifications.
- MultiPath Network Analysis: The MultiPath network combines skip connections, foveal regions, and integral loss, with each modification contributing roughly 1 AP point.Together, the modifications increase AP by 2.7 points to 27.9 and AP50 by 1.4 points to 44.8.
- MultiPath Network Analysis: Integral loss improves overall AP by approximately 1 over the u = 50 model while maintaining a slightly higher AP50 than the u = 60 model.Using 6 heads achieves the highest AP, and the integral-loss experiments use 280K training iterations.
- DeepMask Proposals: DeepMask proposals raise Fast R-CNN accuracy from 19.3 AP with Selective Search to 25.2 AP, while MultiPath with DeepMask reaches 27.9 AP.Using MultiPath with the same Selective Search proposals reaches 22.8 AP, isolating gains from the classifier and proposal method.
- DeepMask Proposals: 400 DeepMask proposals per image saturate accuracy, while approximately 50 DeepMask proposals match the accuracy of 2000 Selective Search proposals.DeepMask proposals produce higher accuracy with fewer proposals.
- DeepMask Proposals: Box regression increases AP by only 1.1 points with MultiPath and DeepMask, compared with 3.5 points for Fast R-CNN with Selective Search.The table caption attributes the smaller gain to already well-localized detections from the MultiPath and DeepMask pipeline.
5 COCO 2015 Results
The system was evaluated in the COCO 2015 detection and segmentation challenges after adding inference, training-data, and ensemble enhancements. It placed second in both tracks, while small-object detection remained challenging.
- COCO 2015 Results: Additional training data, horizontal flipping, fractional max pooling, and model ensembling raise validation AP from 27.9 to 33.2.These enhancements were applied before submitting results to the COCO leaderboard.
- COCO 2015 Results: Segmentation results are obtained by feeding the bounding-box regression outputs back into the DeepMask segmentation system.The pipeline therefore extends its detection outputs to produce segmentation results.
- COCO 2015 Results: The system placed second in both the COCO 2015 bounding-box and segmentation tracks.Only the deeper ResNet classifier outperformed the approach.
- COCO 2015 Results: Compared with baseline Fast R-CNN, the system improved AP on small objects by 4× and AP75 by 82%.The reported gains were largest for small objects and localization.
- COCO 2015 Results: Most remaining overall error comes from false positives and false negatives, with little inter-class classification error.Despite improvements, small-object detection remains quite challenging.
6 Conclusion
The paper concludes that combining three Fast R-CNN modifications with DeepMask proposals substantially improves COCO detection over the baseline. Qualitative and error analyses still show missed objects, false positives, and challenging small-object detection.
- 66% improvement over baseline Fast R-CNN with Selective Search summarizes the reported performance gain of the MultiPath classifier with DeepMask proposals.The conclusion reports this overall improvement but does not restate the absolute AP values.
- Selected COCO examples contain many good detections and segmentations despite missed objects and false positives.
- Localization errors are the dominant analyzed source of potential AP improvement, while similar- and other-class confusion offer only slight gains.The analysis states that removing localization errors would raise AP10 to 58.8.
- Detection performance varies with image scale and NMS threshold, as examined in the accompanying analysis.
Appendix: Additional Analysis
The appendix evaluates training, inference, scale, suppression, and localization choices, reporting several incremental gains and some unsuccessful enhancements. It also identifies settings that improve AP and remaining trade-offs or limitations.
- The appendix reports additional enhancements, parameter analyses, and negative results intended to benefit the community.
- Adding validation data to training improved accuracy by 2.3 points AP.The held-out 5K validation images were excluded, and the proposals were not retrained on train+val.
- Horizontal-flip averaging improved AP by 0.6 points, while fractional max pooling improved AP by 0.7 points.Both modifications average outputs from multiple image or pooling views while computing the network trunk once for FMP.
- An ensemble of 6 similar models boosted AP by 1.7 points to 33.2.
- Increasing image scale improved accuracy up to approximately 800px, while increasing computation time.The chosen 800px scale improved AP by 0.5 points over the 600px scale used by Fast R-CNN on PASCAL.
- An NMS threshold of 50 performed best for the model, improving AP by 0.4 points.The authors suggest higher object density in COCO may explain the difference from Fast R-CNN’s threshold of 30.
- Setting weight decay to 0 improved results by 1.1 AP50 and 0.5 AP.The authors retain dropout at 0.5 and suggest one regularization form may be sufficient.
- ImageNet detection-data augmentation mostly failed to improve accuracy, and iterative localization did not substantially improve AP.ImageNet augmentation helped the hair dryer class but left other classes unchanged or worse; iterative localization was attributed to already strong proposals and localization.