Source-linked AI summary
segDeepM: Exploiting Segmentation and Context in Deep Neural Networks for Object Detection
Yukun Zhu, Raquel Urtasun, Ruslan Salakhutdinov, Sanja Fidler
TL;DR
Object detection remained limited by candidate hypotheses and the modest gains from stronger segmentation proposals. segDeepM formulates detection as MRF inference that combines appearance, context, and selected segmentation proposals, achieving a 4.1% improvement over R-CNN on PASCAL VOC 2010 test and placing at the top of the leaderboard.
Problem
Object detection depends on candidate-box quality, while stronger segmentation proposals had produced only small accuracy improvements.
Method
segDeepM uses an MRF in which detection hypotheses score appearance and context and select class-specific object segments, with iterative box repositioning.
Results
4.1% improvement over R-CNN on PASCAL VOC 2010 test is reported, placing segDeepM at the top of the current PASCAL leaderboard.
Takeaways & Limitations
The approach significantly improves detection by incorporating segmentation and contextual evidence into detection hypotheses.
Takeaways & Limitations
Evaluation is affected by missing PASCAL ground-truth annotations, including small objects, ambiguous object definitions, and labeling mistakes.
Abstract
from arXiv · showhide
In this paper, we propose an approach that exploits object segmentation in order to improve the accuracy of object detection. We frame the problem as inference in a Markov Random Field, in which each detection hypothesis scores object appearance as well as contextual information using Convolutional Neural Networks, and allows the hypothesis to choose and score a segment out of a large pool of accurate object segmentation proposals. This enables the detector to incorporate additional evidence when it is available and thus results in more accurate detections. Our experiments show an improvement of 4.1% in mAP over the R-CNN baseline on PASCAL VOC 2010, and 3.4% over the current state-of-the-art, demonstrating the power of our approach.
1. Introduction
Object detection depends heavily on candidate-box quality, yet better segmentation proposals have yielded only small accuracy gains. segDeepM combines appearance, context, and selected segmentation proposals in an MRF, while iterative box adjustment reduces dependence on initial candidates.
- CNN-based detection typically generates many candidate boxes and classifies them, making object-hypothesis quality central to detector accuracy.
- Better bottom-up segmentation proposals have produced only small improvements in object-detection accuracy.
- segDeepM models detection as MRF inference, scoring appearance and contextual information while allowing each hypothesis to select and score an accurate object segment.
- The model places more accurate bounding boxes where object-segmentation hypotheses or strong contextual cues are available.
- A sequential strategy alternates box adjustment and content classification, reducing dependence on initial candidate boxes.
- 4.1% improvement over R-CNN and 1.4% over the current state-of-the-art are reported on PASCAL VOC 2010 test.
2. Related Work
Prior work used segmentation to refine detection through voting, localization refinement, background masking, and joint formulations. segDeepM transfers segmentation and contextual-cue ideas into the R-CNN framework using an MRF designed to exploit segmentation efficiently.
- Earlier approaches used segmentation for object-location voting, DPM-based localization refinement, background masking, or joint segmentation-detection formulations.
- segDPM improved detection by augmenting DPM with simple segmentation features and placing accurate boxes where class-specific segmentation was available.
- A later approach augmented DPM with deformable contextual parts that scored contextual information.
- segDeepM builds on R-CNN by combining appearance features, rich contextual appearance around objects, and an MRF that exploits segmentation more efficiently than segDPM.
3. Our Approach
segDeepM combines appearance, segmentation, and contextual evidence in a Markov Random Field, while iteratively refining candidate boxes and rescoring them. Its design uses class-specific segment selection, learned segmentation potentials, expanded-box CNN features, and latent-variable learning.
- Segmentation: Class-specific segment selection uses a large proposal set and features that capture spatial arrangement, overlap, background, and predicted segment class.The segmentation term decomposes across classes, allowing different segments to be selected according to detector class and context.
- Model: segDeepM frames detection as a Markov Random Field over candidate boxes, class-specific segments, and contextual information.Each class variable selects a segment whose agreement and contextual relationships influence the candidate box score.
- Model: The model combines CNN appearance features with learned segmentation and contextual potential functions.Appearance uses 4096-dimensional fc7 features, while segmentation potentials measure box–segment agreement and are jointly weighted.
- Context: Context is modeled by enlarging candidate boxes, retaining their object labels, and fine-tuning CNNs on the expanded regions.The expanded regions provide scene information beyond the original detection box, including contextual objects and surroundings.
- Inference and learning: Inference efficiently selects the best segment independently for each class, while learning treats segment assignments as latent variables in a latent SVM.Greedy optimization is guaranteed to find the global maximum because the segmentation features have no pairwise relationships among segment variables.
- Iterative refinement: Iterative bounding-box prediction alternates box correction, feature re-extraction, and model scoring, reprocessing boxes that change by more than 20%.The procedure converges to stable boxes after two iterations.
4. Experimental Evaluation
The evaluation studies segDeepM on PASCAL VOC through ablations, error analysis, and comparisons with R-CNN and state-of-the-art detectors. Results examine segmentation proposals, contextual features, iterative box prediction, missing annotations, and held-out test performance.
- Experimental setup: The experiments evaluate segDeepM on PASCAL VOC using ablations of segmentation, contextual, and iterative bounding-box potentials.The study also compares the method with R-CNN, related baselines, and state-of-the-art systems.
- Ablation analysis: A 1% improvement over R-CNN comes from adding CPMC segments, while segDPM+CNN improves by 2.1% and the proposed segmentation features add 1.1%.The proposed segmentation features require only 223 additional parameters.
- Context analysis: A 1.5% detection gain occurs with 10% box expansion in each direction, peaking at expansion ratio ρ = 0.5 before declining.Using the full image as context yields less than a 1% gain.
- Bounding-box prediction: Iterative box prediction improves over original R-CNN by 1.4% after one iteration, while a second iteration adds only about 0.1% and performance saturates after two iterations.The first and second iterations affect 52% and 42% of boxes, respectively.
- Error analysis: Missing annotations produce a non-negligible number of apparent false positives, mainly involving small objects, ambiguous object definitions, and labeling mistakes.The paper notes that this issue is increasingly relevant as detector performance improves.
- Test-set comparison: On PASCAL VOC 2010 test, segDeepM improves over R-CNN by 4.1% and over the best reported 7-layer method by 1.4%, while a 16-layer version achieves 67.2% mean AP.The 16-layer version outperforms other methods in 20 out of 20 object classes.
5. Conclusion
segDeepM combines appearance and context scoring with segment selection in an MRF, and uses sequential localization to improve detection over R-CNN. It achieves 4.1% and 4.3% improvements on PASCAL VOC 2010 in 7-layer and 16-layer settings, respectively.
- segDeepM models detection with an MRF that scores appearance and context while letting each candidate box select and score a segment.
- The method adds sequential localization by iterating between model scoring and repositioning candidate boxes.
- 4.1% improvement over R-CNN is achieved on PASCAL VOC 2010 test in the 7-layer setting.
- 4.3% improvement over R-CNN is achieved on PASCAL VOC 2010 test in the 16-layer setting.