Source-linked AI summary
Simultaneous Detection and Segmentation
Bharath Hariharan, Pablo Arbeláez, Ross Girshick, Jitendra Malik
TL;DR
The paper defines Simultaneous Detection and Segmentation as detecting each object instance while marking its pixels, combining requirements beyond bounding boxes or semantic labels. It evaluates a pipeline producing scored segmentation hypotheses and reports gains on SDS, semantic segmentation, and object detection, alongside diagnostic tools.
Problem
Simultaneous Detection and Segmentation requires detecting individual instances and marking their pixels, unlike bounding-box detection or semantic segmentation alone.
Method
The pipeline evaluates scored object hypotheses with predicted segmentations using an average-precision metric that requires separately and completely segmenting each instance.
Results
The pipeline improves performance on SDS, semantic segmentation, and object detection, including a 5-point semantic-segmentation advance and higher mean APb than R-CNN.
Takeaways & Limitations
The paper supports a unified evaluation of detection and instance segmentation while providing diagnostic tools for analyzing SDS error modes.
Abstract
from arXiv · showhide
We aim to detect all instances of a category in an image and, for each instance, mark the pixels that belong to it. We call this task Simultaneous Detection and Segmentation (SDS). Unlike classical bounding box detection, SDS requires a segmentation and not just a box. Unlike classical semantic segmentation, we require individual object instances. We build on recent work that uses convolutional neural networks to classify category-independent region proposals (R-CNN [16]), introducing a novel architecture tailored for SDS. We then use category-specific, top- down figure-ground predictions to refine our bottom-up proposals. We show a 7 point boost (16% relative) over our baselines on SDS, a 5 point boost (10% relative) over state-of-the-art on semantic segmentation, and state-of-the-art performance in object detection. Finally, we provide diagnostic tools that unpack performance and provide directions for future work.
1 Introduction
Simultaneous Detection and Segmentation (SDS) unifies instance detection with pixel-level delineation, requiring each object instance to be detected and segmented. The proposed pipeline classifies region proposals using box and foreground features, refines them with category-specific predictions, and improves SDS, semantic segmentation, and detection results.
- Task motivation: SDS detects every category instance and marks the pixels belonging to each instance, producing richer output than boxes or semantic labels alone.Semantic segmentation does not identify how many objects are present or the precise extent of each instance.
- Pipeline: The pipeline starts with category-independent region proposals and classifies each using convolutional features from both the region bounding box and foreground.The approach uses region proposals because SDS requires segmentations rather than only bounding boxes.
- Evaluation: SDS evaluation treats a hypothesis as correct when its predicted segmentation overlaps a ground-truth instance by more than 50%, then computes average precision from the precision-recall curve.The resulting metric is APr, distinguished from traditional bounding-box APb.
- Results: 49.5% APr is achieved while APb improves from 51.0% for R-CNN to 53.0%.APr measures segmentation accuracy while requiring instances to be detected separately and completely.
- Results: 41.4% APr is reported, while APb improves from 41.9% for R-CNN to 44.2% and semantic segmentation rises from 47.9% to 52.6%.The paper also provides diagnostic tools for analyzing SDS error modes and releases the algorithm, benchmark, and tools publicly.
2 Related work
Prior work largely used segmentation to improve detection localization or used detections to guide segmentation, while CNNs were becoming central to both tasks. The paper builds on this context to address detection and segmentation jointly.
- Detection-guided segmentation: Several methods use object detections as cues for semantic segmentation through figure-ground masks, region scoring, or higher-order CRF potentials.These approaches incorporate detections to guide category labeling and region selection.
- Segmentation-guided detection: Other methods use segmentation or appearance models to improve detection localization, but color and texture assumptions are weak for categories such as people.The cited approaches include GrabCut-style refinement and instance or category appearance models.
- Joint cues: Some approaches treat segmentation as a stronger detection cue, including semantic-segmentation scoring and detectors based on non-rectangular patches.This extends beyond using segmentation only after detection.
- CNN context: CNNs had recently become widely used for detection and semantic segmentation after earlier successes in image classification.The paper places its CNN-based approach within this broader shift in computer vision.
3 Our approach
The approach adapts region-proposal detection to SDS by combining box and foreground CNN pathways, jointly training them, and refining bottom-up regions with category-specific figure-ground predictions. This addresses candidate regions that undershoot or overshoot object support.
- Proposal generation: MCG supplies category-independent segment proposals, chosen because SDS evaluates segments rather than only boxes.MCG generates candidates through multiscale segmentation hierarchies and improves APb slightly over Selective Search in the reported experiments.
- Feature extraction: The baseline feature extractor concatenates CNN features from a cropped bounding box and the same box with the region background masked out.This adds foreground information that bounding-box features alone lack for judging segmentation overlap.
- Feature extraction: The architecture uses disjoint box and region pathways whose features are concatenated at the final classifier layer.The box pathway sees cropped boxes, while the region pathway sees cropped boxes with background masked.
- Feature extraction: Jointly training the two pathways as one network yields feature extractor C, initialized from separately fine-tuned box and region networks.At test time, the final classification layer is discarded and the concatenated penultimate-layer output is used.
- Classification: A linear SVM scores regions after iterative positive-set re-estimation, followed by strict non-maximum suppression and retention of the top 20,000 detections per category.The positive set is updated using the highest-scoring candidate overlapping each ground truth by more than 50%.
- Region refinement: Region refinement combines a coarse 10 × 10 top-down figure-ground mask with bottom-up candidate information projected onto superpixels.The refinement targets bottom-up undershooting and overshooting, while superpixels help preserve contours and thin structures.
4 Experiments and results
Experiments show that the proposed SDS pipeline improves segmentation, detection, and semantic-segmentation performance over the evaluated baselines, while diagnostics identify localization as a major error source. Gains persist across overlap thresholds and are statistically significant.
- Results on APr and APr: C+ref improves APr by 2 points and APrvol by 1.2 percentage points over C through category-specific mask refinement.The authors attribute the gain to refining high-quality MCG regions with category-specific information.
- Results on APr and APr: Each proposed improvement increases mean APr across all overlap thresholds, indicating no overfitting to a particular evaluation regime.The threshold analysis varies the criterion for counting a detection as correct.
- Results on APr and APr: 50.3% mean APr for C+ref exceeds the 31.3% upper bound for SegDPM on PASCAL VOC2010 val.The comparison is restricted to VOC2010 val because SegDPM detections are unavailable elsewhere.
- Producing diagnostic information: Mislocalization is the predominant false-positive source for person, while cat retains false positives from misclassification after localization improvements.Removing mislocalizations raises person precision to essentially 100%, whereas cat benefits less from better localization.
- Results on APb and APb: The pipeline improves both APb and APbvol over R-CNN, with the larger improvement on APbvol at stricter overlap thresholds.Performance is worse than R-CNN at low thresholds but substantially better at higher thresholds, partly reflecting improved localization.
- Results on APb and pixel IU: 50.7 mean APb on PASCAL VOC 2012 test improves over R-CNN's 49.6 and SegDPM's 40.7, while semantic segmentation advances the state of the art by about 5 points.The APb comparison uses systems without bounding box regression; the semantic-segmentation result uses C+ref with a pasting scheme.