Source-linked AI summary
Learning to Segment Object Candidates
Pedro O. Pinheiro, Ronan Collobert, Piotr Dollar
TL;DR
Object proposal methods must achieve high recall efficiently and accurately, but prior segmentation approaches rely on low-level cues. The paper introduces a ConvNet that jointly predicts class-agnostic masks and object scores, then applies it densely at multiple scales. It reports higher recall with fewer proposals, generalization to unseen categories, and strong detection results.
Problem
Object proposal methods need high recall with few, accurately matching regions, while prior segmentation proposal approaches rely on low-level cues such as edges or superpixels.
Method
A ConvNet jointly predicts a class-agnostic segmentation mask and the likelihood that an image patch fully contains a centered object, using shared features for both tasks.
Results
The method surpasses prior proposal methods across box and segmentation settings, including higher average recall with fewer proposals and improved detector mAP with 100 proposals.
Takeaways & Limitations
Segmentation object proposals can be learned directly from raw image pixels without edges, superpixels, or other low-level segmentation.
Takeaways & Limitations
Performance is weaker at very high IoU, likely because the method outputs downsampled masks at each location and scale.
Abstract
from arXiv · showhide
Recent object detection systems rely on two critical steps: (1) a set of object proposals is predicted as efficiently as possible, and (2) this set of candidate proposals is then passed to an object classifier. Such approaches have been shown they can be fast, while achieving the state of the art in detection performance. In this paper, we propose a new way to generate object proposals, introducing an approach based on a discriminative convolutional network. Our model is trained jointly with two objectives: given an image patch, the first part of the system outputs a class-agnostic segmentation mask, while the second part of the system outputs the likelihood of the patch being centered on a full object. At test time, the model is efficiently applied on the whole test image and generates a set of segmentation masks, each of them being assigned with a corresponding object likelihood score. We show that our model yields significant improvements over state-of-the-art object proposal algorithms. In particular, compared to previous approaches, our model obtains substantially higher object recall using fewer proposals. We also show that our model is able to generalize to unseen categories it has not seen during training. Unlike all previous approaches for generating object masks, we do not rely on edges, superpixels, or any other form of low-level segmentation.
1 Introduction
The paper proposes a ConvNet-based object proposal method that jointly predicts class-agnostic segmentation masks and object-centeredness scores. It targets high recall with fewer, more accurate proposals and reports strong performance, including on unseen categories.
- Object proposal methods should maximize object recall while using few regions that accurately match objects.These properties are motivated by both efficiency and detection performance.
- The paper introduces a ConvNet object proposal algorithm designed to satisfy these proposal requirements better than existing approaches.ConvNets are presented as trainable feature hierarchies that perform strongly in large-scale recognition tasks.
- Given an image patch, the model jointly predicts a class-agnostic segmentation mask and a score for whether a centered object is fully contained.Most network layers are shared between the mask and score tasks, with only the final layers specialized.
- The method beats prior object proposal approaches by a large margin while using fewer proposals across multiple scenarios.It also generalizes to object categories not seen during training and avoids edges, superpixels, and other low-level segmentation.
2 Related Work
Prior object proposal methods commonly use low-level grouping, saliency, or bounding-box generation, whereas this paper learns ranked segmentation proposals directly with a deep network.
- Object detection systems commonly combine salient object proposal selection with a subsequent ConvNet classifier.This two-step approach followed earlier sliding-window detection frameworks.
- Existing proposal methods use objectness scoring, seed segmentation, or superpixel merging to generate candidate regions.These approaches rely on low-level grouping and saliency cues and may produce boxes or segmentation proposals.
- The proposed method generates ranked segmentation proposals with a data-driven discriminative deep-network architecture instead of edges or superpixels.Its high-level goal resembles prior methods, but the proposal-generation mechanism differs substantially.
- Unlike Multibox, which generates class-agnostic bounding-box proposals, this model generates more informative segmentation proposals.The paper also distinguishes its architecture and training scheme from Multibox and proposal reranking methods such as Deepbox.
3 DeepMask Proposals
DeepMask uses one convolutional network to jointly predict class-agnostic segmentation masks and object-centeredness scores from image patches. It applies this model densely across locations and scales to produce efficient full-image proposals.
- Model outputs: DeepMask predicts a class-agnostic segmentation mask and an object-likelihood score for each input image patch.The score estimates whether the patch fully contains a centered object at an appropriate scale.
- Network architecture: A shared convolutional network jointly learns mask and score predictions, with only the final layers specialized for each task.This shared design reduces model capacity and increases full-scene inference speed relative to using two distinct networks.
- Training data: Training examples combine an RGB patch, a binary mask, and a label indicating whether a centered, fully contained object lies within an allowed scale range.Negative patches receive label −1, including patches containing only partially visible objects; their masks are not used.
- Joint learning: The joint loss combines pixel-wise segmentation losses with an object-score loss, while segmentation error is backpropagated only for positive examples.The scoring branch uses balanced positive and negative sampling.
- Full-scene inference: At test time, DeepMask is applied densely at multiple image locations and scales, producing a mask and score for each tested patch.Convolutional computation makes full-image inference efficient, and training uses jittered canonical positives to improve robustness to position and scale variation.
4 Experimental Results
DeepMask is evaluated on PASCAL and COCO using recall, proposal count, object scale, localization, generalization, detection, and speed. It achieves higher recall with fewer proposals, generalizes to unseen categories, improves detector results, and runs efficiently with GPU inference.
- Proposal quality: DeepMask achieves substantially higher average recall than existing proposal methods across PASCAL and COCO while using fewer proposals.With 100 segmentation proposals, DeepMask reaches AR .245 on COCO, while competing methods require nearly 1000 proposals for similar AR.
- Object scale: DeepMaskZoom improves performance especially for small objects, but increases inference time.All models perform poorly on small objects; the additional smaller evaluation scale provides a performance boost.
- Localization: DeepMask has higher recall across nearly all IoU and proposal-count settings, but falls slightly below other models at very high IoU.The authors attribute this limitation to downsampled masks and suggest multiscale processing or skip connections as possible improvements.
- Generalization: The segmentation branch generalizes extremely well to unseen categories, while the discriminatively trained scoring branch causes the reduced-category model's AR drop.DeepMask20* matches the full DeepMask model when evaluated across all 80 COCO categories.
- Detection: With 100 DeepMask proposals, Fast R-CNN reaches 68.2% mAP and exceeds 66.9% mAP from 2000 SelectiveSearch proposals.With 500 DeepMask proposals, performance rises to 69.9% mAP before beginning to degrade.
- Speed: Inference averages 1.6 seconds per COCO image and 1.2 seconds per PASCAL image, with GPU use required for efficient inference.Parallelizing all scales in one batch can reduce inference time by approximately 30%.
5 Conclusion
The paper introduces a pixel-to-segmentation proposal framework that applies DeepMask densely at multiple scales to produce ranked masks. It surpasses prior methods for both box and segmentation proposals, while identifying closer integration with detectors as future work.
- DeepMask generates ranked segmentation proposals directly from image pixels by applying the model densely across an image at multiple scales.
- The approach surpasses prior state-of-the-art methods by a large margin for both bounding-box and segmentation proposal generation.
- Future work will couple the proposal method more closely with state-of-the-art detection approaches.