Source-linked AI summary
Deep Extreme Cut: From Extreme Points to Object Segmentation
Kevis-Kokitsi Maninis, Sergi Caelles, Jordi Pont-Tuset, Luc Van Gool
TL;DR
Dense object segmentation depends on costly pixel-level annotations, while weaker or interactive alternatives have limitations. DEXTR uses four extreme points encoded as a heatmap alongside an RGB crop to guide CNN-based mask prediction. Across semantic, instance, video, and interactive settings, it reports state-of-the-art results and reduces annotation costs, with scope-dependent evaluation caveats.
Problem
Dense segmentation requires expensive pixel-level annotations, while weakly supervised methods lag state-of-the-art performance and semi-automatic methods require interaction.
Method
DEXTR concatenates a Gaussian heatmap centered on an object’s four extreme points with RGB input and predicts an object-membership probability map using a CNN.
Results
DEXTR reports state-of-the-art results across semantic, instance, video, and interactive segmentation, while extreme-click segmentations outperform bounding-box results and additional points further refine quality.
Takeaways & Limitations
DEXTR-generated masks can support semantic-segmentation training at ground-truth-comparable quality with less annotation budget and reduce labeling cost by a factor of 10.
Takeaways & Limitations
Reported performance varies with dataset overlap and training data, and the video analysis assumes DAVIS mask annotation takes the same time as COCO annotation.
Abstract
from arXiv · showhide
This paper explores the use of extreme points in an object (left-most, right-most, top, bottom pixels) as input to obtain precise object segmentation for images and videos. We do so by adding an extra channel to the image in the input of a convolutional neural network (CNN), which contains a Gaussian centered in each of the extreme points. The CNN learns to transform this information into a segmentation of an object that matches those extreme points. We demonstrate the usefulness of this approach for guided segmentation (grabcut-style), interactive segmentation, video object segmentation, and dense segmentation annotation. We show that we obtain the most precise results to date, also with less user input, in an extensive and varied selection of benchmarks and datasets. All our models and code are publicly available on http://www.vision.ee.ethz.ch/~cvlsegmentation/dextr/.
1. Introduction
DEXTR addresses costly dense segmentation supervision and cumbersome test-time interaction by using four extreme clicks to guide object masks. It reports state-of-the-art performance across segmentation settings and makes mask annotation substantially more efficient.
- Dense segmentation requires expensive pixel-level annotations, while weakly supervised methods remain behind the state of the art and semi-automatic methods require test-time interaction.
- DEXTR converts an object’s left-most, right-most, top, and bottom pixels into an object segmentation mask.
- DEXTR reports more accurate semi-automatic segmentations from extreme clicks than bounding boxes on PASCAL, COCO, and Grabcut.
- Additional non-extreme points further refine DEXTR’s segmentation quality on PASCAL.
- DEXTR-generated annotations yield training results as good as ground-truth annotations while requiring substantially less annotation effort.
- Experiments span COCO, PASCAL, Grabcut, DAVIS 2016, and DAVIS 2017, with code, models, and pre-computed results publicly available.
2. Related Work
Prior segmentation work uses increasingly weak supervisory signals, bounding boxes, object proposals, or interactive point labels to reduce annotation demands. DEXTR relates to these approaches by using extreme clicks to group pixels of an object instance.
- Weakly supervised segmentation has used image-level, noisy web, scribble-level, and point-level labels instead of dense masks.
- Instance segmentation methods produce masks through object proposals, bounding-box guidance, or deep architectures trained with strong supervision.
- DEXTR uses extreme clicks to group pixels belonging to the same object instance, with higher accuracy than the related grouping approaches described.
- Interactive point-based segmentation asks users to add labels progressively, extending earlier bounding-box methods such as Grabcut and click-based video refinement.
3. Method
DEXTR turns extreme-point annotations into guided, class-agnostic object masks using a CNN, and applies the same pipeline to interactive segmentation, annotation, and video segmentation. Experiments report accurate masks and lower annotation costs across these uses.
- 3.1. Extreme points: Bounding-box annotation is cognitively demanding, whereas extreme clicks provide an efficient alternative using the object’s top, bottom, left-most, and right-most points.
- 3.2. Segmentation from Extreme Points: A 2D Gaussian centered on each extreme point forms a heatmap that is concatenated with the RGB image as a four-channel CNN input.
- 3.2. Segmentation from Extreme Points: The extreme-point crop is relaxed by several pixels to retain contextual information around the object.
- 3.2. Segmentation from Extreme Points: DEXTR uses a ResNet-101 backbone with altered pooling and atrous convolutions to preserve resolution while maintaining receptive-field size.
- 3.2. Segmentation from Extreme Points: The CNN outputs a pixelwise object-membership probability map and is trained with frequency-aware cross-entropy loss under strong mask-level supervision.
- 3.2. Segmentation from Extreme Points: Object-centered crops contain more foreground than background, making balanced loss beneficial for training.
- 3.3. Use cases for DEXTR: DEXTR performs class-agnostic instance segmentation, and its reported performance drop on a different training dataset is very small.
4. Experimental Validation
DEXTR is extensively evaluated as a class-agnostic segmentation method using extreme-point guidance, with ablations, human-input validation, cross-dataset tests, interactive refinement, and applications to annotation and video segmentation. Across these settings, the method improves segmentation quality or reduces annotation effort, while its video advantage is smaller when masks contain multiple instances.
- Ablation Study: Adding extreme points as a fourth input channel increases performance by +3.1% in the tested architecture.The network uses extreme-point information to guide segmentation in addition to the cropped image and contextual margin.
- Ablation Study: Human-provided and simulated extreme points produce comparable DEXTR results in the evaluated datasets.The paper therefore uses simulated extreme points for most subsequent experiments.
- Class-agnostic Instance Segmentation: DEXTR outperforms the extreme-point GrabCut baseline by +6.5% and remains above SharpMask baselines by 10.8% and 2.1% on PASCAL.The SharpMask comparisons use the best-overlapping proposal and an oracle-selected upper bound, respectively.
- Class-agnostic Instance Segmentation: On Grabcut, DEXTR achieves a 2.3% error rate, 1.1% below the runner-up, corresponding to a 32% relative improvement.The evaluation measures the percentage of misclassified pixels within the provided bounding boxes.
- Generalization: DEXTR generalizes across datasets and unseen categories, with roughly unchanged COCO mini-val performance across tested categories and 81.75% mIoU on PASCAL-Context background classes.The reported cross-dataset and category results support class-agnostic behavior, while qualitative results also suggest generalization to background classes.
- Annotation: DEXTR-generated masks match ground-truth training performance at equal image counts and reach 70% IoU in 7 minutes versus 46% at the same ground-truth annotation budget.The same 70% accuracy requires 1 hour 10 minutes with ground-truth annotations in the reported comparison.
- Video Object Segmentation: In video segmentation, DEXTR matches one ground-truth mask using five times less annotation budget, but cannot match multiple ground-truth masks on DAVIS 2016.The authors attribute this limitation to sequences containing multiple semantic instances per mask, whereas the gap is smaller on DAVIS 2017.
- Interactive Segmentation: Adding a fifth boundary point with online hard example mining raises difficult-object performance to 73.2%, a +4.2% improvement.The difficult subset contains 338 objects, or 59.6% of the selected objects.
5. Conclusions
DEXTR is a CNN architecture that converts four extreme-point clicks, encoded as an extra heatmap channel, into accurate object masks across multiple segmentation scenarios. Experiments across five datasets report state-of-the-art results, and DEXTR can reduce mask-labeling costs by a factor of 10.
- DEXTR converts four extreme-point annotations into accurate object masks using an extra heatmap input channel.The four points are the left-most, right-most, top, and bottom object locations.
- DEXTR is evaluated for semantic, instance, video, and interactive segmentation across five datasets.
- DEXTR obtains state-of-the-art results in all evaluated scenarios.
- DEXTR reduces mask annotation costs by a factor of 10.