Source-linked AI summary
Deep Interactive Object Selection
Ning Xu, Brian Price, Scott Cohen, Jimei Yang, Thomas Huang
TL;DR
Interactive object selection traditionally requires substantial user interaction, and semantic-segmentation approaches have difficulty with instance-level selection and unseen objects. The paper encodes clicks as distance maps for FCNs, trains on synthetically sampled interactions, and adds graph-cut boundary refinement. Experiments report high-quality, generalized segmentations with little user effort, often only a few clicks.
Problem
Existing interactive methods require substantial interactions, while semantic-segmentation approaches struggle with instance-level selection and unseen objects.
Method
The method converts positive and negative clicks into Euclidean distance maps, concatenates them with RGB images, fine-tunes FCNs on randomly sampled pairs, and uses graph cuts for boundary refinement.
Results
The method achieves high-quality segmentations with a small amount of user effort, often just a few clicks, and is reported to generalize to new classes.
Takeaways & Limitations
Deep models can combine user interactions with object and semantic understanding to support interactive segmentation across seen and unseen objects.
Takeaways & Limitations
In areas where the FCN predictions are insufficiently accurate, graph-cut refinement is affected in producing the final result.
Abstract
from arXiv · showhide
Interactive object selection is a very important research problem and has many applications. Previous algorithms require substantial user interactions to estimate the foreground and background distributions. In this paper, we present a novel deep learning based algorithm which has a much better understanding of objectness and thus can reduce user interactions to just a few clicks. Our algorithm transforms user provided positive and negative clicks into two Euclidean distance maps which are then concatenated with the RGB channels of images to compose (image, user interactions) pairs. We generate many of such pairs by combining several random sampling strategies to model user click patterns and use them to fine tune deep Fully Convolutional Networks (FCNs). Finally the output probability maps of our FCN 8s model is integrated with graph cut optimization to refine the boundary segments. Our model is trained on the PASCAL segmentation dataset and evaluated on other datasets with different object classes. Experimental results on both seen and unseen objects clearly demonstrate that our algorithm has a good generalization ability and is superior to all existing interactive object selection approaches.
1. Introduction
Interactive object selection aims to let users accurately select objects for applications such as localized editing and image/video composition, while reducing the substantial interactions required by earlier methods. The paper proposes a deep-learning system that encodes clicks for FCNs, synthesizes training pairs, and combines FCN outputs with graph cuts for boundary refinement.
- Interactive object selection uses user inputs such as strokes and bounding boxes to select objects accurately for localized editing and image/video composition.
- Earlier interactive methods require substantial user interactions to estimate foreground and background distributions, whereas this approach targets a few clicks.The paper reports that one or two clicks usually give reasonably good results.
- The method transforms positive and negative clicks into separate Euclidean distance maps and concatenates them with RGB channels to form training inputs.
- Graph cut optimization is integrated with FCN output probability maps to refine segmentation boundaries.
- Random sampling strategies model users’ click behaviors and provide inexpensive training pairs for fine-tuning deep fully convolutional networks.
- The system is reported to operate in real time when given a high-end GPU.
2. Related works
Prior interactive segmentation methods rely on low-level features and include contour-, bounding-box-, stroke-, graph-cut-, geodesic-, and random-walk-based approaches. Deep semantic-segmentation methods improve related tasks, but direct application to interactive segmentation faces user-response, instance-level, and unseen-object limitations.
- Previous interactive approaches include contour-based, bounding-box, stroke-based, normalized-cut, graph-cut, geodesic, and random-walk methods.
- These methods estimate foreground and background distributions from low-level features, which are insufficient for distinguishing foreground and background.
- FCN boundary-refinement methods improve semantic-segmentation IU accuracy from 62.2% to 71.6% and 72%, but their results are less acceptable for interactive segmentation.
3. The proposed algorithm
The proposed algorithm combines click-derived distance maps with RGB images to fine-tune FCNs, then uses graph cut optimization to refine segmentation boundaries. Randomly sampled interaction pairs model diverse click patterns, while incremental FCN-8s and graph-cut updates support interactive use.
- 3.1. Transforming user interactions: Positive and negative clicks are transformed into separate Euclidean distance channels and concatenated with the image’s RGB channels.The resulting pair is supervised with the selected object’s ground-truth mask.
- 3.1. Transforming user interactions: The distance channels encode each pixel’s minimum Euclidean distance to the corresponding positive or negative click set.Distances are truncated to 255, and an empty negative-click set assigns 255 to every negative-channel pixel.
- 3.2. Simulating user interactions: Random sampling generates training pairs by selecting separated, boundary-aware positive clicks and negative clicks from three complementary strategies.Negative clicks are sampled near the object, on other objects, or sequentially to cover outside boundaries.
- 3.2. Simulating user interactions: Combining all three negative-click strategies performs better than relying on any single strategy, so each object’s sampled pairs use the strategies with equal probability.The strategies are intended to represent the complexity of users’ click patterns.
- 3.3. Fine tuning FCN models: FCN-32s, FCN-16s, and FCN-8s are fine-tuned on image–interaction pairs, with FCN-16s improving boundary regions over FCN-32s while FCN-16s and FCN-8s have similar accuracy.Training each model takes about 20 epochs; fine-tuning requires approximately three days for FCN-32s and five days for FCN-16s and FCN-8s.
- 3.4. Graph cut optimization: The FCN-8s probability map is integrated into graph-cut energy minimization because direct thresholding at 0.5 produces coarse masks.Graph cut refines the final segmentation using regional and boundary terms.
- 3.4. Graph cut optimization: After each new click, the algorithm recomputes both distance maps and obtains a new FCN-8s probability map, while graph cut updates the segmentation without recomputing everything from scratch.On high-end GPUs, probability-map computation takes less than 100 milliseconds, and graph cut is efficient on modern CPUs.
4. Experiments
Across multiple datasets, the algorithm achieves accurate interactive segmentation with fewer clicks, generalizes to unseen objects, supports object parts, and improves boundaries through graph-cut refinement.
- Settings: The algorithm is evaluated on Pascal, Grabcut, Berkeley, and MS COCO using sequential user interactions and IU accuracy.MS COCO is evaluated separately on seen and unseen categories.
- Results: On Pascal, the algorithm outperforms all other methods on the validation set containing categories seen during training.The evaluation uses 20 Pascal object categories.
- Results: On Grabcut and MS COCO, the algorithm performs strongly with few clicks and remains superior across both seen and unseen MS COCO categories.Its advantage is smaller on Grabcut because many images have easily separated foreground and background distributions.
- Results: The algorithm requires the least number of clicks to reach a given IU accuracy and achieves more accurate results with less interaction than competing methods.Table 1 reports the mean clicks required at specified IU accuracies across datasets.
- Results: Given identical interaction sequences, the method can segment much of an object after one click, whereas other approaches may recover only part of it or fail.The reported explanation attributes this to the FCN models’ high-level understanding of objectness and semantics.
- Limitations: Thin structures and fine details remain a failure case because FCN-8s probabilities are insufficiently accurate there, limiting graph-cut refinement.The limitation is illustrated by a failed segmentation result.
- Comparisons to semantic segmentation approaches: FCNs and CRF-RNN segment known people but cannot isolate an overlapping person instance or recognize an unseen banana category under the comparison strategy.The comparison uses connected components containing the user click as foreground.
- Results: The method generalizes to clothing parts despite training only on whole objects, and graph cut improves boundary localization over thresholded FCN-8s outputs.Fashionista examples include shirts, skirts, and jackets; graph-cut refinement produces more satisfactory boundaries.
5. Conclusion
The proposed deep interactive segmentation algorithm combines user interactions with deep learning and graph cut optimization to produce high-quality segmentations with few clicks.
- The algorithm combines user interactions with deep learning models for interactive segmentation.It transforms user clicks into Euclidean distance maps and trains FCN models on synthesized interaction-image pairs.
- Graph cut optimization is combined with FCN-8s outputs to refine segmentation boundaries.
- High-quality segmentations can often be achieved with just a few clicks.