Deep GrabCut for Object Selection

1 Introduction

• Hard-box assumptions fail when rectangles sit inside objects or leave too little background, while box-confined instance methods can lose object pixels and useful context.The paper identifies both undersized or oversized interactive inputs and detection boxes that omit parts of an object or exclude contextual image information.Source: §1 · pp. 1–2
• A Euclidean distance map makes the rectangle a soft cue while preserving the full image for a CEDN trained on sloppy rectangles.The model concatenates the distance map with the RGB image and predicts an object mask.Source: §1 · p. 2
• The model also supports hand-drawn curves without retraining and uses a dense CRF to turn overlapping detections into instance-level semantic labels.The CRF resolves overlap among segments from detections that may cover the same object.Source: §1 · p. 2

2 Related works

• Interactive rectangle methods iteratively estimate foreground and background color models, but rely on color and edge cues rather than higher-order object structure and shape.Later variants jointly optimize segmentation and color models or add a spatial extent prior.Source: §2 · p. 2
• Figure 2 marks the rectangle in both the image and distance map, with ⊕ denoting their concatenation.The caption identifies the framework diagram's input pairing.Source: Figure 2 · p. 3
• Neural detection-based methods segment within bounding boxes and are placement-sensitive; the paper contrasts them with loose-rectangle segmentation and CRF labeling without overlapping object masks.The related-work passage also distinguishes its approach by not requiring preliminary segmentation or multiple processing iterations.Source: §2 · p. 3

3 Segmentation from rectangles

• The image and rectangle distance map are concatenated as input to a CEDN model that predicts the object mask.The approach is intended for both interactive segmentation and instance segmentation.Source: §3 · p. 3

3.1 Rectangle sampling

• Randomly jittering each instance’s ground-truth box generates varied training rectangles, augmenting a small dataset while keeping the model free from overfitting.The variation level is controlled by v; the four coordinate perturbations are standard Gaussian variables.Source: §3.1 · pp. 3–4

3.2 Rectangle transformation

• The signed distance map assigns values by pixel location relative to the rectangle edge and is concatenated with the RGB image.The authors use signed rather than unsigned distances to better capture rectangle context, then truncate map values to 0–255 for storage.Source: §3.2 · p. 4

3.3 CEDN model training

• A CEDN encoder-decoder predicts binary instance masks, using a VGG-16-initialized encoder and a more concise decoder to reduce parameters and speed training.Dropout after decoder convolutional layers and random resampling each epoch address overfitting.Source: §3.3 · p. 4

• The method converts detection boxes into instance-level semantic segmentation and resolves overlap between segments to produce a pixel labeling.Detections are first filtered with class-agnostic non-max suppression, then each remaining box is segmented independently.Source: §4 · p. 5
• Each detection’s largest categorical confidence score weights its foreground map, while background maps are aggregated across detections and normalized with foreground probabilities.The combined maps provide the probability inputs for pixel labeling.Source: §4 · p. 5
• A fully connected CRF assigns pixel labels using negative log-probability unary potentials and pairwise potentials that penalize disagreement between nearby, similarly colored pixels.The solution yields both pixel-level instance labeling and semantic labeling.Source: §4 · p. 5

5.1 Implementation

• The experiments used PASCAL- and COCO-trained models and a matched-structure deconvolutional baseline that takes only RGB image patches inside rectangles.Training used four randomly sampled boxes per instance per iteration, with inputs resized to 320×320.Source: §5.1 · p. 5

5.2 Rectangle Segmentation

• On GrabCut, the PASCAL-trained model generalized to unseen objects, while the COCO-trained model achieved better results.The evaluation used tight boxes for objects in 50 test images and measured misclassified pixels within the boxes.Source: §5.2 · p. 6, Table 1 · p. 6
• Across rectangle scales, our method stayed consistently effective while prior methods and the deconvolutional baseline deteriorated outside their required box-boundary conditions.Under random detection jitter, it also degraded less as the jitter parameter v increased than two other methods.Source: §5.2 · pp. 6, 8, Figure 4 · p. 6, Figure 5 · p. 6
• Given the same detections, our method had the best results under all evaluation metrics; in the overall comparison, it led under two metrics more related to segmentation.The comparisons are reported on the PASCAL validation dataset.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Instance-level semantic segmentation

• For instance-level labeling, our approach addresses the naive baseline’s loss of small-score objects, confusion from overlapping detections, and inconsistent nearby-pixel labels.The comparison uses segmentations from.Source: §5.3 · p. 8, Figure 8 · p. 8

5.4 Handling arbitrary closed curves

• Although trained only on rectangles, the model accepts arbitrary closed curves without retraining; a loosely drawn curve around either half of a stone accurately selects that half, unlike a bounding box.Source: §5.4 · p. 9

• The proposed neural segmentation method uses rectangles as input and is robust to their placement while incorporating global context.Source: Conclusions · p. 10
• The model generalizes to curve-based inputs, and a CRF-based labeling approach assigns each pixel a unique instance-level label.Source: Conclusions · p. 10
• Benchmark experiments demonstrate effectiveness on both interactive segmentation and instance segmentation.Source: Conclusions · p. 10