PaperlayerView exact revision on arXiv

Source-linked AI summary

Iteratively Trained Interactive Segmentation

Sabarinath Mahadevan, Paul Voigtlaender, Bastian Leibe

arXiv:1805.04398v1cs.CV

TL;DR

Manual segmentation labeling is expensive, motivating interactive click-based methods. This paper trains a convolutional network with iterative error-based clicks and reports that the resulting method surpasses the state of the art across two segmentation settings.

  • Problem

    Object segmentation requires extensive manual annotation, which is costly, while deep learning requires large amounts of annotated training data.

    Source: §1 · p. 1

  • Method

    The authors introduce ITIS, training a click-based segmentation network by iteratively adding correction clicks from predicted-mask errors, with an optional existing-mask input.

    Source: §3 · pp. 3–4, §3.2 · p. 4

  • Results

    The method surpasses the state of the art for both creating segmentations from scratch and correcting masks obtained by video object segmentation, and is robust to test-time click-sampling variations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that ITIS performs better than the current state of the art on a variety of tasks and make its code and models publicly available.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Deep learning requires large amounts of training data to be effective. For the task of object segmentation, manually labeling data is very expensive, and hence interactive methods are needed. Following recent approaches, we develop an interactive object segmentation system which uses user input in the form of clicks as the input to a convolutional network. While previous methods use heuristic click sampling strategies to emulate user clicks during training, we propose a new iterative training strategy. During training, we iteratively add clicks based on the errors of the currently predicted segmentation. We show that our iterative training strategy together with additional improvements to the network architecture results in improved results over the state-of-the-art.

1 Introduction

Deep-learning segmentation depends on extensive annotation, making interactive methods relevant when manual labeling is costly. This paper introduces iterative click-based training and reports improved state-of-the-art results across two segmentation settings.

1 Introduction

  • Manual segmentation labeling is costly, unlike image-classification labeling, for which large datasets such as ImageNet are available.The supplied introduction passage ends before completing its discussion of segmentation annotation effort.Source: §1 · p. 1
  • Training clicks are added according to errors in the current predicted mask, aligning training with test-time correction clicks rather than independent heuristic sampling.The authors state that results are robust to variations in the test-time click-sampling strategy.Source: §1 · pp. 1–2
  • The proposed iterative click-sampling strategy and network improvements surpass the state of the art both for segmenting from scratch and correcting video-segmentation masks.The authors also report robustness to variations in the click-sampling strategy used at test time.Source: §1 · p. 2
  • The work introduces ITIS, a framework for interactive click-based image segmentation, and makes code and models publicly available.It also compares alternative representations of click and mask inputs.Source: §1 · p. 2

2 Related Work

Interactive segmentation approaches differ in the user input they accept and in how they turn that input into masks. The related work spans graph cuts, learned click-based methods, and polygon or extreme-point approaches.

2 Related Work

  • Graph-cut methods use foreground/background marks or a loose rectangle to produce object segmentations, with GrabCut extending the graph-cut approach.LooseCut further optimizes interactive segmentation.Source: §2 · p. 2
  • Deep-learning interactive methods learn objectness representations that can reduce the user interactions needed for high-quality annotations.Earlier approaches include scribble-based supervision and click-based iFCN, which encodes positive and negative clicks as Euclidean distance maps.Source: §2 · p. 2
  • RIS-Net [17] combines regional and global context with graph-cut optimization, while the authors claim their method performs better without those components.The paper says these components could be combined with its method for further improvement.Source: §2 · p. 3
  • DEXTR uses exactly four extreme-point clicks encoded as Gaussians, making refinement with additional clicks difficult when its initial mask is poor.Source: §2 · p. 3
  • Polygon-RNN allows interactive polygon correction but requires a bounding box and is poorly suited to correcting existing pixel masks or objects with multiple components.Source: §2 · p. 3

3 Proposed Method

The method uses click-based inputs, optionally adds an existing mask, and trains by iteratively adding correction clicks from the network’s errors. Its architecture and training procedure are designed for interactive segmentation.

3 Proposed Method

  • ITIS uses positive and negative clicks and optionally an existing segmentation estimate as inputs to a click-based interactive object-segmentation network.The click channels are represented as Gaussians with small variance, and the optional mask channel is used when refining an existing mask.Source: §3 · pp. 3–4

3.1 Network Architecture

  • DeepLabV3+ [8] supplies the encoder-decoder architecture, combining spatial pyramid pooling with a decoder that recovers fine structures.It also uses depth-wise separable convolutions for a faster network with fewer parameters; pretrained weights are used except for the replaced two-class output layer.Source: §3.1 · p. 4

3.2 Iterative Training Procedure

  • Iterative training adds correction clicks from prediction errors, aligning training more closely with how an annotator uses the system.The authors use random sampling to keep training effort manageable.Source: §3.2 · p. 4
  • Training combines initial click sampling, intended to expose the network to varied patterns, with later clicks that teach it to correct predicted-mask errors.The initial clicks use multiple strategies modeled on iFCN.Source: §3.2 · pp. 4–5
  • Clicks are periodically reset to initial samples during training, with the optional mask reset to empty, to retain examples with fewer clicks or lower-quality masks.The reset also adds randomness that the authors say reduces overfitting.Source: §3.2 · pp. 4–5
  • Each correction click targets a largest connected component of misclassified pixels, choosing a location farthest from the component boundary and existing clicks.The click is positive or negative according to whether its target pixel lies on the object; Figure 2 illustrates the progression from error pixels to clusters and a click.Source: §3.2 · p. 5, Figure 2 · p. 5

4 Experiments

Experiments evaluate interactive segmentation from scratch across PASCAL VOC, GrabCut, and KITTI, and correction of automatic video-segmentation masks on DAVIS. They compare ITIS with prior methods, test sampling robustness, and ablate design choices.

4.1 Datasets

  • Experiments cover PASCAL VOC, GrabCut, and KITTI for click-based segmentation from scratch, plus DAVIS for correcting automatic video-segmentation masks.PASCAL VOC training uses more than 20,000 object instances across 20 categories; KITTI evaluation uses 741 cars.Source: §4 · pp. 5–6, §4.1 · p. 6

4.2 Experimental Setup

  • Evaluation measures mIoU and the number of clicks needed to reach a target score, simulating corrective clicks up to 20 per instance.Training uses bootstrapped cross-entropy on the worst 25% of pixel predictions, Adam, and Gaussian-encoded clicks.Source: §4.2 · p. 6

4.3 Comparison to State of the Art

  • ITIS requires the fewest clicks on PASCAL VOC under per-instance scoring and remains strongest there under validation-wide scoring; it is slightly better than RIS-Net on GrabCut under the latter protocol.On GrabCut, ITIS is strongest for the initial few clicks but later performs similarly to RIS-Net; reaching 90% mIoU requires slightly more clicks than RIS-Net.Source: Table 1 · p. 7, Figure 3 · p. 7, §4.3 · p. 7

4.4 Generalisation to Other Sampling Strategies

  • ITIS achieves similar mIoU with random test-time clicks, while cluster-based sampling can help at low click counts and its advantage fades as clicks accumulate.Cluster sampling selects a mislabelled connected component in proportion to its size and clicks its centre; random sampling chooses a pixel from the misclassified region.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 Ablation Study

  • 3.8 vs 5.4 clicks are needed to reach 85% mIoU with iterative training on PASCAL VOC.Replacing Gaussian click inputs with distance transforms raises the required clicks from 5.4 to 6.5; the optional mask channel reduces performance in the first 10 clicks but boosts maximum mIoU at 20 clicks.Source: §4.5 · p. 8, Figure 5 · p. 9

4.6 Correcting Masks for Video Object Segmentation

  • ITIS-VOS, which adds a mask input to refine OSVOS predictions, produces better results than the compared methods, especially at 1 and 4 clicks.The evaluation refines existing video-segmentation masks rather than creating segmentations from scratch.Source: Table 2 · p. 9, §4.6 · p. 9

4.7 Annotating KITTI Instances

  • On 741 KITTI cars, ITIS catches up with Polygon-RNN and performs better across a large range of click counts, though both converge to similar values with many clicks.Polygon-RNN starts with reasonable results using ground-truth bounding boxes, while ITIS works on the whole image; correcting an error region requires less precise clicking than marking a polygon outline.Source: §4.7 · pp. 9–10, Figure 6 · p. 10

5 Conclusion

The paper concludes that ITIS combines click-based interactive segmentation with iterative training and reports better-than-state-of-the-art results across a variety of tasks.

5 Conclusion

  • ITIS is a framework for interactive click-based segmentation with a novel iterative training procedure.Source: §5 · p. 10
  • The authors report results better than the state of the art on a variety of tasks.Source: §5 · p. 10
  • The authors say they will make the code, including an annotation tool, publicly available.They hope it will be used to annotate large datasets.Source: §5 · p. 10

A Initial Click Sampling

Initial click channels are populated using sampled positive and negative clicks, with negative clicks generated by a randomly selected strategy.

A Initial Click Sampling

  • The positive-click count is sampled from [1,Npos], and clicks are sampled from object pixels subject to spacing from one another and the object boundary.Source: A Initial Click Sampling · p. 13
  • The negative-click count is sampled from [0,Ni] after randomly selecting one strategy from S = {s1,s2,s3}.Source: A Initial Click Sampling · p. 13
  • The negative-click strategies sample background near the object boundary, sample negative objects, or cover object boundaries; boundary coverage helps train the network faster.Source: A Initial Click Sampling · p. 13

B Implementation Details

Training uses object-containing random crops, gamma augmentation, and a staged learning-rate schedule, while testing uses the original image resolution without cropping.

B Implementation Details

  • Training uses 350 × 350-pixel crops, upscaling images with a smaller side below 350 pixels and otherwise retaining their original resolution before cropping.Source: B Implementation Details · p. 13
  • Gamma augmentation is the only data augmentation, and the learning rate drops from 10^-5 to 10^-6 at epoch 10 and 3·10^-7 at epoch 15.Source: B Implementation Details · p. 13
  • At test time, images are used at original resolution without resizing or cropping; initial click-sampling settings include Npos = 5, dm = 5, ds = 40, do = 40, N1 = 10, N2 = 5, and N3 = 10.Source: B Implementation Details · p. 13

C Qualitative Results

The qualitative results include single-click examples, and the paper reports that ITIS often produces good-quality segmentations from one click.

C Qualitative Results

Loading 1805.04398v1…