PaperlayerView exact revision on arXiv

Source-linked AI summary

Reviving Iterative Training with Mask Guidance for Interactive Segmentation

Konstantin Sofiiuk, Ilia A. Petrov, Anton Konushin

arXiv:2102.06583v1cs.CV

TL;DR

Inference-time optimization has produced strong interactive-segmentation results but requires backward passes and is difficult to deploy on forward-only mobile frameworks. The paper evaluates design choices, introduces a feedforward model using previous masks, and finds that COCO+LVIS training yields superior performance. Its conclusions emphasize that feedforward design and dataset quality are central to interactive-segmentation performance.

  • Problem

    Inference-time optimization makes click-based interactive segmentation more computationally expensive and harder to deploy on mobile frameworks supporting only forward passes.

  • Method

    The paper develops a feedforward click-based model that uses previous-step masks, supports correcting existing masks, and trains on an iterative COCO+LVIS dataset.

  • Results

    The model achieves or surpasses current state-of-the-art results without additional optimization and sets a new state-of-the-art on common interactive-segmentation benchmarks.

  • Takeaways & Limitations

    A large, diverse COCO+LVIS dataset with high-quality annotations significantly improves generalization and pushes the model's state-of-the-art results further.

Abstract

from arXiv · show

Recent works on click-based interactive segmentation have demonstrated state-of-the-art results by using various inference-time optimization schemes. These methods are considerably more computationally expensive compared to feedforward approaches, as they require performing backward passes through a network during inference and are hard to deploy on mobile frameworks that usually support only forward passes. In this paper, we extensively evaluate various design choices for interactive segmentation and discover that new state-of-the-art results can be obtained without any additional optimization schemes. Thus, we propose a simple feedforward model for click-based interactive segmentation that employs the segmentation masks from previous steps. It allows not only to segment an entirely new object, but also to start with an external mask and correct it. When analyzing the performance of models trained on different datasets, we observe that the choice of a training dataset greatly impacts the quality of interactive segmentation. We find that the models trained on a combination of COCO and LVIS with diverse and high-quality annotations show performance superior to all existing models. The code and trained models are available at https://github.com/saic-vul/ritm_interactive_segmentation.

1. Introduction

Interactive segmentation lets users control predictions across multiple iterations, correct errors, and select previously unseen object classes. The paper argues that feedforward models and diverse training data can achieve strong performance without inference-time optimization.

  • Interactive segmentation uses iterative user input to select objects, correct prediction errors, and segment previously unseen classes.
  • Click-based interaction is widely studied because its training and evaluation protocols are established and its input is intuitive.
  • Interactive segmentation can simplify mask annotation and support photo editing, including object selection on smartphones with limited input.
  • Inference-time optimization improves quality but requires backward passes, limiting deployment on mobile frameworks that generally support forward passes.
  • Models trained with standard random point sampling can achieve state-of-the-art performance, while diverse large datasets with fine masks strongly affect model quality.
  • The proposed approach uses previous-step masks, supports modifying existing instance masks, improves stability when clicks are added, and combines LVIS with COCO for training.

2. Related Work

Related work spans optimization-based, CNN-based, and feedback-based interactive segmentation methods. Existing approaches refine predictions through architectural mechanisms, alternative user inputs, or inference-time optimization, often with associated constraints.

  • Interactive segmentation evolved from graph-based energy minimization to CNN models using click simulation, mask diversity, attention, and test-time refinement.
  • Backpropagating Refinement Scheme methods optimize input distance maps or network parameters after clicks, improving quality at increased computational cost.
  • Bounding boxes provide coarse object localization but lack specific object reference and mask-correction interfaces; combining boxes with clicks addresses these limitations.
  • Extreme-point interaction is compact and limited to four clicks, but point placement is harder and it does not support corrections like bounding-box interaction.
  • Mask-refinement methods generally make local boundary corrections without globally including or excluding large object regions, either within architectures or as postprocessing.

3. Proposed Method

The proposed method revises click-based interactive segmentation around standard segmentation backbones, improved click encoding, iterative interaction simulation, previous-mask input, and normalized focal loss.

  • Backbone architecture: HRNet is preferred over DeepLabV3+ as the backbone for this interactive segmentation task.The paper evaluates both architectures and reports HRNet as more preferable according to experiments.
  • Click encoding: Disk click encoding outperforms distance-transform encoding, likely because adding or moving clicks changes disks locally rather than altering the map drastically.The authors associate the local changes of disk maps with more stable network inputs when new clicks are added.
  • Feeding clicks to the backbone: Conv1S adds a convolutional block whose output is summed with the first backbone layer, preserving pretrained backbone weights while allowing a separate learning rate.The added block matches the first backbone block’s tensor shape and is used to incorporate encoded clicks.
  • Iterative sampling strategy: Iterative sampling simulates clicks in erroneous prediction regions, combining random initialization with additional iteratively simulated clicks during training.The adopted procedure samples from morphologically eroded mislabelled regions and varies the number of iterative clicks.
  • Mask guidance: Previous-step masks are supplied as an additional input channel alongside positive and negative click channels, with an empty mask used for initial interactions.Using a previous mask requires iterative interaction simulation during training.
  • Normalized focal loss: Normalized focal loss is selected because normalization prevents its total gradient from fading over time, supporting faster convergence and better accuracy than BCE.The paper compares NFL with BCE, focal, and soft IoU losses using the COCO+LVIS-trained HRNet-18+OCR baseline.

4. Dataset for Interactive Segmentation

The paper argues that training data strongly affects interactive segmentation and combines LVIS with COCO to address LVIS’s limited coverage of common object categories.

  • Dataset limitations: Existing interactive segmentation models commonly train on SBD, Pascal VOC, or their combination, whose 20 classes cover only general object types.The augmented SBD/Pascal VOC dataset contains 10,582 images and 25,832 instance-level masks.
  • Dataset limitations: LVIS offers a stronger training choice but has a long-tailed distribution that lacks general object categories.The missing common categories may affect model accuracy and generalization.
  • COCO+LVIS: The proposed COCO+LVIS dataset augments LVIS labels with COCO masks because the datasets share the same images.COCO contributes 1.2M instance masks from 118k training images across 80 object classes.
  • Conclusion: The authors conclude that COCO+LVIS is the best training dataset choice for interactive segmentation based on their comparative experiments.The dataset combines diverse, high-quality annotations across broader class distributions.

5. Experiments

Experiments evaluate architectures, datasets, convergence, and iterative training. They show that mask-guided iterative models improve stability and accuracy, while dataset choice and backbone capacity materially affect results.

  • Setup: The experiments use five interactive-segmentation benchmarks, standard NoC@85 and NoC@90 metrics, and multiple DeepLab-V3+ and HRNet+OCR backbones.NoC measures clicks needed to reach a predefined IoU threshold.
  • Backbone comparison: HRNet-18 outperforms DeepLabV3+ with ResNet-34 while using several times fewer parameters and FLOPs.The comparison includes model complexity at 400 × 400 input resolution.
  • Training datasets: COCO+LVIS-trained models outperform alternatives across benchmarks, although SBD-specific models perform best on SBD because their training and testing distributions are similar.The dataset comparison reports NoC@90 on Berkeley, SBD, and DAVIS.
  • Iterative training ablations: N_iters = 3 is optimal for iterative sampling, whereas values above 4 cause training instability and worse results.Models with N_iters = 5 or 6 sometimes collapsed after 10–20 training epochs on COCO+LVIS.
  • Mask guidance: Feeding the previous mask improves stability by preventing accuracy drops when additional clicks are added and enables correction of inaccurate external masks.Without the previous mask, iterative and non-iterative models can show declining mean IoU as clicks increase.
  • Comparison with previous works: The proposed mask-guided iterative method achieves state-of-the-art results on all five benchmarks, while the smallest HRNet-18s+OCR model performs on par with heavier models.The latter result supports use on devices with low computational capability.
  • Convergence: Mean IoU plots show improved accuracy and stability, with the proposed method avoiding accuracy drops after clicks and converging to better results.The plots cover GrabCut, Berkeley, DAVIS, and SBD.

6. Conclusion

The paper concludes that pure feedforward interactive segmentation can match or surpass state-of-the-art methods while supporting both new-object segmentation and mask correction. It also identifies COCO+LVIS training data as a major source of improved generalization.

  • Conclusions: A pure feedforward model with a modern backbone achieves or surpasses current state-of-the-art interactive segmentation results.The approach avoids the inference-time optimization schemes used by many recent methods.
  • Conclusions: The model can segment new objects without prior masks and modify existing segmentation masks interactively.This extends click-based interaction to external-mask correction.
  • Training data: Combining COCO and LVIS yields a large, diverse dataset with high-quality annotations that significantly improves generalization ability.Training on the combined dataset further improves the state-of-the-art results achieved by the model.
Loading 2102.06583v1…