Source-linked AI summary
Large-scale interactive object segmentation with human annotators
Rodrigo Benenson, Stefan Popov, Vittorio Ferrari
TL;DR
Manual instance-mask annotation is costly, motivating interactive human–model correction. The paper studies interactive segmentation in simulation, collects 2.5M OpenImages masks, and introduces quality estimation from annotation signals. It reports instance segmentation that is 3× faster than polygon tools while producing better-quality masks.
Problem
Manual instance segmentation requires expensive pixel-level mask annotation for every object instance.
Method
The paper combines simulated design-space studies with interactive annotation, where humans correct model-generated masks across rounds, and uses correction sequences for quality estimation.
Results
3× faster annotation and better quality than traditional polygon drawing tools are reported, alongside 2.5M masks collected across 300 OpenImages categories.
Takeaways & Limitations
Interactive segmentation is presented as a compelling strategy for producing large-scale instance masks and releasing a large OpenImages annotation resource.
Takeaways & Limitations
Annotation-policy definition can bottleneck deployment, requiring extensive class-specific guidance and reuse across similar classes.
Abstract
from arXiv · showhide
Manually annotating object segmentation masks is very time consuming. Interactive object segmentation methods offer a more efficient alternative where a human annotator and a machine segmentation model collaborate. In this paper we make several contributions to interactive segmentation: (1) we systematically explore in simulation the design space of deep interactive segmentation models and report new insights and caveats; (2) we execute a large-scale annotation campaign with real human annotators, producing masks for 2.5M instances on the OpenImages dataset. We plan to release this data publicly, forming the largest existing dataset for instance segmentation. Moreover, by re-annotating part of the COCO dataset, we show that we can produce instance masks 3 times faster than traditional polygon drawing tools while also providing better quality. (3) We present a technique for automatically estimating the quality of the produced masks which exploits indirect signals from the annotation process.
1. Introduction
Instance segmentation is costly because masks require pixel-level annotation for every object. The paper proposes interactive human–model correction, evaluates it at scale, and reports faster, higher-quality masks than polygon tools.
- Manual instance segmentation is expensive because each object requires a pixel mask identifying its instance membership.
- Interactive annotation lets humans correct model-generated masks over multiple rounds instead of creating complete masks from scratch.
- The paper systematically studies deep interactive-segmentation design choices through extensive simulations and reports associated insights and caveats.
- 3× faster annotation with better quality than traditional polygon drawing tools is reported for instance segmentation.
- 2.5M masks across 300 OpenImages categories were collected with real annotators, addressing the limited scale of prior human experiments.
2. Overall system design
The system generates an initial mask from an image and bounding box, then repeatedly incorporates annotator corrections. A ranking model uses correction sequences to estimate mask quality and prioritize further work.
- The image+box→mask model Mb generates an initial mask from each existing bounding box.
- Annotators iteratively correct Mb+c outputs, and the image+box+corrections→mask model Mb+c uses those corrections to refine masks.
- The final segmentation model uses bounding boxes and corrections supplied throughout the annotation process as privileged information.
- Mr ranks instances from the time sequence of annotator corrections to predict generated-mask quality and prioritize low-quality instances.
- Correction-focused annotation concentrates collected effort on regions not already captured by the models, while easy instances may require no edits.
3. Simulations
The simulations address the system’s many design parameters within a unified blueprint. They are used to explore interactive-segmentation design choices systematically.
- The generic interactive system has many free design parameters, and prior studies report contradicting results.
- Section 3 combines an implementation blueprint with simulation experiments to explore the system’s design space.
3.1. Experiments blueprint
The experiments use noisy simulated interactions and per-instance mIoU evaluation on large COCO objects, with class-agnostic models trained on ADE20k-derived data. The blueprint also makes simulation assumptions explicit.
- The evaluation uses approximately 88k large COCO instances, with results reported on large objects from the COCO 2014 validation set.
- The models are trained on approximately 134k filtered instances from 255 ADE20k object classes, generally treating all instances as one generic class.
- Mean intersection-over-union (mIoU) is the main evaluation metric, averaged across instances while ignoring class labels.
- Simulation outcomes can be over-optimistic when training and testing use the same annotator noise model, and results depend on correction assumptions.
- Simulation adds noise to bounding boxes and clicks, and by default runs 3 rounds with up to 3 clicks per round.
- Mb and Mb+c use augmented Deeplabv2 ResNet101 architectures, while Mr uses a decision forest.
3.2. Simulation results
The simulations show that corrective region clicks, responsive multi-round interaction, and simple binary-disk encoding improve segmentation, while annotation noise materially affects mIoU. Class-specific models offer little lasting advantage after iterative correction.
- 3.2.1 Mb baselines: 65% mIoU was obtained for initial masks from Mb on COCO, versus 50% for raw noisy boxes and 59% for Grabcut.This class-agnostic transfer from ADE20k to COCO served as the starting point for annotation simulations.
- 3.2.2 Boundary click or region click?: 80% mIoU after three rounds was achieved with region clicks, compared with 77% for boundary clicks.The authors attribute this to greater robustness to click perturbations and more explicit information about pixels to add or remove.
- 3.2.3 Annotation noise: 3% and 7% mIoU drops resulted from click-noise standard deviations of 3 and 6 pixels, respectively.Ignoring regions smaller than x = 20 or x = 30 caused additional 3% or 8% mIoU drops, respectively.
- 3.2.4 Clicks encoding: Simple binary disks outperformed Gaussian and distance-transform click encodings, while disk diameter had little effect.Adding the previous-round mask as an input channel did not improve results.
- 3.2.5 Number of clicks and rounds: For approximately 9 clicks, distributing them across rounds, such as 3 × 3 or 4 × 2, outperformed using 8 or 9 clicks in one round.Increasing total clicks improved mIoU with diminishing returns after approximately 15 clicks.
- 3.2.6 Class-agnostic or class-specific?: After three rounds, class-specific and class-agnostic variants were within 2 percentage points of one another in mIoU.In-domain models were only slightly better than out-of-domain models, and the authors conclude that class-specific models are unnecessary.
4. Large-scale annotation campaign
The campaign uses interactive corrective clicks, explicit annotation policies, and reference masks to collect and evaluate large-scale instance segmentation data. A ranking model estimates mask quality from annotation-process signals without using class labels or image features.
- 4. Large-scale annotation campaign: The campaign annotates OpenImages instances with interactive corrective clicks and uses a smaller set of free-painting masks as reference for quality evaluation.Corrective-click and free-painting annotations are produced by human annotators under the campaign’s annotation setup.
- 4. Large-scale annotation campaign: The interface displays the object, original bounding box, current mask, corrective clicks, and the class-specific annotation policy.The policy is shown in the interface for fast consultation during annotation.
- 4. Large-scale annotation campaign: The campaign uses three annotation rounds with up to four clicks per round, following simulation results for the interactive setup.The selected 4 × 3 setup separates rounds by multiple days.
- 4. Large-scale annotation campaign: The ranking model estimates mask IoU from five annotation-process features, including clicks, round number, IoU change, and click distances.The model is trained as a random decision forest to regress to ground-truth mask IoU.
- 4. Large-scale annotation campaign: ∆IoU and average click distance provide the strongest predictive power because small changes and small corrected regions indicate higher mask quality.The ranker relies only on indirect annotation-process signals and excludes class labels and image features, supporting generalization across categories and datasets.
- 4. Large-scale annotation campaign: Annotation policies are a deployment bottleneck, so the campaign reuses policies across similar classes and combines class-specific and class-agnostic policies.The authors created 42 class-specific policies covering 200 classes, while a class-agnostic policy covered 100 classes.
5. Analysis of human annotations
The annotation campaign reveals how annotator behavior relates to time and quality, while corrective clicks provide a faster, higher-quality alternative to polygon drawing and enable mask ranking.
- Annotator behavior: 2.7% of instances were skipped, 2.1% received zero clicks, and 70.0% received four clicks, reflecting annotators’ quality-focused correction behavior.Annotators also became stricter across rounds and often corrected very small missing details.
- Annotation time: 11.4s was the average annotation-interface time per instance, with faster classes averaging under 8.7s per round and slower classes over 12.7s.Average time increased from 10.8s in round 1 to 11.9s in round 3 as errors became smaller and harder to find.
- Time versus quality: After three corrective-click rounds, masks reached 84% mIoU in 34s per instance, versus 82% mIoU in 108s for COCO polygons.Boundary F-measure was also higher for corrective-click masks: 75% versus 65% for COCO polygons.
- Time versus quality: Corrective-click masks were 3× faster to make than COCO polygons while achieving higher quality, using 10.7 clicks per instance across three rounds.The comparison measures mIoU against the free-painted COCO annotations.
- Downstream utility: The semantic-segmentation models trained from corrective-click masks and COCO polygons performed comparably, at 53% and 52% mIoU respectively.Training and evaluation ignored small instances and focused on objects larger than 80 × 40 pixels.
- Mask ranking: Mr effectively ranks masks by quality, allowing selection of the top 70% at 90% mIoU or targeting the bottom 30% at 70% mIoU.The ranking can also be used to weight training samples according to rank.
6. Conclusion
The paper concludes that interactive segmentation is compelling for instance segmentation at scale, combining systematic model analysis, large-scale annotation, and automatic mask-quality estimation.
- Conclusion: Interactive segmentation produced 2.5M high-quality instance masks on OpenImages and introduced automatic estimation of individual mask quality.The reported quality was 84% mIoU and 75% boundary quality.