Source-linked AI summary
Annotating Object Instances with a Polygon-RNN
Lluis Castrejon, Kaustav Kundu, Raquel Urtasun, Sanja Fidler
TL;DR
Large-scale segmentation datasets need precise annotations, but producing them is time consuming and expensive. Polygon-RNN predicts object polygons sequentially from image crops and lets annotators correct vertices during prediction. It achieves a 4.74× annotation speed-up with 78.4% IoU agreement on Cityscapes, matching typical human-annotator agreement, and generalizes across datasets.
Problem
Large-scale segmentation datasets require precise ground-truth annotations, but creating them is time consuming and expensive.
Method
Polygon-RNN takes an image crop within a ground-truth bounding box and sequentially predicts polygon vertices, allowing annotators to correct vertices interactively.
Results
4.74× annotation speed-up is achieved while matching typical human-annotator agreement, with 78.4% IoU agreement across Cityscapes classes.
Takeaways & Limitations
The approach provides a generic annotation tool that produces structurally plausible object annotations with a desired accuracy using only a few annotator clicks.
Takeaways & Limitations
Pixel-level annotation approaches struggle to incorporate shape priors and typically produce outputs too imprecise to serve as official benchmark ground truth.
Abstract
from arXiv · showhide
We propose an approach for semi-automatic annotation of object instances. While most current methods treat object segmentation as a pixel-labeling problem, we here cast it as a polygon prediction task, mimicking how most current datasets have been annotated. In particular, our approach takes as input an image crop and sequentially produces vertices of the polygon outlining the object. This allows a human annotator to interfere at any time and correct a vertex if needed, producing as accurate segmentation as desired by the annotator. We show that our approach speeds up the annotation process by a factor of 4.7 across all classes in Cityscapes, while achieving 78.4% agreement in IoU with original ground-truth, matching the typical agreement between human annotators. For cars, our speed-up factor is 7.3 for an agreement of 82.2%. We further show generalization capabilities of our approach to unseen datasets.
1. Introduction
The paper targets faster, precise object-instance annotation by predicting polygons from image crops and allowing annotators to correct vertices interactively. On Cityscapes, this approach substantially reduces annotation effort while retaining human-level agreement with ground truth.
- Motivation: Large-scale segmentation datasets require annotations that are both time consuming and expensive to produce.The paper links this need to data-hungry neural segmentation methods and the demand for large training datasets.
- Results: 4.74× fewer annotation clicks are required by the proposed method.Figure 1 describes automatic polygon prediction from a bounding box with user corrections when needed.
- Contribution: 4.7× annotation-time reduction is achieved while producing highly accurate and structurally coherent object annotations.Polygon annotation typically uses around 30–40 clicks per object, motivating a more efficient interactive method.
- Method: Polygon-RNN takes an image crop within a ground-truth bounding box and sequentially predicts vertices outlining the object instance.The model casts segmentation as polygon prediction rather than pixel labeling.
- Results: 78.4% IoU agreement with original ground truth matches typical agreement between human annotators.The reported agreement accompanies the 4.7× annotation speed-up.
2. Related Work
Related work includes semi-automatic pixel-level annotation, superpixel-based tools, and polygon-oriented instance segmentation. These approaches improve efficiency or exploit shape structure, but pixel-level methods often struggle with shape priors and produce outputs too imprecise for official ground truth.
- Semi-automatic annotation: Semi-automatic annotation methods use scribbles, motion cues, or graph cuts to reduce pixel-labeling effort.These approaches provide weaker supervision or interactive seeds for foreground and background segmentation.
- Semi-automatic annotation: Pixel-level graphical models make shape priors difficult to incorporate and may produce regions with holes.Their mistakes can also leave annotators with tedious corrections, limiting their use as official benchmark ground truth.
- Annotation tools: Superpixel-level annotation improves efficiency but depends on superpixel scale and can merge small objects or object parts.Multiple superpixel scales were proposed to address this issue in video labeling.
- Object instance segmentation: Object-instance segmentation approaches predominantly label pixels inside boxes or patches using CNNs.Related polygon methods instead detect edge fragments and link them into coherent cycles.
3. Polygon-RNN
Polygon-RNN predicts object polygons as ordered vertex sequences from image crops, while allowing annotators to correct vertices during inference. Its CNN–ConvLSTM design uses image features and polygon history to produce coherent boundaries and supports efficient annotation.
- Given a bounding box and image crop, Polygon-RNN predicts a closed polygon outlining the object with a recurrent neural network.
- The model feeds CNN image features and previously predicted vertices into the RNN, using recent history to follow polygon orientation and the first vertex to detect closure.
- The decoder is a two-layer ConvLSTM that preserves spatial information and predicts one polygon vertex at each time step.
- The first-vertex module jointly predicts object boundaries and polygon vertices on the output grid.
- Annotators can replace any predicted vertex during inference, after which the corrected vertex is fed into the next step to redirect the model.
4. Results
On Cityscapes, Polygon-RNN predicts object polygons and supports vertex corrections, outperforming baselines on most categories while exposing size-dependent limitations and annotation-speed gains.
- Evaluation setup: The evaluation assumes a ground-truth object box and measures per-instance IoU plus simulated vertex corrections.The correction procedure counts clicks when predicted vertices exceed a chessboard-distance threshold from ground truth.
- Prediction mode: Without annotator interaction, Polygon-RNN predicts complete polygons automatically in the prediction mode.The model generates the full polygon before any corrections are applied.
- Prediction mode: Polygon-RNN outperforms baselines in 6 of 8 Cityscapes categories and in the all-class average, with its strongest gains for cars, persons, and riders.It exceeds SharpMask by 12% for cars, 7% for persons, and 6% for riders despite using VGG rather than ResNet.
- Effect of object size: For larger instances, baselines benefit from higher output resolution, and Polygon-RNN obtains lower IoU for classes such as bus and train.The model performs significantly better on small instances, while quantization and output resolution constrain larger-object accuracy.
- Annotator in the loop: With annotator corrections, car instances reach 82% agreement using 4.6 clicks on average, corresponding to a 7.3-fold annotation speed-up.Across all Cityscapes categories, 9.39 clicks yield 78.40% IoU agreement and a 4.74-fold speed-up.
- Generalization: A Cityscapes-trained model generalizes to KITTI, whose instances are usually larger and make DeepMask and SharpMask perform very similarly.The evaluation uses 741 annotated KITTI instances.
5. Conclusion
Polygon-RNN facilitates object-instance annotation by predicting polygons and incorporating annotator corrections, achieving rapid, accurate, and structurally plausible annotations across datasets.
- Polygon-RNN predicts a polygon outlining an object and incorporates corrections from an annotator in the loop.
- 4.74× annotation speed-up matches the agreement between human annotators.
- A few annotator clicks can achieve a desired annotation accuracy while preserving structurally plausible object shapes.
- Cross-dataset generalization supports using the approach as a generic annotation tool.