Source-linked AI summary
Fast Interactive Object Annotation with Curve-GCN
Huan Ling, Jun Gao, Amlan Kar, Wenzheng Chen, Sanja Fidler
TL;DR
Manual object-boundary tracing is laborious, while sequential interactive polygon methods limit scalability and correction flexibility. Curve-GCN predicts all contour vertices simultaneously with a graph convolutional network, supports polygons or splines, and outperforms prior approaches while running at 29.3 ms automatically and 2.6 ms interactively. Its evaluation and representation assume target shapes can be modeled with fixed-topology control points and, for the stated Cityscapes training procedure, disconnected components are handled through staged training.
Problem
Manually tracing object boundaries is laborious, and sequential polygon prediction makes complex-shape annotation harder to scale and correct interactively.
Method
Curve-GCN represents an object as a fixed-topology graph and predicts all vertices simultaneously with an end-to-end differentiable Graph Convolutional Network, using polygon or spline representations.
Results
Curve-GCN outperforms Polygon-RNN++ and PSP-Deeplab/DEXTR in automatic and interactive settings, with 29.3 ms automatic inference and 2.6 ms per interactive correction.
Takeaways & Limitations
The framework provides efficient interactive annotation with local correction effects and supports cross-domain annotation beyond Cityscapes.
Takeaways & Limitations
The approach assumes target shapes can be represented by N control points connected into a cycle, and Cityscapes occlusions split objects into disconnected components that require staged training.
Abstract
from arXiv · showhide
Manually labeling objects by tracing their boundaries is a laborious process. In Polygon-RNN++ the authors proposed Polygon-RNN that produces polygonal annotations in a recurrent manner using a CNN-RNN architecture, allowing interactive correction via humans-in-the-loop. We propose a new framework that alleviates the sequential nature of Polygon-RNN, by predicting all vertices simultaneously using a Graph Convolutional Network (GCN). Our model is trained end-to-end. It supports object annotation by either polygons or splines, facilitating labeling efficiency for both line-based and curved objects. We show that Curve-GCN outperforms all existing approaches in automatic mode, including the powerful PSP-DeepLab and is significantly more efficient in interactive mode than Polygon-RNN++. Our model runs at 29.3ms in automatic, and 2.6ms in interactive mode, making it 10x and 100x faster than Polygon-RNN++.
1. Introduction
Curve-GCN targets the labor of manual boundary tracing by replacing sequential polygon prediction with simultaneous graph-based vertex prediction. It supports polygon and spline representations for interactive annotation and reports strong performance across settings.
- Motivation: Manual object-boundary tracing can take up to 40sec per object, motivating interactive annotation methods.Existing interactive techniques aim to accelerate this process.
- Prior approach: Polygon-RNN predicts polygon vertices sequentially and lets annotators correct errors during the recurrent process.Corrections condition subsequent predictions, with worst-case effort bounded by roughly 30-40 vertices for most objects.
- Curve-GCN: Curve-GCN predicts all vertices simultaneously with a Graph Convolutional Network and is trained end-to-end.The framework represents objects with either polygons or splines at high output resolution.
- Results: Curve-GCN outperforms Polygon-RNN++ and PSP-Deeplab/DEXTR in automatic and interactive Cityscapes annotation settings.The paper also reports gains for cross-domain annotation across general scenes, aerial imagery, and medical imagery.
2. Related Work
Prior interactive segmentation methods use pixels, contours, or sequential polygon tracing, each imposing interaction or scalability constraints. Curve-GCN instead predicts polygon or spline contours simultaneously and supports local, faster corrections.
- Pixel-wise methods: Pixel-wise methods require foreground-background labeling through boxes, scribbles, or clicks, with ambiguous boundaries often requiring many interactions.DEXTR uses four extreme-point clicks and may require additional boundary clicks.
- Contour-based methods: Contour-based methods trace boundaries with level sets or seeded minimal-cost paths, requiring additional user input when errors occur.Intelligent Scissors extends the contour from the last seed to the cursor along a minimum-cost path.
- Polygon-RNN: Polygon-RNN uses a CNN-RNN to output one polygon vertex at a time, limiting scalability and slowing inference as vertex counts grow.Curve-GCN predicts all polygon vertices simultaneously and is designed to be invariant to their order.
- Related representations: Curve-GCN differs from Pixel2Mesh by using spline parametrization and a task-specific loss for 2D human-in-the-loop annotation.Pixel2Mesh does not address the human-in-the-loop scenario described here.
- Spline models: Compared with active shape models, Curve-GCN is end-to-end and repredicts after every spline modification for faster interactive annotation.Active shape models require aligned shapes to compute a PCA basis.
3. Object Annotation via Curve-GCN
Curve-GCN frames object annotation as simultaneous regression of graph-structured control points, supporting polygon and spline contours. A CNN supplies image features, while iterative GCN updates and human corrections refine the predicted boundary.
- Representation: Splines approximate curved contours with fewer control points than polygons, and Curve-GCN supports both representations.The framework uses centripetal Catmull-Rom splines whose control points lie along the curve, making edits more direct.
- Graph formulation: Given a bounding-box crop, Curve-GCN predicts all control-point locations simultaneously from a fixed-topology graph.The graph connects each vertex to four neighbors, enabling information exchange between nodes.
- Prediction: CNN features, edge predictions, vertex outputs, node coordinates, and bilinearly interpolated local features form the inputs used to predict control-point offsets.Each predicted offset updates a node location, after which features are re-extracted for iterative refinement.
- Training: The model uses ordered point matching and differentiable rendering losses to align predicted contours with ground-truth shapes.Ordered matching discourages self-intersections, while rendering loss compares masks and addresses overly smooth predictions.
- Interactive annotation: InteractiveGCN learns to correct neighboring control points after an annotator moves an erroneous point, using iterative training with gradients preserved through the procedure.Experiments set k = 2, and the annotator is modeled as correcting the currently worst predicted point.
4. Experimental Results
Curve-GCN is evaluated for automatic, interactive, metric-sensitive, ablation, additional-input, and cross-domain object annotation. It generally improves accuracy and efficiency over prior methods, while disconnected multi-component objects remain a limitation.
- Experimental setup: Curve-GCN evaluates Polygon-GCN and Spline-GCN on Cityscapes, comparing automatic performance with Polygon-RNN++ and PSP-DeepLab and interactive performance with Polygon-RNN++ and DEXTR.The experiments use IoU and boundary F scores, with separate automatic and interactive regimes.
- Automatic mode: Spline-GCN is on par with PSP-DeepLab in IoU and significantly better in the more precise boundary F score.The authors attribute the boundary advantage to more accurate alignment with object boundaries.
- Limitations: Curve-GCN struggles with train instances split into multiple disconnected components because it predicts only a single connected component.This limitation is especially relevant for occluded and broken Cityscapes trains.
- Ablation study: Iterative inference significantly boosts performance, while adding a boundary prediction branch further improves the model.The ablation evaluates both Polygon-GCN and Spline-GCN with three iterative inference steps.
- Inference time: 29.3 ms per object makes Curve-GCN an order of magnitude faster than Polygon-RNN++ at 298.0 ms in automatic mode.Interactive corrections require 2.6 ms for Curve-GCN versus 270 ms for Polygon-RNN.
- Cross-domain evaluation: Fine-tuning with 10% of data from a new domain effectively adapts Curve-GCN and supports better generalization than PSP-DeepLab.Cross-domain evaluation covers general, aerial, and medical imagery.
5. Conclusion
Curve-GCN enables efficient interactive annotation, improves over the state of the art, and supports local corrections that give annotators greater control.
- Curve-GCN improves over the state of the art while making interactive annotation significantly faster.
- Local interactive corrections affect only nearby regions, giving annotators more control during annotation.
- The resulting annotation strategy improves overall efficiency for collecting computer vision datasets.
- The authors plan to release an annotation tool running Curve-GCN to facilitate faster dataset collection.