PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++

David Acuna, Huan Ling, Amlan Kar, Sanja Fidler

arXiv:1803.09693v1cs.CV

TL;DR

Manually labeling object masks is time-consuming, motivating interactive polygon annotation. Polygon-RNN++ improves Polygon-RNN with a redesigned encoder, reinforcement-learning training, and graph-based high-resolution refinement. It improves Cityscapes annotation in automatic and interactive modes, generalizes across domains, and benefits from online fine-tuning.

  • Problem

    Collecting ground-truth instance masks is extremely time consuming, typically requiring 20-30 seconds per object from human annotators.

  • Method

    Polygon-RNN++ combines a redesigned CNN encoder, reinforcement-learning training, and a Graph Neural Network for higher-resolution interactive polygon annotation.

  • Results

    10% mean IoU improvement on Cityscapes and 50% fewer clicks than Polygon-RNN are reported, alongside stronger cross-domain performance and reduced annotation time after online fine-tuning.

  • Takeaways & Limitations

    Polygon-RNN++ generalizes to novel domains and can be adapted with simple online fine-tuning for efficient annotation.

  • Takeaways & Limitations

    The training setup addresses limitations of maximum-likelihood and teacher-forcing approaches, including metric mismatch and exposure bias.

Abstract

from arXiv · show

Manually labeling datasets with object masks is extremely time consuming. In this work, we follow the idea of Polygon-RNN to produce polygonal annotations of objects interactively using humans-in-the-loop. We introduce several important improvements to the model: 1) we design a new CNN encoder architecture, 2) show how to effectively train the model with Reinforcement Learning, and 3) significantly increase the output resolution using a Graph Neural Network, allowing the model to accurately annotate high-resolution objects in images. Extensive evaluation on the Cityscapes dataset shows that our model, which we refer to as Polygon-RNN++, significantly outperforms the original model in both automatic (10% absolute and 16% relative improvement in mean IoU) and interactive modes (requiring 50% fewer clicks by annotators). We further analyze the cross-domain scenario in which our model is trained on one dataset, and used out of the box on datasets from varying domains. The results show that Polygon-RNN++ exhibits powerful generalization capabilities, achieving significant improvements over existing pixel-wise methods. Using simple online fine-tuning we further achieve a high reduction in annotation time for new datasets, moving a step closer towards an interactive annotation tool to be used in practice.

1. Introduction

Polygon-RNN++ addresses the time-consuming creation of instance masks by improving interactive polygon annotation and adapting across domains. Its architectural, training, and resolution advances improve Cityscapes accuracy, reduce clicks, and support online fine-tuning for novel datasets.

  • 20-30 seconds per object is typically required for human annotators to collect ground-truth instance masks.
  • Polygon-RNN predicts sparse polygon vertices instead of pixel-wise masks, enabling efficient human interaction during annotation.
  • Polygon-RNN++ improves the original model through a redesigned neural architecture, reinforcement-learning training, and higher-resolution polygon output using a Graph Neural Network.
  • 10% mean IoU improvement over the original approach is achieved in fully automatic Cityscapes annotation.
  • 50% fewer clicks are required in interactive mode, while cross-domain evaluation and online fine-tuning improve annotation of novel datasets.

2. Related Work

Prior interactive segmentation methods use scribbles, boxes, or pixel-wise labeling, whereas Polygon-RNN uses sparse polygons and an RNN to produce editable object outlines. Polygon-RNN++ extends this polygon-based interactive labeling approach.

  • Earlier interactive methods used scribbles or bounding boxes to guide pixel-wise foreground/background segmentation.
  • Polygon representations are sparse, allowing annotators to correct large image regions by moving only a few vertices.
  • An RNN represents object shape sequentially, which helps model boundaries in ambiguous regions such as shadows and saturation.
  • Other polygon methods link detected boundary fragments or combine polygonal superpixels, while Polygon-RNN targets interactive labeling with neural polygon generation.

3. Polygon-RNN++

Polygon-RNN++ uses an encoder-decoder architecture to generate object polygons from an annotator-provided bounding box. Candidate selection, visual attention, human corrections, and graph-based refinement support interactive high-resolution annotation.

  • The model takes an annotator-provided bounding box, crops an enlarged image region, and decodes polygon vertices sequentially from CNN features.
  • A redesigned encoder predicts the first vertex, while the recurrent decoder uses visual attention to generate subsequent polygon vertices.
  • A learned evaluator selects the best polygon from decoder candidates before a Graph Neural Network refines it with additional high-resolution vertices.
  • Human annotators can correct an erroneous vertex, and the corrected vertex is fed back to improve later predictions.

3.1. Residual Encoder with Skip Connections

The residual encoder increases feature-map resolution while preserving receptive fields and combines low-level detail with high-level semantics through skip connections. Its outputs feed separate graph and recurrent components.

  • Reducing network stride and adding dilation increase output feature-map resolution without reducing individual neurons’ receptive fields.
  • Skip connections combine low-level edges and corners with high-level semantic information.
  • The residual encoder architecture feeds one tensor to the Graph Neural Network and another to the RNN decoder.
  • Feature maps with different spatial dimensions are bilinearly upsampled before concatenation into the final skip features.

3.2. Recurrent Decoder

Polygon-RNN++ uses a ConvLSTM-based recurrent decoder to generate polygon vertices sequentially, with attention-weighted visual features and a learned first-vertex prediction branch.

  • Recurrent decoding: The ConvLSTM decoder models the sequence of 2D polygon vertices and uses attention while predicting the first vertex within the same network.The decoder uses two ConvLSTM layers and represents each output as a grid location or an end-of-sequence token.
  • Attention-weighted features: At each time step, hidden-state tensors are mapped to the spatial grid and combined with skip features through a Hadamard-product attention mechanism.The attention mechanism produces a weighted feature map from the skip feature tensor and the ConvLSTM hidden states.
  • Decoder evaluation: The evaluator network is depicted as predicting the quality of polygon outputs produced by the RNN decoder.The figure caption identifies the evaluator’s role but does not specify its inputs or selection procedure.
  • First-vertex prediction: A separate branch predicts the first vertex because the initial polygon location is not uniquely determined by a previous vertex and direction.This branch follows the skip-layer architecture and predicts edge and vertex information at resolution D × D.

3.3. Training using Reinforcement Learning

The authors use maximum-likelihood training only for initialization, then fine-tune the polygon decoder with reinforcement learning so it optimizes IoU while learning from its own predictions.

  • Motivation and strategy: MLE training is used only for initialization before reinforcement-learning fine-tuning addresses loss mismatch and teacher-forcing exposure bias.The RL phase optimizes IoU directly and exposes the model to predictions generated by its own policy.
  • Policy formulation: The recurrent decoder acts as a sequential decision-making agent whose policy selects vertices and receives a final reward.The policy is parameterized by the encoder-decoder parameters θ.
  • Reward: The reward is the IoU between the mask enclosed by the generated polygon and the ground-truth mask.This makes the reinforcement-learning objective correspond to the final evaluation metric.
  • Gradient estimation: REINFORCE estimates the policy gradient, while a learned baseline reduces the variance of single-sample Monte-Carlo estimates.The self-critical baseline uses the model’s test-time inference reward.
  • Self-critical training: The self-critical gradient compares the sampled polygon’s reward with the reward from greedy decoding, and temperature τ controls vertex-sampling randomness.The experiments set τ = 0.6 for policy sampling.

3.4. Evaluator Network

The evaluator network addresses uncertainty in the first vertex by scoring multiple candidate polygons, allowing inference to select the candidate predicted to have the highest IoU.

  • Candidate generation: The first vertex strongly influences subsequent predictions, especially for occluding objects, so the model generates multiple candidate starting points.The preferred starting point is far from occlusion boundaries so the polygon follows the object of interest.
  • Evaluator design: The evaluator network estimates polygon quality by predicting IoU from skip features, the final ConvLSTM state, and the predicted polygon.It is implemented as another branch with two 3×3 convolutional layers and a fully connected layer.
  • Evaluator training: The evaluator is trained separately after reinforcement-learning fine-tuning converges, using mean squared error against the sampled polygon’s IoU.Training samples polygons with τ = 0.3 to encourage diversity, and the evaluator is not used as the RL baseline.
  • Inference: At inference, beam search generates one polygon for each of the K top first-vertex predictions, and the evaluator selects the best-scoring polygon.The experiments use K = 5; using the evaluator at every decoding step would make inference impractically long.
  • Interactive correction: During interactive annotation, each corrected vertex is fed back to the model, which re-predicts the remaining polygon.This follows the protocol used in the original Polygon-RNN work.

3.5. Upscaling with a Graph Neural Network

Polygon-RNN++ uses a GGNN to refine RNN-generated polygons at higher resolution by adding midpoint nodes, propagating messages over a cyclic graph, and predicting relative vertex offsets.

  • Motivation: The original model predicts polygons at D × D resolution with D = 28, while the GGNN generates polygons at much higher resolution.The higher-resolution graph model is introduced to overcome the original model’s output-resolution limitation.
  • Resolution and topology: Higher-resolution ground-truth polygons can contain additional vertices because the original training polygons remove co-linear vertices at the target resolution.This may change the topology of the polygon at higher resolution.
  • Graph construction: The upscaling model converts RNN vertices into graph nodes, inserts midpoint nodes between consecutive vertices, and connects the sequence into a cycle.Neighboring nodes use three edge types with separate message-propagation weights.
  • Propagation and prediction: The GGNN propagates information between graph nodes before predicting each node’s relative offset at higher resolution.Its graph consists of nodes and edges, with propagation and output models for message passing and prediction.
  • Node observations: Node observations are extracted from patches around scaled vertex locations, and each output is predicted on a relative D′ × D′ spatial grid.The observations come from a 112 × 112 × 256 feature map produced by an added branch.
  • Implementation: The implementation uses S = 1, D′ = 112, 256-dimensional GGNN hidden states, and T = 5 propagation steps.During training, vertices deviating by more than 3 cells are replaced with ground-truth vertices.

3.6. Annot. New Domains via Online Fine-Tuning

The paper proposes online fine-tuning for applying a model trained on one dataset to novel datasets. As new data arrives, annotators correct predictions, and the model incrementally updates using corrected and previously seen data.

  • Online fine-tuning adapts the model to novel datasets as new data arrives and annotators correct wrong predictions.The procedure is designed for annotation scenarios in which the model is used on a new dataset.
  • Corrected targets for MLE training are smoothed with a Manhattan distance transform truncated at distance 2.
  • Each data chunk is read, annotated and corrected, augmented with sampled previously seen data, then used for sequential MLE, RL, and evaluator training.The updated model becomes the starting point for processing the next chunk.

4. Experimental Results

Polygon-RNN++ is evaluated for automatic and interactive annotation, robustness to box noise, cross-domain generalization, online fine-tuning, and human annotation speed. Across these settings, it improves segmentation quality or reduces annotation effort, while relying on user-provided bounding boxes.

  • Scope and Limitations: The evaluation assumes user-provided ground-truth boxes, and interactive annotation would benefit from allowing humans to add or remove points.The study also analyzes robustness to noise in these boxes.
  • In-Domain Annotation: Polygon-RNN++ outperforms Polygon-RNN by almost 10% IoU in automatic Cityscapes evaluation and achieves the best performance for each class.It also exceeds reported human agreement on cars and reaches human-level performance for trucks and buses using car agreement as a proxy.
  • In-Domain Annotation: At T2 = 0.7, Polygon-RNN++ achieves over 80% mIoU with only 5 clicks per object, reducing clicks by more than 50% versus Polygon-RNN.At T2 = 0.8, it remains more accurate than Polygon-RNN at T2 = 1.0.
  • In-Domain Annotation: With 5-15% bounding-box noise, Polygon-RNN++ outperforms previous baselines reported with perfect bounding boxes, demonstrating robustness to moderate and large noise.The model is also described as very robust to 0-5% noise.
  • Instance-Level Segmentation: Full-image instance segmentation with Faster R-CNN boxes achieves 22.8% AP and 42.6% AP50, rising to 25.49% AP and 45.47% AP50 after semantic post-processing.The post-processing combines the predicted class-semantic map and instance prediction with a logical “and” operation.
  • Out-of-Domain Imagery: Across out-of-domain datasets, online fine-tuning overcomes domain shifts with one data chunk and reduces clicks by at least 65%, nearly 100% on Sunnybrook Cardiac MR.The reported chunks contain 40 Sunnybrook, 3 ssTEM, 200 ADE, and 20 Aerial images.
  • Interaction with Human Annotators: Human annotators are 3x faster with the tool, with only slightly lower IoU agreement than ground truth.The evaluated tool was a basic version with scope for engineering improvements.

5. Conclusion

Polygon-RNN++ extends Polygon-RNN into an interactive annotation model with improvements that enhance automatic and interactive performance, robustness, and cross-domain adaptation.

  • Polygon-RNN++ significantly outperforms Polygon-RNN in both automatic and interactive annotation modes.The model also shows robustness to noisy annotators and generalizes to novel domains.
  • The model can adapt effectively to novel, out-of-domain datasets through a simple online fine-tuning scheme.

6. Appendix

The appendix evaluates Polygon-RNN++ across training dynamics, decoding, output resolution, full-image segmentation, interactive correction, and out-of-domain imagery, while documenting important limitations.

  • Training with RL: Directly optimizing IoU with reinforcement learning increases mean IoU while reducing polygon length and self-intersections.
  • Evaluator Network: Beam search improves greedy decoding by 0.5%, while the evaluator network adds a further 0.2%.
  • Output Resolution and Sensitivity to T of GGNN: The 224 × 224 GGNN output is only 0.02% better than 112 × 112 but requires longer training, motivating the 112 × 112 choice.
  • Full-image Instance-Level Segmentation on Cityscapes: Polygon-RNN++ uses predicted boxes from FasterRCNN to produce full-image instance segmentation polygons.
  • Limitations: The model predicts only one polygon per bounding box, so multi-component masks from occlusion are heavily penalized.Semantic information can be combined with instance polygons, yielding 25.49% AP and 45.47% AP50 on the test set.
  • Interactive mode in Cityscapes: The interactive simulation reports required clicks and achieved IoU, with most predictions corrected successfully using five clicks.
  • Automatic Mode in Out-of-Domain Imagery: Out-of-domain qualitative evaluations use a Cityscapes-trained model without fine-tuning across KITTI, general-scene, aerial, and medical datasets.
  • Automatic Mode and Online Fine-Tuning: Online fine-tuning compares predictions before and after adaptation, with fine-tuned results generated after the final data chunk.
Loading 1803.09693v1…