Source-linked AI summary
ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation
Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, Jian Sun
TL;DR
Precise mask annotation is laborious, so the paper trains semantic-segmentation convolutional networks directly from sparse scribbles. A graphical model propagates scribble information to unmarked pixels, producing competitive PASCAL VOC results and strong PASCAL-CONTEXT performance with inexpensive extra scribbles.
Problem
Precise mask-level annotation for large-scale image data is painstaking and inefficient, while interactive segmentation still requires annotators to make many touch-ups.
Method
The method jointly propagates information from scribbles using a graphical model and learns convolutional-network parameters for semantic segmentation.
Results
Scribble supervision achieves higher accuracy than image-level or box-level weak supervision on PASCAL VOC and higher PASCAL-CONTEXT accuracy than methods unable to use scribbles.
Takeaways & Limitations
Scribbles provide an inexpensive annotation approach that supports competitive object segmentation and effective annotation of stuff categories with no well-defined shape.
Abstract
from arXiv · showhide
Large-scale data is of crucial importance for learning semantic segmentation models, but annotating per-pixel masks is a tedious and inefficient procedure. We note that for the topic of interactive image segmentation, scribbles are very widely used in academic research and commercial software, and are recognized as one of the most user-friendly ways of interacting. In this paper, we propose to use scribbles to annotate images, and develop an algorithm to train convolutional networks for semantic segmentation supervised by scribbles. Our algorithm is based on a graphical model that jointly propagates information from scribbles to unmarked pixels and learns network parameters. We present competitive object semantic segmentation results on the PASCAL VOC dataset by using scribbles as annotations. Scribbles are also favored for annotating stuff (e.g., water, sky, grass) that has no well-defined shape, and our method shows excellent results on the PASCAL-CONTEXT dataset thanks to extra inexpensive scribble annotations. Our scribble annotations on PASCAL VOC are available at http://research.microsoft.com/en-us/um/people/jifdai/downloads/scribble_sup
1. Introduction
Large-scale mask annotation is accurate but time-consuming, motivating sparse scribbles as a more efficient and user-friendly alternative for training segmentation models. Scribble supervision avoids precise boundary tracing while providing location information and supporting objects with ambiguous shapes.
- Motivation: Precise mask annotation can take experienced annotators minutes per image, limiting the amount of available mask-level training data.Interactive tools still require drawing polygons along object boundaries, sometimes with dozens of vertices.
- Motivation: Scribbles are widely used for interactive image segmentation and are recognized as a user-friendly form of interaction.The paper points to both academic research and Adobe Photoshop’s Quick Selection tool as examples.
- Motivation: Scribbles let annotators mark confident object regions without carefully outlining boundaries or performing inefficient mask touch-ups.Annotators can drag the cursor through object centers and provide only a few easy-to-draw scribbles.
- Motivation: Scribbles are also suitable for stuff such as water, sky, and grass, whose boundaries may be ambiguous or lack a well-defined shape.This makes scribble annotation applicable beyond objects with clear contours.
- Approach: The proposed method trains convolutional networks from scribbles by propagating their information to unmarked pixels through a graphical model.The model uses spatial constraints, appearance, and semantic content, while the network and graphical model provide supervision to each other through alternating optimization.
- Results: On PASCAL VOC, scribble supervision degrades reasonably relative to mask supervision and outperforms methods using image-level or box-level annotations.Additional inexpensive scribbles on VOC 2007 also improve PASCAL-CONTEXT results, which includes both objects and stuff.
2. Scribble-Supervised Learning
Scribble-supervised learning jointly propagates sparse scribble labels to unknown pixels and trains a fully convolutional network. A graphical model combines scribble constraints, network predictions, and appearance-based pairwise terms, optimized by alternating label propagation and network learning.
- Problem setup: Scribbles label only sparse pixels, leaving unannotated pixels unknown and requiring propagation to obtain full image annotations.A scribble is a set of category-labeled pixels; the resulting labels are propagated to other pixels during training.
- Graphical model: The graphical model represents training images as super-pixel graphs whose vertices are super-pixels and whose edges encode similarity between adjacent regions.Unary terms operate on individual super-pixels, while pairwise terms model relationships between neighboring super-pixels.
- Objective functions: The unary cost combines scribble constraints with fully convolutional network predictions, while pairwise costs propagate labels using spatially adjacent regions and appearance similarity.Color and texture histograms define appearance similarity; differing labels cost more when adjacent super-pixels look more alike.
- Optimization: Alternating optimization fixes network parameters to infer super-pixel labels, then fixes labels to train the fully convolutional network.Graph cuts solve the labeling step, and the resulting full-image annotations supervise FCN training; network outputs then update the graph's unary term.
- Optimization: Three iterations empirically suffice, after which additional iterations provide negligible improvements.As network predictions become more reliable, propagated labels become more accurate and in turn improve network learning.
- Inference: At inference, only the trained FCN is applied, because the super-pixels and graphical model are used during training בלבד.The method additionally uses a CRF to post-process inference results.
3. Experiments
Experiments evaluate scribble annotation, propagation, robustness, and performance across PASCAL VOC and PASCAL-CONTEXT. The method remains competitive with mask-, box-, and point-supervised alternatives while exploiting inexpensive scribbles for additional training data.
- Annotating Scribbles: Scribble annotations substantially reduce annotation effort, with average scribble length around 70% of the longer side of an object's bounding box.Annotations were collected through Amazon Mechanical Turk on PASCAL VOC and PASCAL-CONTEXT.
- Strategies of utilizing scribbles: 63.1% mIoU is achieved by joint propagation and network learning, about 10% higher than two-step mask-generation solutions.Traditional interactive segmentation methods model low-level color and spatial information, whereas network feedback provides semantic information during propagation.
- Strategies of utilizing scribbles: 60.5% is obtained without pairwise terms, showing that pairwise information is important for propagating labels beyond the scribbles.Without pairwise terms, propagation reduces to winner-take-all selection based on network predictions.
- Sensitivities to scribble quality: 51.6% remains attainable when scribbles are shortened to spots, indicating graceful degradation and robustness to scribble quality.Shortened scribbles are synthesized by reducing their length; a ratio of 0 corresponds to spots.
- Comparisons with other weakly-supervised methods: On PASCAL VOC, scribble supervision reaches accuracy on par with box supervision and exceeds the reported 46.1% point-supervised result with spots.The paper reports 60.6% for WSSL box supervision, 62.0% for BoxSup, and 51.6% for the spot-only result.
- Comparisons with using masks: 73.1% is obtained on the PASCAL VOC 2012 test set using mask annotations plus VOC 2007 scribbles, without the extra COCO mask data.On PASCAL-CONTEXT, the method reaches 42.0% with 5k masks and 10k scribbles, compared with 40.5% for BoxSup using 133k extra boxes.
4. Conclusion and Future Work
The paper presents scribble-supervised semantic segmentation through graphical-model propagation and reports test-set accuracies above 74.8% and 74.1%. The formulation also applies to other weak supervision types, while the figures compare mask, scribble, and mixed annotation settings.
- The method optimizes a graphical model that propagates information from scribbles to support semantic segmentation training.
- Test-set accuracy exceeds 74.8% and 74.1% in the reported evaluations.
- The VOC 2012 and PASCAL-CONTEXT figures compare mask-only, scribble-only, and mixed training annotations, including extra VOC 2007 scribbles.
- The formulation is applicable to other weak supervision types, including box-level and image-level annotations.