PaperlayerView exact revision on arXiv

Source-linked AI summary

What's the Point: Semantic Segmentation with Point Supervision

Amy Bearman, Olga Russakovsky, Vittorio Ferrari, Li Fei-Fei

arXiv:1506.02106v5cs.CV

TL;DR

Semantic segmentation requires balancing annotation cost against model accuracy, because detailed pixel labels are expensive while cheaper image-level labels are less accurate. The paper trains CNNs with point supervision and an objectness prior, achieving higher accuracy than alternative supervision types under comparable annotation budgets. On PASCAL VOC 2012, the combined approach improves mIOU over image-level supervision.

  • Problem

    Semantic segmentation needs accurate per-pixel training labels, but obtaining detailed annotations is time-consuming and cheaper image-level supervision is less accurate.

  • Method

    The paper trains a CNN using point-level supervision and incorporates an objectness prior directly into the training loss to guide object-extent prediction.

  • Results

    12.9% mIOU improvement over image-level labels is reported on PASCAL VOC 2012, while point supervision outperforms image-level, squiggle-level, and full supervision by 2.7 −20.8% mIOU at a fixed annotation budget.

  • Takeaways & Limitations

    Point-level supervision provides a time-efficient alternative that trains more accurate segmentation models than other popular supervision forms under the same annotation-time budget.

  • Takeaways & Limitations

    The AllPoints annotation task left 7.9% of ground-truth instances unannotated and placed 14.8% of clicks on the wrong object class.

Abstract

from arXiv · show

The semantic image segmentation task presents a trade-off between test time accuracy and training-time annotation cost. Detailed per-pixel annotations enable training accurate models but are very time-consuming to obtain, image-level class labels are an order of magnitude cheaper but result in less accurate models. We take a natural step from image-level annotation towards stronger supervision: we ask annotators to point to an object if one exists. We incorporate this point supervision along with a novel objectness potential in the training loss function of a CNN model. Experimental results on the PASCAL VOC 2012 benchmark reveal that the combined effect of point-level supervision and objectness potential yields an improvement of 12.9% mIOU over image-level supervision. Further, we demonstrate that models trained with point-level supervision are more accurate than models trained with image-level, squiggle-level or full supervision given a fixed annotation budget.

1 Introduction

Semantic segmentation faces a trade-off between annotation cost and accuracy. The paper proposes point-level supervision with an objectness prior, reporting stronger accuracy than cheaper or equally budgeted alternatives.

  • Motivation: Per-pixel annotations support accurate segmentation but are time-consuming, while image-level labels reduce effort at the cost of accuracy.A single object outline takes 54–79 seconds, and typical indoor scenes contain 23 objects.
  • Motivation: Point annotation offers a natural step toward stronger supervision with negligible additional time compared to image-level labels.The paper notes that pointing is a natural and consistent way for humans to refer to objects.
  • Approach: The method incorporates one annotated point per object class into a CNN training loss for semantic segmentation.The approach extends a state-of-the-art CNN framework and was evaluated using points collected on PASCAL VOC 2012.
  • Approach: An objectness prior is added directly to the loss to help the CNN distinguish objects from background and infer object extent.The prior provides a probability that a pixel belongs to an object.
  • Results: 12.9% mIOU improvement is achieved over image-level labels on PASCAL VOC 2012 from the combined point-supervision and objectness contributions.The reported metric is mean intersection over union (mIOU).
  • Results: 2.7 −20.8% mIOU gains are reported over image-level, squiggle-level, and full supervision under a fixed annotation budget.The comparison is made across models trained with different supervision types while holding annotation budget fixed.

2 Related Work

Prior work reduces semantic-segmentation annotation effort through multiple weak or semi-supervised signals, but supervision strength remains tied to an accuracy–cost trade-off. This paper positions point supervision as an intermediate alternative used directly during CNN training.

  • Types of Supervision: Weak and semi-supervised segmentation has used eye tracks, squiggles, web tags, size constraints, heterogeneous annotations, image-level labels, and bounding boxes.These approaches target reduced up-front annotation time.
  • Annotation Trade-off: Higher supervision levels generally produce better models but require costly and scarce strongly supervised datasets.The related-work discussion frames annotation time and accuracy as a trade-off.
  • Point Supervision: Point supervision is proposed as an intermediate form that adds negligible annotation time to image-level labels while achieving better results.The paper contrasts its approach with prior point supervision that used points as a unary potential in a CRF.
  • Interactive Segmentation: Interactive segmentation collects extra annotations at test time, whereas this method collects user points once and uses them only during training.Prior interactive methods used points or free-form squiggles to refine test-time segmentation.

3 Semantic Segmentation Method

The method extends CNN semantic segmentation training from image-level labels to point-level supervision and adds an objectness prior to help infer object extent.

  • CNN Framework: The CNN maps an input image of size W × H to a W × H × N score map, which is converted into per-pixel predictions at test time.Predictions can use the highest-scoring class at each pixel or more complex post-processing.
  • Point-Level Supervision: Point-level supervision labels a small set of supervised pixels with object classes while other pixels are only known to belong to image-level class sets.The framework generalizes image-level losses to use sparse point annotations.
  • Point-Level Supervision: The point-supervision loss weights each supervised pixel through α_i, supporting uniform, annotator-confidence, or point-order weighting.When annotators mark every instance, earlier points receive higher weight because they are more likely to indicate larger objects.
  • Objectness Prior: The objectness prior estimates whether each pixel belongs to any object rather than background, guiding the model toward plausible object extents.At high objectness probabilities, the loss favors object classes; at low probabilities, it favors background.
  • Objectness Prior: The objectness term is incorporated directly into CNN training and requires no additional human supervision beyond pre-training the generic objectness detector.This distinguishes the approach from prior uses of objectness during region selection, graphical-model inference, or post-CNN inference.

4 Crowdsourcing Annotation Data

The study measures annotation effort and quality for point- and squiggle-level supervision alongside image-level and full supervision on PASCAL VOC 2012. Point annotations are nearly as fast as image-level labels and substantially faster than full supervision, while squiggles take longer.

  • Annotation Time: 20.0 sec/img is the estimated annotation time for image-level supervision across 20 PASCAL VOC object classes.The estimate assumes 1 second per class.
  • Annotation Time: 239.7 sec/img is the estimated annotation time for full supervision in PASCAL VOC 2012.The estimate combines absent-class labeling with segmentation of 2.8 object instances per image at 79 seconds per object.
  • Point-Level Supervision: 22.1 seconds per image is the labeling time for 1Point supervision, only 1.1–1.2× slower than image-level labels and more than 10× cheaper than full supervision.1Point records one click per present object class.
  • Annotation Quality: 7.2% of 1Point clicks were on a different class label, with another 0.8% on unclassified difficult pixels.For AllPoints, 7.9% of ground-truth instances were missed and 14.8% of clicks were on the wrong object class.
  • Squiggle-Level Supervision: 1.6× more time is required for 1Squiggle supervision than for 1Point supervision, and 1.7× more than for image-level labels.Drawing one squiggle per present class takes 34.9 seconds per image.
  • Annotation Quality: 6.3% of squiggle-labeled pixels were on the wrong object class, and an additional 1.4% were on difficult pixels.The study used quality control procedures similar to those for point-level supervision.

5 Experiments

Experiments evaluate point supervision and an objectness prior for semantic segmentation, including their accuracy effects, supervision variations, stronger-supervision hybrids, and fixed-budget tradeoffs. Results on PASCAL VOC 2012 show complementary benefits from point annotations and objectness, with hybrid supervision outperforming squiggle-level supervision under the reported annotation costs.

  • Experimental setup: The experiments use PASCAL VOC 2012 with 10,582 training, 1,449 validation, and 1,456 test images, reporting mIOU over 21 classes.The model is a fully convolutional VGG-based network producing dense pixel predictions.
  • Point-level supervision: 5.3% mIOU: adding one annotated point per class raises image-level supervision from 29.8% to 35.1% mIOU.Only 1.5 pixels per image are supervised on average; all other pixels remain unsupervised.
  • Objectness prior: 2.4% mIOU: the objectness prior raises image-level supervision from 29.8% to 32.2% mIOU by helping infer object extent.Image-level models tend to overestimate the area occupied by objects.
  • Synergy: +13% mIOU: combining point supervision and objectness improves performance from 29.8% to 42.7% mIOU.With point supervision, objectness raises 35.1% to 42.7% mIOU; conversely, one point per class raises the objectness model 32.2% to 42.7%.
  • Point-level supervision variations: 42.7% mIOU: using all instance points without weighting does not improve over one point per class, while weighting them yields 43.4% mIOU.The weighting assigns the first instance weight 1, the second 1/2, and so on.
  • Stronger supervision: 53.1% mIOU: a hybrid using 100 fully supervised and 10,482 point-supervised images is 9.8x cheaper than full supervision and outperforms squiggle-level supervision at 49.1% mIOU.The hybrid costs 24.5 seconds per image versus 35.2 seconds for squiggle-level supervision and 239.7 seconds for full supervision.
  • Fixed-budget comparisons: 43.6% mIOU: point-level supervision improves accuracy over the cited image-level weak-supervision methods, whose results range from 25.7% to 40.6% mIOU.The comparison includes methods using approximately 67 to 400 hours of annotation.

6 Conclusions

The paper proposes time-efficient point supervision for semantic segmentation and reports better accuracy than other supervision types under equal annotation budgets.

  • Point-level supervision is presented as a time-efficient approach for semantic image segmentation.
  • The method enables more accurate segmentation models than other popular supervision forms given the same annotation time budget.
  • An objectness prior is introduced directly into the CNN loss function to help infer object extent.
  • The approach is evaluated on the PASCAL VOC 2012 dataset.
  • The authors release their dataset and annotation interfaces for future large-scale semantic segmentation efforts.
Loading 1506.02106v5…