PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation

George Papandreou, Liang-Chieh Chen, Kevin Murphy, Alan L. Yuille

arXiv:1502.02734v3cs.CV

TL;DR

Pixel-level annotation makes semantic segmentation expensive, motivating learning from weak labels or few strong labels. The paper develops EM-based training methods that achieve competitive PASCAL VOC 2012 performance, nearly matching full supervision in semi-supervised settings while reducing annotation effort.

  • Problem

    Training DCNN segmentation models typically requires expensive, time-consuming pixel-level annotations, while weak-label methods have lagged fully supervised performance.

  • Method

    The paper develops online EM algorithms that estimate latent pixel labels under weak-label constraints and optimize DCNN parameters with stochastic gradient descent.

  • Results

    68.5% is achieved using 2.9k pixel-level and 9k image-level annotated images, only 2% below full supervision; combining PASCAL and MS-COCO annotations reaches 73.9% IoU.

  • Takeaways & Limitations

    Weak annotations can support competitive segmentation, especially when combined with a small amount of pixel-level supervision or annotations from another dataset.

  • Takeaways & Limitations

    The EM-Fixed variant assumes that log P(z|y) factorizes over pixel positions.

Abstract

from arXiv · show

Deep convolutional neural networks (DCNNs) trained on a large number of images with strong pixel-level annotations have recently significantly pushed the state-of-art in semantic image segmentation. We study the more challenging problem of learning DCNNs for semantic image segmentation from either (1) weakly annotated training data such as bounding boxes or image-level labels or (2) a combination of few strongly labeled and many weakly labeled images, sourced from one or multiple datasets. We develop Expectation-Maximization (EM) methods for semantic image segmentation model training under these weakly supervised and semi-supervised settings. Extensive experimental evaluation shows that the proposed techniques can learn models delivering competitive results on the challenging PASCAL VOC 2012 image segmentation benchmark, while requiring significantly less annotation effort. We share source code implementing the proposed system at https://bitbucket.org/deeplab/deeplab-public.

1. Introduction

The paper addresses the annotation bottleneck in DCNN-based semantic segmentation by developing EM methods that use bounding boxes, image-level labels, or mixtures with pixel-level annotations. The approach achieves competitive PASCAL VOC 2012 performance, including 73.9% IOU when combining PASCAL and MS-COCO annotations.

  • Motivation: DCNNs achieve the best reported results on the challenging Pascal VOC 2012 semantic segmentation benchmark.Semantic segmentation assigns a semantic label to every pixel.
  • Motivation: Pixel-level annotation is expensive, whereas bounding boxes and image-level labels are easier to collect.Bounding boxes around each class instance are about 15 times faster/cheaper to collect than pixel-level labeling.
  • Results: 62.2% IOU is achieved on the PASCAL VOC 2012 test set using foreground/background segmentation of bounding box annotations.The result is obtained without requiring fully pixel-level annotated training images.
  • Method: The paper develops online EM algorithms for weakly supervised and semi-supervised DCNN segmentation training.The algorithms alternate between estimating latent pixel labels under weak-annotation constraints and optimizing DCNN parameters with SGD.
  • Results: 73.9% IOU is reached on PASCAL VOC 2012 by combining annotations from the PASCAL and MS-COCO datasets.With 2.9k pixel-level images and 9k image-level annotated images, the method can almost match the fully supervised system, though the passage is truncated.

2. Related work

Prior work has explored weakly supervised semantic segmentation using image-level labels, bounding boxes, or both. Unlike related MIL-based DCNN methods, this work uses EM to infer latent segmentations and shows that foreground/background segmentation methods can support DCNN training.

  • Image-level weak supervision: Recent weakly supervised DCNN methods use MIL-based algorithms, whereas this work employs EM to infer latent image segmentations from weak labels.The EM approach is described as similar to prior work that incorporates weak labels during latent-segmentation inference.
  • Bounding-box supervision: Bounding boxes have been used for semantic segmentation, jointly with image-level labels, and in interactive segmentation.Prior work also attained human-level car-segmentation accuracy using 3D bounding boxes.
  • Bounding-box supervision: Foreground/background segmentation methods can estimate object segments accurately enough to train a DCNN semantic segmentation system.A closely related method used MCG, which requires pixel-level training, to infer object masks from bounding-box labels during DCNN training.

3. Proposed Methods

The proposed methods train DeepLab DCNNs for semantic segmentation from weak image-level or bounding-box labels, and from mixtures of weak and strong annotations, using EM-based latent segmentation estimation. A fully connected CRF is used only at test time in the presented training procedures.

  • Base model: DeepLab predicts per-pixel label distributions with a DCNN, followed by a fully connected CRF that smooths predictions while preserving image edges.The methods train only the DCNN parameters from weak labels and use the CRF at test time.
  • Image-level supervision: For image-level labels, hard EM treats pixel segmentations as latent, estimates them in the E-step, and optimizes the resulting pseudo-ground-truth segmentations by mini-batch SGD.The expected complete-data log-likelihood is approximated by assigning each pixel its most likely latent label, then updating θ using that segmentation.
  • Image-level supervision: EM-Fixed factorizes the observation model over pixels, using fixed biases that favor present foreground classes over background to encourage object coverage and avoid all-background assignments.The method sets b_l = b_fg for foreground classes and b_0 = b_bg, with b_fg > b_bg > 0.
  • Image-level supervision: EM-Adapt uses cardinality potentials and adaptive image- and class-dependent biases to enforce prescribed foreground and background area proportions, preventing background domination.For present class l, it encourages at least a ρ_l portion of the image to receive that class, while excluding absent classes.
  • Bounding-box supervision: For bounding boxes, Bbox-Rect labels box interiors as positive, Bbox-Seg filters background with CRF-based foreground/background segmentation, and Bbox-EM-Fixed refines these estimates during training.Bbox-EM-Fixed is an EM-Fixed variant that boosts present foreground object scores only within the bounding-box area.
  • Semi-supervised training: For semi-supervised training, each mini-batch contains a fixed proportion of strongly and weakly annotated images, with EM estimating latent segmentations for weak images at every iteration.This combines the methods for strong and weak supervision in a single training procedure.

4. Experimental Evaluation

Experiments on PASCAL VOC 2012 evaluate weakly supervised and semi-supervised training using mean pixel intersection-over-union across 21 classes. The results show substantial gains from EM-based learning, limited strong annotations, bounding-box processing, and cross-dataset augmentation.

  • Experimental setup: Evaluation uses mean pixel intersection-over-union averaged across 21 classes on PASCAL VOC 2012 validation and test sets.The dataset includes 1,464 train, 1,449 validation, and 1,456 test images, plus augmented sets of 10,582 and 12,031 images.
  • Strong supervision baseline: 70.3% test IOU is achieved by the reproduced DeepLab-CRF baseline trained with strong pixel-level annotations.The corresponding validation score is 67.6%.
  • Image-level supervision: 38.2% validation IOU for EM-Adapt versus 20.8% for EM-Fixed shows that EM-Adapt performs much better with only image-level annotations.Adding 1,464 pixel-level and 9,118 image-level annotations raises EM-Fixed to 64.6%, compared with 62.5% using only the strong annotations.
  • Image-level supervision: 39.6% test IOU for EM-Adapt improves over MIL-FCN by 13.9 percentage points using image-level annotations.MIL-sppxl reaches 35.8% with additional low-level superpixels, while 2.9k strong plus 9k image-level annotations yields 68.5%.
  • Bounding-box supervision: 62.2% test IOU for Bbox-Seg substantially exceeds Bbox-Rect at 54.2%, while 1.4k strong annotations raise performance to 66.6%.Bbox-Seg improves over Bbox-Rect by 8.1% on validation, and the 1.4k-strong result slightly exceeds BoxSup’s 66.2%.
  • Cross-dataset training: 71.7% test IOU is obtained by Cross-Joint training with all 123,287 strong MS-COCO annotations, while cross-dataset augmentation improves 4.1% over the best PASCAL-only result.The top multi-scale model reaches 73.9%, and joint PASCAL–MS-COCO training improves further to 74.7% with end-to-end DCNN and CRF learning.

5. Conclusions

The paper studies training state-of-the-art semantic image segmentation models with weak or partial annotations, evaluating the approach extensively on PASCAL VOC 2012. Image-level labels alone appear insufficient for high-quality models, whereas bounding boxes combined with careful training-set segmentation inference suffice for competitive performance.

  • Image-level weak annotation alone appears insufficient to train a high-quality semantic segmentation model.
  • Weak bounding-box annotation plus careful segmentation inference on training images suffices to train a competitive model.
  • Extensive experiments on the challenging PASCAL VOC 2012 dataset support these conclusions.

A. E-Step with Cardinality Constraints: Details of our EM-Adapt Algorithm

EM-Adapt trains segmentation models from image-level annotations by using cardinality potentials and adaptively chosen label biases in the E-step. The procedure approximately enforces prescribed area constraints, suppresses absent classes, and runs in linear time for single-label images.

  • Model representation: Image-level annotations encode which labels are present, while the latent variable y specifies the segmentation label at each image position.z_l = 1 indicates that label l appears somewhere in the image.
  • Cardinality constraints: The cardinality potential encourages each present class to occupy at least its prescribed image-area proportion and assigns no pixels to absent classes.The parameters are ρ_l = ρ_fg for foreground classes and ρ_0 = ρ_bg for the background.
  • Adaptive E-step: In the E-step, EM-Adapt assigns each pixel the label maximizing its network score plus an adaptively selected bias.The biases are chosen so prescribed proportions are assigned to present background and foreground classes.
  • Single-label case: For a single present label, the method sets other biases to zero, computes per-position maximum competing scores, and chooses the label bias from a prescribed score-difference percentile.This enforces at least ρ_l of the image area for the present label.
  • Computational procedure: O(M) is the cost of the single-label procedure, linear in the number of pixels.With multiple present labels, the procedure is applied sequentially, visiting background first and then foreground labels in random order; absent labels receive bias −∞.

B. Effect of Field-Of-View

The study examines how Field-Of-View (FOV) affects DeepLab-CRF training under weak and semi-supervised settings. Smaller FOV performs better with limited supervision, while larger FOV becomes advantageous with more annotations and consistently helps when combining PASCAL VOC and MS-COCO data.

  • Method: The atrous algorithm controls FOV by adjusting the input stride at VGG-16’s first fully connected layer.A larger input stride increases the model’s FOV.
  • EM-Adapt: A 3×3 kernel matches 7×7 performance while training 3.4 times faster.This kernel size was therefore used to explore varying FOV values with EM-Adapt (Weak).
  • Network architectures: The small-FOV architecture uses a 4×4 kernel and input stride 4 for a 128×128 FOV, whereas the large-FOV architecture uses a 3×3 kernel and input stride 12 for a 224×224 FOV.These are the two architectures compared in the experiments.
  • Image-level annotations: Small FOV performs better with limited image-level and pixel-level supervision, but large FOV performs better when more than 750 pixel-level annotations are available.This comparison concerns semi-supervised settings combining weak image-level and varying strong pixel-level annotations.
  • Bounding box annotations: With bounding-box annotations, small FOV is better for Bbox-Rect (Weak), Bbox-EM-Fixed (Weak), and Bbox-Rect (Semi-1464 strong), while large FOV is better in the other cases.The comparison covers varying numbers of pixel-level annotations alongside weak bounding-box annotations.
  • Annotations across datasets: At least 3%, large FOV consistently improves over small FOV when PASCAL VOC strong annotations are combined with weak or strong MS-COCO annotations.This improvement holds across all reported cross-dataset cases.

C. Detailed test results

The section reports detailed PASCAL VOC 2012 results for all methods, including test-set results and field-of-view analyses under varying strong and weak annotation settings.

  • Detailed test results: Detailed results for all reported methods are presented on the PASCAL VOC 2012 test set.These results appear in Tables 11, 12, and 13.
  • Effect of Field-Of-View: Field-of-view effects are evaluated on VOC 2012 validation data while varying pixel-level strong and bounding-box weak annotations.This analysis corresponds to Section 4.4 of the main paper.
  • Effect of Field-Of-View: Field-of-view effects are also evaluated using all 10,582 augmented PASCAL training images alongside varying strong and weak MS-COCO annotations.This analysis corresponds to Section 4.5 of the main paper.
Loading 1502.02734v3…