PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting saliency for object segmentation from image level labels

Seong Joon Oh, Rodrigo Benenson, Anna Khoreva, Zeynep Akata, Mario Fritz, Bernt Schiele

arXiv:1701.08261v2cs.CV

TL;DR

Pixel-wise semantic segmentation from image-level labels is difficult because labels provide discriminative regions but not full object extents, especially under co-occurrence ambiguity. The paper combines classifier-derived seeds with class-agnostic saliency in a Guided Segmentation architecture. Its weakly supervised results reach 80% of fully supervised quality and are reported as state of the art, while performance remains tied to saliency and seeder quality.

  • Problem

    Image-level labels can identify discriminative object regions but do not specify full object extents, making weakly supervised pixel-wise segmentation ambiguous under co-occurring categories.

  • Method

    Guided Segmentation combines image-label-derived object seeds with class-agnostic saliency to produce guide masks for training a segmentation convnet.

  • Results

    80% of fully supervised quality is reached, with the method reported as the best performer among known image-level-supervision methods.

  • Takeaways & Limitations

    Saliency is a viable additional source of object-extent information for weakly supervised semantic segmentation.

  • Takeaways & Limitations

    Further improvements are expected from better understanding of seeder methods and improvements to the saliency model.

Abstract

from arXiv · show

There have been remarkable improvements in the semantic labelling task in the recent years. However, the state of the art methods rely on large-scale pixel-level annotations. This paper studies the problem of training a pixel-wise semantic labeller network from image-level annotations of the present object classes. Recently, it has been shown that high quality seeds indicating discriminative object regions can be obtained from image-level labels. Without additional information, obtaining the full extent of the object is an inherently ill-posed problem due to co-occurrences. We propose using a saliency model as additional information and hereby exploit prior knowledge on the object extent and image statistics. We show how to combine both information sources in order to recover 80% of the fully supervised performance - which is the new state of the art in weakly supervised training for pixel-wise semantic labelling. The code is available at https://goo.gl/KygSeb.

1. Introduction

The paper addresses weakly supervised semantic segmentation from image-level labels, where image labels locate discriminative object regions but do not determine full extents. It combines these seeds with class-agnostic saliency to guide pixel-wise labelling and reaches state-of-the-art weakly supervised performance.

  • Motivation: Pixel-wise semantic labelling requires costly pixel-level annotations, motivating weaker supervision with lower annotation effort.The introduction frames the goal as approaching fully supervised quality with minimal supervision.
  • Motivation: Image-level supervision is ambiguous because co-occurring categories such as trains and rails cannot be separated without additional information.Prior work addresses this ambiguity using size priors, external images, or human corrections.
  • Approach: The method decomposes segmentation into finding object locations and finding object extents, with extent estimation equivalent to identifying background.This separates the discriminative localization problem from the harder boundary and coverage problem.
  • Approach: Image classifiers provide high-confidence object seeds and background regions, but struggle with nondiscriminative fine details of object instances.Seeds therefore provide precise but incomplete object evidence.
  • Approach: Class-agnostic saliency supplies noisy information about object extent beyond the seeds, and its model is trained using bounding boxes rather than pixel-wise annotations.The paper uses saliency as an additional source of object-extent information.
  • Results: 80% of fully supervised quality is reached, establishing the reported best performance among methods using image-level supervision with or without external data.The result is reported against a fully supervised DeepLabv1 model.

2. Related work

Related work spans increasingly weak forms of supervision and methods for deriving object evidence from image labels. This paper distinguishes its saliency-guided approach from prior propagation, proposal, objectness, and externally augmented methods.

  • Supervision: Weakly supervised semantic labelling has used image labels, points, scribbles, and bounding boxes as alternative supervision forms.This paper focuses on image labels as its main supervision.
  • Object seeds: Object-seed methods use image-level classifiers to identify high-precision, low-recall class regions through occlusion, gradients, modified back-propagation, or per-image optimization.These methods localize discriminative areas without necessarily covering full objects.
  • Pixel labelling: Pixel-labelling methods have combined multiple-instance learning, expectation-maximization, superpixels, size priors, CRFs, and segment proposals with image-level supervision.Without additional priors, early approaches obtained poor results.
  • Saliency-guided methods: The proposed method differs from a prior saliency approach by using better cues, avoiding roughly 40k crawled class-specific images, and reporting significantly better results.The comparison highlights the role of saliency as additional information.
  • Saliency-guided methods: Unlike diffuse objectness maps derived from bounding boxes, the paper's saliency maps are described as having sharp object boundaries for more precise guidance.The distinction concerns the precision of guidance supplied to the semantic labeller.
  • Saliency: Saliency is used here as a class-agnostic binary mask for the object a person is most likely to look at, and the model is trained weakly from bounding boxes.The paper treats improved saliency models as a route to improved segmentation results.

3. Guided Segmentation architecture

Guided Segmentation uses a guide labeller to combine classifier-derived seeds with saliency before training a separate segmentation convnet. The architecture keeps the final segmenter’s training procedure fully supervised with the generated guide mask.

  • Design rationale: The architecture focuses on saliency as prior knowledge while keeping the overall architecture simple.This contrasts with prior work using sophisticated losses or more involved pipelines.
  • Architecture: Guided Segmentation is a two-module system consisting of a guide labeller and a segmenter convnet.The guide labeller combines cues before the segmenter is trained.
  • Guide labeller: Given an image and image-level labels, the guide labeller combines seeder and saliency outputs into a rough segmentation mask called the guide.The seeder supplies discriminative object regions, while saliency extends coverage toward object extents.
  • Segmenter: The segmenter convnet is trained using the generated guide mask as supervision with per-pixel softmax cross-entropy loss.The final training procedure is fully supervised with respect to the produced guide mask.

4. Finding goods seeds

The paper compares classifier-based seed generators for weakly supervised segmentation, finding that GAP architecture choices strongly affect localisation while classification remains stable. GAP-HighRes is selected for subsequent experiments.

  • Seed methods: Classifier-based localisation methods generate high-confidence object regions but may have low foreground recall, motivating direct comparison of GAP and back-propagation seeders.The evaluation includes GAP variants, back-propagation methods, and a centre mean shape lower bound.
  • Evaluation: Mean precision (mP) averages foreground precision at 20% recall and background precision at 80% recall to summarize localisation performance.The foreground threshold emphasizes precise discriminative regions, while the background threshold reflects the larger and more diverse background region.
  • Seed methods: GAP variants generally provide greater precision than back-propagation variants at the same recall rate.Guided back-propagation reaches its highest precision only at approximately 5% recall, which is too low for practical utility.
  • Architecture: 80.7 mP for GAP-HighRes versus 76.5 mP for GAP-LowRes shows that higher output resolution improves localisation.The paper reports that classification performance remains stable across these architectural choices.
  • Architecture: 57.7 mP for GAP-DeepLab versus 87.0 mP for GAP-HighRes shows that dilated convolutions significantly hurt GAP localisation performance.GAP-HighRes and GAP-ROI provide higher precision over a wide recall range, whereas GAP-DeepLab performs substantially worse than other GAP variants.
  • Architecture: GAP-HighRes is chosen as the seeder module for the following sections.The implementation treats every non-foreground region as background.

5. Finding the object extent

The paper uses class-agnostic saliency to estimate object extent beyond discriminative seeds, addressing ambiguities that image-level labels cannot resolve alone. Saliency is useful despite noise, but it can merge instances, highlight irrelevant objects, or miss salient regions.

  • Motivation: Image-level labels alone cannot reliably learn object extent because co-occurring categories and systematically occluded parts create ambiguity.Examples include trains with rails and missing feet in images containing people.
  • Results: The saliency approach improves mIoU by +17 points for GAP-HighRes and +18 points for GAP-ROI, compared with smaller and inconsistent CRF gains.CRF gives +13 points for GAP-HighRes and +7 points for GAP-ROI under the same comparison.
  • Saliency: Class-agnostic object saliency supplies foreground extent information beyond the high-precision, low-recall regions identified by image-label seeds.The saliency model is trained using bounding-box annotations rather than accurate pixel-wise annotations.
  • Limitations: Saliency-guided labelling can merge distinct objects, focus on an irrelevant salient category, or fail to identify any salient region.The paper illustrates these difficulties with person-bike merging, a shirt selected instead of a person, and an image without a detected salient region.
  • Saliency quality: Only about 20% of validation images have saliency foreground quality above IoU 0.6, yet this limited signal still helps weakly supervised learning.The paper reports that its convnet saliency model outperforms hand-crafted methods despite this coverage.
  • Guide labels: The guide-label pipeline combines binary class-agnostic saliency masks with class-specific seed labels, evaluating seed-free baselines G0 and G1 before fusion strategy G2.G0 randomly assigns image-label classes to salient foreground, while G1 classifies each connected component.
  • Fusion: The seed-saliency fusion assigns labels using seed and foreground-component intersections, propagating touching seed labels and marking uncertain cases as ignore.The strategy also treats seeds as reliable small predictors while allowing saliency to activate on irrelevant objects.

6. Experiments

The experiments evaluate saliency- and seed-based guide labellers on Pascal VOC 2012, comparing their supervision quality and performance with prior weakly supervised methods. Combining seeds and saliency achieves the best reported results, reaching 80% of fully supervised quality.

  • Evaluation uses Pascal VOC 2012, reporting intermediate results on the 1,449-image validation set and the final system result on the test set.
  • Guide strategies: 38.7 mIoU results from using seeds directly as guide labels, whose foreground recall is only 37%.Saliency-only G0 reaches 46.0 mIoU with 52% foreground recall at comparable precision.
  • Guide strategies: 45.8 → 46.2 mIoU shows that adding a classifier on top of saliency provides only a negligible improvement from G0 to G1.
  • Guide strategies: 5 pp (46.2 → 51.2) is gained when classifier-generated seeds replace image-level scoring in the transition from G1 to G2.The result indicates that how the classifier is used substantially affects guide quality.
  • Guide strategies: 51.2 → 56.9 mIoU with ideal saliency shows that saliency quality remains an important source of potential improvement.
  • Comparison with prior work: G2 obtains the best reported results on this task and reaches 80% of fully supervised quality compared with a fully supervised DeepLabv1 model.The comparison includes methods using saliency, additional images, or human annotations.

7. Conclusion

The paper concludes that image-level labels provide high-quality object seeds but do not adequately reveal full object extents. Its Guided Segmentation architecture combines seeds and saliency to train a segmentation convnet, achieving state-of-the-art performance at 80% of the fully supervised case.

  • Image labels provide high-quality discriminative object seeds, whereas learning complete object extents remains difficult.
  • The guide labeller combines seed and saliency cues to train a segmentation convnet successfully.
  • 80% of the fully supervised case is reached by the weakly supervised results.
  • Further improvements are expected from deeper understanding of seeder methods and better saliency models.
  • The reported comparison excludes methods using MCG scores trained on ground-truth Pascal segments because the results are not considered comparable.

Supplementary Materials

The supplementary materials provide implementation, experiment, architecture, and qualitative-result details supporting the main paper. They also document a linearity-based equivalence in the placement of the GAP layer for GAP-ROI.

  • Supplementary materials include architecture details for GAP-LowRes, GAP-HighRes, GAP-DeepLab, and GAP-ROI.
  • They document CRF experiments and parameters, plus additional qualitative saliency results and guide-labeller examples.
  • Additional training details cover the Seeder, Classifier, and Segmenter networks.
  • For GAP-ROI, placing GAP after the final linear layer is functionally identical to placing it after the penultimate layer.GAP sums spatial dimensions linearly and the final layer combines channels linearly, so the operations can be swapped.

B.2. Training GAP

GAP variants use fixed SGD training, but their qualitative outputs differ substantially: GAP-DeepLab produces repeating artifacts, unlike the other variants.

  • All GAP variants use SGD with minibatch size 15, momentum 0.9, weight decay 5 × 10−4, and learning rate drops every 2 000 iterations.Training stops at 8 000 iterations.
  • GAP-LowRes, GAP-HighRes, and GAP-ROI produce qualitatively similar outputs, whereas GAP-DeepLab has significantly lower quality.GAP-DeepLab shows repeating patterns in its output.

C. CRF

CRF-based refinement can substantially improve segmentation, but its gains depend strongly on CRF parameters and seed choice; saliency produces larger, more stable improvements.

  • 50.4 mIoU results from combining crf-loss and crf-postproc with the GAP-HighRes seed, a 12.9 mIoU boost over the vanilla seed.Changing CRF parameters loses 5.2 mIoU, while changing the seed to GAP-ROI loses 5.4 mIoU.
  • The 12.9 mIoU CRF boost is fragile because it disappears substantially when CRF parameters or seed type changes.The comparison uses CRF parameter settings v1 and v2 and GAP-HighRes versus GAP-ROI seeds.
  • Saliency gives a consistent ≥4 mIoU gain over the best CRF combination regardless of seed type, indicating greater performance and stability.Combining crf-loss with saliency was excluded because preliminary experiments reduced performance relative to saliency alone.
  • The experiments consider combinations of CRF units, crf-seed, crf-loss, and crf-postproc, using standard or alternative CRF parameter settings.The default parameters follow DeepLab-LargeFOV, while some experiments use parameters from [18].

E. G2 guide labeller algorithm

The G2 guide labeller combines seed localization with saliency foreground components to produce class-wise training labels, using ignore regions when evidence is ambiguous.

  • G2 guide labeller algorithm: G2 assigns labels by intersecting saliency foreground components with zero, one, or multiple foreground seed categories.Zero-category regions are ignored, one-category regions receive the seed class, and multi-category regions use dense CRF inference.
  • G2 guide labeller algorithm: Saliency foreground components are connected regions covering at least 1% of the image, while seed components are computed separately.The procedure then evaluates their intersections to generate pixel-wise labels.
  • G2 guide labeller algorithm: The algorithm marks seed pixels outside intersecting saliency regions as ignore rather than forcing a label.This follows the assumption that saliency better delineates the object extent.
  • G2 guide labeller algorithm: Seeds touching multiple foreground regions propagate their label to all of them, while isolated seeds are retained as foreground missed by saliency.These rules preserve seed evidence when saliency merges or omits relevant regions.
  • G2 guide labeller algorithm: G2 produces much more precise labelling than G0 and G1 by using richer localization information from the seeds.The comparison is qualitative and shown in the guide-labelling examples.

F. Convnet training details

The system uses specified DeepLab and VGG-16 components, compares multiple GAP architectures, and illustrates how seeds and saliency are combined into guided segmentation.

  • Convnet training details: The saliency network is DeepLab-v2 ResNet, while the segmenter is DeepLab-v1 trained with SGD using the stated optimization schedule.The classifiers are VGG-16 networks trained with SGD.
  • Convnet training details: GAP-LowRes, GAP-HighRes, GAP-ROI, and GAP-DeepLab are the four fully convolutional architectures evaluated with a GAP layer.Their architectural differences are summarized in the paper's architecture table.
  • Convnet training details: GAP-DeepLab produces repeating stride patterns, whereas GAP-LowRes, GAP-HighRes, and GAP-ROI are qualitatively similar.The artifact is attributed to repeating learned filters associated with dilated convolutions.
  • Convnet training details: The saliency model is trained with MSRA box annotations, excluding samples corresponding to Pascal categories.The training samples are therefore not drawn from Pascal-category examples.
  • Convnet training details: Seeds have high precision and low recall, while saliency supplies class-agnostic object extent information for G2 guide labelling.The resulting guide labels can remain noisy, but the segmenter convnet produces more precise predictions.
Loading 1701.08261v2…