PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features

Xiang Wang, Shaodi You, Xi Li, Huimin Ma

arXiv:1806.04659v1cs.CV

TL;DR

Weakly supervised segmentation from image tags is difficult because classification networks yield coarse, discriminative localization rather than complete pixel-level regions. MCOF iteratively mines common object features, refines regions with saliency, and trains segmentation networks; on PASCAL VOC 2012, it outperforms prior methods by large margins.

  • Problem

    Image-tag supervision leaves classification networks producing coarse, inaccurate discriminative regions that are inadequate for pixel-wise semantic segmentation.

  • Method

    MCOF alternates bottom-up common-feature mining and top-down segmentation training, using initial seeds, saliency-guided refinement, and iteratively predicted masks.

  • Results

    MCOF outperforms previous methods by large margins and achieves state-of-the-art performance on PASCAL VOC 2012.

  • Takeaways & Limitations

    Iterative mining progressively expands object regions, corrects inaccurate localization, and optimizes the segmentation network.

  • Takeaways & Limitations

    Saliency-guided refinement is limited to the first iteration because later saliency maps have limited accuracy and could introduce additional noise.

Abstract

from arXiv · show

Weakly-supervised semantic segmentation under image tags supervision is a challenging task as it directly associates high-level semantic to low-level appearance. To bridge this gap, in this paper, we propose an iterative bottom-up and top-down framework which alternatively expands object regions and optimizes segmentation network. We start from initial localization produced by classification networks. While classification networks are only responsive to small and coarse discriminative object regions, we argue that, these regions contain significant common features about objects. So in the bottom-up step, we mine common object features from the initial localization and expand object regions with the mined features. To supplement non-discriminative regions, saliency maps are then considered under Bayesian framework to refine the object regions. Then in the top-down step, the refined object regions are used as supervision to train the segmentation network and to predict object masks. These object masks provide more accurate localization and contain more regions of object. Further, we take these object masks as initial localization and mine common object features from them. These processes are conducted iteratively to progressively produce fine object masks and optimize segmentation networks. Experimental results on Pascal VOC 2012 dataset demonstrate that the proposed method outperforms previous state-of-the-art methods by a large margin.

1. Introduction

Image-level supervision makes semantic segmentation difficult because classification networks provide coarse, inaccurate discriminative regions rather than pixel-wise masks. MCOF iteratively mines common object features, expands regions, refines them with saliency, and trains segmentation networks.

  • Image tags provide limited supervision, requiring pixel-wise segmentation without pixel-wise annotations.
  • Classification networks produce coarse and inaccurate discriminative regions that do not satisfy pixel-wise segmentation requirements.
  • MCOF mines common object features from initial localization to progressively expand object regions despite inaccurate seeds.
  • The framework trains a region classification network, applies saliency-guided refinement, and uses the resulting regions to supervise segmentation masks.
  • MCOF combines iterative bottom-up and top-down learning, with saliency refinement supplementing non-discriminative regions.

2. Related Work

Related work spans fully supervised region- and pixel-based segmentation and weakly supervised methods using bounding boxes, scribbles, or image-level labels. MCOF addresses limitations of localization-based approaches by iteratively mining common object features and correcting regions.

  • Fully supervised segmentation methods use pixel-wise annotations and include region-based and pixel-based network designs.
  • Pixel-based networks are more powerful for semantic segmentation, while region-based networks can learn common object features for supervision.
  • Weak supervision includes bounding boxes, scribbles, and image-level labels; this paper focuses on image-level supervision.
  • Image-level methods are broadly divided into MIL-based approaches and localization-based approaches.
  • Localization-based methods use classification-derived cues, but sequential erasing can accumulate errors and produce coarse boundaries.

3. Architecture of the Proposed MCOF

MCOF alternates bottom-up region expansion with top-down segmentation training. Saliency refinement is used only initially, while later iterations use predicted masks to refine supervision and update seeds.

  • The architecture alternates a bottom-up step that mines common features with a top-down step that trains the segmentation network.
  • At initialization, training images are represented by superpixels, and RegionNet is trained from initial object seeds.
  • RegionNet predictions produce object regions, which are saliency-refined during the first iteration before PixelNet training.
  • PixelNet predicts object masks that replace the seeds for subsequent iterations, progressively correcting regions and producing finer masks.
  • Saliency maps are used only in the first iteration because later predicted masks are more accurate and saliency accuracy is limited.
  • Only the final-iteration PixelNet is used for inference, making inference efficient despite iterative training.

4. Mining Common Object Features

The method builds object regions from coarse CAM localization by mining common features with RegionNet, then supplements missing regions using saliency-guided Bayesian refinement. These refined regions support segmentation training and iterative improvement.

  • Initial Object Seeds: CAM heatmaps are averaged within superpixels, then local maxima and above-threshold regions are selected as initial object seeds.The seeds capture discriminative key parts but remain coarse and incomplete.
  • Mining Common Object Features: Initial seeds from different images expose shared object attributes, enabling RegionNet to learn common features and predict expanded object regions.RegionNet can recover initially background-labeled regions and suppress some noisy predictions.
  • Mining Common Object Features: Training images are represented by superpixel regions with one-hot class labels, and RegionNet models each region’s probability of belonging to a class.The network uses a mask-based Fast R-CNN framework with superpixel-aware RoI pooling.
  • Saliency-Guided Supplement: Saliency maps supplement mined regions under a Bayesian framework, while CRF binarization produces refined object regions.The saliency prior is used cautiously because saliency can include noise or objects outside the target semantic classes.
  • Iterative Optimization: Predicted masks are reused as seeds in later iterations, progressively expanding and correcting object regions while improving segmentation supervision.Saliency-guided refinement is applied only to images with a single object class; multi-class regions remain unchanged.

5. Iterative Learning Framework

The framework alternates bottom-up object-region expansion with top-down segmentation training. Refined regions supervise PixelNet, whose masks become seeds for further feature mining and iterative refinement.

  • Top-down training: Refined object regions supervise the weakly supervised segmentation network without requiring class labels because wrong-class regions have been removed.The framework uses localization cues alone as supervision for segmentation training.
  • Top-down training: The segmentation network models per-location class probabilities and is trained with cross-entropy to match refined object regions.The supervision set S_c contains locations labelled with class c.
  • Mask prediction: PixelNet uses whole-image context, unlike the region classification network, and predicts masks that include more object regions while suppressing noisy regions.The predicted masks provide improved localization for subsequent iterations.
  • Iterative refinement: Predicted segmentation masks become object seeds for repeated mining and training, progressively expanding regions, correcting inaccuracies, and improving segmentation.Each iteration supplies more robust common object features and better supervision.

6. Experiments

Experiments on PASCAL VOC 2012 evaluate MCOF against prior weakly supervised methods and through ablations of its iterative components. MCOF achieves state-of-the-art results, with performance gains from feature mining, saliency refinement, PixelNet training, and iterative training.

  • Experimental setup: MCOF is evaluated on PASCAL VOC 2012 using mIoU on validation and test sets, with 10,582 augmented training images.The benchmark contains 20 object classes and one background class.
  • State-of-the-art comparison: MCOF is compared with prior image-level weakly supervised segmentation methods using DeepLab-LargeFOV architectures built on VGG16 and ResNet101.The comparison includes methods such as AE-PSL, SEC, STC, and MIL-sppxl.
  • State-of-the-art comparison: 1.2% and 1.9%: MCOF-VGG16 outperforms AE-PSL on the PASCAL VOC 2012 validation and test sets, respectively.The reported metric is mIoU.
  • State-of-the-art comparison: 5.3% and 5.5%: MCOF-ResNet101 improves over the comparison methods on the validation and test sets, respectively.The reported metric is mIoU, and the method uses the same 10K-scale training regime as most compared methods.
  • Ablation studies: 48.4%: PixelNet performance rises from 14.27% initial seeds to 29.1% after RegionNet learning and 34.8% after saliency-guided refinement.Later iterations continue to improve performance gradually.
  • Iterative training comparison: MCOF improves rapidly across iterations, whereas direct iterative training increases slowly and reaches low accuracy.MCOF mines common object features before expanding regions and retraining the segmentation network.
  • Saliency refinement: Saliency-guided refinement supplements missing non-discriminative object regions, while omitting saliency maps leaves regions missing and limits performance.Saliency maps are used in the first iteration but omitted later to avoid additional noise.

7. Conclusion

MCOF bridges high-level semantics and low-level appearance through iterative bottom-up feature mining and top-down network optimization. It progressively corrects localization and achieves state-of-the-art weakly supervised segmentation performance.

  • Conclusion: MCOF mines common object features from coarse seeds, expands object regions, and iteratively optimizes segmentation networks.The framework alternates bottom-up region expansion with top-down mask prediction and supervision.
  • Conclusion: Saliency-guided refinement supplements non-discriminative regions omitted by initial classification-based localization.Predicted masks then provide more complete regions for later feature mining.
  • Conclusion: The iterative process progressively corrects inaccurate localization and produces more accurate object regions for semantic segmentation.The paper reports new state-of-the-art performance on PASCAL VOC 2012.
Loading 1806.04659v1…