PaperlayerView exact revision on arXiv

Source-linked AI summary

FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference

Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, Sungroh Yoon

arXiv:1902.10421v2cs.CV

TL;DR

Weakly supervised segmentation struggles to recover object locations and boundaries from image-level annotations because classifier localization maps emphasize small discriminative regions. FickleNet uses stochastic hidden-unit selection to generate and aggregate diverse localization maps from one image, then uses them as segmentation pseudo-labels. It reports better performance than other methods in weakly supervised and semi-supervised Pascal VOC 2012 segmentation.

  • Problem

    Image-level annotations lack object locations and boundaries, while classifier localization maps typically focus on small discriminative object regions.

  • Method

    FickleNet randomly selects hidden units during training and inference, aggregates diverse localization maps, and uses the resulting map as pseudo-labels for segmentation.

  • Results

    FickleNet achieves better results than other methods in both weakly supervised and semi-supervised segmentation, including state-of-the-art Pascal VOC 2012 performance.

  • Takeaways & Limitations

    A single network can produce larger and more consistent object-part regions than an equivalent deterministic technique without additional training steps.

  • Takeaways & Limitations

    Dropout is applied during both training and inference, so the inference process remains stochastic.

Abstract

from arXiv · show

The main obstacle to weakly supervised semantic image segmentation is the difficulty of obtaining pixel-level information from coarse image-level annotations. Most methods based on image-level annotations use localization maps obtained from the classifier, but these only focus on the small discriminative parts of objects and do not capture precise boundaries. FickleNet explores diverse combinations of locations on feature maps created by generic deep neural networks. It selects hidden units randomly and then uses them to obtain activation scores for image classification. FickleNet implicitly learns the coherence of each location in the feature maps, resulting in a localization map which identifies both discriminative and other parts of objects. The ensemble effects are obtained from a single network by selecting random hidden unit pairs, which means that a variety of localization maps are generated from a single image. Our approach does not require any additional training steps and only adds a simple layer to a standard convolutional neural network; nevertheless it outperforms recent comparable techniques on the Pascal VOC 2012 benchmark in both weakly and semi-supervised settings.

1. Introduction

FickleNet addresses weakly supervised segmentation's limited pixel-level information by using stochastic hidden-unit combinations to generate broader localization maps from one image. It reports state-of-the-art Pascal VOC 2012 performance in weakly and semi-supervised settings.

  • Weakly supervised methods use easier annotations, but image-level labels provide no object locations or boundaries and current results remain inferior.
  • FickleNet randomly selects hidden units at each sliding-window position to generate diverse localization maps that can delineate object regions more sharply.Deterministic selection can activate foreground and background together, whereas stochastic selection produces more flexible region shapes.
  • The method aggregates pixels discovered across multiple stochastic localization maps into one map, which becomes pseudo-labels for training a segmentation network.
  • FickleNet is realized with dropout and adds a feature-map expansion method that makes the algorithm faster with only a small GPU-memory cost.
  • FickleNet achieves state-of-the-art performance on Pascal VOC 2012 in both weakly supervised and semi-supervised settings.

2. Related Work

Related approaches expand classifier localization maps through image manipulation, feature processing, multiple dilation rates, or region growing, but often require extra classifiers, training, or networks. FickleNet is presented as an alternative based on stochastic feature selection and efficient map expansion.

  • CAMs identify classification-relevant hidden units but usually focus on small discriminative object regions, making them unsuitable alone for semantic segmentation.
  • Image-level processing: Image-level erasure methods seek complementary object regions, but Adversarial Erasing requires multiple classification networks and repeated classification-erasure steps.
  • Feature-level processing: Feature-level complementary methods use a second classifier and training step, while erased features may confuse that classifier.
  • Feature-level processing: Multi-dilated convolution aggregates CAMs across dilation rates but requires separate training for each rate and is limited to integer dilation rates.
  • Region growing: AffinityNet expands CAMs using learned semantic affinities and random walks, but learning those affinities requires an additional network.
  • FickleNet implementation: FickleNet's map expansion avoids overlapping sliding kernels and reduces dropout and convolution calls to one each per forward pass.

3. Proposed Method

FickleNet trains a stochastic hidden-unit classifier, generates localization maps during inference, and uses those maps as pseudo-labels for segmentation training.

  • FickleNet is trained for multi-class classification using stochastic selection of hidden units.
  • Generated localization maps are used as pseudo-labels to train a semantic image segmentation network.
  • The procedure takes an image, ground-truth class label, and dropout rate, and outputs a classification score and localization maps.
  • The method includes stochastic hidden-unit selection and inference CAM generation as distinct procedural stages.

3.1. Stochastic Hidden Unit Selection

FickleNet uses spatially varying stochastic hidden-unit selection to explore diverse receptive fields, implemented efficiently through feature-map expansion and dropout during both training and inference.

  • Stochastic Hidden Unit Selection: Random selection produces receptive fields with many shapes and sizes, including fields resembling standard dilated convolutions at different rates.This provides varied receptive-field patterns without separately training convolutional blocks for each dilation rate.
  • Feature Map Expansion: The expansion method reduces the naive w × h repeated function calls to one convolution and one dropout call per forward pass.The method requires more GPU memory, but the number of trainable parameters remains constant and GPU load does not increase significantly.
  • Feature Map Expansion: Feature-map expansion prevents sliding-window overlap, allowing the stochastic selection and convolution operations to use optimized framework functions.The expanded map has size k × (sh) × (sw), where s is the convolution-kernel size.
  • Stochastic Hidden Unit Selection: FickleNet applies spatial dropout independently at each sliding-window position to sample hidden-unit combinations and discover relationships between object parts.Unlike standard dropout, selection varies across positions; the kernel center is retained to relate it to other locations.
  • Center-preserving Spatial Dropout: Dropout is applied uniformly across channels and used during both training and inference to generate stochastic hidden-unit selections.The center of each kernel is not dropped, preserving center-to-location relationships.
  • Classification: Applying convolution with kernel size s and stride s to the dropped expanded feature map yields a c × w × h output, which global average pooling and sigmoid convert into class scores.The network is trained with sigmoid cross-entropy for multi-label classification.

3.2. Inference Localization Map

FickleNet converts each stochastic hidden-unit selection into a Grad-CAM localization map and aggregates maps from multiple selections into one map.

  • Map Generation: Each random hidden-unit combination produces a distinct classification score and localization map for the same image.FickleNet constructs N maps, M[i], from N random selections before aggregation.
  • Grad-CAM: Grad-CAM uses gradients of the target-class score with respect to the pre-expansion feature map to weight channels before summing them.This yields a class-specific localization map based on each hidden unit’s contribution to the classification score.
  • Map Aggregation: The localization maps from multiple random selections are aggregated into a single localization map.The paper denotes the map from selection i as M[i] for 1 ≤ i ≤ N.

3.3. Training the Segmentation Network

FickleNet localization maps provide pseudo-label seed cues for weakly supervised segmentation, while semi-supervised training adds a loss from fully annotated data.

  • Weakly Supervised Training: For weakly supervised segmentation, FickleNet localization maps are used as seed cues and supplied to DSRG to train the segmentation network.The method uses the same background cues as DSRG.
  • Semi-supervised Training: Semi-supervised learning introduces an additional loss derived from images with person-provided full annotations.The training loss is defined for classes present in the image.
  • Semi-supervised Training: The semi-supervised loss combines DSRG’s balanced seeding loss and boundary loss with a term based on ground-truth masks.H_u,c denotes the class-c probability at location u, and F_c denotes the ground-truth mask.

4. Experiments

Experiments on PASCAL VOC 2012 show that FickleNet improves weakly and semi-supervised segmentation while remaining efficient and stable under stochastic inference.

  • Weakly supervised segmentation: FickleNet achieved mIoU values of 61.2 and 61.9 on weakly supervised validation and test images, respectively.These results improved on the DSRG backbone by 2.2% and 1.5%, respectively.
  • Weakly supervised segmentation: FickleNet achieved mIoU values of 64.9 and 65.3 on validation and test images, respectively, using DeepLab-v2-ResNet101.These results improved on DSRG by 3.5% and 2.1%, respectively, without additional training steps or networks.
  • Semi-supervised segmentation: With pixel-level annotations for only 13.8% of PASCAL images, FickleNet reached an mIoU of 65.8, or 97.3% of fully supervised DeepLab.Its validation performance was 1.5% better than DSRG.
  • Iterative inference: mIoU increased with the number of localization maps and converged to 61.2 when N exceeded 200.Additional random selections identify more object regions for the aggregated localization map.
  • Iterative inference: Segmentation performance converged as N increased, while standard deviations of mIoU, recall, and precision decreased further with larger N.The reported variability across five feature-selection runs was very small.
  • Dropout analysis: A dropout rate p of 0.9 covered larger target-object regions than deterministic DSRG localization maps.Higher dropout makes non-discriminative object parts more likely to be considered during random selections.
  • Step effectiveness: The best mIoU was obtained when stochastic selection was used during both training and inference.“Train S + infer D” also outperformed the deterministic training-and-inference setting.

5. Conclusions

FickleNet addresses image-level semantic segmentation by aggregating localization maps generated through random feature selection during training and inference. The resulting maps cover larger, more consistent object regions, and the method outperforms other state-of-the-art approaches in weakly and semi-supervised settings.

  • 5. Conclusions: FickleNet randomly selects features during training and inference, generates multiple localization maps from one image, and aggregates them into one map.The aggregated map is used to address segmentation with image-level annotations.
  • 5. Conclusions: The aggregated localization map contains larger and more consistent object regions than an equivalent deterministic technique.The method is implemented efficiently by expanding feature maps to avoid overlaps between sliding convolution kernels.
  • 5. Conclusions: FickleNet produces better results than other state-of-the-art approaches in both weakly supervised and semi-supervised segmentation.The conclusion covers both supervision settings using image-level annotations.

A.1. Additional Results

Additional evaluations show FickleNet outperforms DSRG with and without CRF and across most evaluated classes. Qualitative examples compare localization maps across dropout rates and predicted masks using two backbones.

  • Additional quantitative results: FickleNet outperforms DSRG with and without CRF, while showing the least difference between the two post-processing settings.The reproduced DSRG result is 58.9 mIoU without CRF versus a reported 59.0 with CRF.
  • Per-class results: FickleNet exceeds DSRG in 17 of 21 evaluated classes by per-class mIoU.
  • Qualitative results: Figure 7 compares DSRG and FickleNet localization maps across various dropout rates, including deterministic inference at p = 0.
  • Qualitative results: Figure 8 presents FickleNet predicted segmentation masks on Pascal VOC 2012 validation images using VGG-16 and ResNet-101.
Loading 1902.10421v2…