PaperlayerView exact revision on arXiv

Source-linked AI summary

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation

Jungbeom Lee, Jooyoung Choi, Jisoo Mok, Sungroh Yoon

arXiv:2110.06530v1cs.CVcs.LG

TL;DR

Weakly supervised semantic segmentation must obtain pixel-level localization from cheaper image-level labels, but classifiers focus on only small discriminative object regions. The paper reduces the final-layer information bottleneck by removing sigmoid or softmax and adding non-discriminative region pooling. The resulting localization maps improve significantly across PASCAL VOC 2012 and MS COCO 2014, with state-of-the-art weakly supervised segmentation performance.

  • Problem

    Image-level labels are cheap but provide no explicit localization, causing classifier maps to focus on small discriminative regions and sometimes spuriously correlated background.

  • Method

    RIB removes the final sigmoid or softmax activation and adds global non-discriminative region pooling to transmit more non-discriminative object information.

  • Results

    RIB significantly improves localization maps on PASCAL VOC 2012 and MS COCO 2014, achieving state-of-the-art performance; on VOC, it surpasses IRN-based CONTA by 3.0%p.

  • Takeaways & Limitations

    Reducing the final-layer information bottleneck improves classifier-derived localization for weakly supervised semantic segmentation.

  • Takeaways & Limitations

    Saliency supervision can provide more precise boundaries while causing non-salient objects to be ignored, whereas RIB identifies those objects; jointly achieving both remains future work.

Abstract

from arXiv · show

Weakly supervised semantic segmentation produces pixel-level localization from class labels; however, a classifier trained on such labels is likely to focus on a small discriminative region of the target object. We interpret this phenomenon using the information bottleneck principle: the final layer of a deep neural network, activated by the sigmoid or softmax activation functions, causes an information bottleneck, and as a result, only a subset of the task-relevant information is passed on to the output. We first support this argument through a simulated toy experiment and then propose a method to reduce the information bottleneck by removing the last activation function. In addition, we introduce a new pooling method that further encourages the transmission of information from non-discriminative regions to the classification. Our experimental evaluations demonstrate that this simple modification significantly improves the quality of localization maps on both the PASCAL VOC 2012 and MS COCO 2014 datasets, exhibiting a new state-of-the-art performance for weakly supervised semantic segmentation. The code is available at: https://github.com/jbeomlee93/RIB.

1 Introduction

Weakly supervised semantic segmentation reduces the need for costly pixel-level annotations by learning from cheaper image-level labels, but classifiers tend to localize only discriminative object regions. The paper attributes this bottleneck to the final activated DNN layer and proposes removing that activation plus non-discriminative region pooling.

  • Motivation: Pixel-level semantic segmentation annotations are costly, motivating weakly supervised learning with cheaper image-level class labels.Annotating one Cityscapes image takes more than 90 minutes.
  • Challenge: Image-level labels provide no explicit object localization, so classifiers often produce maps focused on small discriminative regions.The resulting maps can also highlight spuriously correlated background, reducing precision.
  • Analysis: The paper argues that sigmoid or softmax in the final DNN layer creates a prominent information bottleneck that excludes non-discriminative object information.This interpretation connects the classifier’s localization behavior to information flow through the network.
  • Method: RIB reduces the bottleneck by removing the final activation function and introducing pooling that processes more non-discriminative features.The method is intended to transmit more target-object information into classification and attribution maps.
  • Results: The method significantly improves classifier localization maps and achieves state-of-the-art weakly supervised segmentation performance on PASCAL VOC 2012 and MS COCO 2014.The contribution statement reports this result at the paper level without specifying benchmark values here.

2 Preliminaries

The preliminaries frame DNN layers as an information-processing chain in which input information is progressively compressed while task-relevant classification information is retained. They also introduce minimum sufficient representations and CAMs as the theoretical and localization foundations for the paper.

  • Information Bottleneck: DNN layers form a Markov chain, so the data processing inequality represents non-increasing mutual information from the input toward the output.The inequality is I(X; T1) ≥ I(X; T2) ≥ ··· ≥ I(X; TL) ≥ I(X; Ŷ).
  • Information Bottleneck: This layered information flow implies that information about the input is compressed as it passes through the DNN.The compression is described as part of the network’s sequential processing.
  • Information Bottleneck: Classification training seeks minimum sufficient features that compress the input while preserving information needed for classification.The information bottleneck objective balances compression I(X; T) against classification information I(T; Y).
  • Information Bottleneck: Double-sided saturating nonlinearities such as sigmoid and tanh are associated with stronger compression than single-sided nonlinearities such as ReLU.The cited analysis reports a compression phase mainly for networks using double-sided saturating nonlinearities.
  • Class Activation Mapping: CAM identifies image regions emphasized by a classifier by using class-specific contributions from the final feature map after global average pooling.Spatial maxima are used to normalize the resulting map.
  • Weakly Supervised Semantic Segmentation: Weakly supervised segmentation methods use classifier-derived CAM or Grad-CAM maps to construct initial pixel-level localization seeds.The preliminaries situate erasure and multi-context methods as related ways to improve these maps.

3 Proposed Method

The method interprets weakly supervised localization failure as an information bottleneck concentrated in the classifier’s final layer, then reduces it by removing the final sigmoid or softmax and pooling non-discriminative regions. Toy experiments support the diagnosis, while RIB improves information transmission and localization maps.

  • Motivation: The final layer exhibits the strongest bottleneck because sigmoid or softmax is a double-sided saturating activation that limits information transmitted from the final feature map.Earlier layers use ReLU in common architectures, whereas the final layer uses sigmoid or softmax.
  • Motivation: Toy experiments show that gradients in non-discriminative but class-relevant regions nearly disappear between the penultimate and final layers, while HGR trends confirm their disproportionate compression.The analysis distinguishes discriminative regions RD, non-discriminative class-relevant regions RND, and background RBG.
  • Motivation: CAM-based localization maps focus on small discriminative regions because the final classification layer filters out much of the non-discriminative information.The toy experiment measures this loss using gradient maps and HGR values across network layers.
  • RIB: RIB removes the final sigmoid or softmax by replacing sigmoid BCE with a loss that operates on classification logits, reducing the final-layer information bottleneck.Training from scratch with the replacement loss is unstable, so the method first trains an initial classifier and then fine-tunes it for each image.
  • RIB: RIB increases information transmitted from both discriminative and non-discriminative regions while keeping background transmission relatively constant during iterations.This behavior is reported from G6 and HGR measurements averaged over 100 images.
  • GNDRP: GNDRP selectively pools spatial locations with CAM scores below threshold τ, unlike GAP, to encourage activation of non-discriminative regions.Previous alternative pooling methods emphasize discriminative parts, whereas GNDRP excludes highly activated regions.

4 Experiments

RIB improves localization seeds, pseudo-ground-truth masks, and final segmentation across PASCAL VOC 2012 and MS COCO 2014. Ablations attribute later-iteration gains to GNDRP and show robustness to several hyper-parameter choices, while saliency supervision sharpens boundaries but can miss non-salient objects.

  • Quality of the initial seed and pseudo ground truth: 7.7%p improves PASCAL VOC 2012 initial seeds over original CAMs, outperforming the other compared methods.The initial seeds also outperform SEAM, which refines CAMs using pixel relationships.
  • Quality of the initial seed and pseudo ground truth: 68.6 mIoU with PSA and 70.6 with IRN are achieved by the refined PASCAL VOC 2012 pseudo-ground-truth masks.These results outperform other methods by a large margin under both seed-refinement techniques.
  • Performance of weakly supervised semantic segmentation: 68.3 and 68.6 mIoU are obtained on PASCAL VOC 2012 validation and test images, surpassing image-level-label competitors.Against CONTA implemented with IRN, the method achieves a 3.0%p advantage, 68.3 versus 65.3 mIoU on the reported comparison.
  • Performance of weakly supervised semantic segmentation: 70.2 and 70.0 mIoU are achieved on PASCAL VOC 2012 validation and test images with additional salient-object supervision.These results outperform recently introduced methods under the same supervision level.
  • Performance of weakly supervised semantic segmentation: 2.4%p improves MS COCO 2014 validation mIoU over IRN, reaching 43.8 from 41.4.The reported relative improvement exceeds CONTA’s 0.8%p improvement, 32.6 to 33.4, in the cited comparison.
  • Ablative Studies: GNDRP enables later RIB iterations to improve localization more effectively than GAP, while changing K above 10 causes less than a 0.8%p mIoU drop.Increasing the GNDRP threshold τ from 0.3 to 0.5 changes mIoU by less than 1%p, indicating robustness to τ.

5 Conclusions

The study analyzed why classifiers trained with image-level labels localize only small object regions and addressed this through two training modifications. The resulting method improved localization maps and achieved state-of-the-art performance on PASCAL VOC 2012 and MS COCO 2014.

  • 5 Conclusions: The information bottleneck analysis attributes small-region localization to limited information delivery through the DNN’s final layer.The analysis identifies the final layer as largely determining how much information from an input image reaches the output classification.
  • 5 Conclusions: Removing the final non-linear activation and introducing a new pooling method reduced the information bottleneck.These were presented as two simple modifications to the existing training scheme.
  • 5 Conclusions: The method significantly improved classifier localization maps on PASCAL VOC 2012 and MS COCO 2014.The reported outcome was new state-of-the-art performance on both datasets.
  • 5 Conclusions: Object segmentation without pixel-level annotation may save resources, while potentially requiring annotation companies to change their business models.The paper identifies possible usefulness in costly-annotation fields such as medicine and a possible impact on annotation providers.

Checklist

The checklist records affirmative responses for reproducibility, training-detail reporting, error bars, compute disclosure, asset attribution, and discussion of limitations and societal impacts. The paper reports that theoretical assumptions and proofs are addressed where applicable, while several human-subject and data-consent items are marked not applicable.

  • Checklist: The paper reports that its main claims reflect the stated contributions and scope and that limitations and negative societal impacts are discussed.These items are marked yes in the checklist.
  • Checklist: The theoretical-results checklist reports stated assumptions and complete proofs, with the assumptions and proofs described as widely accepted or already established in prior literature.These responses concern theoretical results where applicable.
  • Checklist: The paper reports including code, data, instructions, training details, error bars, and compute-resource information needed to reproduce the experiments.The checklist directs readers to the Appendix and Section 4 for these materials and details.
  • Checklist: Existing assets are cited, while asset licensing and new-asset release are marked not applicable.The checklist identifies Section 4 for citations and reports no applicable licensing or new-asset issue.
  • Checklist: Human-subject instructions, participant risks, compensation, consent, personally identifiable information, and offensive-content discussions are marked not applicable.The checklist marks the relevant crowdsourcing and human-subject items as N/A.

A.1 Implementation Details

The implementation uses dataset-specific optimization settings and a pseudo-label retraining procedure for semantic segmentation. Experiments ran on four NVIDIA Quadro RTX 8000 GPUs, with the PASCAL VOC RIB process taking 32 minutes on that setup.

  • A.1 Implementation Details: PASCAL VOC training uses batch size 10, 30K iterations, and learning rate 2.0 × 10^-4.These settings are reported for semantic-segmentation optimization.
  • A.1 Implementation Details: MS COCO training uses batch size 10, 100K iterations, and learning rate 2.5 × 10^-4.These settings differ from those reported for PASCAL VOC.
  • A.1 Implementation Details: The retraining procedure thresholds IRN random-walk maps, ignores intermediate pixels initially, then fills and trains on all pseudo labels.Values above 0.3 are foreground, values below 0.2 are background, and the remaining pixels form Pignore.
  • A.1 Implementation Details: Without retraining, the method obtains 67.83 mIoU and outperforms all methods in Table 2 by a large margin.Retraining is not used for RIB with saliency or for the MS COCO dataset.
  • A.1 Implementation Details: The experiments use four NVIDIA Quadro RTX 8000 GPUs, and the PASCAL VOC RIB process takes 32 minutes on them.The same process takes 43 minutes on four NVIDIA Tesla V100 GPUs.

A.2 Additional Analysis

Additional analyses examine training stability, activation-function saturation, seed variability, batch-size sensitivity, and qualitative localization and segmentation examples. They report instability when training LRIB from scratch, activation-dependent bottlenecks, stable repeated mIoU, and saturation beyond a suitable batch size.

  • A.2 Additional Analysis: Training a classifier with LRIB from scratch causes instability because the loss diverges to −∞ after a few iterations.The analysis attributes this to a non-saturating loss gradient and illustrates it with learning-rate-dependent loss curves.
  • A.2 Additional Analysis: Tanh saturates fastest, while softsign behaves most linearly, indicating the largest bottleneck for tanh and smallest for softsign.The paper states that Table 5(a) supports this activation-dependent ordering.
  • A.2 Additional Analysis: 56.44 ± 0.05 mIoU is obtained when the RIB process is repeated five times across random seeds.This measures sensitivity to the initial seed.
  • A.2 Additional Analysis: Using additional B − 1 samples to construct a batch is more effective than using only the target image, with performance saturating above a certain B.The analysis compares batch size B = 1 with larger batches for PASCAL VOC initial-seed mIoU.
  • A.2 Additional Analysis: Figures A2 and A3 show localization maps refined through RIB for PASCAL VOC and MS COCO, while Figure A4 shows predicted segmentation masks.Figure A4 compares masks from IRN and the proposed method on validation images.
  • A.2 Additional Analysis: Tables A2 and A3 report per-class mIoU scores for the proposed method and other methods on PASCAL VOC and MS COCO.The tables provide class-level comparisons for both datasets.
Loading 2110.06530v1…