PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-Stage Semantic Segmentation from Image Labels

Nikita Araslanov, Stefan Roth

arXiv:2005.08104v1cs.CVcs.LG

TL;DR

Weakly supervised semantic segmentation from image-level labels remains difficult because attention maps are incomplete and coarse, while high-performing alternatives often require complex multi-stage pipelines. The paper develops a single-stage segmentation network and self-supervised scheme guided by local consistency, semantic fidelity, and completeness. It reports state-of-the-art-level segmentation quality despite using one network trained in one round.

  • Problem

    Image-level supervision must support pixel-precise semantic segmentation, but attention maps are coarse and incomplete and stronger methods often use complex multi-stage pipelines.

  • Method

    The method trains a segmentation-based network in one stage using normalised Global Weighted Pooling, Pixel-Adaptive Mask Refinement, and a Stochastic Gate for self-supervision.

  • Results

    The approach achieves segmentation quality on par with the state of the art and outperforms a range of recent multi-stage methods using additional data or saliency supervision.

  • Takeaways & Limitations

    A single segmentation network trained in one round can provide a practical alternative to substantially more complex weakly supervised pipelines.

Abstract

from arXiv · show

Recent years have seen a rapid growth in new approaches improving the accuracy of semantic segmentation in a weakly supervised setting, i.e. with only image-level labels available for training. However, this has come at the cost of increased model complexity and sophisticated multi-stage training procedures. This is in contrast to earlier work that used only a single stage $-$ training one segmentation network on image labels $-$ which was abandoned due to inferior segmentation accuracy. In this work, we first define three desirable properties of a weakly supervised method: local consistency, semantic fidelity, and completeness. Using these properties as guidelines, we then develop a segmentation-based network model and a self-supervised training scheme to train for semantic masks from image-level annotations in a single stage. We show that despite its simplicity, our method achieves results that are competitive with significantly more complex pipelines, substantially outperforming earlier single-stage methods.

1. Introduction

Weakly supervised segmentation from image-level labels is practical but challenging: attention maps are incomplete and coarse, while stronger methods often require complex multi-stage pipelines. The paper proposes a single-stage alternative guided by local consistency, semantic fidelity, and completeness.

  • Motivation: Pixel-level semantic segmentation is valuable for scene understanding, but supervised methods depend on costly pixel-level annotations.Learning from image-level labels alone is harder than settings with bounding boxes, scribbles, or points.
  • Motivation: Attention maps localize discriminative regions but violate boundaries, miss object extent, and mislabel parts of objects.These limitations arise because attention maps were not designed for segmentation.
  • Motivation: Modern weakly supervised methods exceed 80% of fully supervised accuracy, but typically rely on multiple models, training cycles, or saliency methods.Earlier single-stage approaches were superseded by these more complex pipelines.
  • Contribution: The proposed approach trains one segmentation network in a single stage without saliency estimation or additional data.Its model uses normalised Global Weighted Pooling, Pixel-Adaptive Mask Refinement, and a Stochastic Gate for self-supervised segmentation.
  • Design principles: The paper defines local consistency, semantic fidelity, and completeness as desirable properties for segmentation-aware training from image labels.The properties address inconsistent neighboring labels, reliable classification-supporting masks, and full object coverage.

2. Related Work

Prior weakly supervised segmentation methods commonly refine localization cues through additional networks, training rounds, saliency supervision, or auxiliary data. This work instead develops a competitive single-network alternative trained in one cycle.

  • Single-stage methods: Early single-stage methods used MIL, expectation-maximisation, or joint attention and segmentation models to learn from image labels.These approaches aggregated pixel predictions, inferred pseudo ground truth, or combined top-down and bottom-up cues.
  • Seed and expand: Seed-and-expand methods improve coverage by expanding CAM-derived localization cues toward segment boundaries.Their localization cues are pre-computed by a standalone classification network, so better localization can improve segmentation.
  • Erasing: Erasing methods remove high-confidence CAM regions and retrain classification models to discover additional object cues.Related variants use decoder branches and saliency assistance.
  • Multiple training rounds: Multiple-round methods sequentially train segmentation networks and refine intermediate predictions with tools such as GrabCut or saliency maps.These pipelines increase task difficulty or repeatedly provide predictions as supervision.
  • Additional supervision and refinement: Other approaches add video data, saliency supervision, saliency detectors, or multiple dilation rates to improve attention-map recall and mask accuracy.These methods extend supervision or refinement beyond image-level labels alone.
  • Positioning: The paper offers a competitive alternative using one network trained in one cycle, in contrast to related methods requiring standalone segmentation networks or cascades.The design prioritizes practicality while retaining weak supervision from image labels.

3. Model

The model jointly learns classification and segmentation from image-level labels using segmentation-aware score aggregation, appearance-based mask refinement, and stochastic feature regularization. These components target semantic fidelity, local consistency, and completeness while retaining a single-stage segmentation architecture.

  • Overview: The network extends a fully convolutional segmentation model with softmax outputs and skip connections, enabling single-pass inference and use of pretrained classification models for initialization.The architecture is designed as a straightforward extension of segmentation-based networks.
  • Overview: Three components address the task’s main requirements: nGWP for class aggregation, PAMR for local mask refinement, and a stochastic gate for feature regularization.The components are introduced as a new class aggregation function, a local mask refinement module, and a stochastic gate.
  • Classification scores: CAM-GAP is ill-suited to segmentation because its masks are unbounded, lack pixel-level class competition, and can disadvantage small segments through context reliance.CAM-GAP encourages all feature-map pixels to identify with the target class, which can reduce mask precision.
  • Classification scores: nGWP predicts pixelwise class scores, adds a background channel, applies pixelwise softmax masks, and aggregates scores using pixel-level confidence weights.The aggregation mapping supports segmentation-aware classification while retaining the baseline classification loss.
  • Classification scores: nGWP is invariant to mask size, while a logarithmic penalty encourages larger positive-class masks and separates class-score influence from mask-size influence.The penalty is controlled by λ > 0 and focuses strongly on near-zero masks.
  • Pixel-Adaptive Mask Refinement: PAMR iteratively updates masks using image-dependent pixel affinities, then supplies high-confidence refined pixels as pseudo ground truth for self-supervised segmentation.PAMR is parameter-free and recurrent; ten refinement steps provide an empirically sufficient efficiency–accuracy trade-off, with one iteration adding less than 1% of baseline GPU memory.
  • Stochastic gate: The stochastic gate counters pseudo-supervision errors by exchanging deep and shallow representations while preserving deep features as the expected value.Global Cue Injection enriches shallow features with global cues extracted from deep features through Adaptive Instance Normalisation.

4. Experiments

Experiments show that the single-stage model benefits substantially from its focal penalty, PAMR, and stochastic gate, and produces competitive segmentation masks without extra models or cues. Ablations and comparisons indicate strong mask quality and state-of-the-art-level accuracy despite the method’s simpler training setup.

  • 4.1. Setup: The Pascal VOC experiments use 10,582 training images with image-level annotations and 1,449 validation images.The model uses a WideResNet-38-based DeepLabv3+ implementation and trains for 20 epochs, enabling self-supervised segmentation after five classification-only epochs.
  • 4.2. Ablation study: 62.2% IoU is achieved with focal-penalty settings p = 3 and λ = 0.01 plus CRF refinement, versus 60.5% for p = 0.The focal penalty improves the baseline at negligible computational cost, although larger p values do not consistently benefit CRF processing.
  • 4.2. Ablation study: PAMR is crucial: removing it drops segmentation accuracy from 59.4% to 31.8%.Larger affinity receptive fields improve mask quality, while moderate changes to kernel shape have only small effects.
  • 4.2. Ablation study: Disabling the stochastic gate reduces mask IoU from 59.8% to 55.6%, while deterministic mixing reaches only 57.5% IoU.Global Cue Injection generally improves results and provides more stable performance across mixing rates.
  • 4.3. Comparison to the state of the art: The single-stage masks improve CAM baseline IoU by 18.9% and outperform recent multi-stage methods by 7.2% and 0.4% IoU.This mask-quality evaluation uses ground-truth image labels to remove masks for absent classes and requires neither additional models nor saliency detection.
  • 4.3. Comparison to the state of the art: IRNet exceeds the method by only 0.5% IoU, while SSDD exceeds it by 1.2% IoU at considerably greater model complexity.The approach also surpasses several multi-stage methods and previous single-stage methods without additional data, saliency supervision, or multiple decoder passes.
  • 4.3. Comparison to the state of the art: Qualitative results show masks aligning well with object boundaries and generalising to challenging scenes with varying object scales and semantic content.Observed failure modes resemble those of fully supervised segmentation systems, including fine-grained detail errors and occlusion-related mislabelling.

5. Conclusion

The paper concludes that a single segmentation network trained in one round can achieve state-of-the-art-level weakly supervised segmentation from image labels alone. Its design combines components targeting local consistency, semantic fidelity, and completeness while avoiding the complexity of many multi-stage methods.

  • 5. Conclusion: The proposed practical approach trains a single segmentation network in one round for weakly supervised semantic segmentation.It introduces a class aggregation function, local mask refinement module, and stochastic gate to address the three targeted mask properties.
  • 5. Conclusion: The method achieves segmentation accuracy on par with the state of the art and outperforms recent multi-stage methods using extra data or saliency supervision.The authors further state that auxiliary supervision may benefit the model and that it can suit downstream tasks without considerable deployment effort.
  • 5. Conclusion: The conclusion presents simplicity and one-round training as practical characteristics of the proposed approach.The paper frames these characteristics alongside competitive accuracy and reduced reliance on additional training resources.

– Supplemental Material –

The supplied supplemental-material passage identifies the paper’s authors and their institutional affiliation.

  • – Supplemental Material –: The paper lists Nikita Araslanov and Stefan Roth as authors.
  • – Supplemental Material –: The authors are affiliated with the Department of Computer Science.
  • – Supplemental Material –: The listed institution is TU Darmstadt.

A. On Training Stages

The paper distinguishes training stages by whether independent model parameters are learned, with offline intermediate outputs passed to later stages. This framing situates the proposed single-stage approach within the broader evolution toward increasingly complex pipelines.

  • A. On Training Stages: A training stage is defined as learning an independent set of model parameters with intermediate results saved offline for the next stage.Under this definition, extracting CAM seeds and then training a pixel-affinity network count as separate stages.
  • A. On Training Stages: The stage-counting convention makes explicit why methods that train standalone segmentation networks on pseudo-labels are multi-stage.Several cited approaches generate intermediate masks before training a final segmentation network.

B.1. Classification loss

The classification loss combines multi-label soft-margin supervision with normalized Global Weighted Pooling and a focal mask penalty. A positive offset ϵ improves stability and reduces negative-class bias while preserving the intended positive/negative score directions.

  • Classification loss: The multi-label soft-margin loss encourages positive classes to have yc > 0 and negative classes to have yc < 0.Its classwise loss saturates toward zero as positive-class scores approach infinity or negative-class scores approach negative infinity.
  • Score construction: The model defines class confidence at each pixel with a softmax over class score maps plus a fixed background channel y0,:,: ≡ 1.Pixel scores yc,i,j are aggregated into the class score yc, while the confidence map supplies the weights used by normalized Global Weighted Pooling.
  • Role of ϵ: The positive offset ϵ > 0 prevents division by zero and removes the discontinuity that otherwise destabilizes normalized Global Weighted Pooling near saturated negative-class scores.With ϵ = 0, the relevant multivariable limit does not exist when score values differ across pixel sites.
  • Negative classes: For negative classes, the focal penalty is emphasized because relative pixel weighting is meaningful for positive classes but not for negative ones.The authors observed that with ϵ > 0, the focal penalty contributes more to negative-class scores than normalized Global Weighted Pooling.
  • Negative classes: The loss counters a negative-class bias in which saturated negative scores continue decreasing while positive scores increase only marginally.The authors identify this behavior as potentially undesirable for real-world deployment and use ϵ = 1 in all experiments.
  • Segmentation loss: The segmentation loss is a weighted cross-entropy over pseudo-ground-truth pixels, with class weights compensating for unequal class-specific supervision.The aggregated loss is a weighted mean over batch samples, and a constant in the denominator provides numerical stability.

C. Quantitative Analysis

Per-class Pascal VOC results show that no prior method dominates every category, while the proposed single-stage method compares favorably with more complex approaches. It substantially improves over the previous best single-stage baseline on both validation and test sets.

  • Pascal VOC results: 9.9 and 10.6 percentage points: the proposed method improves over CRF-RNN’s 52.8% validation and 53.7% test IoU, respectively.CRF-RNN is identified as the best prior single-stage method using only image-level annotations.
  • Pascal VOC results: No previous method, including the state of the art, outperforms other pipelines across all Pascal VOC class categories.FickleNet leads only on bottle, bus, car, and tv, while SSDD has the highest mean IoU but is inferior on 10 of 21 classes.
  • Pascal VOC results: The proposed single-stage method compares favorably with multi-stage approaches that use saliency supervision or additional data.The comparison includes AffinityNet, which predicts pixel-level affinities from CAMs before training a segmentation network in a pseudo-fully-supervised regime.

D. Ablation Study: PAMR Iterations

PAMR iteration count controls a trade-off between mask quality, computational expense, and segmentation accuracy. Ten iterations provide the reported balance, while intermediate visualizations show refinement of coarse boundaries into pseudo-ground truth.

  • PAMR iterations: 10 PAMR iterations strike a balance between computational expense and segmentation accuracy.Fewer iterations decrease mask quality, while increasing the iteration count further provides diminishing benefits.
  • PAMR iterations: The ablation reports IoU on Pascal VOC validation with and without CRF refinement while varying the PAMR iteration number.The corresponding comparison is summarized in Table 7.
  • PAMR iterations: PAMR mitigates coarse early-stage mask boundaries by exploiting visual cues through pixel-adaptive convolutions.The revised masks are then used as pseudo-ground truth for self-supervised segmentation.

E. Pseudo Labels

Pseudo labels generated by the proposed method can support a separate fully supervised segmentation stage. Under equivalent image-level supervision, the resulting two-stage pipeline outperforms prior multi-stage frameworks and matches a framework using additional data and saliency detection.

  • Pseudo-label pipeline: The last-stage segmentation network is agnostic to pseudo-label generation; pseudo-label quality and ease of acquisition determine its usefulness.This motivates evaluating the generated masks as supervision for a separately trained segmentation model.
  • Pseudo-label pipeline: The pseudo labels enable DeepLabv3+ to be trained in a fully supervised regime as a simple two-stage pipeline.The pseudo ground truth is obtained from the proposed method with CRF refinement on the Pascal VOC train split.
  • Results: The resulting two-stage approach outperforms previous multi-stage frameworks under equivalent image-level supervision.The comparison is reported using mean IoU on Pascal VOC validation and test sets.
  • Results: The method attains mask accuracy on par with Frame-to-Frame despite that framework using saliency detectors, 15K extra data, and sophisticated network models.Frame-to-Frame is described as relying on PWC-Net and FickleNet in addition to those external resources.

F. Exchanging Backbones

The segmentation method generalises across VGG16, ResNet-50, ResNet-101, and WideResNet-38 backbones under compatible evaluation settings. Across all backbones, it improves over the CAM baseline in validation IoU, with further gains from CRF refinement.

  • VGG16, ResNet-50, and ResNet-101 serve as drop-in alternatives to WideResNet-38 for testing backbone generalisation.These models are trained on 448 × 448 image crops with the same data augmentation.
  • Validation IoU is measured on masks with false positives removed using ground-truth labels, enabling a focus on segmentation accuracy.WideResNet-38 is re-evaluated using the same multi-scale inference settings for compatibility across backbones.
  • 16.6% IoU average improvement over the CAM baseline is achieved without CRF post-processing, rising to 20.2% IoU with CRF refinement.The improvement occurs across all evaluated backbones.
Loading 2005.08104v1…