PaperlayerView exact revision on arXiv

Source-linked AI summary

GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models

Chen Liang, Wenguan Wang, Jiaxu Miao, Yi Yang

arXiv:2210.02025v1cs.CV

TL;DR

Dense softmax segmentation models focus on p(c|x) and neglect class-conditional feature densities, limiting their treatment of out-of-distribution data. GMMSeg combines EM-optimized per-class Gaussian mixtures with discriminatively trained representations, outperforming softmax counterparts on three closed-set datasets and performing promisingly in open-world settings without modification.

  • Problem

    Dense softmax segmentation models learn p(c|x) without modeling the underlying p(x|c), and therefore struggle to recognize out-of-distribution data.

  • Method

    GMMSeg models each class with a Gaussian mixture optimized by EM while training the dense feature representation end-to-end with discriminative loss.

  • Results

    GMMSeg outperforms softmax counterparts on ADE20K, Cityscapes, and COCO-Stuff, and reports promising closed-set and open-world performance using one model instance without modification.

  • Takeaways & Limitations

    GMMSeg demonstrates that generative classification can be integrated with discriminative representation learning for semantic segmentation across closed-set and open-world settings.

  • Takeaways & Limitations

    Standard EM can converge slowly and produce unsatisfactory results because its parameters are sensitive to initialization.

Abstract

from arXiv · show

Prevalent semantic segmentation solutions are, in essence, a dense discriminative classifier of p(class|pixel feature). Though straightforward, this de facto paradigm neglects the underlying data distribution p(pixel feature|class), and struggles to identify out-of-distribution data. Going beyond this, we propose GMMSeg, a new family of segmentation models that rely on a dense generative classifier for the joint distribution p(pixel feature,class). For each class, GMMSeg builds Gaussian Mixture Models (GMMs) via Expectation-Maximization (EM), so as to capture class-conditional densities. Meanwhile, the deep dense representation is end-to-end trained in a discriminative manner, i.e., maximizing p(class|pixel feature). This endows GMMSeg with the strengths of both generative and discriminative models. With a variety of segmentation architectures and backbones, GMMSeg outperforms the discriminative counterparts on three closed-set datasets. More impressively, without any modification, GMMSeg even performs well on open-world datasets. We believe this work brings fundamental insights into the related fields.

1 Introduction

GMMSeg replaces dense softmax classification with class-conditional Gaussian mixture modeling while retaining end-to-end discriminative representation learning. It reports gains over softmax counterparts across three closed-set datasets and promising performance in open-world settings without modification.

  • Motivation: Softmax segmentation models learn posterior p(c|x) and decision boundaries, but do not model the underlying data distribution or within-class variation.They use one weight vector per class, assuming unimodal class structure, and can struggle with out-of-distribution data.
  • Approach: GMMSeg models each class with a separate Gaussian mixture for p(x|c), optimized online with momentum Sinkhorn EM.The classifier remains synchronized with the evolving feature space during training.
  • Approach: GMMSeg jointly uses generative GMM classification and end-to-end cross-entropy training of the dense feature extractor.This combines density modeling with discriminative representation learning.
  • Properties: GMMSeg supports multimodal class densities and can reject abnormal inputs without architectural changes, retraining, or post-calibration.These properties follow from its mixture-based, distribution-preserving design.
  • Results: 0.6%–1.5%, 0.5%–0.8%, and 0.7%–1.7% mIoU gains are reported over softmax counterparts on ADE20K, Cityscapes, and COCO-Stuff, respectively.The method is evaluated with multiple segmentation architectures and backbones.
  • Results: GMMSeg reports promising results on both closed-set and open-world scenarios using a single model instance.The paper presents this as evidence for generative classifiers in dense segmentation.

2 Related Work

Related work contrasts discriminative segmentation, which focuses on class boundaries, with generative density modeling. GMMSeg combines EM-optimized class densities with discriminatively trained representations and applies this framework to anomaly segmentation without modification.

  • Semantic Segmentation: FCN-style segmentation commonly learns dense representations and categorization end-to-end through discriminative softmax classifiers.Prior work mainly modifies architectures to improve receptive fields, context, or related representation properties.
  • Discriminative vs Generative Classifiers: Discriminative classifiers learn class boundaries p(c|x), whereas generative classifiers learn class densities p(x|c).Generative classification first models p(x,c) and then applies Bayes rule.
  • GMMSeg: GMMSeg optimizes its GMM via EM while training the deep representation with discriminative loss, forming a hybrid generative-discriminative model.The related-work passage distinguishes this from GMM classifiers trained entirely discriminatively.
  • Anomaly Segmentation: GMMSeg handles anomaly segmentation through its generative classifier without external outlier datasets or additional image-resynthesis models.This differs from uncertainty-estimation methods that post-process softmax segmentation outputs.
  • Anomaly Segmentation: GMMSeg reports promising closed-set and open-world results with one model instance and no changes to network architecture, training, or inference protocols.The paper positions this as a distinction from prior density-estimation and anomaly-segmentation approaches.

3 Methodology

GMMSeg reframes semantic segmentation as dense generative classification by modeling class-conditional feature distributions with GMMs, while retaining discriminative end-to-end representation learning. Its hybrid design addresses limitations of softmax-based classifiers, supports multimodal class densities, and can handle off-manifold inputs.

  • Existing segmentation solutions: Modern segmentation models typically combine an encoder-decoder feature extractor with a softmax classifier that directly models p(c|x).The feature extractor and classifier are jointly optimized with cross-entropy, making the overall model a dense discriminative classifier.
  • Existing segmentation solutions: Softmax-based segmentation mainly learns class decision boundaries, assumes unimodal class distributions, and struggles to identify unseen out-of-distribution pixels.These limitations arise from modeling p(c|x) without explicitly modeling the input distribution or class-conditional densities.
  • GMMSeg generative classification: GMMSeg models p(x|c) with a mixture of M multivariate Gaussians for each class and derives p(c|x) using Bayes rule.The mixture components have independent covariance structures, allowing flexible local modeling across feature dimensions.
  • GMMSeg generative classification: GMMSeg couples EM-based generative optimization of the GMMs with discriminative end-to-end learning of the dense feature extractor.This separates class-density estimation from representation learning while allowing both processes to cooperate during training.
  • GMMSeg generative classification: Standard EM can converge slowly and produce unsatisfactory results, so GMMSeg uses a constrained Sinkhorn EM procedure that is less prone to local optima.The method introduces uniform mixture-weight constraints and uses Sinkhorn-Knopp iteration for the E-step.
  • GMMSeg generative classification: The hybrid design preserves compatibility with modern segmentation architectures and gives class-conditional likelihoods for handling off-manifold examples.The authors describe the generative and discriminative components as independently optimized but closely collaborative.

4 Experiments

GMMSeg is evaluated across standard semantic segmentation, anomaly segmentation, and diagnostic settings. It improves segmentation performance, supports anomaly rejection without modification, and benefits from hybrid training and mixture modeling with minimal inference overhead.

  • Experimental Setup: Experiments cover semantic segmentation on three datasets using four architectures and diverse backbones, plus anomaly segmentation and diagnostic studies.The semantic segmentation evaluation uses ADE20K, Cityscapes, and COCO-Stuff; anomaly experiments use Fishyscapes Lost&Found and Road Anomaly.
  • Semantic Segmentation: GMMSeg outperforms discriminative counterparts across datasets, backbones, and network architectures in the reported mean-IoU comparisons.Table 1 reports results on ADE20K validation, Cityscapes validation, and COCO-Stuff test sets.
  • Anomaly Segmentation: GMMSeg outperforms competitors for anomaly segmentation without external out-of-distribution data, extra resynthesis, or post-processing.Its likelihood-based confidence scores support direct anomaly scoring, while the same Cityscapes-trained models are used for evaluation.
  • Anomaly Segmentation: Qualitative anomaly heatmaps show softmax predictions ignoring anomalies while GMMSeg rejects them as high-anomaly regions.The comparison is between MSP-DeepLabV3+ and GMMSeg-DeepLabV3+ on Fishyscapes Lost&Found.
  • Diagnostic Experiments: Online hybrid training raises ADE20K mIoU from 31.6% to 46.0% compared with fitting a GMM onto features pretrained with a softmax classifier.The result supports jointly updating the discriminative feature extractor and generative classifier.
  • Diagnostic Experiments: Increasing Gaussian components per class improves mIoU from 44.2% with one component to 46.0% with five components.The one-component setting corresponds to a single-Gaussian baseline, while the mixture model represents multimodal class densities.
  • Runtime: GMMSeg runs at 13.37 fps versus 14.16 fps for its softmax counterpart on one NVIDIA GeForce RTX 3090 GPU.The reported measurement uses batch size one.

5 Conclusion

The paper presents GMMSeg as a generative neural framework for semantic segmentation that models feature distributions with GMMs while learning representations discriminatively. The framework is reported as applicable to both closed-set and open-world settings.

  • GMMSeg explicitly models data distributions as Gaussian mixture models for semantic segmentation.Its generative classifier is jointly optimized with end-to-end discriminative representation learning.
  • The framework is reported to address limitations of softmax-based discriminative segmentation in both closed-set and open-world settings.The conclusion describes the model as principled and applicable across both settings.
  • The authors identify image classification and trustworthy AI as future application areas.These are presented as future work rather than evaluated results in the paper.

SUMMARY OF THE APPENDIX

The appendix provides additional training information, experimental results, and qualitative visualizations to supplement the main paper.

  • The appendix includes detailed training parameters.
  • The appendix includes more experimental results.
  • The appendix includes more qualitative visualizations.

A Detailed Training Parameters

The appendix evaluates GMMSeg on six base segmentation architectures, adding FCN and Mask2Former to the four architectures presented in the main paper. It follows the official Mask2Former and MMSegmentation training settings for these models.

  • The supplementary material evaluates GMMSeg on six base segmentation architectures.The four main-paper architectures are DeepLabv3+, OCRNet, Swin-UperNet, and SegFormer; FCN and Mask2Former are additional architectures.
  • FCN and Mask2Former are the two additional architectures supplied in the supplementary material.
  • Training follows the official Mask2Former codebase and MMSegmentation settings for the respective architectures.

B More Experimental Results

Additional experiments show that replacing softmax with GMMSeg’s generative GMM classifier improves diverse segmentation formulations and provides strong anomaly-segmentation performance. Performance also improves as EM memory capacity increases until the stored samples sufficiently represent the training distribution.

  • Additional base architectures: GMMSeg boosts both pixel-classification and mask-classification architectures, achieving 56.7%/83.8%/52.0% mIoU with Mask2Former-SwinLarge on ADE20K/Cityscapes/COCO-Stuff.The classifier replacement preserves the base architecture while changing the discriminative softmax into a generative GMM.
  • Anomaly segmentation: GMMSeg surpasses state-of-the-art anomaly-segmentation methods without post-calibration, with test-set gains of +24.58%/+14.91% in AP and +22.91%/+3.68% in FPR95 on Lost&Found/Static.The reported gains are listed in the order Fishyscapes Lost&Found and Static.
  • Anomaly segmentation: GMMSeg outperforms benchmark methods using additional training networks or data on Fishyscapes Lost&Found test.The authors attribute this robustness to accurate data-density modeling.
  • Memory capacity: Increasing EM memory capacity improves performance until stored pixel samples sufficiently represent the whole training-set distribution.With zero memory, EM estimates the distribution only within mini-batches and performs worse.

C More Qualitative Visualization

Qualitative visualizations compare GMMSeg with baseline segmentation models across three semantic-segmentation datasets and with MSP on Fishyscapes. The comparisons emphasize more precise category predictions and improved anomaly identification.

  • Semantic segmentation: On ADE20K, Cityscapes, and COCO-Stuff, GMMSeg produces more precise predictions and is less confused by object categories than SegFormer.The comparisons use SegFormer-MiTB5 and GMMSeg equipped with the same backbone.
  • Anomaly segmentation: On Fishyscapes Lost&Found validation images, GMMSeg successfully identifies anomalies rather than being overwhelmed by overconfident predictions like MSP.The visualization compares MSP-DeepLabV3+ with GMMSeg-DeepLabV3+ and includes anomaly heatmaps.
Loading 2210.02025v1…