PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Semi-supervised Segmentation Beyond Accuracy: Reliability and Robustness

Steven Landgraf, Markus Hillemann, Markus Ulrich

arXiv:2506.05917v1cs.CVcs.AIcs.LG

TL;DR

Semi-supervised segmentation evaluation largely overlooks reliability and robustness despite their importance for deployment. The paper introduces RSS, a harmonic-mean metric combining accuracy, calibration, and uncertainty quality, and finds that UniMatchV2 improves predictive performance and robustness but often remains less reliable than a supervised counterpart.

  • Problem

    Semi-supervised segmentation evaluations focus mainly on accuracy while overlooking reliability and robustness needed for real-world deployment.

    Source: §1 · p. 2

  • Method

    The paper introduces RSS, which combines accuracy, calibration, and uncertainty quality through a harmonic mean that penalizes deficiencies in any component.

    Source: §3 · p. 5

  • Results

    UniMatchV2 achieves superior predictive performance and robustness but is often less calibrated and produces less reliable uncertainty estimates than its supervised counterpart.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Semi-supervised segmentation should be evaluated with holistic metrics such as RSS alongside accuracy to better reflect real-world deployment needs.

    Source: §1 · p. 2, §6 · p. 12

Abstract

from arXiv · show

Semantic segmentation is critical for scene understanding but demands costly pixel-wise annotations, attracting increasing attention to semi-supervised approaches to leverage abundant unlabeled data. While semi-supervised segmentation is often promoted as a path toward scalable, real-world deployment, it is astonishing that current evaluation protocols exclusively focus on segmentation accuracy, entirely overlooking reliability and robustness. These qualities, which ensure consistent performance under diverse conditions (robustness) and well-calibrated model confidences as well as meaningful uncertainties (reliability), are essential for safety-critical applications like autonomous driving, where models must handle unpredictable environments and avoid sudden failures at all costs. To address this gap, we introduce the Reliable Segmentation Score (RSS), a novel metric that combines predictive accuracy, calibration, and uncertainty quality measures via a harmonic mean. RSS penalizes deficiencies in any of its components, providing an easy and intuitive way of holistically judging segmentation models. Comprehensive evaluations of UniMatchV2 against its predecessor and a supervised baseline show that semi-supervised methods often trade reliability for accuracy. While out-of-domain evaluations demonstrate UniMatchV2's robustness, they further expose persistent reliability shortcomings. We advocate for a shift in evaluation protocols toward more holistic metrics like RSS to better align semi-supervised learning research with real-world deployment needs.

1 Introduction

Semi-supervised segmentation reduces dense-labeling costs, but existing evaluation emphasizes accuracy while overlooking reliability and robustness needed for real-world deployment. The paper proposes RSS and evaluates UniMatchV2 to promote more holistic protocols.

  • A Cityscapes image with 19 classes takes around 1.5 hours to label, making dense annotation a major deployment obstacle.Source: §1 · p. 1
  • Semi-supervised segmentation leverages many unlabeled images alongside a small labeled set to alleviate the cost of fully supervised learning.Source: §1 · p. 2
  • Reliability concerns whether confidence reflects correctness and uncertainty aligns with predictive ambiguities and errors, while robustness concerns performance under perturbations, noise, or distribution shifts.Source: §1 · p. 2
  • Existing semi-supervised segmentation evaluations have overlooked reliability and robustness despite their importance for real-world deployment.Source: §1 · p. 2
  • RSS combines accuracy, calibration, and uncertainty quality through a harmonic mean, while evaluations compare UniMatchV2 with UniMatchV1 and a supervised baseline.Source: §1 · p. 2
  • The study reports that semi-supervised methods may sacrifice calibration and uncertainty quality for accuracy, while UniMatchV2 may be more robust but retain reliability shortcomings.Source: §1 · p. 2

2 Related Work

Related work establishes reliability as calibration and uncertainty quality, and robustness as maintaining effectiveness under varied perturbations and shifts. The paper identifies a gap: these issues have been studied for segmentation, but not for semi-supervised segmentation.

  • Reliability includes calibration of predicted probabilities and uncertainty quality aligning confidence with predictive ambiguities and errors.Source: §2.2 · p. 3
  • Reliability shortcomings pose risks in safety-critical contexts such as autonomous driving, where models must remain reliable under unexpected scenarios.Source: §2.2 · pp. 3–4
  • Robustness includes effectiveness under perturbations, noise, distribution shifts, adversarial attacks, and natural changes in weather, lighting, or geography.Source: §2.2 · p. 4
  • Prior analyses examine reliability and robustness in semantic segmentation, but none considered semi-supervised learning.Source: §2.2 · p. 4
  • The paper asks whether semi-supervised evaluation should prioritize reliability and robustness alongside accuracy.Source: §2.2 · p. 4

3 Reliable Segmentation Score

The Reliable Segmentation Score combines segmentation accuracy, calibration, and uncertainty quality into a harmonic-mean metric. Its components are intended to capture complementary properties and prevent strong performance in one dimension from masking weakness in another.

  • mIoU measures pixel-wise segmentation accuracy, whereas ECE measures the mismatch between confidence and empirical accuracy across probability bins.Source: §3 · p. 4
  • Equal weighting is proposed so calibration and uncertainty quality are represented alongside accuracy in safety-critical applications.Source: §3 · p. 5
  • RSS integrates mIoU, calibration, and uncertainty quality through a harmonic mean that penalizes poor performance in any component.Source: §3 · p. 5
  • The four metrics are treated as complementary because high mIoU can coexist with poor calibration, while uncertainty metrics assess certainty utility and error detection.Source: §3 · pp. 5–6
  • p(acc|cer) measures whether low-uncertainty predictions are correct, while p(unc|inacc) measures whether incorrect predictions receive high uncertainty.Source: §3 · pp. 5–6
  • mIoU and ECE primarily use the highest softmax probability, missing whether probability mass is narrowly peaked or spread across classes.Source: §3 · p. 6

4 Experimental Setup

The study compares UniMatchV1 and UniMatchV2, which share weak-to-strong consistency but differ in architectural and augmentation designs, using in-domain and out-of-domain evaluations. It measures segmentation accuracy, calibration, and uncertainty quality with established metrics and Shannon-entropy uncertainty estimates.

Methodological Background

  • UniMatchV1 extends FixMatch with weak-image pseudo-labels, strong-image supervision, feature perturbation through dropout, and a dual-stream learning setup.Source: §4.1 · p. 6
  • UniMatchV2 retains weak-to-strong consistency while replacing ResNet encoders with pretrained Vision Transformers and adding Complementary Dropout across strongly augmented views.The design unifies UniMatchV1’s image- and feature-level augmentations into one stream, encouraging feature diversity while reducing training cost.Source: §4.1 · pp. 6–7

Datasets

  • Experiments use Cityscapes and Pascal VOC2012 for in-domain evaluation, with Foggy and Rainy Cityscapes providing progressively perturbed out-of-domain conditions.Source: §4.2 · p. 7

Evaluation Metrics

  • Evaluation follows the UniMatchV2 protocol for mIoU and uses 15-bin Expected Calibration Error to assess calibration.Source: §4.3 · p. 7

Uncertainty Measurement

  • Pixel-wise uncertainty is computed with Shannon entropy, then thresholded by each image’s median uncertainty to calculate p(acc|cer) and p(unc|inacc).Source: §4.3 · p. 7

5 Experiments

Experiments compare supervised, UniMatchV1, and UniMatchV2 models across in-domain datasets and adverse-weather distribution shifts using accuracy, calibration, uncertainty, and RSS. UniMatchV2 generally improves predictive performance and out-of-domain robustness, but the supervised baseline often remains more reliable because UniMatchV2 underestimates uncertainty for incorrect predictions.

In-Domain Evaluation

  • UniMatchV2 achieves higher mIoU than both baselines, yet the supervised baseline generally leads reliability metrics and RSS on Cityscapes, while UniMatchV2 leads RSS across Pascal VOC2012 label regimes.UniMatchV1 exceeds UniMatchV2 in RSS on 4 of 5 Cityscapes splits, whereas UniMatchV2’s Pascal VOC2012 advantage reflects stronger segmentation quality while remaining competitive on reliability metrics.Source: §5.1 · p. 8, Table 1 · p. 9
  • Qualitative Cityscapes examples show similar predictions, but UniMatchV2 assigns low uncertainty to multiple-human predictions in an unlabeled region where the supervised model is noisy yet appropriately uncertain.Source: §5.1 · p. 8, Figure 1 · p. 10

Qualitative Evaluation

  • On Pascal VOC2012 and perturbed Cityscapes examples, UniMatchV2 sometimes segments objects more accurately but assigns lower uncertainty to errors that the supervised model flags.Source: §5.1 · p. 8, §5.2 · p. 11, Figure 3 · p. 12

Out-of-Domain Evaluation

  • Under increasing fog and rain without fine-tuning, UniMatchV2 maintains higher mIoU and smaller RSS degradation, indicating stronger out-of-domain robustness.Calibration worsens under fog but is comparatively stable under rain, while uncertainty quality changes little under fog and improves under rain, especially for UniMatchV2.Source: §5.2 · p. 10, Figure 2 · p. 11
  • UniMatchV2 remains less reliable out of domain because p(unc|inacc) barely exceeds 0.6, compared with approximately 0.85 for the supervised baseline.Source: §5.2 · p. 10

6 Conclusion

The paper addresses the underassessment of reliability and robustness in semi-supervised segmentation by introducing RSS and evaluating UniMatchV2 across in-domain and out-of-domain settings. UniMatchV2 improves predictive performance and robustness but is often less calibrated and produces less reliable uncertainty estimates than the supervised baseline.

Conclusion

  • RSS combines accuracy, calibration, and uncertainty quality through a harmonic mean, penalizing poor performance in any component.Source: §6 · p. 12
  • Across in-domain and out-of-domain evaluations, UniMatchV2 achieves stronger predictive performance and robustness but is often less calibrated and less reliable in its uncertainty estimates than the supervised counterpart.Source: §6 · p. 12
  • The findings question whether incremental segmentation-accuracy gains represent meaningful progress toward reliable and robust deployment.Source: §6 · p. 12

Supplementary Material

During training, UniMatchV2’s mIoU improves more consistently than RSS, while out-of-domain evaluations show smaller RSS degradation under Foggy and Rainy Cityscapes perturbations.

Predictive Performance and Reliability during Training

  • 0.65 to around 0.84 mIoU on Cityscapes, while RSS fluctuates between 0.76 and 0.91 without consistent improvement.The Cityscapes model used only 1/16th of available labels; the RSS behavior suggests reliability metrics may degrade despite accuracy gains.Source: Supplementary Material · p. 20
  • 0.80 to 0.88 mIoU on Pascal VOC2012, while RSS rises from approximately 0.92 to 0.95 but less strongly than mIoU.The model was trained with 183 labeled images, and the weaker RSS gain indicates that improved segmentation quality does not proportionally improve reliability.Source: Supplementary Material · p. 20

Quantitative Out-of-Domain Analysis

  • UniMatchV2 is more robust under Foggy and Rainy Cityscapes perturbations, showing smaller RSS degradation across label fractions and encoder sizes without fine-tuning.Tables 2 and 3 evaluate the strongest perturbation settings and compare performance, calibration, and uncertainty robustness with original Cityscapes results.Source: Supplementary Material · p. 21, Table 2 · p. 21, Table 3 · p. 22
Loading 2506.05917v1…