Source-linked AI summary
ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, Patrick Pérez
TL;DR
Semantic segmentation models can lose substantial performance under source-target domain shifts, motivating unsupervised adaptation for settings such as synthetic-to-real transfer. The paper proposes direct entropy minimization and adversarial weighted-self-information adaptation, achieving state-of-the-art results on two synthetic-to-real benchmarks and promising detection results. Its main scope boundary is the target-pseudo-label assumption used to formulate the self-training connection.
Problem
Source-trained semantic segmentation models often generalize poorly to different target domains, causing severe performance loss in real-world and synthetic-to-real settings.
Method
The paper proposes complementary direct entropy-loss and adversarial weighted-self-information objectives for unsupervised domain adaptation in semantic segmentation.
Results
The models achieve state-of-the-art performance on the two main synthetic-to-real semantic-segmentation benchmarks, with AdvEnt reaching 43.8 mIoU on GTA5→Cityscapes.
Takeaways & Limitations
Entropy-based adaptation improves semantic segmentation UDA across the two challenging synthetic-to-real benchmarks, and the approach also shows a promising result for object detection.
Takeaways & Limitations
The self-training formulation assumes that high-scoring target pixels are correct with high probability, using them as pseudo-labels.
Abstract
from arXiv · showhide
Semantic segmentation is a key problem for many computer vision tasks. While approaches based on convolutional neural networks constantly break new records on different benchmarks, generalizing well to diverse testing environments remains a major challenge. In numerous real world applications, there is indeed a large gap between data distributions in train and test domains, which results in severe performance loss at run-time. In this work, we address the task of unsupervised domain adaptation in semantic segmentation with losses based on the entropy of the pixel-wise predictions. To this end, we propose two novel, complementary methods using (i) entropy loss and (ii) adversarial loss respectively. We demonstrate state-of-the-art performance in semantic segmentation on two challenging "synthetic-2-real" set-ups and show that the approach can also be used for detection.
1. Introduction
The paper targets severe source-to-target generalization loss in semantic segmentation UDA, especially synthetic-to-real settings. It adapts entropy minimization through direct entropy and adversarial objectives, reporting cleaner outputs and state-of-the-art benchmark performance.
- Motivation: Synthetic-to-real domain shifts can cause drastic performance drops when segmentation models trained on source data are applied to real target scenes.The challenge is particularly important for systems operating across varied urban environments and adverse weather.
- Motivation: Source-only models tend to produce low-entropy predictions on source-like images but noisy, high-entropy predictions on target-like images.The paper uses this contrast as the basis for adapting target predictions.
- Approach: The proposed UDA methods directly minimize target prediction entropy or indirectly match source and target distributions through adversarial weighted self-information.The direct method constrains independent pixel-wise predictions, whereas the adversarial method targets global distribution structure.
- Approach: The entropy loss penalizes low-confidence target predictions without significant overhead, while adversarial training also adapts target output structure.Additional practices include selecting specific entropy ranges and incorporating class-ratio priors.
- Results: The models outperform state-of-the-art approaches on several semantic-segmentation UDA benchmarks, especially GTA5→Cityscapes and SYNTHIA→Cityscapes.The objectives are described as pushing decision boundaries toward low-density regions in target prediction space and producing cleaner outputs.
2. Related works
Prior semantic-segmentation UDA work mainly reduces source-target discrepancy, uses self-training, or applies generative adaptation. This paper positions entropy-based UDA as a complementary approach and reports it as the first successful application yielding competitive segmentation performance.
- Discrepancy and adversarial methods: UDA methods commonly reduce discrepancy between source and target feature or output distributions using MMD or adversarial training.These methods may align representations at a single level or across multiple levels.
- Generative methods: Generative approaches adapt domains at the pixel level, feature level, or through channel-wise feature alignment while preserving source semantic content.Examples include CycleGAN-based image translation, feature reconstruction, and generated images combining source content with target style.
- Self-training: Self-training uses predictions from an ensemble or previous model state as pseudo-labels for unlabeled target data, sometimes with class balancing and spatial priors.The paper notes an explicit connection between self-training and entropy minimization.
- Other approaches: Other UDA methods combine adversarial and generative losses or enforce consistency across local and global label distributions.These approaches broaden adaptation beyond a single discrepancy objective.
- Entropy-based adaptation: Entropy minimization has been used in semi-supervised learning, clustering, and classification domain adaptation, but this paper reports its first successful application to competitive semantic-segmentation UDA.The novelty claim is specifically about applying entropy-based UDA training to semantic segmentation.
3. Approaches
The paper presents two complementary entropy-based UDA approaches for semantic segmentation: directly minimizing target prediction entropy and adversarially aligning weighted self-information across domains.
- Approaches: The framework adds a domain-adaptation branch to existing semantic segmentation models while retaining supervised source-domain training.Target images lack annotations, so adaptation relies on unsupervised constraints rather than the source segmentation loss.
- 3.1. Direct entropy minimization: Direct entropy minimization reduces the sum of normalized pixel-wise Shannon entropies on target predictions.The entropy map contains independent pixel-wise entropies normalized to [0, 1], and its loss sums them over the image.
- 3.1. Direct entropy minimization: Entropy minimization is a soft-assignment analogue of pseudo-label cross-entropy that avoids threshold-scheduling procedures.Unlike self-training, the approach does not require selecting high-confidence pixels with a fixed or scheduled threshold.
- 3.2. Minimizing entropy with adversarial learning: The adversarial approach aligns source and target distributions in weighted self-information space using a discriminator and a segmentation network trained to fool it.Weighted self-information vectors disentangle the Shannon entropy, while the discriminator classifies source versus target maps.
- 3.2. Minimizing entropy with adversarial learning: During adversarial training, the discriminator distinguishes source and target maps, while the segmentation network is optimized to make them indistinguishable.The discriminator and segmentation network are alternatively optimized using separate objectives, with λadv weighting the adversarial term.
- 3.3. Incorporating class-ratio priors: A source-label class prior can penalize excessive discrepancies between target expected class probabilities and the source class distribution.The relaxation parameter μ accounts for the fact that a single target image’s class distribution need not match the source prior.
4. Experiments
Experiments evaluate entropy-based adaptation on synthetic-to-real semantic segmentation benchmarks using GTA5 or SYNTHIA as sources and Cityscapes as the unlabeled target. MinEnt and AdvEnt achieve state-of-the-art results, with performance depending on the base CNN, entropy selection, and model combination.
- Experimental details: Experiments use GTA5→Cityscapes and SYNTHIA→Cityscapes, training on annotated synthetic images with unlabeled Cityscapes images and evaluating mIoU on Cityscapes validation data.GTA5 provides 24,966 frames and SYNTHIA provides 9,400 images; both setups use 2,975 unlabeled Cityscapes training images and 500 validation images.
- Experimental details: The models use DeepLab-V2 with ASPP and compare VGG-16 and ResNet-101 base CNNs, while the adversarial discriminator classifies source versus target weighted self-information maps.The discriminator uses four convolutional layers with leaky-ReLU activations and a classifier output for domain identification.
- Results: The proposed models achieve state-of-the-art performance across the two UDA semantic-segmentation benchmarks, with behavior varying across training sets and base CNNs.The experiments compare direct entropy minimization and adversarial entropy adaptation against multiple baselines.
- Discussion: 43.1% mIoU is obtained on GTA5→Cityscapes by MinEnt trained on the top 30% highest-entropy pixels, while that selection gains 0.8% mIoU over vanilla MinEnt.The authors note that high-entropy pixels can include correct but low-confidence predictions for the ResNet-101-based model, an assumption not holding for VGG-16.
- Results: 43.8 mIoU is achieved by AdvEnt on GTA5→Cityscapes, exceeding MinEnt by 3.3% with VGG-16 and 1.5% with ResNet-101.The authors associate the larger VGG-16 gain with the value of adversarial regularization on structured outputs for a lower-capacity base model.
- Results: 45.5% mIoU is achieved by combining MinEnt and AdvEnt on Cityscapes, while the SYNTHIA ensemble reaches 41.2% and 48.0% mIoU on 16- and 13-class subsets.The combined models are described as complementary because entropy loss acts at pixel level whereas adversarial loss operates more on image-level scene topology.
5. Conclusion
The paper addresses unsupervised domain adaptation for semantic segmentation with two complementary entropy-based approaches. They achieve state-of-the-art results on two synthetic-to-real benchmarks, improve further when ensembled, and show a promising object-detection result.
- The proposed models achieve state-of-the-art performance on the two challenging synthetic-to-real semantic-segmentation benchmarks.
- Ensembling the two entropy-based models improves performance beyond either single model.
- The approach also produces a promising result for unsupervised domain adaptation in object detection.
A. Entropy-based UDA for object detection
For object detection, the paper applies entropy-based adaptation to SSD-300 with a VGG-16 backbone. It minimizes target detection entropy directly or adapts weighted self-information maps adversarially alongside supervised source detection losses.
- The detection experiments use SSD-300 with VGG-16 as the backbone, producing dense predictions across feature maps and anchor boxes before non-maximum suppression.
- Direct entropy minimization: Direct entropy minimization computes normalized box-level entropies across anchor boxes and feature-map resolutions, then sums them into the target entropy loss.
- Direct entropy minimization: The model jointly optimizes supervised source detection losses and the unsupervised target entropy loss.
- Entropy minimization with adversarial learning: The adversarial approach applies the semantic-segmentation adversarial framework to weighted self-information maps derived from soft-detection maps.