PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Adaptation for Semantic Segmentation with Maximum Squares Loss

Minghao Chen, Hongyang Xue, Deng Cai

arXiv:1909.13589v1cs.CV

TL;DR

Pixel-level annotation makes semantic segmentation expensive, motivating transfer from labeled synthetic data to unlabeled real-world data through UDA. The paper proposes maximum squares loss and image-wise weighting to address entropy-gradient and target-class imbalances, and reports state-of-the-art performance in synthetic-to-real and cross-city adaptation.

  • Problem

    In semantic-segmentation UDA, entropy minimization gives larger gradients to high-probability, easy-to-transfer target samples, creating probability imbalance.

  • Method

    The paper introduces maximum squares loss with a linearly increasing gradient, image-wise class weighting, and multi-level self-produced guidance for unlabeled target data.

  • Results

    The method achieves state-of-the-art performance in synthetic-to-real and cross-city adaptation experiments without a discriminator in adversarial learning methods.

  • Takeaways & Limitations

    The approach addresses easy-sample gradient dominance and target-domain class imbalance without additional structure, a discriminator, or pseudo-label computation.

  • Takeaways & Limitations

    The related CBST-SP method assumes shared spatial priors between source and target domains, an assumption that may not hold across different datasets.

Abstract

from arXiv · show

Deep neural networks for semantic segmentation always require a large number of samples with pixel-level labels, which becomes the major difficulty in their real-world applications. To reduce the labeling cost, unsupervised domain adaptation (UDA) approaches are proposed to transfer knowledge from labeled synthesized datasets to unlabeled real-world datasets. Recently, some semi-supervised learning methods have been applied to UDA and achieved state-of-the-art performance. One of the most popular approaches in semi-supervised learning is the entropy minimization method. However, when applying the entropy minimization to UDA for semantic segmentation, the gradient of the entropy is biased towards samples that are easy to transfer. To balance the gradient of well-classified target samples, we propose the maximum squares loss. Our maximum squares loss prevents the training process being dominated by easy-to-transfer samples in the target domain. Besides, we introduce the image-wise weighting ratio to alleviate the class imbalance in the unlabeled target domain. Both synthetic-to-real and cross-city adaptation experiments demonstrate the effectiveness of our proposed approach. The code is released at https://github. com/ZJULearning/MaxSquareLoss.

1. Introduction

Pixel-level annotation makes real-world semantic segmentation costly, motivating UDA from labeled synthetic to unlabeled real-world data. The paper identifies probability and class imbalance in entropy-minimization UDA and proposes maximum squares loss with image-wise weighting.

  • Motivation: Pixel-wise labels require substantial manual effort, while synthetic datasets offer more samples but transfer poorly to real-world images.UDA uses unlabeled target data to reduce the performance gap between labeled synthetic source data and unlabeled real-world target data.
  • Problem: Entropy minimization produces larger gradients for higher-probability target samples, allowing easy-to-transfer areas to dominate training.The paper terms this imbalance probability imbalance and notes that scaled entropy introduces a difficult-to-select hyper-parameter.
  • Maximum Squares Loss: Maximum squares loss uses a linearly increasing gradient to reduce excessive gradients from confident areas and train difficult samples more efficiently.It is defined as the negative sum of squared probabilities and is presented as a replacement for entropy minimization.
  • Image-wise Weighting: The paper introduces an image-wise weighting factor based on class percentages to address class imbalance in unlabeled target images.This weighting is described as more suitable for UDA than conventional class weighting factors.
  • Results: The approach reports competitive results with state-of-the-art methods across multiple UDA settings without additional structure or a discriminator.Unlike self-training, it does not require redundant computation to generate pseudo-labels.

2. Related Work

Related work addresses synthetic-to-real segmentation through domain alignment, output or conditional-distribution adaptation, style transfer, and semi-supervised techniques. Entropy minimization and pseudo-labeling are prominent strategies for using unlabeled target data.

  • Semantic Segmentation: Synthetic datasets reduce labeling costs, but rendering differences create a performance gap when transferring segmentation models to real-world datasets.Examples include SYNTHIA and GTA5 as synthetic sources and Cityscapes as a real-world target.
  • Unsupervised Domain Adaptation: Traditional UDA minimizes divergence between labeled source and unlabeled target feature distributions using methods such as MMD or adversarial learning.The goal is to learn domain-invariant features.
  • UDA for Semantic Segmentation: Segmentation UDA requires methods suited to higher-dimensional feature spaces, including global feature alignment, label-statistic matching, and structured output adaptation.Other approaches align conditioned distributions or transfer image style while preserving semantic labels.
  • Semi-supervised Learning Based Methods: Semi-supervised UDA methods use unlabeled data to align domains, including class-balanced self-training based on class-wise pseudo-label thresholds.These methods assume higher prediction probabilities indicate greater accuracy.
  • Semi-supervised Learning Based Methods: Entropy minimization makes unlabeled predictions less ambiguous by minimizing target prediction entropy and encouraging more separable classes.ADVENT applies this semi-supervised learning idea to UDA for semantic segmentation.

3. Methods

The method addresses probability and class imbalance in target-domain adaptation by combining maximum squares loss, image-wise weighting, and multi-level self-produced guidance.

  • 3.2. Maximum Squares Loss: Entropy minimization gives high-probability target samples disproportionately large gradients, allowing easy-to-transfer classes to dominate training.This probability imbalance hinders training for difficult-to-transfer classes.
  • 3.2. Maximum Squares Loss: Maximum squares loss uses a linearly increasing gradient to reduce the dominance of high-confidence areas while preserving their larger gradients.The authors report that this balances training across target samples and exceeds entropy minimization by a large margin in experiments.
  • 3.2.2 Interpretation from f-divergence View: From an f-divergence perspective, optimizing maximum squares loss maximizes Pearson χ2 divergence with the uniform distribution for class-wise distribution alignment.The loss pushes target features away from the decision boundary toward the corresponding source feature distribution.
  • 3.3. Image-wise Class-balanced Weighting Factor: Because target class labels are unavailable and target frequencies may differ from source frequencies, the method estimates class percentages separately for each target image.The loss averages are interpolated using image-level class counts to improve stability under inaccurate predictions.
  • 3.4. Multi-level Self-produced Guidance for UDA: The multi-level objective combines final and low-level target losses, using high-level ensemble predictions as self-produced guidance for low-level feature training.The guidance averages final and low-level output maps, with δ set to 0.95; λ_low is fixed at 0.1.

4. Experiment

The experiments compare entropy minimization with maximum squares loss on classification and evaluate the approach in synthetic-to-real and cross-city semantic-segmentation adaptation settings.

  • 4. Experiment: Experiments cover classification, synthetic-to-real adaptation, and cross-city adaptation to evaluate maximum squares loss for unsupervised domain adaptation.The paper states that these experiments demonstrate the effectiveness of the approach.

4.1. Datasets

The experiments cover classification and semantic-segmentation domain adaptation across synthetic-to-real and cross-city settings, with datasets, domains, and evaluation metrics specified for each task.

  • Classification: Office-31 contains 4,652 images across 13 categories and three domains: Amazon, Webcam, and DSLR.
  • Classification: The Office-31 evaluation covers six directed adaptation tasks among Amazon, Webcam, and DSLR.
  • Semantic Segmentation: Synthetic-to-real segmentation uses GTA5 or SYNTHIA as the labeled source and Cityscapes as the unlabeled target.
  • Semantic Segmentation: Segmentation performance is measured using per-class IoU and mIoU, with 19 classes for GTA5-to-Cityscapes and 16- or 13-class settings for SYNTHIA-to-Cityscapes.
  • Cross-City Adaptation: Cross-city adaptation transfers from labeled Cityscapes to four unlabeled NTHU city domains, evaluated on shared 13-class IoU and mIoU.

4.2. Implementation Details

Implementation uses standard backbone-based adaptation setups for classification and semantic segmentation, with source-target batches and task-specific optimization details.

  • Classification: Classification applies entropy minimization and maximum square loss to ImageNet-pretrained ResNet-50 models using SGD.Batches contain equal numbers of source and target samples; λT is 0.3 for maximum square loss and 0.03 for entropy minimization.
  • Semantic Segmentation: Semantic-segmentation experiments use Deeplabv2 with ImageNet-pretrained ResNet-101 backbones as the base model.
  • Semantic Segmentation: Before adaptation, the segmentation network is pretrained on the source domain for 70k steps.Training uses batch size 2, with one source and one target image, on a single NVIDIA 1080Ti GPU.
  • Semantic Segmentation: Segmentation optimization uses SGD with learning rate 2.5 × 10^-4, momentum 0.9, weight decay 5 × 10^-4, and a poly learning-rate schedule.Random mirroring and Gaussian blur are used for augmentation.
  • Semantic Segmentation: The segmentation loss weight is fixed at λT = 0.1, while image-wise weighting experiments use α = 0.2.

4.3. Experiments on Classification

On Office-31, maximum square loss substantially outperforms entropy minimization and DANN, with the classification analysis focusing on transfer difficulty.

  • Overall Results: Maximum square loss exceeds entropy minimization and DANN by a large margin on Office-31, despite being uncompetitive with state-of-the-art methods.
  • Difficulty Analysis: MaxSquare and entropy minimization achieve similar accuracy on the top set, while MaxSquare is much more accurate on the bottom set.The reported pattern attributes MaxSquare’s main improvement to difficult samples.

4.4. GTA5 to Cityscapes

Synthetic-to-real and cross-city experiments show that maximum square loss benefits difficult samples, image-wise weighting improves class balance, and multi-level guidance further improves performance.

  • Overall Results: MaxSquare+IW+Multi achieves state-of-the-art performance on GTA5-to-Cityscapes with a ResNet-101 backbone.MaxSquare+IW improves transfer for small object classes, while MaxSquare improves several hard-to-transfer classes over MinEnt†.
  • Overall Results: CBST-SP achieves similar results but assumes shared spatial priors between source and target domains, an assumption that may not hold across datasets.
  • Ablation Study: Adding image-wise weighting improves entropy minimization and maximum squares performance by nearly 1 point on GTA5-to-Cityscapes.The study describes image-wise weighting as a robust solution to class imbalance in the unlabeled target domain.
  • Ablation Study: Multi-level self-produced guidance can significantly improve final performance.
  • Parameter Sensitivity: Different α values preserve MaxSquare+IW’s advantage over MaxSquare, while δ has little effect and λT is empirically set to 0.1.

4.5. SYNTHIA to Cityscapes

On SYNTHIA-to-Cityscapes adaptation, the proposed methods achieve competitive results against existing approaches, with MaxSquare+IW improving several small-object classes over MaxSquare.

  • MaxSquare and MaxSquare+IW achieve competitive results against other methods on shared-class IoU and mIoU evaluations.
  • MaxSquare+IW surpasses MaxSquare on several small-object classes, including traffic light, traffic sign, and motorbike.

4.6. Cross City Adaptation

Cross-city adaptation on the NTHU dataset shows consistent gains for MaxSquare+IW under smaller domain shifts, while the method remains robust across transfer settings.

  • About 1 point: MaxSquare+IW outperforms other advanced methods in all four Cityscapes-to-NTHU city transfers.
  • These cross-city results support the effectiveness of both maximum squares loss and image-wise weighting.
  • Unlike self-training, the approach does not assume shared spatial priors between source and target domains.
  • The method is therefore reported as robust to various transfer settings.

5. Conclusion

The paper identifies probability imbalance in entropy minimization for segmentation UDA and proposes maximum squares loss with image-wise class weighting. Synthetic-to-real and cross-city experiments show state-of-the-art performance without adversarial discriminators.

  • The paper demonstrates a probability imbalance problem when entropy minimization is applied to UDA for semantic segmentation.
  • Maximum squares loss prevents easy-to-transfer classes from dominating target-domain training.
  • Optimizing maximum squares loss is equivalent to maximizing Pearson χ2 divergence with the normal distribution.
  • Image-wise class weighting is computed from each image’s predicted quantity for each class to address target-domain class imbalance.
  • Synthetic-to-real and cross-city experiments show state-of-the-art performance without an adversarial-learning discriminator.
Loading 1909.13589v1…