PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Fewer Annotations: Active Learning via Region Impurity and Prediction Uncertainty for Domain Adaptive Semantic Segmentation

Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xinjing Cheng

arXiv:2111.12940v2cs.CVcs.LG

TL;DR

Domain-adaptive semantic segmentation faces costly annotation and generalization challenges. RIPU selects target regions using impurity and prediction uncertainty, and the authors report state-of-the-art results comparable to supervised counterparts.

  • Problem

    Domain-adaptive semantic segmentation seeks good target-domain performance with few annotations, while annotation is costly and generalization across testing situations remains challenging.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    RIPU selects target regions using region impurity and prediction uncertainty, alongside source-domain local consistency and target-domain negative learning.

    Source: §1 · p. 2

  • Results

    RIPU achieves new state-of-the-art results and performs comparably to supervised counterparts, with significant gains across two domain-adaptation benchmarks using DeepLab-v2 and DeepLab-v3+.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors report that image-based and point-based selection strategies are not effective for active domain adaptation in semantic segmentation.

    Source: §1 · p. 2

Abstract

from arXiv · show

Self-training has greatly facilitated domain adaptive semantic segmentation, which iteratively generates pseudo labels on unlabeled target data and retrains the network. However, realistic segmentation datasets are highly imbalanced, pseudo labels are typically biased to the majority classes and basically noisy, leading to an error-prone and suboptimal model. In this paper, we propose a simple region-based active learning approach for semantic segmentation under a domain shift, aiming to automatically query a small partition of image regions to be labeled while maximizing segmentation performance. Our algorithm, Region Impurity and Prediction Uncertainty (RIPU), introduces a new acquisition strategy characterizing the spatial adjacency of image regions along with the prediction confidence. We show that the proposed region-based selection strategy makes more efficient use of a limited budget than image-based or point-based counterparts. Further, we enforce local prediction consistency between a pixel and its nearest neighbors on a source image. Alongside, we develop a negative learning loss to make the features more discriminative. Extensive experiments demonstrate that our method only requires very few annotations to almost reach the supervised performance and substantially outperforms state-of-the-art methods. The code is available at https://github.com/BIT-DA/RIPU.

1. Introduction

Noisy self-training pseudo-labels can favor frequent classes and leave domain-adaptive segmentation behind supervised performance. RIPU queries diverse, uncertain image regions and combines region and pixel annotation with additional training mechanisms; experiments report gains across two domain-adaptation benchmarks.

1. Introduction

  • Self-training pseudo-labels are noisy under class imbalance, emphasizing frequent classes and sacrificing rare classes or small objects.The resulting performance lags behind supervised learning counterparts.Source: §1 · p. 1
  • Image-based selection can spend annotations on redundant areas, while point-based uncertainty sampling can miscalibrate or select redundant category-specific points under domain shift.Both alternatives underuse spatial proximity within an image.Source: Figure 1 · p. 1, §1 · p. 2
  • RIPU selects regions that combine spatial diversity with prediction uncertainty, using region impurity and mean pixel-prediction entropy.The method derives pseudo-labels, identifies candidate regions with the k-square-neighbors algorithm, then combines the two scores.Source: §1 · p. 2
  • The method offers Region-based Annotating at about 2.2% ground-truth pixels and Pixel-based Annotating at about 40 pixels, alongside source consistency and target negative learning.The authors report near-supervised performance with dramatically reduced human labeling costs.Source: §1 · p. 2
  • The authors benchmark prior active domain-adaptation methods for segmentation and find image-based or point-based selection strategies ineffective.The cited comparisons motivate region-based selection rather than labeling whole images or isolated points.Source: §1 · p. 2
  • RIPU brings significant performance gains across GTAV → Cityscapes and SYNTHIA → Cityscapes with DeepLab-v2 and DeepLab-v3+ models.The conclusion reports new state-of-the-art results and performance comparable to supervised counterparts.Source: §1 · p. 2

2. Related Work

Prior domain-adaptive segmentation work includes feature alignment and self-training, while active learning offers a route to reduce annotation effort. Existing image- and point-selection approaches leave spatial structure underused, motivating region-based selection.

2. Related Work

  • Domain-adaptive segmentation methods commonly use adversarial appearance transfer, feature matching, or output-space alignment.Earlier domain-adaptation work also minimizes source-target feature discrepancy.Source: §2 · p. 2
  • Self-training uses target pseudo-labels, but their noise and dependence on initialization motivate active learning as an alternative with minimal annotation workload.Other approaches add weak labels or dense depth information.Source: §2 · p. 2
  • Active learning aims to reduce labeling effort while maximizing model performance, using strategies such as uncertainty or representative sampling.Dense prediction annotation is more laborious than image-classification labeling, and comparatively little work addresses it.Source: §2 · p. 2
  • Transferring annotations from synthetic to real-world data remains relatively underexplored despite occurring frequently in practice.The dataset shift makes this an active domain-adaptation setting.Source: §2 · p. 2
  • In active domain-adaptive segmentation, image selection labels entire images and point selection neglects contextual structure and spatial contiguity.RIPU instead favors diverse, uncertain regions to seek high information content with low labeling costs.Source: §2 · p. 3

3. Approach

RIPU addresses domain shift by selecting informative target-image regions for annotation, then trains with labeled source and target data. It combines spatial class diversity and prediction uncertainty in selection, alongside source-domain consistency regularization and a target-domain negative-learning loss.

3.1. Preliminaries and Motivation

  • RIPU frames adaptation as learning target-domain segmentation from labeled source data and a small number of target annotations.Source-trained models may generalize poorly under domain shift, while self-training relies on noisy, confidence-filtered target pseudo-labels.Source: §3.1 · p. 3

3.2. Region Generation

  • Candidate regions are regularly shaped squares centered on pixels, though the method also permits other region divisions.This operational region differs from semantic regions defined by pixels belonging to one class.Source: §3.2 · p. 3

3.3. Region Impurity and Prediction Uncertainty

  • RIPU scores regions by combining impurity in their pseudo-label class composition with prediction uncertainty, then queries the highest-scoring regions or pixels.For region-based annotating, uncertainty is the average pixel entropy within the region; for pixel-based annotating, it is the entropy map itself.Source: §3.3 · p. 4
  • The acquisition score is the element-wise product of impurity and uncertainty, and selected regions or pixel neighborhoods do not overlap.The two criteria favor regions that are diverse in spatial adjacency and uncertain in prediction.Source: §3.3 · p. 4

3.4. Training Objectives

  • Source-domain consistency regularization aligns each pixel prediction with its neighborhood mean to promote smoother predictions and avoid source-data overfitting.The neighborhood used for the mean is 3×3.Source: §3.4 · pp. 4–5
  • The training objective combines supervised source and target losses with source consistency and a negative-learning loss using available binary negative pseudo-labels.Negative pseudo-labels mark target classes whose predicted probability falls below threshold τ, set to 0.05 in this work.Source: §3.4 · pp. 4–5

4. Experiments

Across GTAV→Cityscapes and SYNTHIA→Cityscapes, experiments evaluate active annotation using Cityscapes validation mIoU. RIPU outperforms prior methods and performs comparably to fully supervised training.

4.1. Comparisons with the state-of-the-arts

  • Against LabOR on GTAV→Cityscapes, Ours (PA) improves by 2.0 mIoU and Ours (RA) by 3.0 mIoU; both methods also beat corresponding state-of-the-art methods on SYNTHIA→Cityscapes.Overall, RIPU performs favorably against existing ADA and competing DA methods and comparably to the fully supervised model; it exceeds that model on some categories, including rider, bus, and motor.Source: §4.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.2. Qualitative results

  • RIPU predictions are smoother and contain fewer spurious areas than Source Only predictions, especially for hard classes.Figure 3 places the target image, ground truth, Source Only, Ours (RA), and Ours (PA) results side by side.Source: §4.2 · p. 6, Figure 3 · p. 6
  • RAND samples uniformly, while Uncertainty Only clusters nearby regions and Impurity Only favors multi-category areas; Ours (RA) selects diverse regions across the image.At the limited 2.2% budget, selected regions can look like edges; the authors also use Canny to select uncertain pixels from detected edges.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

4.3. Ablation Study

  • Region impurity and prediction uncertainty each improve on RAND, and combining them gives further gains; the full method performs best with complementary training losses.The authors report that impurity’s spatial-adjacency property can alleviate class imbalance; source consistency and target negative-learning losses each add gains over the combined selection criterion.Source: §4.3 · p. 7, Table 3 · p. 7

4.4. Further Analysis

  • With a 40-pixel budget, Ours (PA) gains 4.6 mIoU over RAND, while entropy and softmax-confidence selection fall behind RAND.For the 2.2% region budget, the authors report consistent increases over competing selection methods.Source: §4.4 · pp. 7–8, Table 4 · p. 7
  • For the 2.2% regions selected by Ours (RA), the authors report more regions containing rare classes than in the Cityscapes training-set distribution, particularly light, sign, rider, bus, train, and motor.Source: §4.4 · p. 8
  • For RA, k = 1 performs best because smaller regions use the budget more finely; for PA, k = 32 performs best to reduce repeated selections in one area.The authors recommend selecting pixels that cover the whole image to make full use of the limited budget.Source: §4.4 · p. 8, Figure 6 · p. 8
  • Adaptive k-square-neighbors regions outperform fixed rectangles and SEEDS superpixels on GTAV→Cityscapes, whose fixed shapes do not fit changing region importance.The authors describe the adaptive regions as allowing selection of diverse and uncertain image regions.Source: §4.4 · p. 8, Table 6 · p. 8

5. Conclusion

RIPU combines region-based acquisition with source-domain consistency regularization and target-domain negative learning, achieving state-of-the-art results comparable to supervised counterparts. The paper also describes its potential annotation-cost impact and reports the setup for its hyperparameter sensitivity experiments.

5. Conclusion

  • RIPU achieves new state-of-the-art results and performs comparably to supervised counterparts, using region selection alongside source consistency regularization and target negative learning.The experiments and ablation studies are presented as verifying the method’s effectiveness.Source: §5 · p. 8

A. Societal impact

  • The authors state that RIPU may reduce the cost of annotating out-of-domain data and report no observed societal issues in their benchmark experiments.They note that the work does not involve sensitive attributes.Source: A. Societal impact · p. 12

B. Sensitivity to different hyper-parameters

  • Hyperparameter sensitivity experiments use DeepLab-v2 with ResNet-101 on GTAV →Cityscapes and Ours (RA), with defaults τ=0.05, α1=0.1, and α2=1.0.Each sensitivity test varies one hyperparameter while fixing the others at their default values.Source: B. Sensitivity to different… · p. 12

B.1. Effect of the negative threshold τ

The sensitivity experiments identify a suitable range for the negative threshold and a best-performing consistency-loss weight. Performance is reported as stable across negative-learning-loss weights.

B.1. Effect of the negative threshold τ

  • Ours (RA) achieves consistent results for τ≤0.10, but performance drops at τ=0.20.The tested thresholds are 0.01, 0.02, 0.05, 0.08, 0.10, and 0.20.Source: §B.1 · p. 12

B.2. Effect of the consistency regularization loss weight α1

  • The best performance is achieved at α1=0.1; smaller or larger values induce weaker or stronger consistency constraints.α1=0.0 removes the source-data consistency constraint.Source: §B.2 · p. 12

B.3. Effect of the negative learning loss weight α2

  • Performance is stable across α2 values, which the authors interpret as indicating equal magnitude between negative-learning and supervised-learning losses.At α2=0.0, training omits the target-data negative-learning loss.Source: §B.3 · p. 12

C. Comparison with a simple edge detector

Compared with Canny plus uncertainty sampling, RIPU’s region-based selection performs better under both reported annotation budgets. The visual comparison also indicates that RIPU avoids selecting some edges inside objects.

C. Comparison with a simple edge detector

D.1. Details about comparison methods and performance per class

The comparison evaluates random, entropy-based, and softmax-confidence selection, alongside per-class intersection-over-union results. Ours (RA) is reported to work especially well for several rare object categories.

D.1. Details about comparison methods and performance per class

  • RAND samples pixels and regions randomly per target image; ENT selects highest-entropy pixels or regions by average pixel entropy, while SCONF uses softmax confidence scores.For SCONF region-based selection, regions with the largest average pixel score are selected; larger scores indicate less confidence.Source: §D.1 · p. 13
  • Ours (RA) works especially well for rare categories including fence, pole, sign, and rider, although the methods select more regions or pixels from majority classes.The authors describe the rare-category performance as a side effect of directly optimizing per-class IoU and mean IoU.Source: §D.1 · p. 13

D.2. t-SNE visualization

The t-SNE visualization compares sampled target-image pixels with the remaining pixels across selection strategies. RIPU (RA) samples regions that combine diversity and uncertainty, and is reported as the best option for performance gain and visual presentation.

D.2. t-SNE visualization

  • In Fig. 8, sampled pixels are shown as large triangles and remaining pixels as small points in the target-image feature visualization.The passage describes the visualization as a t-SNE view of feature representations.Source: §D.2 · pp. 13–14
  • RAND can spend annotation budget on redundant areas, while ENT and SCONF favor uncertain regions but seldom select infrequent categories such as signs, fences, and poles.RIPU (RA) can select those infrequent categories.Source: §D.2 · p. 14
  • Across strategies, RIPU (RA) samples diverse regions with more object categories and uncertain clusters around unpredictable object boundaries.Source: §D.2 · p. 14
  • Ours (RA) is reported as the best option among selection strategies for both performance gain and visual presentation.The performance comparison is identified as Table 11; the visualization comparison is Fig. 8.Source: §D.2 · p. 14

E. Extension of RIPU to source-free scenario

The paper extends RIPU to source-free domain adaptation, adapting a source-pretrained model with labeled active samples and unlabeled target data. With a 2.2% region budget, it reports improvements over existing SFDA approaches on both benchmarks, alongside slight performance degradation without source data during training.

E. Extension of RIPU to source-free scenario

  • The SFDA extension starts from a source-domain-pretrained model and optimizes with cross-entropy on active samples and negative learning on target data, without using the source domain.Source: E. Extension of RIPU to source-free… · p. 14
  • RIPU is evaluated on GTAV →Cityscapes and SYNTHIA →Cityscapes with DeepLab-v2 and ResNet-101, following the main paper’s training procedure.The reported training details include the learning-rate schedule, batch size, maximum iteration, and input size.Source: E. Extension of RIPU to source-free… · p. 14
  • 13.7∼22.0 mIoU on GTAV →Cityscapes and 16.7∼29.1 mIoU on SYNTHIA →Cityscapes are reported as improvements over existing SFDA approaches.The experiments use a 2.2% region-per-image annotation budget.Source: E. Extension of RIPU to source-free… · p. 14
  • The authors report slight performance degradation when source data does not participate in training.Source: E. Extension of RIPU to source-free… · p. 14

F. Additional qualitative results

The additional qualitative results show segmentation predictions and queried regions for GTAV and SYNTHIA adaptation to Cityscapes. The queried-region visualizations compare RIPU (RA) with random, uncertainty-only, and impurity-only selection.

F. Additional qualitative results

  • The additional comparisons cover GTAV →Cityscapes segmentation and queried-region results, as well as SYNTHIA →Cityscapes segmentation and queried-region results.Source: F. Additional qualitative results · p. 14
  • The GTAV →Cityscapes segmentation visualization presents the target image, ground truth, Source Only prediction, Ours (RA), and Ours (PA).Source: Figure 9 · p. 15
  • The SYNTHIA →Cityscapes segmentation visualization presents the target image, ground truth, Source Only prediction, Ours (RA), and Ours (PA).Source: Figure 10 · p. 16
  • For both GTAV and SYNTHIA →Cityscapes, queried-region visualizations show 2.2% annotations and report that Ours (RA) selects the most diverse and uncertain regions versus RAND, Uncertainty Only, and Impurity Only.Source: Figure 11 · p. 17, Figure 12 · p. 18

Related papers

Loading 2111.12940v2…