PaperlayerView exact revision on arXiv

Source-linked AI summary

SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation

Binhui Xie, Shuang Li, Mingjia Li, Chi Harold Liu, Gao Huang, Guoren Wang

arXiv:2204.08808v2cs.CV

TL;DR

Domain adaptive semantic segmentation must produce dense predictions on unlabeled target domains despite source–target domain shift and dispersed target representations. SePiCo addresses this with one-stage semantic-guided pixel contrast using centroid- and distribution-aware concepts, and reports strong results across synthetic-to-real and daytime-to-nighttime adaptation.

  • Problem

    Source-trained segmentation models struggle on realistic target domains, while prior methods may leave target representations dispersed and rely on staged training with initialized pseudo-labels.

  • Method

    SePiCo is a one-stage framework that guides cross-domain pixel contrast with category centroids or estimated category distributions, using a closed-form upper bound for efficient optimization.

  • Results

    SePiCo achieves superior performance across synthetic-to-real and daytime-to-nighttime benchmarks, including mIoUs of 61.0%, 58.1%, and 45.4% on three reported adaptation tasks.

  • Takeaways & Limitations

    Semantic concepts can guide more effective domain alignment and improve the generalization capacity of pixel representations and self-training.

  • Takeaways & Limitations

    Prior centroid-based statistics can omit semantic diversity, while memory-bank representations become inefficient despite covering more information as its size grows.

Abstract

from arXiv · show

Domain adaptive semantic segmentation attempts to make satisfactory dense predictions on an unlabeled target domain by utilizing the supervised model trained on a labeled source domain. In this work, we propose Semantic-Guided Pixel Contrast (SePiCo), a novel one-stage adaptation framework that highlights the semantic concepts of individual pixels to promote learning of class-discriminative and class-balanced pixel representations across domains, eventually boosting the performance of self-training methods. Specifically, to explore proper semantic concepts, we first investigate a centroid-aware pixel contrast that employs the category centroids of the entire source domain or a single source image to guide the learning of discriminative features. Considering the possible lack of category diversity in semantic concepts, we then blaze a trail of distributional perspective to involve a sufficient quantity of instances, namely distribution-aware pixel contrast, in which we approximate the true distribution of each semantic category from the statistics of labeled source data. Moreover, such an optimization objective can derive a closed-form upper bound by implicitly involving an infinite number of (dis)similar pairs, making it computationally efficient. Extensive experiments show that SePiCo not only helps stabilize training but also yields discriminative representations, making significant progress on both synthetic-to-real and daytime-to-nighttime adaptation scenarios.

1 INTRODUCTION

Domain adaptive semantic segmentation addresses severe performance degradation when models trained on synthetic or labeled source data encounter realistic unlabeled target domains. SePiCo uses semantic-guided pixel contrast within one-stage adaptation to learn more discriminative, cross-domain pixel representations and improve self-training.

  • Motivation: Synthetic-trained segmentation models often drop substantially in realistic scenarios because pixel-level annotation is expensive and source–target domain shift remains.The motivation includes diverse weather and illumination conditions in autonomous driving.
  • Limitations of prior work: Existing alignment methods can bring domains closer globally without ensuring that target representations from different classes are well separated.Visual variation across images, including color, scale, and illumination, can hinder continual cross-domain pixel-representation learning.
  • Approach: SePiCo contrastively matches pixels to semantic concepts, attracting same-concept features and separating different-concept features across domains.The framework explores global prototypes, local category centroids stored in a memory bank, and category distributions as semantic descriptions.
  • Limitations of prior work: Self-training methods still face dispersed target representations, noisy pseudo-labels, and stage-wise dependence on a well-initialized model.Several approaches combine adversarial training or auxiliary tasks with self-training to learn more discriminative target representations.
  • Approach: A distribution-aware contrastive objective uses category statistics to represent richer semantic variation and derives a closed-form upper bound involving infinitely many pairs.This formulation is designed to improve efficiency while retaining broad semantic information.
  • Results: SePiCo reports mIoUs of 61.0%, 58.1%, and 45.4% on GTAV → Cityscapes, SYNTHIA → Cityscapes, and Cityscapes → Dark Zurich, respectively.With the latest Transformer, it further improves by 9.3%, 5.6%, and 8.0%, respectively, setting new state-of-the-art results.
  • Scope: The framework is presented as a general adaptation method that also generalizes to unseen target domains and can be applied to object detection.It is described as robust across daytime and nighttime segmentation situations.

2 RELATED WORK

Prior work in semantic segmentation and domain adaptation relies on increasingly capable architectures, adversarial alignment, self-training, and representation learning. SePiCo instead targets semantic cross-domain pixel structure with one-stage contrastive adaptation applicable to daytime and nighttime settings.

  • Semantic segmentation: Modern semantic segmentation architectures improve context modeling but require large labeled datasets, limiting their use under annotation scarcity.The related work discusses fully convolutional networks, DeepLab, and Transformer-based segmentation.
  • SePiCo: SePiCo maps pixels from different semantic classes into a distinctive feature space, producing source-discriminative and cross-domain-aligned pixel features.Its one-stage end-to-end framework is designed to optimize pixel embeddings without separate preprocessing stages.
  • SePiCo: Unlike specialized daytime or nighttime methods, SePiCo is presented as a universal framework for both appearance-shifted segmentation settings.The paper positions this as a distinction from methods designed separately for each task.
  • Domain adaptation: Adversarial domain adaptation reduces distribution shift at image, feature, or output levels, while self-training exploits unlabeled target data through pseudo-labels.Self-training methods also use confidence estimation, consistency regularization, class balance, and label denoising.
  • Domain adaptation: Existing self-training approaches can produce dispersed target representations, depend on warmup models, and often require separate stages to initialize pseudo-labels.These issues make initial pseudo-label reliability difficult to tune.
  • Representation learning: Dense contrastive-learning methods generally learn category correspondence across views or supervised settings rather than semantic concepts across domains.SePiCo constructs contrastive pairs using semantic information to bridge domain shift.

3 METHODOLOGY

The methodology section introduces the framework's overall idea before detailing semantic statistics, the framework itself, and the training algorithm. This organization proceeds from background and conceptual setup to implementation and optimization.

  • 3.1 Background and overall idea: Section 3.1 introduces the background and illustrates SePiCo's overall idea.This establishes the conceptual setup before the framework details.
  • 3.2–3.3 Method details: Section 3.2 explains semantic-statistics calculation, while Section 3.3 elaborates the framework.The passage presents these as the central methodological detail sections.
  • 3.4 Training procedure: Section 3.4 presents the training procedure and SePiCo algorithm.It completes the methodology progression from concept and statistics to training.

3.1 Background

SePiCo builds on teacher-student self-training for domain-adaptive semantic segmentation, adding semantic-guided pixel contrast and class-balanced cropping to address domain alignment and class imbalance.

  • Problem formulation: The model uses an encoder, segmentation head, and projection head within a teacher-student architecture for labeled source and unlabeled target images.The teacher supplies source feature maps and target pseudo labels, while the student produces trainable representations and predictions.
  • Overall motivation: SePiCo combines self-training losses with centroid-aware or distribution-aware pixel contrast in a one-stage framework.The projection head is discarded after training, leaving the encoder and segmentation head for segmentation.
  • Self-training domain adaptation revisit: Source and target images are processed by student and momentum-updated teacher networks to obtain feature maps and pixel-level predictions.No gradients are back-propagated into the teacher network.
  • Self-training domain adaptation revisit: Self-training minimizes source cross-entropy and trains the student on target data using teacher-generated pseudo labels and confidence weights.Pseudo labels are computed online, and the ratio of sufficiently confident pixels provides the target confidence weight.
  • Overall motivation: Traditional self-training neglects explicit domain alignment, leaving same-class features across domains potentially far apart and target representations dispersed.SePiCo introduces semantic-guided representation learning to address this limitation.

3.2 Semantic Statistics Calculation

SePiCo represents semantic categories through source-domain statistics, progressing from centroids to dynamically estimated distributions that capture richer feature variation.

  • Local semantic statistics: Source feature maps are partitioned by semantic masks, and local category centroids are computed from the features belonging to each class.The centroid for category k is calculated over the corresponding masked feature set.
  • Centroid-aware statistics: Global category prototypes are estimated online by aggregating class means across source images.The resulting prototype set contains one mean vector for each of the K categories.
  • Centroid-aware statistics: A memory bank stores local centroids from recent source images in K class-specific queues, replacing old centroids with current ones.This increases the available semantic samples but can suffer class bias when some categories rarely appear.
  • Centroid-aware statistics: Global prototypes capture common category appearance but may omit variation, whereas memory banks provide more semantic information at higher storage and efficiency costs.The discussion motivates a distributional representation as a more comprehensive alternative.
  • Distribution-aware statistics: Distribution-aware statistics use class means and covariance matrices estimated from labeled source features to represent category variation.These statistics are dynamically updated with source feature maps from the momentum-updated teacher network.

3.3 Semantic-Guided Pixel Contrast

Semantic-guided pixel contrast organizes source and target embeddings around class semantics, using centroids or estimated distributions to form positive and negative relationships efficiently.

  • Pixel contrast formulation: Pixel contrast assigns positive relationships to same-class representations and negative relationships to representations from different classes across source and target domains.The unified loss aims to pull same-category pixels together and separate different categories.
  • Pixel contrast formulation: Applying contrastive loss in both domains improves source pixel discriminability while reducing target intra-category discrepancy and increasing inter-category margins.This supports cross-domain knowledge transfer at the pixel level.
  • Regularization: A diversity-promoting regularizer operates on the mean feature representation of each image to keep embedding-space representations globally diverse and smooth.It helps avoid the trivial solution in which all unlabeled target data receive the same encoding.
  • Centroid-aware Pixel Contrast: ProtoCL uses one global prototype as a positive pair and the K−1 other category prototypes as negative pairs.BankCL instead uses multiple positive and negative local centroids stored in a memory bank.
  • Centroid-aware Pixel Contrast: Centroid-aware contrast benefits from semantic anchors, but its pair count limits how comprehensively it represents the embedding space.The distributional formulation treats infinitely many positive and negative pairs through estimated semantic distributions.
  • Distribution-aware Pixel Contrast: Distribution-aware contrast replaces explicit infinite sampling with a closed-form upper bound, yielding SePiCo (DistCL) with analytically solvable gradients.The bound follows from Jensen’s inequality and implicitly explores infinite samples.

3.4 Training Procedure

SePiCo is optimized end to end in one stage by combining self-training, semantic-guided contrastive losses, and class-balanced cropping for more stable training.

  • Training procedure: The training procedure jointly optimizes cross-entropy, self-supervised, contrastive, and regularization losses in a one-stage pipeline.The contrastive and regularization weights are both set to 1.0 without tuning.
  • Class-balanced cropping: Class-balanced cropping is applied to source and target images to reduce overfitting toward majority classes under distribution shift.The method addresses target-domain imbalance without requiring target annotations.
  • Training procedure: The algorithm initializes the encoder from ImageNet and randomly initializes the segmentation and projection heads before sampling source-target image pairs.Both images undergo class-balanced cropping before feature extraction.
  • Training procedure: Source and target feature maps are separated into pixel representations using their corresponding masks during training.The masks are source ground truth and target pseudo-label masks.
  • Training procedure: After contrastive training, the final network retains the encoder and segmentation head while omitting the projection head.These retained components produce the final segmentation network.

4 EXPERIMENT

This section validates SePiCo across synthetic-to-real and daytime-to-nighttime adaptation tasks using comparisons, qualitative analyses, and architectural evaluations.

  • Experimental scope: Experiments cover two synthetic-to-real tasks and one challenging daytime-to-nighttime adaptation task.The section also reports analyses across diverse datasets and architectures.
  • Evaluation design: The evaluation compares SePiCo with leading approaches across datasets and architectures, including Vision Transformer-based models.Qualitative results and detailed analyses are included alongside quantitative comparisons.
  • Evaluation scope: The experiments conclude with analyses intended to provide a complete picture of SePiCo’s performance.The reported analyses include qualitative results and component-focused investigations.

4.1 Experimental Setups

The experiments use synthetic and real urban-scene datasets for domain adaptation, with DeepLab-V2 and ResNet-101 as the main CNN comparison setup.

  • Datasets: GTAV supplies 24,966 rendered city-scene images sharing 19 classes with Cityscapes.These images are used as source-domain training data.
  • Datasets: SYNTHIA-RAND-CITYSCAPES contributes 9,400 synthetic images with 16 semantic annotations shared with Cityscapes.The selected images have resolution 1280×760 and serve as source data.
  • Datasets: Cityscapes contains finely annotated real urban scenes from 50 cities, including 2,975 training and 500 validation images.The dataset represents the real urban-scene domain used in the adaptation benchmarks.
  • Network architecture: CNN-based experiments use DeepLab-V2 with a ResNet101 backbone, while a 512-d ℓ2-normalized projection head maps pixel embeddings.The implementation is based on the mmsegmentation toolbox.
  • Comparison protocol: Table 1 compares GTAV →Cityscapes methods using DeepLab-V2 with ResNet-101 for a fair comparison.The best result is highlighted in bold.

4.2 Experimental Results

SePiCo performs strongly across synthetic-to-real, daytime-to-nighttime, generalization, and detection evaluations, with DistCL often combining strong accuracy and lower training burden.

  • Synthetic-to-real adaptation: 61.0% mIoU on GTAV →Cityscapes gives SePiCo (DistCL) a +22.4% mIoU gain over Source Only.SePiCo (ProtoCL) is reported as setting the new state of the art overall.
  • Variant comparison: 59.5% and 60.4% mIoUs are achieved by ProtoCL and BankCL respectively, while DistCL surpasses BankCL with less memory and slower-training burden.The results associate higher-quality features with semantic concepts having greater diversity.
  • Synthetic-to-real adaptation: 58.1% mIoU and 66.5% mIoU∗ on SYNTHIA →Cityscapes place SePiCo (DistCL) among the best methods.The gains over Source Only are +24.6% mIoU and +27.9% mIoU∗.
  • Variant comparison: SePiCo (DistCL) provides more class-balanced performance than ProtoCL and BankCL on SYNTHIA →Cityscapes.The comparison reports that the variants tie or underperform DistCL in mIoU while remaining comparable in mIoU∗.
  • Daytime-to-nighttime adaptation: On Cityscapes →Dark Zurich, SePiCo (BankCL) and SePiCo (DistCL) outperform comparison methods by a large margin.ProtoCL also wins by a narrow margin over the previous methods.
  • Generalization: SePiCo (DistCL) reaches 56.9% and 40.6% mIoUs on Nighttime Driving and BDD100k-night, respectively, setting records on both unseen domains.The authors also report improved source-domain segmentation relative to a source-only model.

4.3 Ablation Studies

Ablations show that SePiCo’s self-training, contrastive objectives, regularization, class balancing, teacher network, memory-bank design, and feature selection each affect performance. Distribution-aware contrast and appropriate feature choices provide particularly strong benefits.

  • Semantic-guided pixel contrast: Contrastive learning improves mIoU from 48.5% without self-training to 52.1% with self-training, while the unified pipeline adds +7.2% mIoU.The results support combining representation learning with self-training.
  • Regularization: Lreg adds +0.7% mIoU and +1.1% mIoU in the two evaluated settings, supporting representation smoothing.
  • Class-balanced cropping: CBC balances IoUs across categories without annotations, whereas RCS mainly benefits minority classes such as rider, mbike, and bike.CBC can slightly reduce IoU for some categories while substantially increasing it for others.
  • Teacher network: The teacher network contributes more than +3.0% mIoU, and β is fixed at 0.999 for regularization.
  • Bank size and distribution modeling: DistCL uses distributions to simulate infinite bank size on-the-fly, avoiding outdated representations and exceeding BankCL by a considerable margin.Larger banks improve diversity but can retain stale representations.
  • Feature selection: Layer 4 features are best overall for all SePiCo variants, while fusing all CNN features slightly degrades results; DistCL remains nearly equally effective across selected features.The authors attribute the robustness to increased feature diversity from distribution modeling.

4.4 Further Analysis

Further analyses examine representation discrimination, embedding structure, parameter sensitivity, transferability, and computational cost. SePiCo produces more structured pixel representations, remains robust across settings, and adds only slight training computation while outperforming comparison methods.

  • Pixel-wise discrimination distance: High PDD indicates dense same-class clustering and relatively large between-class distances; SePiCo achieves much higher PDD values in most categories.PDD uses cosine similarity over pixel representations grouped by semantic class.
  • Embedding visualization: SePiCo’s ProtoCL, BankCL, and DistCL objectives produce well-structured embedding spaces in the t-SNE analysis.
  • Parameter sensitivity: SePiCo (DistCL) remains superior to the previous state of the art across varied loss-weight compositions and is relatively robust to the contrastive-learning start iteration.Performance peaks at Lw = 3000 in the reported sensitivity analysis.
  • Model transferability: LogME is used to assess transferability from GTAV →Cityscapes by measuring label evidence from extracted pixel features.Higher LogME values indicate likely better transfer performance.
  • Throughput: SePiCo introduces only slight extra computation per training iteration on CNN-based and Transformer-based networks while significantly surpassing comparison methods.Throughput is evaluated on GTAV →Cityscapes with batch size 1 using V100-32G.

5 CONCLUSION

SePiCo is an end-to-end pixel-contrast adaptation framework that strengthens self-training through semantic concepts and an efficiently computable distributional objective. Experiments demonstrate superiority on daytime and nighttime segmentation benchmarks.

  • Conclusion: SePiCo combines pixel-level contrastive learning with self-training and implicitly learns from infinitely many similar and dissimilar pixel pairs.An upper bound converts the originally intractable loss into a practical implementation.
  • Conclusion: Experiments demonstrate SePiCo’s superiority on both daytime and nighttime segmentation benchmarks.
Loading 2204.08808v2…