PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Aware Mixup for Domain Adaptive Semantic Segmentation

Qianyu Zhou, Zhengyang Feng, Qiqi Gu, Jiangmiao Pang, Guangliang Cheng, Xuequan Lu, Jianping Shi, Lizhuang Ma

arXiv:2108.03557v3cs.CV

TL;DR

Domain adaptive semantic segmentation must bridge synthetic-to-real shifts, but existing UDA methods sparsely exploit shared contextual relationships. CAMix uses contextual masks to guide three-level domain mixup and adds significance-reweighted consistency; experiments report effectiveness, consistent state-of-the-art improvements, and strong ablation gains on UDA benchmarks.

  • Problem

    Synthetic-to-real domain gaps cause significant performance drops, while cross-domain context-dependency remains sparsely exploited in UDA semantic segmentation.

  • Method

    CAMix generates contextual masks from source spatial distributions and target contextual relationships to guide input-, output-, and significance-mask-level mixup with significance-reweighted consistency.

  • Results

    CAMix achieves consistent improvements over state-of-the-art methods on two challenging UDA benchmarks and improves ablation performance to 55.2% and 59.7% mIoU across reported settings.

  • Takeaways & Limitations

    Explicitly transferring shared context and reweighting consistency by significance supports target-domain adaptation while alleviating early performance degradation and training instability.

Abstract

from arXiv · show

Unsupervised domain adaptation (UDA) aims to adapt a model of the labeled source domain to an unlabeled target domain. Existing UDA-based semantic segmentation approaches always reduce the domain shifts in pixel level, feature level, and output level. However, almost all of them largely neglect the contextual dependency, which is generally shared across different domains, leading to less-desired performance. In this paper, we propose a novel Context-Aware Mixup (CAMix) framework for domain adaptive semantic segmentation, which exploits this important clue of context-dependency as explicit prior knowledge in a fully end-to-end trainable manner for enhancing the adaptability toward the target domain. Firstly, we present a contextual mask generation strategy by leveraging the accumulated spatial distributions and prior contextual relationships. The generated contextual mask is critical in this work and will guide the context-aware domain mixup on three different levels. Besides, provided the context knowledge, we introduce a significance-reweighted consistency loss to penalize the inconsistency between the mixed student prediction and the mixed teacher prediction, which alleviates the negative transfer of the adaptation, e.g., early performance degradation. Extensive experiments and analysis demonstrate the effectiveness of our method against the state-of-the-art approaches on widely-used UDA benchmarks.

I. INTRODUCTION

Domain adaptive semantic segmentation faces a synthetic-to-real domain gap, while existing adaptation methods largely underuse shared cross-domain context. CAMix addresses this gap with contextual mask-guided mixup and a significance-reweighted consistency loss, achieving improvements on UDA benchmarks.

  • Synthetic-to-real domain gaps cause significant performance drops when source-trained segmentation models are applied directly to unlabeled target data.
  • Existing UDA methods reduce shifts at pixel, feature, or output levels but sparsely exploit shared context-dependency across domains.
  • CAMix treats cross-domain context-dependency as explicit prior knowledge in a fully end-to-end trainable domain adaptation framework.
  • Its contextual mask uses source spatial distributions and target contextual relationships to guide mixup at input, output, and significance-mask levels.
  • The significance-reweighted consistency loss penalizes mixed student–teacher prediction inconsistency and alleviates adverse adaptation effects such as early performance degradation.
  • CAMix achieves consistent improvements over state-of-the-art methods, can plug into DACS and DAFormer, and performs effectively on two challenging UDA benchmarks.

II. RELATED WORK

UDA semantic segmentation addresses domain shifts between labeled source and unlabeled target data, but prior work has largely underused shared contextual relationships. CAMix explicitly transfers this context through guided mixup and significance-reweighted consistency.

  • UDA bridges domain shifts between labeled source and unlabeled target domains for semantic segmentation.
  • Existing approaches reduce shifts at pixel, feature, or output levels using adversarial learning, self-training, consistency regularization, and entropy minimization.
  • Context dependency across domains remains underexplored, despite shared relationships such as riders with bicycles and sidewalks beside roads.
  • CAMix uses contextual mask generation to guide domain mixup at input, output, and significance-mask levels.
  • The framework combines source images and labels with unlabeled target images to synthesize mixed images, pseudo-labels, and significance masks.
  • Significance-reweighted consistency loss reduces over-alignment and negative transfer during online adaptation.

B. Contextual Mask Generation

Contextual mask generation represents shared scene structure using source spatial distributions and target semantic relationships. The resulting binary mask selects contextually appropriate target categories for transfer onto source images.

  • CAMix uses source spatial contexts and target category relationships as explicit prior knowledge for adaptation.
  • A spatial prior tensor Q records source-domain class frequencies at each image location, regularizing target teacher predictions.
  • Target outdoor-scene categories are organized hierarchically, with multiple fine categories belonging to one coarse category.
  • Categories within a meta-class are copied together from target images and pasted onto source images to avoid inappropriate semantic contexts.
  • The method spatially modulates target pseudo-labels, randomly selects half the present classes, and checks them against the meta-class list.
  • The binary mask M assigns one to selected classes and zero elsewhere, then guides image, label, and significance-mask mixing.

C. Input-level and Output-level Domain Mixup

CAMix synthesizes source-target training examples by applying a contextual mask to images and labels, while using an EMA teacher for target pseudo-labels. Its target-to-source direction incorporates spatial and contextual constraints when target predictions are uncertain.

  • Input level: At input level, source and target images are synthesized into a mixed image using the contextual mixup procedure.
  • Teacher update: The teacher model is updated as an exponential moving average of student weights, with α controlling the update rate.
  • Output level: At output level, source labels and target teacher pseudo-labels are mixed into a combined label.
  • Mixing direction: Unlike prior directions, CAMix copies target categories and pastes them onto source images rather than mixing source content into target images.
  • Mixing direction: The target-to-source direction accommodates spatial and contextual relationships because target predictions lack sufficient supervision.

D. Significance-mask Level Domain Mixup

The significance-mask level mixup filters unreliable mixed teacher predictions using predictive entropy and a dynamic threshold, then reweights consistency with contextual guidance.

  • Uncertainty estimation: Stochastic forward passes estimate pixel-wise predictive entropy for the mixed teacher predictions.The predictive probabilities are averaged across repeated forward passes before entropy is calculated.
  • Dynamic threshold: A dynamic threshold R filters unreliable predictions, increasing slowly early in training and faster later.The threshold is determined from predictive entropy rather than softmax probabilities.
  • Significance mask: The significance mask UT retains only pixels whose predictive entropy is below R.These retained high-confidence target pixels are used to reweight the consistency loss.
  • Significance-mask construction: SigMask level mixup uses contextual mask M to guide mixing source and target significance masks into UM.The mixed mask provides additional supervisory signals for domain mixup.
  • Consistency reweighting: SRC penalizes inconsistency between mixed teacher and student predictions using the mixed significance mask UM.The loss is normalized by the sum of UM pixels to reduce adverse impacts and negative transfer.

E. End-to-End Training and Inference

CAMix jointly trains its loss components end to end on source segmentation, while inference uses the EMA teacher model for predictions.

  • End-to-end training: The source segmentation loss Lseg is a cross-entropy objective optimized on source images.Ground-truth source labels supervise the source segmentation output.
  • End-to-end training: All models operating at the three mixup levels are jointly trained in an end-to-end manner.The framework integrates its loss functions during optimization.
  • Inference: The teacher model is an exponential moving average of the student model and is used for inference.The paper states that the EMA teacher performs slightly better than the student.

F. Discussions on differences from related work UACR [73]

The discussion contrasts CAMix with UACR by emphasizing explicit cross-domain context as prior knowledge, whereas UACR focuses on unreliable teacher guidance.

  • Different motivations: UACR addresses unreliable Mean Teacher guidance by using uncertainty to recalibrate teacher predictions.Its motivation differs from CAMix’s context-focused adaptation strategy.
  • Different motivations: CAMix explicitly exploits contexts as prior knowledge for improving adaptability toward the target domain.The supplied discussion presents this as the central distinction from UACR.
  • Experimental comparison: Table II reports mIoU comparisons with state-of-the-art methods for GTAV to Cityscapes.The caption identifies the benchmark and metric but does not provide individual table values here.

IV. EXPERIMENTS

The experiments evaluate CAMix on two widely used UDA benchmarks, then use ablations and visualizations to analyze its effectiveness and component contributions.

  • Benchmark evaluation: CAMix is evaluated on GTAV → Cityscapes and SYNTHIA → Cityscapes.These are described as two widely used UDA benchmarks.
  • Benchmark evaluation: The experiments demonstrate the effectiveness of the proposed framework on both benchmarks.The evaluation is presented alongside implementation and setup details.
  • Analysis: Ablation studies analyze the contribution of each component, while visualizations provide additional analysis.The paper places these analyses after the benchmark demonstrations.

A. Datasets

Experiments use synthetic GTAV and SYNTHIA as labeled source domains and Cityscapes as the unlabeled real target, with DeepLabV2 and SegFormer implementations.

  • Cityscapes provides 2,975 training images and 500 validation images at 2048 × 1024 resolution for target-domain evaluation.
  • GTAV and SYNTHIA serve as synthetic source domains, while Cityscapes serves as the unlabeled real target domain.
  • The method is implemented with DeepLabV2 using a ResNet-101 backbone and with SegFormer using an MiT-B5 encoder and context-aware feature fusion decoder.
  • CAMix uses eight stochastic forward passes for SigMask-level mixup and defaults to β = 0.75 and γ = −5.

C. Comparison with the State-of-the-Art Methods

CAMix consistently improves strong UDA baselines across GTAV→Cityscapes and SYNTHIA→Cityscapes, without requiring adversarial discriminators, offline self-training, or image-to-image translation.

  • CAMix improves DACS by 3.1% and 4.9% mIoU and improves DAFormer by 1.7% and 1.8% mIoU across the two benchmarks.
  • Against adversarial approaches, CAMix outperforms competing methods by more than 6% with DACS and 20% with DAFormer.
  • CAMix achieves higher performance than SIBAN while using contextual masks and Eq. 5–Eq. 8 rather than an additional significance-aware model.
  • CAMix surpasses offline self-training methods by at least 5% with DACS and 16% with DAFormer through end-to-end online consistency regularization.
  • CAMix surpasses translation-based methods by around 5%–8% without style transfer, spectral transfer, or image-to-image translation networks.
  • CAMix explicitly explores cross-domain context in image space and remains end-to-end trainable, unlike cited feature-space context methods.

D. Comparison with related Domain Mixup methods

CAMix outperforms related domain mixup methods by using contextual guidance across multiple levels, reducing label contamination and improving adaptation performance.

  • CAMix outperforms related domain mixup methods under both DeepLabV2 and SegFormer basic architectures.
  • CowMix can produce partial objects, while CutMix, DACS, and DAFormer tend to cause severe label contamination in mixed images.
  • Adding spatial prior and contextual relationship components improves the iDACS baseline by 1.6% and 1.4%, reaching 53.1% and 54.5%.
  • Replacing SRC with MSE reduces performance to 44.5%, while CE reaches 54.2%, remaining 1.0% below SRC on GTAV→Cityscapes.
  • The best performance occurs with the first two meta class groups because too little context provides insufficient supervision and too much imposes excessive constraints.

F. Visualization

Visual and training analyses show that CAMix preserves contextual relationships, supports infrequent categories, and reduces instability and early performance degradation.

  • DACS mixed samples show label contamination and category confusion, whereas CAMix mitigates these problems by respecting shared cross-domain context.
  • In SYNTHIA→Cityscapes qualitative results, the comparison includes RGB input, ground truth, DACS predictions, and CAMix predictions.
  • CAMix provides sufficient training data for selected infrequent categories and alleviates over-fitting during adaptation.
  • The SRC-based consistency curve eases training instability and early performance drop compared with conventional consistency regularization.
  • The highest mIoU occurs near β = 0.75 and γ = −5, which are therefore used across experiments.
  • CAMix uses CMG to guide input-, output-, and significance-mask-level mixup and SRC to penalize mixed student–teacher inconsistency.
Loading 2108.03557v3…