Contextual-Relation Consistent Domain Adaptation for Semantic Segmentation

1 Introduction

• Dense pixel annotations are expensive, and models trained on synthesized images usually lose performance on real-world images because of domain bias and shift.Synthesized images with automatically generated labels are presented as one way to reduce annotation demands.Source: §1 · pp. 1–2
• Local contextual relations capture compatibility and incompatibility between neighboring categories, including pole–sidewalk co-occurrence and person-above-car patterns.The paper describes such relations as important segmentation cues that are largely neglected in unsupervised domain adaptation.Source: §1 · p. 3
• CrCDA learns contextual relations from labeled-source regions clustered into indexed groups, then transfers them to the unlabeled target domain.A classifier learns the relations in source feature space and enforces them in the target domain.Source: §1 · p. 3
• Adaptive entropy max-minimizing aligns local contextual relations without a discriminator, avoiding the redundant computation of many discriminators.The scheme estimates prototypical relation representations and alternately congregates neighboring target samples toward approximated correct source prototypes.Source: §1 · pp. 3–4
• CrCDA achieves superior segmentation performance consistently on GTA5 → Cityscapes and SYNTHIA → Cityscapes.The method can be integrated into existing domain adaptation techniques without extra overhead except two classifiers.Source: §1 · p. 4

2 Related works

• UDA segmentation methods are grouped into adversarial-learning, image-translation, and pseudo-label retraining approaches.The paper positions CrCDA outside discriminator-based global or class-wise alignment and target-data retraining.Source: §2 · pp. 4–5
• Adversarial methods use discriminators to align source and target feature or output spaces, including global structure and scene layout.Other work combines adversarial learning with co-training by maximizing the discrepancy between two classifiers’ outputs.Source: §2 · p. 4
• Image-translation methods generate target-style images from source images to reduce domain discrepancy before segmentation training.The cited approaches include CycleGAN-based input adaptation and channel-wise feature alignment that preserves spatial structure and semantic concepts.Source: §2 · pp. 4–5
• Pseudo-label retraining uses confident target predictions to fine-tune source-trained models, while CrCDA uses multi-scale entropy max-minimizing instead.The paper describes CrCDA as an end-to-end multi-scale UDA network evaluated on two challenging tasks.Source: §2 · p. 5

3.1 Problem Definition

• Global image-level discriminator alignment can deconstruct existing local contextual alignment, a limitation the method targets for dense pixel prediction.The paper calls this lack of local consistency, or local contextual inconsistency.Source: §3.1 · pp. 5–6

3.2 Overview of Network Architecture

• The model uses a shared feature extractor with a pixel-scale segmentation classifier and a region-scale contextual-relations classifier, trained adversarially for local consistency.Source data supplies segmentation and contextual-relation losses; target data supplies the local entropy objective.Source: §3.2 · pp. 6–7, Figure 2 · p. 6

3.3 Contextual-relation consistent domain adaptation

• Region-scale contextual-relation pseudo-labels are created by cropping source pixel labels into regions and clustering their gradient histograms with Dbscan.These indexed region labels support local-scale alignment.Source: §3.3 · p. 7, Figure 3 · p. 7
• AEMM uses classifier entropy maximization to estimate target contextual-relation prototypes and feature-extractor entropy minimization to adapt target features toward them.A regularizer supports prototype estimation where target labels are unavailable.Source: §3.3 · pp. 8–9

3.4 CrCDA with pixel-/global-scale

• Multi-scale CrCDA extends AEMM to pixel and global alignment, using a domain classifier on concatenated pixel- and region-probability maps.The network aligns domain shift at global, local, and pixel scales simultaneously.Source: §3.4 · pp. 9–10

4.1 Datasets

• The GTA5 and SYNTHIA synthetic source datasets contain 24,966 and 9,400 images, respectively, and Cityscapes supplies 2,975 unlabelled target-training images.The evaluation uses 19 common categories for GTA5 and 16 common categories for SYNTHIA.Source: §4.1 · p. 10

4.2 Implementation Details

• Experiments use Deeplab-V2 with ImageNet-pretrained ResNet-101, with VGG-16 also evaluated; multi-scale models use separate local- and global-scale heads.The implementation also describes gradient-reversal training and reports its hyperparameter settings.Source: §4.2 · pp. 10–11

4.3 Comparison with state-of-art

• CrCDA adds over +2.1% mIoU on ResNet-101 versus Adapt-SegMap, category-level adversarial network, and patch-represented global alignment on GTA5→Cityscapes.The comparison is reported in Table 1.Source: §4.3 · p. 11, Table 1 · p. 11
• SYNTHIA→Cityscapes experiments report consistent improvements over state-of-the-art methods.The results are presented for VGG-16 and ResNet-101 architectures, with mIoU and mIoU* calculated over 16 and 13 classes, respectively.Source: §4.3 · p. 12, Table 2 · p. 12

4.4 Ablation Studies and Analysis

• Pixel-scale AEMM improves mIoU by +1.4% over MinEnt, while single-scale CrCDA* beats MinEnt-based contextual-relations alignment by +1.6% on ResNet-101.Table 3 reports the GTA5-to-Cityscapes ablation settings with the ResNet-101 network.Source: §4.4 · p. 12, Table 3 · p. 13
• Three-scale CrCDA reaches 48.6% mIoU on ResNet-101 and improves mIoU by +2.6% over Pixel+Global.CrCDA combines pixel-, local-, and global-scale adaptation.Source: §4.4 · p. 12, Table 3 · p. 13
• Local-scale adaptation aligns groups of pixel predictions, while global-scale adaptation operates at image scale and pixel-scale adaptation aligns individual-pixel feature representations.The paper describes these scales as learning complementary information and reports a complementary study with global-alignment methods in Table 4.Source: §4.4 · p. 13, Table 4 · p. 13

5 Conclusions

• Fig. 5 contrasts CrCDA's alignment of low- and high-level features with AdvEnt's focus on global consistency and high-level information.The caption highlights low-level boundaries, including those of sidewalks, cars, and people.Source: Figure 5 · p. 14
• The t-SNE visualization describes CrCDA's adapted target features as more discriminative and consistent with source features than those produced by traditional global alignment.The caption attributes the difference to CrCDA aligning local contextual-relation consistency as well as global consistency.Source: Figure 6 · p. 14
• CrCDA achieves state-of-the-art performance on two challenging semantic-segmentation UDA tasks while aligning domain shifts at local and global scales.The tasks are GTA5 to Cityscapes and SYNTHIA to Cityscapes.Source: §5 · pp. 13–14

A.1. Method - Details of region-scale/contextual-relation pseudo labels and regularizer weight

• Two independent contextual-relation classifiers handle the two region sizes used for each dataset.The two region sizes are 18 × 32 and 36 × 64 for GTA5, 19 × 32 and 38 × 64 for SYNTHIA, and 16 × 32 and 32 × 64 for Cityscapes.Source: §A.1 · p. 19
• The source input sizes are 720 × 1280 for GTA5 and 760 × 1280 for SYNTHIA, while Cityscapes target inputs are 512 × 1024.Source: §A.1 · p. 19
• The regularizer weight decreases with training iteration according to λR = (1 − iter/max iter)^power, with power = 0.9.Source: §A.1 · p. 19

A.2. Method - Traditional Losses

• Source-domain training minimizes a supervised segmentation loss for model G.Source: §A.2 · p. 19
• Target-domain adversarial training uses a minimax game between G and discriminator D to train G to extract domain-invariant features.Source: §A.2 · p. 19
• The overall UDA loss is formulated as the sum of segmentation and adversarial losses: L(Xs, Xt) = Lseg(G) + Ladv(G, D).Source: §A.2 · p. 19

A.3. Method - Loss in Multi-Scale Adaptation

• Source data contribute to segmentation, contextual-relation, and domain-discriminator losses using pixel labels and region-scale contextual-relation pseudo labels.Source: §A.3 · p. 19
• The source pixel classifier predicts a probability map over C classes for each pixel, while the contextual-relation classifier predicts over N classes for each region.Source: §A.3 · p. 19
• For target images without labels, adversarial training between feature extractor E and classifiers Cseg, Ccr, and CD uses entropy max-minimization to extract discriminative features.Source: §A.3 · p. 20
• The target pixel and region classifiers produce probability maps over C pixel classes and N region classes, respectively.Source: §A.3 · p. 20
• The overall global alignment loss is LD = Lsd + Ltd + Entsd + Enttd.The passage identifies these as the source-domain loss, target-domain loss, and source and target entropy terms.Source: §A.3 · p. 20

• CrCDA uses multi-scale adversarial learning to align low-level features, including object boundaries, alongside high-level features.Source: §A.4 · p. 20, Figure 7 · p. 21
• AdvEnt focuses on high-level information through global alignment and neglects low-level features.Source: §A.4 · p. 20, Figure 7 · p. 21
• CrCDA achieves both local and global segmentation consistencies, whereas AdvEnt achieves global consistency only.Source: §A.4 · p. 20, Figure 7 · p. 21