PaperlayerView exact revision on arXiv

Source-linked AI summary

MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation

Lukas Hoyer, Dengxin Dai, Haoran Wang, Luc Van Gool

arXiv:2212.01322v2cs.CV

TL;DR

UDA struggles with similarly appearing classes because unlabeled target data provide no ground truth for subtle appearance differences. MIC trains masked-target predictions against EMA-teacher pseudo-labels from complete images, learning context relations, and reports consistent gains across tasks and domain gaps, including +2.1 and +3.0 percentage points on GTA→Cityscapes and VisDA-2017.

  • Problem

    UDA lacks target annotations for learning slight appearance differences between similarly looking classes, leaving such classes difficult to distinguish.

  • Method

    MIC masks random target-image patches and enforces consistency with pseudo-labels generated from complete images by an exponential moving average teacher.

  • Results

    MIC consistently improves UDA performance across image classification, semantic segmentation, and object detection, with +2.1 and +3.0 percentage-point gains on GTA→Cityscapes and VisDA-2017.

  • Takeaways & Limitations

    MIC provides a simple, universal UDA plug-in for learning target-domain context relations across recognition tasks and domain-adaptation scenarios.

  • Takeaways & Limitations

    MIC has limited improvement potential when context is less important for recognition, and some autonomous-driving corner cases may still require target-domain annotations.

Abstract

from arXiv · show

In unsupervised domain adaptation (UDA), a model trained on source data (e.g. synthetic) is adapted to target data (e.g. real-world) without access to target annotation. Most previous UDA methods struggle with classes that have a similar visual appearance on the target domain as no ground truth is available to learn the slight appearance differences. To address this problem, we propose a Masked Image Consistency (MIC) module to enhance UDA by learning spatial context relations of the target domain as additional clues for robust visual recognition. MIC enforces the consistency between predictions of masked target images, where random patches are withheld, and pseudo-labels that are generated based on the complete image by an exponential moving average teacher. To minimize the consistency loss, the network has to learn to infer the predictions of the masked regions from their context. Due to its simple and universal concept, MIC can be integrated into various UDA methods across different visual recognition tasks such as image classification, semantic segmentation, and object detection. MIC significantly improves the state-of-the-art performance across the different recognition tasks for synthetic-to-real, day-to-nighttime, and clear-to-adverse-weather UDA. For instance, MIC achieves an unprecedented UDA performance of 75.9 mIoU and 92.8% on GTA-to-Cityscapes and VisDA-2017, respectively, which corresponds to an improvement of +2.1 and +3.0 percent points over the previous state of the art. The implementation is available at https://github.com/lhoyer/MIC.

1 ETH Zurich 2 Max Planck Institute for Informatics, Saarland Informatics Campus 3 KU Leuven

The listed affiliations include ETH Zurich, the Max Planck Institute for Informatics, and KU Leuven.

  • The paper lists affiliations with ETH Zurich, the Max Planck Institute for Informatics, and KU Leuven.

1. Introduction

UDA adapts models from labeled source domains to unlabeled target domains, but similar-looking target classes remain difficult. MIC addresses this gap by training models to use spatial context, improving performance across tasks and adaptation scenarios.

  • UDA adapts models trained on labeled source data to unlabeled target data, which avoids the costly annotation of new target datasets.Semantic-segmentation annotation can take more than one hour per image.
  • Similar-looking target classes, such as road and sidewalk, remain difficult because target ground truth is unavailable for learning slight appearance differences.The sidewalk interior may be segmented as road because of ambiguous local appearance.
  • MIC masks random target-image patches and matches masked-image predictions to pseudo-labels from complete images produced by an EMA teacher.The student must infer masked-region semantics from remaining context, while the teacher uses intact local and contextual information.
  • MIC is designed as a plug-in for UDA across image classification, semantic segmentation, and object detection, spanning synthetic-to-real, clear-to-adverse-weather, and day-to-nighttime settings.The method is presented as applicable across multiple recognition tasks and domain gaps.
  • +2.1 and +3.0 percentage points improve state-of-the-art performance on GTA→Cityscapes and VisDA-2017, respectively.The reported performances are 75.9 mIoU and 92.8%, respectively.

2. Related Work

Related UDA work captures context through network modules, alignment, or consistency training, while masked image modeling traditionally targets self-supervised representation learning. MIC instead uses masking to learn target-domain context relations in prediction space.

  • UDA methods commonly use discrepancy minimization, adversarial training, or self-training across image classification, semantic segmentation, and object detection.
  • Prior UDA methods add spatial attention, cross-domain attention, context-aware fusion, local-context alignment, or multi-crop consistency to capture context.The cited mechanisms still do not capture all relevant context clues in target-domain adaptation.
  • Masked image modeling trains vision networks to reconstruct properties of masked areas, including VAE features, HOG features, or color information.Explored masking schemes include block-wise, random-patch, and attention-guided masking.
  • MIC uses masked images for UDA context learning rather than self-supervised representation pretraining or restoration of predefined pretext targets.It performs reconstruction in the task’s actual prediction space, such as semantic segmentation, and naive ImageNet masked modeling does not improve target-domain performance.

3. Methods

MIC extends unsupervised domain adaptation by explicitly training models to use target-domain context when local appearance is ambiguous. It combines masked-target consistency with pseudo-labels from an EMA teacher and integrates into the standard UDA objective.

  • Unsupervised Domain Adaptation: The source model uses a supervised source loss, while unlabeled target images contribute an unsupervised adaptation loss determined by the chosen UDA strategy.The target loss may implement adversarial training or self-training.
  • Masked Image Consistency: MIC plugs into UDA methods to learn target-domain spatial context for recognizing classes with similar local appearances.It is designed for integration across image classification, semantic segmentation, and object detection.
  • Masked Image Consistency: MIC randomly masks target-image patches and trains predictions from the masked image to match pseudo-labels for the complete image.Because masked predictions lack local information, the network must infer masked-region semantics from remaining context.
  • Pseudo-Labeling: The EMA teacher generates target pseudo-labels from complete images, while its temporal ensemble of student weights improves pseudo-label robustness and stability.The teacher has access to the original target image, unlike the masked student input.
  • Training Objective: MIC adds a weighted consistency loss to the UDA optimization objective alongside the existing target adaptation loss.Pseudo-label quality estimates weight the consistency signal, including maximum softmax confidence for classification and confidence-based measures for segmentation and detection.

4. Experiments

MIC is evaluated across semantic segmentation, image classification, and object detection UDA settings, consistently improving performance across diverse domain shifts and architectures. Ablations and context tests indicate that masked-image consistency is central to the gains, with modest training overhead when an EMA teacher already exists.

  • Semantic Segmentation: MIC improves GTA→CS semantic segmentation by +1.2 to +4.7 mIoU across UDA methods and network architectures.The gains extend to both DAFormer Transformers and DeepLabV2 CNNs, while improvements decrease at higher baseline performance.
  • Semantic Segmentation: MIC improves state-of-the-art segmentation by +2.1 mIoU on GTA→CS, +1.5 on Synthia→CS, +4.3 on CS→DarkZurich, and +2.4 on CS→ACDC.The largest class-wise benefits occur for difficult classes including sidewalk, fence, pole, traffic sign, terrain, and rider.
  • Image Classification: MIC improves VisDA-2017 classification by +2.5 and +3.0 percentage points with ResNet and ViT networks, respectively, and improves Office-Home by +1.9.Difficult classes and difficult domain transfers benefit most.
  • Object Detection: MIC improves CS→Foggy CS object detection by +3.6 mAP over SADA, with consistent gains across categories and the largest benefits for car, motorcycle, and rider.MIC also has an advantage over more recent methods such as SIGMA for most categories.
  • In-Depth Analysis: Masked-image consistency is essential: removing masking reduces GTA→CS performance by -20.0 mIoU, whereas removing color augmentation reduces it by -0.3 mIoU.Replacing EMA predictions reduces performance by -0.7 mIoU, and removing the pseudo-label confidence loss weight reduces it by -1.6 mIoU.
  • In-Depth Analysis: When an EMA teacher is already available, MIC increases training speed cost by 24% and memory footprint by 1%, while adding no inference-time overhead.Without an EMA teacher, training speed decreases by 75% and memory consumption increases by 3%.

5. Conclusions

The paper presents MIC as a simple UDA module that learns target-domain context relations by enforcing consistency between predictions from masked and complete images. Across recognition tasks and domain shifts, MIC yields significant performance improvements, including gains of +2.1 on GTA→CS and +3.0 on VisDA-2017.

  • Conclusions: MIC learns target-domain context relations by enforcing consistency between predictions from partly masked and complete images.The module is presented for image classification, semantic segmentation, and object detection across multiple adaptation scenarios.
  • Conclusions: +2.1 mIoU on GTA→CS and +3.0 percentage points on VisDA-2017 improve the state-of-the-art performance with MIC.The conclusion reports significant improvements across the evaluated UDA tasks.

Supplementary Material

The supplementary material provides implementation resources and extends the empirical analysis of MIC. It covers additional parameter studies, behavioral analyses, expanded segmentation comparisons, qualitative comparisons, and potential limitations.

  • Supplementary Material: The supplement provides source code, further MIC parameter studies, behavioral analyses, expanded semantic-segmentation comparisons, qualitative comparisons, and potential limitations.The implementation is based on HRDA and mmsegmentation for segmentation, SDAT for classification, and SADA for object detection.

C.1. MIC Prediction Region

MIC benefits from applying consistency loss to both masked and unmasked regions, while its loss weight, teacher momentum, and augmentation settings affect performance.

  • Prediction regions: +1.5 mIoU is gained when MIC loss is applied only to masked patches or only to unmasked patches over DAFormer without MIC.The masked-region result reflects learning to predict missing local information from context, while the unmasked-region result reflects robustness to incomplete context.
  • Prediction regions: +0.8 mIoU is gained by applying the default MIC loss to both masked and unmasked regions compared with either single-region setting.The two capabilities are complementary and combine successfully when the loss covers all image patches.
  • Loss weight: λM = 1 achieves the best performance, whereas λM = 10 can reduce performance below the baseline by dominating the total loss.Excessive MIC weighting prevents source and adaptation losses from working effectively.
  • Teacher momentum: α = 0.999 is best for GTA→CS, while α = 0.9 is best for VisDA-2017, showing task-dependent effects of teacher update speed.Both smaller and larger deviations from the default can reduce performance on GTA→CS.
  • Augmentation: Color augmentation improves MIC on GTA→CS but decreases performance on CS→ACDC.The authors attribute the adverse-weather decrease to corruption of already dark, low-contrast nighttime content.

D. Extended Analysis of MIC

Extended analyses examine pseudo-label quality, host-loss requirements, random masking, and MIC's applicability to classification features.

  • Pseudo-label quality: Pseudo-label quality and MIC validation mIoU are clearly correlated throughout training, even with simulated pseudo-label noise.The supplied passage indicates that better pseudo-labels improve the model and EMA updates improve pseudo-labels.
  • Host adaptation loss: +10.5 mIoU is achieved without the host adaptation loss on GTA→CS compared with DAFormer without that loss, although performance drops to 62.5 mIoU.MIC is designed as an orthogonal plug-in but requires a host adaptation loss to work well.
  • Random patch selection: Random masking supplies diverse context combinations without requiring a specific architecture or data priors.Because images contain many objects and context relations, the diverse combinations help the network identify relevant relations.
  • Image classification: MIC models spatial context relations in classification networks' intermediate features despite producing one prediction per image.This can improve distinction between ambiguous object parts and reduce effects of ill-adapted parts on the target domain.

E. Extended Comparison for UDA Semantic Segmentation

Extended semantic-segmentation comparisons show that MIC improves strong UDA methods across Foggy Zurich and multiple benchmark settings, including different backbones and inference strategies.

  • Cityscapes→Foggy Zurich: +3.7 mIoU is achieved by MIC(HRDA) on Cityscapes→Foggy Zurich over HRDA.The model uses annotated Cityscapes as source and unlabeled Foggy Zurich medium fog images as target.
  • Extended comparison: MIC(HRDA) outperforms all previous methods by a large margin in the extended comparison across four semantic-segmentation UDA benchmarks.Another method can perform better for a specific class in some cases, such as DAP for vegetation on Synthia→Cityscapes.
  • MIC with DAFormer: MIC(DAFormer) achieves significant performance improvements over DAFormer across the different datasets.Sliding-window inference can further improve MIC(DAFormer) by matching the training crop input size for the Transformer's learned positional embedding.
  • MIC with DeepLabV2: MIC(HRDADLv2) significantly outperforms recent ResNet-based methods across synthetic-to-real, day-to-nighttime, and clear-to-adverse-weather adaptation.The comparisons include DecoupleNet, DAP, CPSL, HRDADLv2, CCDistill, and DANIA.

F. Further Example Predictions

Examples show MIC resolving several ambiguous segmentation, detection, and classification cases through context, while difficult appearances and crowded or adverse conditions remain failure modes.

  • Synthetic-to-real segmentation: MIC more completely segments ambiguous sidewalk and fence regions than previous methods on GTA→Cityscapes.It reduces segmentation of objects behind fences and recognizes sidewalk segments that other methods miss.
  • Synthetic-to-real segmentation: MIC better segments ambiguous bus textures and close riders, and can recognize distant riders using bicycles as a context clue.These examples illustrate more holistic reasoning about image context.
  • Segmentation failure cases: MIC still struggles with rare vehicle appearances, merged sidewalk-road regions, sidewalk under parked cars, and pedestrians near bicycles.These remain difficult GTA→Cityscapes segmentation cases for UDA methods including MIC.
  • Adverse-weather segmentation: On CS→ACDC, MIC fails on snow-covered sidewalks, dark-scene class distinctions, and motion-blurred dynamic objects.The cited examples identify distinct clear-to-adverse-weather failure cases.
  • Object detection: MIC better detects bus, truck, rider, motorcycle, and bicycle instances on CS→Foggy Cityscapes, but still produces duplicate and missed detections.Confusion of semantically similar objects is also reported as a typical failure case.
  • Image classification: MIC better distinguishes semantically similar vehicle classes on VisDA-2017, yet mostly confuses classes near decision boundaries or co-occurring in one image.The examples include train versus bus, bus versus truck, and truck versus car.
  • Supervised segmentation: MIC improves difficult supervised segmentation regions such as terrain, sidewalk, bus, and rider, though the strong supervised baseline leaves less room for improvement.The smaller improvement potential is also reflected in the quantitative results.

G. Potential Limitations

MIC remains bounded by the relevance and availability of target-domain context, and current UDA performance is not yet reliable enough for safe autonomous-driving deployment.

  • Target-domain annotations may still be necessary for corner cases requiring reliable adaptation.The authors specifically note that a few target annotations may be needed to guide adaptation safely.
  • MIC has limited improvement potential for classes whose recognition depends less on context, such as building or vegetation in synthetic-to-real adaptation.
  • MIC assumes that relevant context relations are available and that context information is a relevant factor for recognition.The supplied passage introduces two assumptions, but only explicitly states the first assumption's content.
  • Failure cases are reported for MIC on GTA→Cityscapes, Cityscapes→ACDC, Cityscapes→Foggy Cityscapes, and VisDA.The supplementary figures identify failure cases for each of these adaptation settings.
Loading 2212.01322v2…