Source-linked AI summary
xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Émilie Wirbel, Patrick Pérez
TL;DR
UDA for 3D semantic segmentation remains limited by predominantly uni-modal methods, despite multi-modal datasets and modality-specific domain shifts. xMUDA uses disentangled 2D and 3D streams with mutual output mimicking, and it substantially outperforms uni-modal UDA across tested scenarios while complementing pseudo-label adaptation.
Problem
Existing UDA approaches are mostly uni-modal, although 3D datasets often provide images and point clouds that can experience different domain shifts.
Method
xMUDA uses separate 2D and 3D streams with mutual mimicking through KL-divergence-based cross-modal learning disentangled from segmentation.
Results
xMUDA largely outperforms uni-modal UDA methods across the tested scenarios and is complementary to pseudo-label learning and fusion.
Takeaways & Limitations
Cross-modal learning provides a way for both modalities to benefit from each other while remaining applicable to target data without annotations.
Takeaways & Limitations
The proposed architecture is constrained by the need to preserve modality-specific private information, since naive output alignment can discard information exclusive to each sensor.
Abstract
from arXiv · showhide
Unsupervised Domain Adaptation (UDA) is crucial to tackle the lack of annotations in a new domain. There are many multi-modal datasets, but most UDA approaches are uni-modal. In this work, we explore how to learn from multi-modality and propose cross-modal UDA (xMUDA) where we assume the presence of 2D images and 3D point clouds for 3D semantic segmentation. This is challenging as the two input spaces are heterogeneous and can be impacted differently by domain shift. In xMUDA, modalities learn from each other through mutual mimicking, disentangled from the segmentation objective, to prevent the stronger modality from adopting false predictions from the weaker one. We evaluate on new UDA scenarios including day-to-night, country-to-country and dataset-to-dataset, leveraging recent autonomous driving datasets. xMUDA brings large improvements over uni-modal UDA on all tested scenarios, and is complementary to state-of-the-art UDA techniques. Code is available at https://github.com/valeoai/xmuda.
1. Introduction
xMUDA addresses multi-modal 3D semantic-segmentation domain adaptation by enabling controlled information exchange between separate 2D and 3D streams. It introduces new cross-modal benchmarks and reports substantial gains over single-modality UDA methods, with extensions to self-training and fusion.
- Motivation: Existing UDA work is mostly 2D and single-modality, despite 3D datasets commonly containing both images and point clouds.Domain shifts can affect modalities differently, making multi-modal adaptation challenging.
- Benchmarks: xMUDA defines new UDA scenarios and corresponding splits for recently published 2D–3D datasets.The scenarios cover different lighting conditions, environments, and sensor setups.
- Architecture: Its disentangled architecture enables cross-modal learning while preserving private and shared information in the 2D and 3D modalities.The architecture uses separate modality streams and separates cross-modal learning from the main segmentation objective.
- Learning scheme: The learning scheme balances mutual cross-modal learning with the main segmentation objective and can be combined with self-training using pseudo-labels.This combination is presented as a way to boost performance.
- Results: xMUDA outperforms single-modality state-of-the-art UDA techniques by a significant margin on the proposed benchmarks.The framework is also extended to late fusion, which produces superior results.
2. Related Work
Prior work largely treats UDA as a single-modality problem and handles 2D images and 3D point clouds through separate technical traditions. xMUDA instead combines independent modality streams with projected feature correspondence and cross-modal output learning.
- Unsupervised Domain Adaptation: UDA methods commonly seek domain-invariant representations through adversarial training, self-training, or related distribution-alignment objectives.These approaches have been developed mainly for 2D perception tasks.
- Unsupervised Domain Adaptation: Very few prior methods address UDA for 3D semantic segmentation, and prior multi-modal settings generally use an extra modality only as source-domain privileged information.xMUDA assumes all modalities are available during training and testing on both domains.
- Multi-Modality Learning: Multi-modality learning often fuses features, but combining images with point clouds is harder because they occupy different metric spaces.Prior solutions include projections into bird’s-eye or front views and lifting image features into 3D.
- 3D Networks for Semantic Segmentation: 3D segmentation networks include voxel-, point-, and graph-based families with different computational representations.The paper selects SparseConvNet as its 3D network.
- xMUDA framework: The xMUDA overview uses separate 2D and 3D streams whose outputs are aligned to the N 3D points for cross-modal prediction learning.The 2D stream processes images, the 3D stream processes point clouds, and the streams produce main and mimicry outputs.
3. xMUDA
xMUDA enables controlled information exchange between independent 2D and 3D streams for unsupervised domain adaptation, while preserving modality-specific information. It combines supervised source segmentation with cross-modal learning on source and target data, and can complement pseudo-label self-training and other UDA methods.
- Architecture: xMUDA uses independent 2D and 3D branches whose outputs are linked through explicit cross-modal mimicry.The architecture uses separate modality-specific networks, while mimicry connects their output probabilities.
- Architecture: A dual-head design separates the main segmentation objective from mimicry, preserving private information while exploiting shared knowledge.Each stream has a main head for its best prediction and a mimicry head estimating the other modality’s output.
- Learning Scheme: xMUDA introduces cross-modal self-training that operates across four subsets: Source 2D, Target 2D, Source 3D and Target 3D.The scheme uses source supervision, target self-supervision and cross-modal predictions from both modalities.
- Learning Scheme: Cross-modal learning uses KL divergence to make each modality estimate the other modality’s prediction on both source and unlabeled target data.The loss transfers information without target annotations and acts as an auxiliary objective alongside source segmentation.
- Learning Scheme: The training objective combines source segmentation loss with cross-modal losses on source and target, weighted by separate hyperparameters.The segmentation loss is supervised on source data, while cross-modal loss is applied to both domains.
- Extensions: Additional pseudo-label self-training is complementary to cross-modal learning, and the resulting xMUDAPL variant retrains using confident target pseudo-labels.Pseudo-labels are extracted offline from predicted class probabilities and used for an additional target segmentation loss.
4. Experiments
The experiments evaluate xMUDA across three real-to-real autonomous-driving adaptation scenarios using synchronized, calibrated camera and LiDAR data. Results compare unimodal UDA, cross-modal learning, pseudo-labeling, and fusion approaches.
- Datasets and scenarios: The benchmarks cover day-to-night, country-to-country, and dataset-to-dataset shifts involving lighting, environment, and sensor setup.The datasets include nuScenes, A2D2, and SemanticKITTI, with front-camera images and LiDAR points projected into the image.
- Evaluation: The evaluation reports 3D semantic-segmentation mIoU separately for 2D and 3D streams and for their softmax-average ensemble.The baseline uses source-only training, while the oracle uses target-only training.
- Main results: xMUDA significantly adapts all three scenarios, often outperforming unimodal UDA baselines and consistently improving both modalities.This holds even when one modality is initially stronger than the other.
- Main results: Combining cross-modal learning with pseudo-labeling yields the best score everywhere except the Day/Night softmax-average result.The two techniques are complementary, and their combination consistently scores higher than either technique separately.
- Fusion: xMUDA Fusion adds modality-specific segmentation heads before a central fusion layer so each stream can mimic the fused prediction.This preserves cross-modal learning while using a late-fusion architecture.
- Fusion: xMUDA Fusion outperforms Deep logCORAL and MinEnt, while xMUDAPL Fusion outperforms all other unimodal baselines.The comparison is conducted in the USA/Singapore scenario.
5. Ablation Studies
The ablations examine fusion and segmentation-head design, showing that uncoupling segmentation from mimicry improves robustness and that cross-modal learning benefits supervised training.
- Fusion and heads: xMUDA Fusion preserves cross-modal learning by adding separate 2D and 3D heads before the fused head.A single fused head would prevent the original cross-modal learning scheme from being applied to the main branches.
- Segmentation heads: Dual-head architectures achieve the best performance, whereas single-head performance drops drastically for high target alignment weight λt.The analysis fixes λs = 1.0 and varies λt in the USA/Singapore scenario.
- Cross-modal loss: 4.8 mIoU on 2D and 4.4 on 3D are gained when cross-modal loss is applied on source as well as target.Training the mimicking head on source stabilizes predictions used during target adaptation.
- Oracle training: Cross-modal learning improves supervised oracle training over the baseline and is proposed as an auxiliary loss that may regularize training.The authors conjecture that it can help prevent overfitting.
6. Conclusion
The paper concludes that xMUDA enables controlled mutual learning between 2D and 3D modalities for target-domain 3D segmentation. It outperforms unimodal UDA, complements pseudo-labeling, and also boosts fusion performance.
- Conclusion: xMUDA uses KL-divergence mutual mimicking between 2D and 3D modalities to improve target-domain performance.Separate main and mimicking heads disentangle segmentation from cross-modal learning.
- Conclusion: Experiments across three new cross-modal UDA scenarios show that xMUDA largely outperforms unimodal UDA.The conclusion also reports an analogous performance boost for fusion.
- Conclusion: Cross-modal learning is complementary to pseudo-labeling, and the paper defines new UDA scenarios and corresponding dataset splits.The contributions include a disentangled architecture and a balanced learning scheme.
- Conclusion: The authors suggest that cross-modal learning may extend beyond UDA and beyond image–point-cloud modality pairs.This is presented as a prospective direction rather than an experimentally established result.
A.1. nuScenes
The nuScenes dataset contains 1,000 driving scenes and 40k annotated keyframes, with official train, validation, and hidden test splits. The paper uses its metadata to construct adaptation splits.
- nuScenes: nuScenes contains 1,000 driving scenes of 20 seconds each, corresponding to 40k annotated keyframes sampled at 2Hz.The official split includes 28,130 training and 6,019 validation keyframes, plus a hidden test set.
- nuScenes: The paper proposes Day/Night and Boston/Singapore source-target splits for domain adaptation using the official validation split.
A.2. A2D2 and SemanticKITTI
A2D2 and SemanticKITTI provide complementary 2D-derived and directly annotated 3D point-cloud data, aligned through 10 shared semantic classes for adaptation experiments.
- A2D2 contains 28,637 frames from 20 drives, with point clouds captured by three 16-layer front LiDARs.Its 38-class semantic labels originate in 2D images and are projected onto the point cloud.
- SemanticKITTI provides directly annotated 3D point clouds from the KITTI Odometry dataset, using a 64-layer LiDAR and 28 classes.Scenes 0, 1, 2, 3, 4, 5, 6, 9, and 10 are used for training, scene 7 for validation, and scene 8 for testing.
- The datasets are harmonized to 10 shared classes: car, truck, bike, person, road, parking, sidewalk, building, nature, and other-objects.
A2D2 - SemanticKITTI Day - Night
The figure compares uni-modal pseudo-label adaptation with xMUDA across 2D and 3D predictions in the A2D2/SemanticKITTI and day/night scenarios.
- In the A2D2/SemanticKITTI scenario, xMUDA stabilizes both modalities and improves predictions for the bike, road, sidewalk, and building.
- In the day/night scenario, the figure separately presents 2D and 3D predictions for the UDA baseline and xMUDA.