Source-linked AI summary
Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Clouds
Li Yi, Boqing Gong, Thomas Funkhouser
TL;DR
Different LiDAR sensors produce sampling patterns that hinder semantic-label transfer across datasets. The paper completes sparse point clouds into surfaces with SVCN, then labels those surfaces, reporting substantial gains over prior domain-adaptation methods.
Problem
Different LiDAR sensors create distinct 3D sampling patterns, limiting transfer of semantic-segmentation networks across datasets.
Method
The complete-and-label approach uses SVCN to recover 3D surfaces, trains segmentation on the completed surfaces, and uses self-supervised completion data without manual labels.
Results
6.0 and 10.4 absolute mIoU points are reported over prior domain-adaptation methods for Waymo→nuScenes and nuScenes→Waymo transfer, respectively.
Takeaways & Limitations
Completed 3D surfaces provide a sensor-agnostic canonical domain for transferring semantic labels across LiDAR scanners.
Takeaways & Limitations
SVCN training can use data different from semantic-segmentation training and relies on self-supervised exemplars such as simulation, multi-view registration, or high-resolution LiDAR.
Abstract
from arXiv · showhide
We study an unsupervised domain adaptation problem for the semantic labeling of 3D point clouds, with a particular focus on domain discrepancies induced by different LiDAR sensors. Based on the observation that sparse 3D point clouds are sampled from 3D surfaces, we take a Complete and Label approach to recover the underlying surfaces before passing them to a segmentation network. Specifically, we design a Sparse Voxel Completion Network (SVCN) to complete the 3D surfaces of a sparse point cloud. Unlike semantic labels, to obtain training pairs for SVCN requires no manual labeling. We also introduce local adversarial learning to model the surface prior. The recovered 3D surfaces serve as a canonical domain, from which semantic labels can transfer across different LiDAR sensors. Experiments and ablation studies with our new benchmark for cross-domain semantic labeling of LiDAR data show that the proposed approach provides 8.2-36.6% better performance than previous domain adaptation methods.
1. Introduction
The paper addresses cross-sensor domain gaps in LiDAR semantic segmentation by completing sparse point clouds into canonical 3D surfaces before labeling them. Its SVCN-based approach avoids manual completion labels and improves cross-dataset adaptation performance.
- LiDAR semantic segmentation assigns a semantic label to every point and supports applications including autonomous driving, semantic mapping, and construction-site monitoring.
- Different LiDAR sensors create different 3D sampling patterns, so networks trained on one dataset often perform poorly on another.
- The approach transforms sampling-driven domain adaptation into 3D surface completion by recovering complete surfaces from sparse LiDAR samples.
- Surface completion is easier to supervise than semantic segmentation because geometric priors support self-supervision or synthetic training without manual labels.
- A Sparse Voxel Completion Network recovers surfaces, followed by a sparse convolutional U-Net that predicts semantic labels for completed voxels.
- 6.0 absolute mIoU points and 10.4 absolute mIoU points are reported for Waymo→nuScenes and nuScenes→Waymo transfer, respectively, over prior domain-adaptation methods.
2. Related Work
Related work spans unsupervised domain adaptation, deep 3D semantic segmentation, and deep 3D shape completion. The paper positions sparse voxel completion as a high-resolution, LiDAR-scale bridge between shape completion and sensor-invariant segmentation.
- Unsupervised domain adaptation: Unsupervised domain adaptation addresses mismatched training and test distributions by exposing learners to unlabeled target examples while retaining source-domain supervision.
- Unsupervised domain adaptation: Prior 3D point-cloud adaptation methods address sensor, representation, synthetic-to-real, or feature-matching discrepancies using dense voxels, bird’s-eye-view images, or alignment techniques.
- Deep 3D semantic segmentation: Deep 3D semantic segmentation must handle multiple data representations, while dense voxel convolutions are computationally expensive and typically limit resolution.
- Deep 3D shape completion: Deep 3D shape completion reconstructs missing geometry, but dense methods scale poorly to large LiDAR clouds and point-cloud upsampling lacks control over output sampling patterns.
- Deep 3D shape completion: The paper introduces sparse voxel completion to produce high-resolution completed LiDAR surfaces while applying shape completion specifically to 3D domain adaptation.
3. Method
The method transforms cross-sensor domain adaptation into surface completion: sensor-specific networks recover dense 3D surfaces, after which a segmentation network operates in a canonical domain. Training data for completion is constructed without manual semantic labels by reconstructing dense surfaces and simulating sensor-specific sampling.
- Overview: The two-stage approach first completes sensor-specific sparse LiDAR clouds and then assigns semantic labels with a sensor-agnostic segmentation network.Completed surfaces form the canonical domain used for cross-sensor label transfer.
- SVCN Architecture: SVCN generates denser voxels representing underlying surfaces, then refines the result by pruning redundant or low-confidence voxels.Its structure generation network extrapolates new voxels, while refinement adds no new voxels and predicts existence confidence.
- Training Data: Dense surface supervision is obtained by aggregating multiple LiDAR frames, reconstructing meshes with Poisson surface reconstruction, and uniformly sampling the surfaces.A SurfaceTrimmer step removes low-density regions before discretization.
- Training Data: A polar sampling scheme transfers domain-specific sensor patterns by matching angular coordinates in a complete scene while accounting for occlusion and sensor location.The method uses nearest neighbors in (θ, φ) space to imitate a reference point cloud’s sampling pattern.
- Training Algorithm: SVCN training uses voxel-wise binary cross-entropy for structure generation and refinement, supplemented by local adversarial learning over surface patches.The adversarial loss models the prior that densified voxels should lie on 3D surfaces.
4. Experiments
Experiments evaluate completion, cross-dataset domain adaptation, and generalization across Waymo, nuScenes-lidarseg, and SemanticKITTI, showing that surface completion improves transfer under sensor-induced sampling differences.
- Experimental setup: The benchmark evaluates three autonomous-driving datasets captured with different LiDAR configurations, using overlapping semantic categories for cross-domain transfer.Waymo-to-nuScenes/SemanticKITTI transfers use vehicles and pedestrians; nuScenes-to-SemanticKITTI additionally uses ten overlapping categories.
- Sparse LiDAR Point Cloud Completion: SVCN is trained to complete sparse LiDAR inputs into dense volumetric surfaces, evaluated with voxel IoU and Chamfer Distance.Training pairs aggregate multiple Waymo frames into 2,400 training and 200 test complete scenes, then generate incomplete inputs with virtual LiDAR sampling.
- Sparse LiDAR Point Cloud Completion: The full SVCN with local adversarial learning outperforms competing completion methods across the sampling patterns of all three datasets.The refinement network improves completion quality, while local adversarial learning further improves completions by modeling surface priors.
- Sparse LiDAR Point Cloud Completion: SVCN recovers underlying surfaces across different input sampling patterns and fills small geometric holes, supporting a more canonical representation for segmentation.Visualizations compare sparse inputs with colored SVCN outputs across datasets.
- Unsupervised Domain Adaptation Results: 2D adaptation baselines perform poorly on 3D point clouds because they do not model local sampling and surface cues or introduce boundary errors after range-image projection.The proposed method outperforms handcrafted sampling-alignment baselines, whose limitations include phantom points and uncompleted occluded regions.
- Domain Generalization Results: Generic SVCN domain generalization performs slightly worse than target-specific adaptation, but both substantially outperform no-adaptation baselines.The reported pattern holds for completion and semantic segmentation on nuScenes-lidarseg and SemanticKITTI.
5. Conclusion
“Complete and label” transforms cross-sensor domain adaptation into surface completion, then performs semantic segmentation on completed surfaces with sensor-agnostic networks.
- The approach recovers complete 3D surfaces from LiDAR point clouds before applying downstream semantic segmentation.
A. Additional Ablation Studies
Additional ablations examine completion quality and handcrafted sampling alignment across Waymo, nuScenes-lidarseg, and SemanticKITTI, supporting the role of learned completion in domain transfer.
- Better scene completion quality leads to better domain transfer performance across additional domain directions.
- Table 6 reports segmentation mIoU for variants of scene completion across nuScenes-lidarseg, Waymo, and SemanticKITTI.
- The nuScenes-lidarseg–SemanticKITTI adaptation uses 10 categories, and the method outperforms both handcrafted baselines and no adaptation by large margins.
- Table 7 compares the method with no adaptation and handcrafted sampling alignment baselines across the three datasets.
B. Loss Function for Training SVCN
SVCN is trained with voxel-existence supervision and local adversarial learning that injects a prior favoring recovered voxels lying on 3D surfaces.
- The structure generation and refinement networks operate across 7 resolution levels with voxel-existence supervision derived from complete output point clouds.
- SVCN uses binary cross entropy to supervise voxel existence predictions for structure generation and refinement.
- Local adversarial learning adds adversarial losses to SVCN’s generation and refinement objectives to model the prior that completed voxels lie on 3D surfaces.
- The discriminators use local receptive fields and fully convolutional architectures instead of a global discriminator.
- Confidence-aware sparse convolution incorporates voxel confidence values into discriminator processing, making perfect 0-or-1 predictions harder to distinguish from realistic scenes.
C. Label Transfer to and from the Canonical Domain
The method transfers source labels into a completed canonical domain for training and projects canonical predictions back onto target-domain points at inference.
- Prop transfers source-domain labels to the canonical domain, while Proj maps canonical predictions back to target-domain labels.
- Both operations use nearest-neighbor mappings between voxelized point clouds and SVCN outputs.
- Source voxel labels are determined by majority voting, and unlabeled canonical voxels are masked during training.
- At inference, each target point receives the fetched label assigned to its containing voxel.
D. Implementation Details
The networks use a shared seven-level encoder–decoder design, with specified filter widths across levels. Training uses 20 cm voxels, LiDAR positions, multiscale voxelization, and a decaying adversarial-learning rate.
- Network architecture: The structure generation, structure refinement, and semantic segmentation networks each use seven encoder–decoder levels with shared filter-count settings.Encoder filter pairs increase from (24, 24) at level 0 to (96, 112) at level 6.
- Voxelization: All experiments use a voxel size of d = 20cm.
- Voxelization: Structure-generation training voxelizes complete point clouds at level-specific resolutions of 2^ld and marks a voxel occupied when it contains at least one point.
- Input representation: The models use LiDAR point positions as input without color or intensity information.
- Optimization: The discriminator learning rate starts at 10−4 and decays by a factor of 0.7 every 200k training steps.