PaperlayerView exact revision on arXiv

Source-linked AI summary

SPG: Unsupervised Domain Adaptation for 3D Object Detection via Semantic Point Generation

Qiangeng Xu, Yin Zhou, Weiyue Wang, Charles R. Qi, Dragomir Anguelov

arXiv:2108.06709v1cs.CV

TL;DR

LiDAR detectors can lose substantial cross-domain accuracy when weather and geography degrade point-cloud quality. The paper introduces SPG, which generates semantic foreground points and merges them with raw clouds; it improves PointPillars and PV-RCNN across UDA and original-domain evaluations, including KITTI.

  • Problem

    The paper addresses unsupervised adaptation of LiDAR 3D detectors across geographic and weather domains, where deteriorating point-cloud quality causes cross-domain performance drops.

  • Method

    SPG predicts foreground regions, generates semantic points to recover missing foreground shape, and merges them with the original cloud for modern detectors.

  • Results

    SPG significantly improves PointPillars and PV-RCNN on the target domain and consistently benefits source-domain detection on Waymo Open Dataset and KITTI.

  • Takeaways & Limitations

    SPG provides a general, lightweight adaptation approach that can be integrated with modern LiDAR detectors while adding only 6% extra points.

Abstract

from arXiv · show

In autonomous driving, a LiDAR-based object detector should perform reliably at different geographic locations and under various weather conditions. While recent 3D detection research focuses on improving performance within a single domain, our study reveals that the performance of modern detectors can drop drastically cross-domain. In this paper, we investigate unsupervised domain adaptation (UDA) for LiDAR-based 3D object detection. On the Waymo Domain Adaptation dataset, we identify the deteriorating point cloud quality as the root cause of the performance drop. To address this issue, we present Semantic Point Generation (SPG), a general approach to enhance the reliability of LiDAR detectors against domain shifts. Specifically, SPG generates semantic points at the predicted foreground regions and faithfully recovers missing parts of the foreground objects, which are caused by phenomena such as occlusions, low reflectance or weather interference. By merging the semantic points with the original points, we obtain an augmented point cloud, which can be directly consumed by modern LiDAR-based detectors. To validate the wide applicability of SPG, we experiment with two representative detectors, PointPillars and PV-RCNN. On the UDA task, SPG significantly improves both detectors across all object categories of interest and at all difficulty levels. SPG can also benefit object detection in the original domain. On the Waymo Open Dataset and KITTI, SPG improves 3D detection results of these two methods across all categories. Combined with PV-RCNN, SPG achieves state-of-the-art 3D detection results on KITTI.

1. Introduction

The paper studies why LiDAR 3D detectors fail across domains and identifies deteriorating point-cloud quality, especially under rainy weather, as the major gap. It proposes SPG to recover missing foreground points and improve detector reliability across domains.

  • Cross-domain generalization remains open because most 3D detection methods assume similar training and test conditions.
  • SPG is designed as a general, lightweight approach that benefits both target- and source-domain detection and integrates with modern LiDAR detectors.
  • A PointPillars model trained on Waymo Open Dataset drops 21.8 points in 3D AP when evaluated on Kirk.
  • Kirk contains roughly twice as many missing LiDAR points per frame as Waymo Open Dataset or Kirk Dry, with vehicles receiving around 27% fewer observations.
  • The major Waymo domain gap is deteriorating point-cloud quality caused by rainy weather, termed the “missing point” problem.
  • SPG predicts foreground regions, generates semantic points there, merges them with the raw cloud, and feeds the augmented cloud to a detector.

2. Related Work

Related work addresses domain adaptation through feature alignment, projection, point-cloud transformation, upsampling, completion, and style transfer. The paper distinguishes SPG by targeting missing foreground regions in full-scene LiDAR detection with a learning-based approach.

  • Most UDA methods align feature representations across domains, with many focusing on 2D detection or object-level 3D tasks.
  • Existing 3D approaches include global or local feature alignment, projection to 2D or BEV views, and adaptation of object features from car models.
  • A prior LiDAR UDA study attributes the KITTI-to-other-dataset gap to vehicle size, whereas this paper identifies point-cloud quality as the major Waymo gap.
  • Point-cloud upsampling can densify observed regions but requires high-density ground truth and does not recover regions with no observations.
  • Completion networks assume manually isolated single objects, while style-transfer methods do not focus on preserving local geometric details for detection.

3. Semantic Point Generation

SPG voxelizes raw point clouds, predicts foreground regions, generates semantic points, and merges them with the original points for LiDAR detection. Its training combines voxel classification and point-feature regression with hidden-point reconstruction and expanded-area supervision to recover missing foreground regions.

  • Pipeline: SPG voxelizes the raw point cloud, predicts foreground probabilities and semantic point features, then merges selected generated points with the original cloud.The augmented cloud is encoded with an additional foreground-confidence channel before being consumed by a detector.
  • Pipeline: The generation area includes occupied or neighboring voxels, while thresholding and top-K selection retain high-confidence semantic points.The practical foreground threshold is Pthresh = 0.5.
  • Model structure: Its encoder-decoder comprises voxel feature encoding, information propagation, and point generation modules.Information propagation transfers semantic information from occupied pillars into neighboring empty pillars, enabling foreground recovery in empty space.
  • Hide and Predict: Hide and Predict masks occupied voxels during training so SPG learns foreground labels and point features for unobserved regions.The method hides regions rather than isolated points, mimicking target-domain missing-point patterns; the reported masking rate is γ = 25.
  • Training targets: SPG constructs training targets by voxelizing the cloud, identifying foreground voxels, assigning labels, and computing foreground point-feature targets.Foreground labels identify voxels inside object bounding boxes; occupied foreground targets use point centroids and mean point properties.
  • Semantic Area Expansion: Semantic Area Expansion supervises empty voxels inside bounding boxes as foreground with reduced weight, while applying point-feature supervision only to occupied foreground voxels.The expansion produced 510% more semantic points on foreground objects and helped fill foreground space affected by interference and occlusions.
  • Objectives: SPG optimizes foreground-area classification with focal loss and semantic point-feature regression with Smooth-L1 loss within the generation area.The classification loss covers occupied, empty-background, empty-foreground, and hidden voxels; regression covers occupied and hidden foreground voxels.

4. Experiments

Experiments evaluate SPG with PointPillars and PV-RCNN on Waymo domain adaptation and KITTI, showing improvements across target, source, object, and difficulty settings with modest overhead.

  • Waymo domain adaptation: SPG improves Kirk target-domain detection for both detectors across object types and difficulty levels.PointPillars vehicle 3D AP increases 6.7%, while PV-RCNN pedestrian 3D AP increases 5.6%, averaged over two difficulty levels.
  • Waymo domain adaptation: SPG also improves source-domain results, unlike UDA methods that optimize only target-domain performance.OD vehicle 3D AP rises 5.4% for PointPillars, and OD pedestrian 3D AP rises 1.6% for PV-RCNN.
  • Alternative strategies: SPG outperforms alternative point-cloud strategies while adding only 8000 semantic points, less than 6% of a single frame.Five-frame aggregation improves the single-frame baseline but increases memory and processing time; ADA improves target AP to 36.34 while reducing source AP by 1.52.
  • KITTI: On KITTI, SPG improves PV-RCNN car detection and reaches 83.84% overall 3D AP on the test set.The reported ranking was first among published methods and fourth among submitted approaches as of March 3, 2021.
  • KITTI: On the KITTI validation set, SPG improves both detectors across object types and difficulty levels.PointPillars car 3D AP gains are 2.02%, 2.97%, and 3.67% across easy, moderate, and hard levels; PV-RCNN pedestrian gains are 5.40%, 5.13%, and 4.48%.
  • Efficiency: SPG contains 0.39 million parameters and adds less than 17 milliseconds of latency on the KITTI validation split.The evaluation uses a 1080Ti GPU with batch size 1.
  • Ablation studies: Probability threshold 0.5 achieves the best ablation result, while over-aggressive point generation does not improve performance.The threshold retains semantic points with foreground probability above 0.5.

5. Conclusions

The paper identifies rainy-weather point-cloud deterioration as a cause of cross-domain detection drops and proposes SPG for UDA without target-domain training data. SPG generates semantic foreground points with about 6% extra points and improves detection on Waymo and KITTI.

  • Conclusions: Rainy weather severely deteriorates LiDAR point-cloud quality and causes drastic performance drops across domains.The UDA setting evaluates detectors without using training data from the new domain.
  • Conclusions: SPG uses Hide and Predict and Semantic Area Generation to recover foreground shape with about 6% extra points.The method is designed for integration with modern LiDAR detectors and is evaluated with PointPillars and PV-RCNN.
  • Conclusions: SPG improves detection quality on the challenging target domain and consistently benefits source-domain detection on Waymo Open and KITTI.These results support its use across the two evaluated detector architectures.

A. Statistics of the Waymo Domain Adaptation Dataset

Waymo point statistics show that weather, rather than collection location, corresponds to the major difference in object point density. Dry datasets have similar counts, rainy datasets have similar counts, and rainy objects have fewer observations.

  • Point-count statistics: Figure 7 plots log10 point counts per vehicle against distance from the LiDAR sensor.The datasets include OD Validation, Kirk Dry, Kirk Training Rainy, and Kirk Validation.
  • Point-count statistics: Datasets with similar weather conditions have similar numbers of points per object across ranges, despite different collection locations.The two dry datasets and the two rainy datasets form matching point-count groups.
  • Point-count statistics: Dry-weather point clouds contain more points per object than rainy-weather point clouds across all ranges.The difference is substantial after applying log10 to the point counts for visualization.

B. The Robustness of the Foreground Voxel Classifier

SPG relies on foreground-voxel classification to generate semantic points reliably across domains. Its classifier is evaluated on both OD and Kirk and achieves high performance in each.

  • Foreground voxel classification: Correct foreground-voxel classification is necessary for reliable semantic-point generation across domains.The classifier identifies voxels inside foreground objects before points are generated.
  • Foreground voxel classification: The classifier is trained on OD and evaluated on both OD validation and Kirk validation using a 0.5 prediction threshold.Accuracy, precision, recall, and AP are computed under this foreground-versus-background criterion.
  • Foreground voxel classification: SPG achieves high foreground-voxel classification performance in both domains.The reported metrics include accuracy, precision, recall, and AP.

C. Dropout Rate of the RndDrop Method

RndDrop randomly removes 17% of source-domain points during training, matching the observed point-count difference between source and target domains.

  • 17% of source-domain points are randomly dropped during training in RndDrop.The ratio matches (N_src − N_tgt)/N_src, using 121.2K source points and 100.4K target points per scene on average.

D. More Results on the Waymo Domain Adaptation Dataset

The Waymo domain-adaptation evaluation reports AP across distance ranges and compares baseline detectors with their SPG-augmented versions on both domains and multiple object categories.

  • AP is reported for 0–30 meters, 30–50 meters, and beyond 50 meters, using 100 recall thresholds.
  • PointPillars and PV-RCNN are each compared with their SPG-augmented versions after training on OD and evaluation on both OD and Kirk.The comparisons cover vehicle and pedestrian 3D AP and BEV AP.

E. More Results on KITTI

Additional KITTI experiments evaluate SPG with PointPillars and PV-RCNN across car, pedestrian, and cyclist detection using multiple AP standards and dataset splits.

  • SPG significantly boosts cyclist detection in addition to improving car and pedestrian detection on KITTI.Results are reported for 3D AP and BEV AP using both R11 and R40 evaluation standards.
  • KITTI validation results use R11 and R40 AP tables, while additional car comparisons appear on the KITTI test set.

F. More Visualization of Semantic Point Generation

The visualizations distinguish raw and generated points and present SPG-related results across Waymo domain-adaptation tables and KITTI comparisons.

  • F. More Visualization of Semantic Point Generation: Raw points are rendered in grey, while generated semantic points are highlighted in red in the augmented point-cloud visualizations.
  • D. More Results on the Waymo Domain Adaptation Dataset: Waymo tables report vehicle and pedestrian 3D AP and BEV AP for OD and Kirk, with distance-based AP breakdowns.
  • E. More Results on KITTI: KITTI validation tables report comparisons using AP with 11 and 40 recall positions.
  • E. More Results on KITTI: The KITTI test-set table compares car detectors using R40 AP across Easy, Moderate, and Hard difficulty levels.Avg. AP averages the AP values for the three difficulty levels.
Loading 2108.06709v1…