PaperlayerView exact revision on arXiv

Source-linked AI summary

ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection

Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, Xiaojuan Qi

arXiv:2103.05346v2cs.CVcs.LG

TL;DR

3D detectors suffer large domain shifts, while adapting them to unlabeled target domains remains difficult. ST3D redesigns self-training with source augmentation, quality-aware pseudo-label updating, and curriculum training, closing 16% ∼75% of source-only-to-oracle gaps across four datasets and surpassing oracle results on Waymo →KITTI when combined with an existing approach.

  • Problem

    3D detectors may generalize poorly across domains, while collecting additional cross-domain annotations is often infeasible or costly.

  • Method

    ST3D pre-trains with random object scaling, then alternates quality-aware triplet-memory-bank pseudo-label updating with curriculum data augmentation.

  • Results

    16% ∼75% of source-only-to-oracle performance gaps are closed across four datasets, and combined ST3D surpasses oracle results on Waymo →KITTI.

  • Takeaways & Limitations

    ST3D improves cross-domain 3D detection across label-rich-to-label-insufficient and differing-LiDAR-beam adaptations.

  • Takeaways & Limitations

    Lyft’s annotation protocol omits many objects on both road sides, making the actual performance boost difficult to evaluate.

Abstract

from arXiv · show

We present a new domain adaptive self-training pipeline, named ST3D, for unsupervised domain adaptation on 3D object detection from point clouds. First, we pre-train the 3D detector on the source domain with our proposed random object scaling strategy for mitigating the negative effects of source domain bias. Then, the detector is iteratively improved on the target domain by alternatively conducting two steps, which are the pseudo label updating with the developed quality-aware triplet memory bank and the model training with curriculum data augmentation. These specific designs for 3D object detection enable the detector to be trained with consistent and high-quality pseudo labels and to avoid overfitting to the large number of easy examples in pseudo labeled data. Our ST3D achieves state-of-the-art performance on all evaluated datasets and even surpasses fully supervised results on KITTI 3D object detection benchmark. Code will be available at https://github.com/CVMI-Lab/ST3D.

1. Introduction

3D object detectors can lose substantial accuracy across sensor, weather, and geographic domains, while unlabeled target data makes adaptation difficult. ST3D redesigns self-training for this setting with source-domain augmentation, quality-aware pseudo-labeling, and curriculum training, yielding strong cross-domain results.

  • Motivation: 3D object detection localizes and classifies objects from point clouds for applications including autonomous driving, robotics, and virtual reality.Point-cloud detection is challenging because 3D point clouds are irregular and sparse.
  • Motivation: Over 45% performance drops when detectors trained on Waymo data are evaluated on KITTI, reflecting shifts in sensors, weather, and geography.Waymo uses LiDAR collected in USA cities, whereas KITTI uses Velodyne LiDAR in European cities.
  • Motivation: Unsupervised domain adaptation targets adaptation from labeled source data to an unlabeled target domain when collecting additional cross-domain annotations is infeasible or costly.This setting is especially relevant to practical 3D detection applications.
  • Related work: Few methods address UDA for 3D detection, and prior SN adaptation requires target-domain object-size statistics while depending strongly on data distributions.Image-based UDA methods are not readily transferable because 3D detection uses different data structures and network architectures.
  • Problem: Naive self-training performs poorly for 3D detection, despite its success in image recognition, because pseudo labels include both oriented 3D boxes and object categories.The pipeline iterates between pseudo-label generation and target-domain model training.
  • Approach: ST3D combines random object scaling, a quality-aware triplet memory bank, and curriculum data augmentation to redesign self-training for 3D UDA.These components address source object-size bias, pseudo-label noise, ambiguous examples, training instability, and overfitting to easy pseudo-labeled data.
  • Results: 16% ∼75% of the performance gaps between source-only and fully supervised oracle results are closed across four evaluated 3D detection datasets.ST3D also outperforms the existing approach on all evaluated settings and, when combined with it, exceeds oracle results on Waymo →KITTI.

3. Method

ST3D adapts a source-trained 3D detector to unlabeled target data through ROS pre-training, QTMB pseudo-label generation, and CDA training. These stages address source object-size bias, pseudo-label quality and consistency, and overfitting to easy pseudo-labeled examples.

  • Overview: ST3D pre-trains the detector on labeled source data with random object scaling (ROS) before target-domain self-training.ROS randomly scales annotated 3D objects to mitigate source-domain object-size bias.
  • Overview: The framework alternates pseudo-label generation and model training on target data until convergence.
  • Model Pre-training with ROS: ROS simulates diverse object sizes, facilitating size-robust detectors that produce more accurate initial pseudo boxes.
  • Pseudo-label Generation with QTMB: QTMB maintains pseudo labels across iterations by combining current detector predictions with historical labels through memory ensemble and voting.
  • Pseudo-label Generation with QTMB: QTMB generates target pseudo labels using IoU-based localization-quality scoring and triplet partitioning that avoids ambiguous predictions.The IoU criterion assesses box quality directly, while the triplet scheme reduces noise from ambiguous examples.
  • Model training with CDA: CDA progressively increases augmentation intensity to generate harder examples without confusing the learner early in training.This curriculum enables learning from challenging samples while keeping example difficulty within the learner’s capability.

4. Experiments

Experiments evaluate ST3D across multiple datasets and adaptation settings, comparing it with source-only, statistical-supervision, and oracle baselines. Ablations show that its components improve pseudo-label quality and target-domain detection performance.

  • Experimental Setup: Experiments cover KITTI, Waymo, nuScenes, and Lyft across label-rich-to-label-insufficient and differing-LiDAR-beam adaptation settings.Evaluation uses car-category AP at IoU 0.7, with ring-view point clouds except for KITTI’s front view.
  • Main Results: ST3D improves over Source Only on every evaluated UDA setting.On Waymo →KITTI and nuScenes →KITTI, AP3D improves by around 34% ∼43%.
  • Main Results: ST3D combined with SN improves SN by 14% with SECOND-IoU and 13% with PV-RCNN in AP3D on Waymo →KITTI.On this setting, ST3D with SN performs on par with the fully supervised detector.
  • Ablation Studies: Random object scaling improves AP3D by around 27.2% on Waymo →KITTI and remains 4.5% below the weakly supervised SN method.The result supports robustness to object-size variation without target-domain size statistics.
  • Ablation Studies: Removing triplet box partition drops AP3D by 3.3% or 5.4% for Tpos = Tneg = 0.6 or 0.25, respectively.The quality-aware criterion, memory ensemble-and-voting, and curriculum data augmentation contribute additional gains in the component analysis.
  • Ablation Studies: Max-score memory merging gains 1.3% over weighted averaging, and removing memory voting reduces performance by over 2.4%.The analysis attributes these differences to heading-angle mismatches and handling unmatched boxes across memories.
  • Ablation Studies: Without data augmentation, performance degrades by over 6.6%, while curriculum data augmentation adds around 0.9% over normal-intensity augmentation.Stronger-than-normal augmentation slightly reduces performance by confusing the deep learner.

5. Conclusion

The paper concludes that ST3D is a redesigned self-training pipeline for unsupervised domain-adaptive 3D object detection from point clouds. It combines three targeted components and substantially advances the state of the art.

  • 5. Conclusion: ST3D combines random object scaling, a quality-aware triplet memory bank, and curriculum data augmentation for unsupervised domain adaptation.These components address challenges arising in self-training for 3D object detection.
  • 5. Conclusion: Experiments demonstrate that ST3D substantially advances the state of the art.

Outline

The supplementary file adds dataset details, domain-difference analyses, implementation information, further experiments, ablations, qualitative results, and additional adaptation tasks. These materials expand the main paper’s empirical and methodological coverage.

  • Supplementary Overview: Section S1 provides dataset details for the domain adaptation tasks.
  • Supplementary Overview: Section S2 analyzes domain differences and systematic bias in pseudo labels.
  • Supplementary Overview: Sections S3–S5 provide implementation details, additional results, and ablation studies.
  • Supplementary Overview: Sections S6–S7 present qualitative results and experiments on other adaptation tasks.

S1. Dataset Overview

The study compares four autonomous-driving LiDAR datasets that differ in sensor characteristics, point-cloud distributions, density, size, and collection locations. Across the evaluated settings, ST3D improves the baseline and outperforms the existing approach.

  • Dataset Overview: Waymo, KITTI, nuScenes, and Lyft differ in LiDAR type, beam angles, point-cloud density, size, and collection locations.Even datasets with the same beam count show different range, vertical, and horizontal point distributions.
  • Dataset Overview: Experiments include Waymo →KITTI, Waymo →Lyft, Waymo →nuScenes, and cross-beam adaptations Waymo →nuScenes and nuScenes →KITTI.
  • Dataset Overview: ST3D improves the baseline and outperforms the existing approach on all evaluated settings.

S2. Domain Difference and Systematic bias

The evaluated domains differ in point-cloud density, annotation protocols, object-size and point-distribution characteristics, creating systematic biases that complicate adaptation and evaluation.

  • Annotation differences: Lyft does not annotate objects on both sides of the road, unlike the other three datasets.This annotation protocol difference is illustrated for Lyft and Waymo scenes.
  • Sensor and point-distribution differences: Waymo, KITTI, Lyft, and nuScenes differ in LiDAR-based point-cloud density, with nuScenes sparsest and Waymo denser than KITTI and Lyft.nuScenes uses 32-beam LiDAR, while the other three use 64-beam LiDAR; Waymo beams cluster at medium vertical angles.
  • Evaluation implications: Waymo-pretrained predictions can identify cars omitted from Lyft annotations, making the actual adaptation improvement difficult to evaluate.The paper reports correct predictions on both sides of Lyft roads that are absent from Lyft ground truth.
  • Systematic bias: The domain gap is attributed to content differences such as object size and point-distribution differences caused by different LiDAR beams.Self-training addresses this gap by reformulating adaptation as target-domain supervised learning with pseudo labels.
  • Systematic bias: Pseudo labels can inherit source-domain annotation-style bias because annotation rules differ in box tightness and which objects are labeled.This bias makes pseudo labels differ from target-domain ground truth.

S3. Implementation details

The supplementary implementation uses staged self-training components, fixed detector and voxel settings, and omits GT sampling to control computational cost and comparison fairness.

  • Training schedule: The Waymo → KITTI setup pre-trains for 30 epochs, self-trains for 30 epochs, and updates pseudo labels every two epochs.ROS uses scaling range [0.75, 1.1], while QTMB thresholds are Tneg=0.25 and Tpos=0.6.
  • Training schedule: CDA divides self-training into six successive epoch stages.The supplied passage begins listing the stages as [0, 5), [5, 10), [10, 15), [15, 20), and [20, 25).
  • Voxel and detector settings: All experiments use voxel size [0.10m, 0.10m, 0.15m] for a fair evaluation across models and frameworks.The choice balances memory and realization under the stated ring-view training setup.
  • Data augmentation: GT sampling is omitted because iterative pseudo-label updates would require repeatedly rebuilding the ground-truth database.Using GT sampling for self-training takes more than 3× the training time.
  • Evaluation protocol: The authors emphasize that comparisons aim to assess ST3D without various additional 3D detection training tricks.This motivation accompanies the reported sub-optimal memory and computational setup.
  • Pseudo-label processing: The implementation includes IoU-based proposal scoring and alternative NMS and bipartite ensembles for combining historical and current pseudo labels.NMS removes matched lower-confidence boxes, while bipartite ensemble uses optimal matching before consistency processing.

S4. Experimental Results with IoU = 0.5

The supplementary experiments report APBEV and AP3D at IoU threshold 0.5 as additional results, while noting that 0.7 is the stricter common car-detection criterion.

  • Reported metrics: APBEV and AP3D are reported with IoU threshold 0.5 in supplementary tables.The paper presents these measurements as a supplement to the main experimental results.
  • Evaluation criterion: IoU threshold 0.7 is described as a stricter criterion widely used for evaluating 3D car detection models.The passage contrasts this threshold with the supplementary 0.5 setting.

S5. Extra Ablation Studies

Extra ablations examine evaluation settings, contemporary comparisons, and confidence criteria, showing benefits from IoU-based confidence and competitive single-frame performance.

  • Ablation setup: The supplementary setting evaluates SECOND-IoU on Waymo → KITTI using moderate-difficulty KITTI cars.The reported AP results use the car category for this adaptation setting.
  • Contemporary comparison: ST3D achieves similar performance to SF-UDA3D using only single-frame point clouds while remaining closer to fully supervised oracle results.SF-UDA3D leverages temporal information from point-cloud sequences.
  • Confidence criterion: IoU confidence improves AP3D by 2.12% for SN, 0.44% for the fully supervised oracle, and up to 3.79% for ST3D with SN.The results compare IoU confidence against classification confidence.
  • Confidence criterion: The reported IoU-confidence gains support using localization quality, rather than classification score alone, when producing pseudo-labeled data.The paper states that the IoU criterion can facilitate higher-quality pseudo labels.

S6. Qualitative Results

Qualitative analyses show that random object scaling improves pre-trained localization without target-domain statistics, while ST3D with SN produces high-quality boxes across varied scenes.

  • Random Object Scaling: Random object scaling largely improves pre-trained-model localization accuracy without using extra target-domain statistical information.The comparison includes Source Only, ROS, SN, and Oracle models.
  • ST3D Qualitative Results: ST3D equipped with SN predicts high-quality object bounding boxes across various KITTI validation scenes.These qualitative results evaluate the adaptation and self-training setting.

S7. Experimental Result on More Tasks.

Additional experiments extend evaluation across practical adaptation settings while excluding ill-posed source-domain choices; ST3D remains promising without hyperparameter tuning.

  • Experimental Scope: The main experiments cover practical scenarios spanning different LiDAR beam configurations and label-rich to label-insufficient domains.Some settings are excluded because KITTI lacks ring-view annotations and Lyft has substantially different annotations.
  • Evaluation Protocol: The adaptation-task results report car APBEV and AP3D at IoU = 0.5, using moderate-case results for KITTI-target tasks and overall results otherwise.The task-specific reporting protocol is summarized in Table S16.
  • Additional Tasks: ST3D achieves promising results on five additional adaptation tasks without hyperparameter tuning.These experiments further validate the method beyond the main evaluated settings.
Loading 2103.05346v2…