PaperlayerView exact revision on arXiv

Source-linked AI summary

Train in Germany, Test in The USA: Making 3D Object Detectors Generalize

Yan Wang, Xiangyu Chen, Yurong You, Li Erran, Bharath Hariharan, Mark Campbell, Kilian Q. Weinberger, Wei-Lun Chao

arXiv:2005.08139v1cs.CV

TL;DR

The paper asks why 3D object detectors generalize poorly across autonomous-driving datasets despite strong within-dataset performance. It evaluates cross-dataset domain differences and applies an aggregate car-size correction, finding that geographic variation in car sizes is the primary hurdle and that the correction substantially narrows the gap. The study provides a simple baseline for cross-country 3D detection adaptation.

  • Problem

    3D detectors may overfit to dataset-specific biases, while autonomous-driving datasets often cover narrow geographic, sensor, and environmental domains, making cross-dataset adaptation difficult.

  • Method

    The paper evaluates LiDAR-based 3D detectors across multiple datasets and corrects domain bias using aggregate statistics of car sizes in annotations and input signals.

  • Results

    The analysis identifies car-size differences as the dominant adaptation factor, and the correction raises 3D easy part average precision by 41.4 points.

  • Takeaways & Limitations

    Aggregate car-size information offers a simple, broadly incorporable baseline for improving cross-dataset 3D object detection performance.

  • Takeaways & Limitations

    Category definitions can differ across datasets, and point resampling or voxelization reduces but does not fully resolve cross-domain point-cloud differences.

Abstract

from arXiv · show

In the domain of autonomous driving, deep learning has substantially improved the 3D object detection accuracy for LiDAR and stereo camera data alike. While deep networks are great at generalization, they are also notorious to over-fit to all kinds of spurious artifacts, such as brightness, car sizes and models, that may appear consistently throughout the data. In fact, most datasets for autonomous driving are collected within a narrow subset of cities within one country, typically under similar weather conditions. In this paper we consider the task of adapting 3D object detectors from one dataset to another. We observe that naively, this appears to be a very challenging task, resulting in drastic drops in accuracy levels. We provide extensive experiments to investigate the true adaptation challenges and arrive at a surprising conclusion: the primary adaptation hurdle to overcome are differences in car sizes across geographic areas. A simple correction based on the average car size yields a strong correction of the adaptation gap. Our proposed method is simple and easily incorporated into most 3D object detection frameworks. It provides a first baseline for 3D object detection adaptation across countries, and gives hope that the underlying problem may be more within grasp than one may have hoped to believe. Our code is available at https://github.com/cxy1997/3D_adapt_auto_driving.

1. Introduction

The paper studies why 3D object detectors trained in one driving environment fail in another and finds that car-size statistics are the dominant adaptation issue. It proposes a simple aggregate-size correction that substantially improves cross-dataset performance.

  • KITTI is a narrow-domain dataset collected in Germany with a fixed sensing setup, clear daytime conditions, and approximately 15K scenes.
  • Cross-dataset evaluation reveals drastic accuracy drops for 3D object detectors trained in one autonomous-driving dataset and tested in another.
  • The analysis identifies statistical differences in car sizes and shapes across locations as the main adaptation hurdle.
  • The proposed correction uses aggregate car-size statistics, such as the mean, to adjust output annotations and input signals.
  • 41.4 points: the correction raises 3D easy part average precision and produces a more robust cross-dataset detector.

2. Related Work

This section reviews LiDAR-based 3D object detection and domain adaptation for street-scene perception. It emphasizes that adaptation research in autonomous driving has mainly addressed 2D segmentation and detection.

  • LiDAR-based object detection: LiDAR-based detectors encode accurate 3D environmental points to predict point labels or 3D bounding boxes.The review notes that some methods also combine LiDAR with images.
  • Domain adaptation: Domain adaptation has mainly been studied for 2D semantic segmentation and 2D object detection in autonomous-driving scenes.The reviewed literature covers adaptation between labeled source domains and unlabeled target domains.
  • Domain adaptation: The common adaptation setting transfers a model from one labeled source domain to an unlabeled target domain.An example pairs synthetic source images with real target images.

3. Datasets

The experiments use five autonomous-driving datasets with ground-truth 3D bounding boxes for several object types. They span different collection locations, sensors, weather conditions, and times of day.

  • Dataset overview: Five datasets—KITTI, Argoverse, Lyft, nuScenes, and Waymo—are used for experiments on 3D object detection.All provide ground-truth 3D bounding-box labels for several kinds of objects.
  • KITTI: KITTI contains 7,481 training images and 7,518 test images for its object-detection benchmark.The passage also notes that KITTI provides imagery from five additional cameras covering 360° information.
  • Lyft: Lyft scenes come from Palo Alto, USA, under clear daytime conditions, with point clouds captured by 40- or 64-beam roof LiDAR and 40-beam bumper LiDAR.Its split includes 12,599 training, 3,024 validation, and 3,011 testing images.
  • Waymo: Waymo scenes come from Phoenix, Mountain View, and San Francisco across multiple weathers and times of day.Waymo provides combined point clouds from five LiDAR sensors and images from four cameras.
  • Waymo: Waymo is subsampled into 12,000 training, 3,000 validation, and 3,000 test images.The original dataset contains 122,000 training, 30,407 validation, and 40,077 test images.

4. Experiments and Analysis

Experiments across five autonomous-driving datasets show substantial cross-dataset performance drops despite strong within-dataset detection. Analyses identify sensor quality, sparse far-range points, and especially mismatched car-size statistics as key domain differences, with errors concentrated in localization.

  • Experimental setup: POINTRCNN and PIXOR are trained on five datasets and evaluated using Car-category APBEV and AP3D under KITTI-style difficulty and range settings.The study uses LiDAR-based detectors without image inputs.
  • Cross-dataset results: Cross-dataset degradation is larger at greater distances, while Argoverse and nuScenes are particularly affected by sparse far-range LiDAR points.nuScenes uses 32 beams and Argoverse uses 64 beams with a stacked configuration.
  • Within-dataset results: Within-dataset performance is generally strong when sensor signals are high quality and labeled instances are sufficient.
  • Cross-dataset results: 45.2% APBEV (Moderate) and 11.9% AP3D are achieved on Waymo by a POINTRCNN trained on KITTI, versus 85.3% AP3D for the Waymo-trained model.The KITTI-trained model is lower by over 40% in APBEV than the Waymo-trained model.
  • Cross-dataset results: KITTI-trained detectors transfer worst overall, whereas target-domain sensor quality appears more important for testing than source-domain sensor quality.The authors also suggest KITTI’s car styles, time, and weather may be too limited or different for transfer.
  • Domain idiosyncrasies: Waymo has ten times as many points per car as nuScenes, while KITTI has the smallest and Waymo the largest bounding-box sizes across the analyzed datasets.The authors associate box-size differences with car styles captured in different datasets.
  • Domain idiosyncrasies: Point resampling and voxelization reduce, but cannot fully resolve, point-cloud differences across domains.
  • Localization analysis: Lowering the IoU threshold makes cross-dataset AP3D models converge below 0.4, indicating comparable detection ability but differing localization quality.Replacing detected box sizes with corresponding ground-truth sizes produces an immediate performance boost.

5. Domain Adaptation Approaches

The paper evaluates few-shot fine-tuning and statistical normalization for cross-dataset adaptation. Statistical normalization modifies source boxes and point clouds using target car-size statistics, improving transfer without labeled target-domain data.

  • Adaptation settings: The adaptation study considers few labeled target scenes and access only to target-domain car-size statistics as two practical scenarios.The authors suggest local annotations or statistics from vehicle offices and car-selling websites as possible sources.
  • Few-shot fine-tuning: Few-shot fine-tuning evaluates KITTI validation performance as the number of KITTI training images increases, with scratch training as a randomly initialized baseline.
  • Statistical normalization: Statistical normalization computes target-minus-source mean car-size differences for height, width, and length, then adjusts source labels and point clouds accordingly.Points inside each original car box are rescaled to fit the adjusted box.
  • Adaptation results: Statistical normalization significantly improves cross-dataset performance without requiring labeled target-domain data.Figures show predicted-box histograms shifting from source-domain toward target-domain sizes after normalization.
  • Adaptation results: The approach targets one domain-gap factor—box-size differences—which the authors find highly effective for closing cross-dataset gaps.The paper presents this factor and solution as additions to 3D domain adaptation research.

6. Conclusion

The paper concludes that cross-dataset 3D detection is difficult, but varying car sizes across geographic regions explain a majority share of the adaptation gap. Matching mean car size during fine-tuning substantially reduces the resulting uncertainty.

  • The paper provides and investigates a standardized form of widely used autonomous-driving 3D object-detection datasets.
  • Varying car sizes across geographic regions are identified as a single dominant factor explaining a majority share of the adaptation gap.
  • Matching the mean size of cars during fine-tuning is reported to reduce bounding-box uncertainty substantially.The conclusion illustrates the issue with a German-trained detector encountering a larger American Ford F-Series truck.

S1. Converting Datasets into KITTI Format

The appendix converts four datasets into KITTI format by standardizing object filtering, categories, 2D boxes, truncation, and occlusion annotations. These procedures align heterogeneous annotations for cross-dataset car detection.

  • S1. Converting Datasets into KITTI Format: Argoverse, nuScenes, Lyft, and Waymo are converted into KITTI format, with image, point-cloud, and calibration formatting treated as straightforward.The appendix focuses on nondeterministic labeling definitions.
  • S1.1. Object filtering: Objects are discarded when all projected 3D-box corners lie outside the frontal image view or when their depth exceeds 70 meters.The procedure follows KITTI’s camera-view setting while allowing truncated objects.
  • S1.2. Matching categories with KITTI: Dataset-specific categories are remapped into KITTI-style car and truck classes, with trucks retained because they count as car false positives.Waymo labels all kinds of vehicles as cars, while other datasets differ in car–truck assignments.
  • S1.2. Matching categories with KITTI: Category mismatches can reduce cross-domain accuracy even when datasets share similar locations and sensors.Argoverse and nuScenes assign some Ford F-Series vehicles differently.
  • S1.3. Handling missing 2D bounding boxes: Missing 2D boxes are reconstructed by projecting the eight 3D-box corners into the image and enclosing their in-frame projections.The resulting box is the smallest rectangle containing the projected corners that fall within the image plane.
  • S1.4. Calculating truncation values: Truncation is estimated from the portion of each object’s uncropped 2D box that lies outside the image boundary.This follows the KITTI formulation of truncation.
  • S1.5. Calculating occlusion values: Occlusion is approximated with 2D boxes by measuring the percentage of pixels covered by boxes from closer objects.Continuous occlusion values from 0 to 1 are discretized into KITTI labels {0, 1, 2, 3} using four equal intervals.
  • S1.5. Calculating occlusion values: The occlusion algorithm processes image pixels and object-box regions to assign occlusion values for each object in a scene.Its inputs are image height, image width, and the object list.

S2. The New Difficulty Metric

The appendix replaces KITTI’s camera-dependent difficulty thresholds with depth-based thresholds to improve comparability across datasets. Results using the original metric are less comparable because datasets partition difficulty differently.

  • S2. The New Difficulty Metric: The new metric defines difficulty with truncation, occlusion, and depth range in meters, removing camera influences.The design aims to produce case partitions similar to KITTI’s original metric.
  • S2. The New Difficulty Metric: 30 meters becomes the Easy-to-Moderate/Hard depth threshold, derived from a 27.03-meter depth corresponding to 40 pixels for a 1.53-meter KITTI car.Cars deeper than 70 meters are excluded from Moderate and Hard cases.
  • S2. The New Difficulty Metric: The Moderate partition contains all Easy cases, while the Hard partition contains all Easy and Moderate cases.These nested partitions preserve the cumulative structure of KITTI’s difficulty categories.
  • S2. The New Difficulty Metric: Using the old metric, Easy-case within- and across-domain detection performance drops for all non-KITTI datasets.Many far-away cars in those datasets are classified as Easy under the old pixel-based rule.
  • S2. The New Difficulty Metric: The new metric selects fewer Easy examples than the old metric on every dataset except KITTI.This reflects larger image objects caused by different focal lengths or resolutions in the other datasets.

S3. Dataset discrepancy

Dataset discrepancy arises from both geometric car-size differences and point-cloud style differences. Bounding-box sizes strongly distinguish KITTI, while PointNet++ nearly perfectly classifies car point clouds by dataset.

  • S3. Dataset discrepancy: KITTI has a large bounding-box size gap relative to the other four datasets, and an RBF-kernel SVM distinguishes KITTI with high confidence.The SVM uses car bounding-box height, width, and length as inputs.
  • S3. Dataset discrepancy: PointNet++ classifies car point clouds by dataset using points inside ground-truth car boxes.The experiment samples 8,000 training instances and 1,000 testing instances per dataset.
  • S3. Dataset discrepancy: The point-cloud classifier can almost perfectly classify the datasets’ car point clouds.This indicates that point-cloud styles, including density and laser-beam count, contribute to dataset discrepancy alongside box sizes.
  • S3. Dataset discrepancy: Point resampling in PointRCNN may reduce cross-domain point-cloud differences by equalizing the number of input points.The operation is applied in both the region proposal and object-proposal stages.
  • S3. Dataset discrepancy: The SVM confusion matrix predicts dataset identity from car-object dimensions, whereas the PointNet++ matrix predicts it from points inside car boxes.Together, the analyses compare geometric size cues with point-cloud appearance cues.

S4. PIXOR Results

PIXOR performs well within each dataset but suffers a clear cross-dataset performance drop. Applying PointRCNN-style resampling improves transfer from Waymo to other datasets, likely by reducing its unusually high point count.

  • S4. PIXOR Results: PIXOR achieves strong within-dataset detection performance, suggesting its design does not over-fit specifically to KITTI.PIXOR uses voxelized tensors rather than point clouds as input.
  • S4. PIXOR Results: Cross-dataset testing produces a clear performance drop for PIXOR, often more severe than for PointRCNN.The authors investigate whether point resampling contributes to this difference.
  • S4. PIXOR Results: Resampling improves Waymo-to-other-dataset transfer for PIXOR.Waymo has the highest average LiDAR point count, so resampling makes its inputs more similar to those of other datasets.
  • S4. PIXOR Results: Tuning the resampling point count may further improve PIXOR’s cross-dataset performance.This expectation is stated as a prospective improvement rather than a reported result.

S5. Additional Results Using POINTRCNN

Additional experiments examine statistical normalization across datasets, including external car-sales statistics, and report its effects on cross-dataset performance. The results also indicate that pedestrian-size differences are smaller than car-size differences.

  • Statistical normalization and output transformation yield smaller improvements, or occasional degradation, among datasets collected within the USA than between German and USA datasets.
  • USA cars average (1.75, 1.93, 5.15) meters in height, width, and length, versus (1.49, 1.79, 4.40) meters in Germany.The corresponding difference is (0.26, 0.14, 0.75).
  • Statistical normalization using car-sales data produces notable improvements over directly applying the source detector, although it performs slightly worse than using dataset statistics.The car-sales correction uses (∆h, ∆w, ∆l) = (0.26, 0.15, 0.75).
  • Pedestrian-size differences are smaller across datasets, so the authors expect statistical normalization to yield smaller improvements for pedestrians.

S6. Qualitative Results

Qualitative and supplementary results illustrate how statistical normalization changes cross-dataset 3D detection outputs. The reported comparisons focus on bounding-box size correction and false-positive reduction, using POINTRCNN and related evaluation settings.

  • Figure S11 compares Waymo-trained POINTRCNN predictions on KITTI before and after statistical normalization refinement.Green boxes denote ground truth, while purple and magenta boxes denote pre- and post-refinement predictions.
  • Statistical normalization improves predicted bounding-box sizes and reduces false-positive rates in the qualitative Waymo-to-KITTI comparison.
  • Supplementary cross-dataset tables report Car APBEV/AP3D at IoU = 0.7 using POINTRCNN across ground-truth-size assignment, few-shot fine-tuning, statistical normalization, and output transformation settings.
  • Table S15 compares statistical normalization based on dataset mean sizes with car-sales data, alongside direct source-detector application.
  • Table S16 provides pedestrian dataset statistics in meters for comparison with the car-focused normalization results.
Loading 2005.08139v1…