PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving

Yinpeng Dong, Caixin Kang, Jinlai Zhang, Zijian Zhu, Yikai Wang, Xiao Yang, Hang Su, Xingxing Wei, Jun Zhu

arXiv:2303.11040v1cs.CVcs.AIcs.CR

TL;DR

Existing 3D detectors lack robustness to real-world corruptions, creating concerns for safe autonomous driving. The paper designs 27 LiDAR- and camera-focused corruptions, builds KITTI-C, nuScenes-C, and Waymo-C, and evaluates diverse detectors. Motion-level corruptions cause the greatest degradation, fusion models are more robust, and camera-only models are especially vulnerable to image corruptions.

  • Problem

    Existing 3D detectors generalize poorly to adverse weather, sensor noise, and other corruptions, despite strong clean-benchmark performance, creating safety concerns for autonomous driving.

  • Method

    The paper designs 27 corruption types across five levels, synthesizes them on public datasets to create KITTI-C, nuScenes-C, and Waymo-C, and evaluates diverse detectors.

  • Results

    Motion-level corruptions degrade detector performance most; LiDAR-camera fusion models are more robust, while camera-only models are especially vulnerable to image corruptions.

  • Takeaways & Limitations

    Motion corruptions pose a significant autonomous-driving threat, and reliable 3D detection benefits from LiDAR point clouds or more robust camera-only models.

  • Takeaways & Limitations

    Synthetic corruptions inevitably differ from all real-world corruption combinations, although synthetic weather results are consistent with real adverse-weather data.

Abstract

from arXiv · show

3D object detection is an important task in autonomous driving to perceive the surroundings. Despite the excellent performance, the existing 3D detectors lack the robustness to real-world corruptions caused by adverse weathers, sensor noises, etc., provoking concerns about the safety and reliability of autonomous driving systems. To comprehensively and rigorously benchmark the corruption robustness of 3D detectors, in this paper we design 27 types of common corruptions for both LiDAR and camera inputs considering real-world driving scenarios. By synthesizing these corruptions on public datasets, we establish three corruption robustness benchmarks -- KITTI-C, nuScenes-C, and Waymo-C. Then, we conduct large-scale experiments on 24 diverse 3D object detection models to evaluate their corruption robustness. Based on the evaluation results, we draw several important findings, including: 1) motion-level corruptions are the most threatening ones that lead to significant performance drop of all models; 2) LiDAR-camera fusion models demonstrate better robustness; 3) camera-only models are extremely vulnerable to image corruptions, showing the indispensability of LiDAR point clouds. We release the benchmarks and codes at https://github.com/kkkcx/3D_Corruptions_AD. We hope that our benchmarks and findings can provide insights for future research on developing robust 3D object detection models.

1. Introduction

3D object detection must remain reliable under real-world corruptions because autonomous driving is safety-critical. The paper introduces comprehensive corruption benchmarks and evaluates diverse detectors to identify robustness patterns.

  • Motivation: 3D object detection predicts object categories and 3D bounding boxes using LiDAR point clouds, camera images, or both modalities.LiDAR provides sparse depth information, while cameras provide abundant scene semantics.
  • Motivation: Existing data-driven detectors often generalize poorly to adverse weather, sensor noise, and uncommon objects, while real-world robustness datasets are typically small.Their limited size reflects the high cost and rarity of collecting corner cases and adverse-weather data.
  • Benchmark design: The benchmark designs 27 corruption types for LiDAR and camera sensors, grouped into weather, sensor, motion, object, and alignment levels.Each corruption has five severities, producing 135 distinct corruption settings.
  • Benchmark design: The corruptions are synthesized on KITTI, nuScenes, and Waymo to establish KITTI-C, nuScenes-C, and Waymo-C.These benchmarks target systematic evaluation of corruption robustness in autonomous-driving 3D detection.
  • Findings: Experiments evaluate 24 diverse detectors and find that motion corruptions are most damaging, fusion models are more resistant, and camera-only models are especially vulnerable to image corruptions.Robustness also correlates strongly with clean accuracy, while data augmentation provides little gain.

2. Related Work

Related work covers LiDAR-only, camera-only, and fusion-based 3D detectors, alongside datasets and synthetic-corruption approaches for robustness evaluation. The paper positions its benchmark as a broader autonomous-driving testbed than prior efforts.

  • 3D object detection: LiDAR-only detectors process sparse, irregular, unordered point clouds with voxel-based, point-based, or point-voxel-based representations.Representative methods include VoxelNet, SECOND, PointPillars, PointRCNN, 3DSSD, and PV-RCNN.
  • 3D object detection: Camera-only detection is challenging because images lack depth information, although image-based methods remain attractive because of their lower cost.Approaches include monocular detectors, transformer-based DETR3D, and multi-view BEVFormer.
  • 3D object detection: LiDAR-camera fusion methods exploit complementary modalities through point-level, proposal-level, or unified-representation fusion.Point-level methods such as PointPainting and EPNet augment point clouds with semantic image features.
  • Robustness evaluation: Robustness studies use adverse-weather and road-anomaly datasets, but these collections are constrained by data-collection costs and rare corner cases.Examples include Seeing Through Fog, CADC, Ithaca365, and road-anomaly datasets.
  • Robustness evaluation: Synthetic-corruption benchmarks have been applied to image classification, 2D detection, and point-cloud recognition, yet many corruptions are unrealistic for autonomous driving.The paper identifies a need for a comprehensive 3D detection benchmark with broader evaluation.

3. Corruptions in 3D Object Detection

The paper constructs 27 controllable corruption types across five levels to model diverse autonomous-driving disturbances affecting LiDAR, cameras, or both. It also discusses the remaining gap between synthetic and real-world corruptions.

  • Corruption taxonomy: The benchmark categorizes 27 corruption types into weather, sensor, motion, object, and alignment levels.Some corruptions affect both modalities, while others target a single sensor; Figure 2 visualizes representative examples.
  • Weather-level corruptions: Weather corruptions model disturbances such as fog that reduce image visibility and scatter LiDAR points through attenuation and backscattering.Four weather-level corruptions target performance under weather changes.
  • Sensor-level corruptions: The benchmark includes 10 practical sensor-level corruptions: seven for point clouds and three for images.These simulate effects from sensor vibration, lighting conditions, reflective materials, and sensor noise.
  • Motion-level corruptions: Motion-level corruptions comprise Motion Compensation, Moving Object, and Motion Blur, designed to represent practical driving disturbances.They simulate ego-pose noise, shifting points and blurred object patches from moving objects, and camera blur from driving too fast.
  • Object-level corruptions: Object-level corruptions alter LiDAR object geometry or point quality through local noise, cutout, density changes, shear, scale, and rotation.The taxonomy contains eight object-level corruptions, with the first five applied only to LiDAR point clouds.
  • Alignment-level corruptions: Alignment-level corruptions address violations of the assumed LiDAR-camera calibration through spatial or temporal misalignment.Spatial misalignment can follow sensor vibration, while temporal misalignment can result from stuck or delayed sensor data.
  • Synthetic versus real corruptions: Synthetic corruptions cannot enumerate every real-world combination, but the benchmark offers a practical controllable testbed and synthetic weather performance is consistent with real adverse-weather data.The authors acknowledge that a gap between synthetic and real corruptions inevitably remains.

4. Corruption Robustness Benchmarks

The paper constructs corruption benchmarks from KITTI, nuScenes, and Waymo data, then evaluates diverse detectors using corruption-averaged metrics and relative performance drops.

  • Benchmark construction: Three benchmarks—KITTI-C, nuScenes-C, and Waymo-C—are created by applying synthetic corruptions to validation sets of widely used autonomous-driving datasets.nuScenes-C and Waymo-C include all 27 corruptions at five severities; KITTI-C excludes three corruptions because required information or coverage is unavailable.
  • Evaluation protocol: KITTI-C measures Average Precision across Car, Pedestrian, and Cyclist categories, difficulty levels, corruption types, and five severities.The metric uses 40 recall positions, with IoU thresholds of 0.7 for cars and 0.5 for pedestrians and cyclists.
  • Evaluation protocol: Corruption robustness is summarized by averaging performance across evaluated corruption types and severities, while RCE measures the percentage drop from clean performance.Because corruption sets differ across detector modalities, overall APcor values are not directly comparable between model kinds; the paper therefore performs fine-grained analysis by corruption.
  • Evaluated models: The evaluation covers 11 KITTI models, 10 nuScenes models, and 3 Waymo models spanning LiDAR-only, camera-only, and LiDAR-camera fusion inputs.Models also vary in representation learning methods and detection heads.
  • Dataset-specific metrics: nuScenes-C uses mAP and NDS over 10 categories, whereas Waymo-C uses mAP and mAPH, with corruption robustness and RCE computed analogously.nuScenes mAP uses ground-plane 2D center distance, and NDS combines mAP with scale, orientation, and other aspects.

5. Benchmarking Results

Experiments on KITTI-C and nuScenes-C show that corruption robustness depends on corruption type, sensor modality, and detector design. Motion and weather corruptions are especially damaging, while LiDAR-camera fusion often improves robustness but can suffer when multiple sensors are corrupted.

  • Benchmark setup: 11 detectors are evaluated on KITTI-C, while 10 detectors are evaluated on nuScenes-C using corruption-specific and aggregate robustness results.KITTI-C reports car performance at moderate difficulty; nuScenes-C reports mAPcor averaged over corruption types.
  • Corruption types: Weather- and motion-level corruptions most strongly affect LiDAR-only and fusion models on KITTI-C, while camera-only models suffer significant drops under all corruptions.Snow and Rain exceed 35% RCE for all models; Moving Object and Shear are also challenging.
  • Detector comparisons: LiDAR-only models outperform fusion models when both modalities are corrupted, whereas fusion models are more competitive against point-cloud-only sensor noise.Accurate images help fusion models resist point-cloud noise, but image corruption exposes them to degradation from both inputs.
  • Detector comparisons: SECOND, PointRCNN, and PV-RCNN achieve RCE values of 13.65%, 13.61%, and 13.99%, while 3DSSD shows a 24.34% performance drop.The results do not show a clear robustness margin between voxel- and point-based or one-stage and two-stage detectors.
  • nuScenes-C: On nuScenes-C, motion-level corruptions cause more than 50% performance drops for LiDAR-only models and about 30% for fusion models.FOV Lost and Motion Compensation are the most damaging corruptions, while most models resist weather-level corruptions in this dataset.
  • nuScenes-C: Camera-only models drop more than 40% under RCE, compared with less than 20% for LiDAR-only and fusion models.Multi-view methods outperform monocular methods; FCOS3D and PGD drop 57% and 54%, versus 46% and 45% for DETR3D and BEVFormer.
  • nuScenes-C: Fusion models trade robustness to image corruptions against robustness to point-cloud corruptions because they rely differently on the two modalities.Under Motion Compensation, FUTR3D reaches 31.87% mAP while TransFusion reaches 9.01%; under image noises, their average drops are 12.9% and 2.5%.

6. Discussion and Conclusion

The benchmarks expose consistent robustness patterns across 3D detectors: motion corruptions are most damaging, fusion models are generally more robust, and camera-only models are especially vulnerable. Data augmentation offers little improvement, leaving robustness enhancement open for future work.

  • 24 diverse 3D object detection models were evaluated across KITTI-C, nuScenes-C, and Waymo-C benchmarks.
  • Corruption robustness is generally correlated with clean detection performance.
  • Motion-level corruptions degrade detector performance most, while weather-level corruptions also affect models trained on normal weather.
  • LiDAR-camera fusion models are more robust, especially when corruption affects only one modality, but dual-sensor corruption still degrades performance.
  • Camera-only models are more easily affected by common corruptions, highlighting LiDAR point clouds’ role in reliable 3D detection.
  • Several tested point-cloud and image augmentation strategies can hardly improve corruption robustness, leaving enhancement an open problem.

A.1. Implementation Details

The benchmark implements 27 corruption types across five severity levels for LiDAR and camera inputs. These corruptions cover weather, sensor, motion, object, and alignment effects using dataset-specific synthetic transformations.

  • Each corruption is assigned five severity levels across the KITTI-C, nuScenes-C, and Waymo-C benchmarks.
  • Weather-level corruptions: Weather corruptions include synthetic snow and rain applied separately to LiDAR and camera inputs.
  • Sensor-level corruptions: Sensor corruptions modify LiDAR density, field of view, crosstalk, point coordinates, or camera noise and intensity.
  • Motion-level corruptions: Motion corruptions simulate pose noise, object motion, and motion blur through geometric, temporal, and image transformations.
  • Object-level corruptions: Object corruptions alter points or annotated box geometry through local deletion, noise, shear, scale, and rotation transformations.
  • Alignment-level corruptions: Alignment corruptions perturb LiDAR-camera calibration or retain previous frames to simulate spatial and temporal misalignment.

A.4. Naturalness of Common Corruptions

Synthetic weather corruptions were compared with real-world weather conditions to assess their naturalness. Synthetic and real rain produced largely consistent model performance, while benchmark data quality was generally checked for human detectability.

  • The benchmark uses synthetic corruptions across validation data to compare robustness under controlled conditions.
  • Model performance under synthetic and real rain weather is largely consistent.
  • Objects were kept detectable for humans by adjusting corruption hyperparameters, except for Cutout and FOV Lost.
  • For Cutout and FOV Lost, some objects may disappear from point clouds, and the original evaluation results were retained because alternative comparisons were difficult for fusion models.

B. Additional Results on KITTI-C

Additional KITTI-C experiments extend the robustness analysis across object classes and difficulty levels. The reported trends remain consistent for cars, while performance generally decreases as corruption severity increases.

  • Results for car detection at easy and hard difficulties are highly consistent with those at moderate difficulty.
  • Six detectors were additionally evaluated on pedestrian and cyclist classes at moderate difficulty.
  • For most corruptions, model performance drops as corruption severity increases.

C. Additional Results on nuScenes-C

The nuScenes-C results are also reported under the NDS metric, and the findings are consistent with those obtained using mAP.

  • The nuScenes-C findings are consistent across the mAP and NDS metrics.Model-performance curves across corruption severities are also provided under NDS.

D. Results on Waymo-C

Waymo-C evaluation covers three detectors and supports a limited comparison in which the LiDAR-camera fusion model TransFusion performs better than the other evaluated models.

  • TransFusion demonstrates better corruption robustness than PointPillars and BEVFormer on Waymo-C.The comparison evaluates only three models, limiting the breadth of the conclusion.

E. Data Augmentation as Potential Defense

The paper tests common point-cloud and image augmentations as defenses against corruption, but gains are inconsistent for LiDAR-only models and multimodal augmentation degrades fusion-model performance.

  • Augmentation setup: LiDAR-only experiments apply PA-AUG, Dropout, and PointCutMix-R to point clouds, alongside Mixup and CutMix for images.The study uses SECOND and PV-RCNN for LiDAR-only evaluation and Focals Conv for fusion evaluation.
  • LiDAR-only models: 0.84 corruption-robustness points is the improvement obtained by Dropout for SECOND on KITTI-C.The augmentations do not improve performance consistently across the evaluated LiDAR-only models.
  • LiDAR-only models: More than 4.68 corruption-robustness points is the drop for PV-RCNN under these augmentations.The reported degradation is attributed to reduced clean-data performance.
  • LiDAR-camera fusion models: Combining point-cloud and image augmentations degrades Focals Conv performance substantially on KITTI-C.The paper identifies improving corruption robustness, especially for LiDAR-camera fusion models, as an open problem.
Loading 2303.11040v1…