Source-linked AI summary
Robo3D: Towards Robust and Reliable 3D Perception against Corruptions
Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, Ziwei Liu
TL;DR
Robo3D addresses the limited evidence about how 3D perception models withstand natural corruptions and sensor failures in deployment-like settings. It introduces a corruption benchmark and robustness-oriented training strategies, finding vulnerability patterns tied to sensor setups and representations while proposing methods to improve resilience.
Problem
Robustness of 3D perception models under real-world corruptions and sensor failures remains obscure despite strong performance on clean benchmarks.
Method
Robo3D benchmarks LiDAR-based detectors and segmentors using eight corruption types, while combining density-insensitive training with flexible voxelization strategies.
Results
Models show vulnerability to corruptions, with resilience varying by sensor setup and data representation; voxel and point-voxel fusion outperform projection-based methods such as range view.
Takeaways & Limitations
Robust 3D perception requires evaluating both detection and segmentation across realistic corruptions and considering sensor, representation, augmentation, and training choices.
Takeaways & Limitations
The proposed out-of-context consistency regularization is conjectured to relieve overfitting and improve robustness against unseen scenarios, rather than established as a guaranteed effect.
Abstract
from arXiv · showhide
The robustness of 3D perception systems under natural corruptions from environments and sensors is pivotal for safety-critical applications. Existing large-scale 3D perception datasets often contain data that are meticulously cleaned. Such configurations, however, cannot reflect the reliability of perception models during the deployment stage. In this work, we present Robo3D, the first comprehensive benchmark heading toward probing the robustness of 3D detectors and segmentors under out-of-distribution scenarios against natural corruptions that occur in real-world environments. Specifically, we consider eight corruption types stemming from severe weather conditions, external disturbances, and internal sensor failure. We uncover that, although promising results have been progressively achieved on standard benchmarks, state-of-the-art 3D perception models are at risk of being vulnerable to corruptions. We draw key observations on the use of data representations, augmentation schemes, and training strategies, that could severely affect the model's performance. To pursue better robustness, we propose a density-insensitive training framework along with a simple flexible voxelization strategy to enhance the model resiliency. We hope our benchmark and approach could inspire future research in designing more robust and reliable 3D perception models. Our robustness benchmark suite is publicly available.
1. Introduction
Robo3D addresses the obscured robustness of LiDAR-based 3D perception under real-world corruptions and sensor failures by benchmarking models across diverse deployment-like conditions. The study identifies sensitivity to sensor setups and data representations, then proposes benchmark-driven strategies for improving resilience.
- Motivation: Clean benchmark performance does not establish robustness, because real-world data collection, processing, weather, and scene complexity introduce inevitable corruptions.The study focuses on performance deterioration under real-world corruption and sensor failure rather than domain transfer or adversarial worst-case inputs.
- Benchmark: Robo3D simulates physically principled corruptions on four large-scale driving datasets, using three severity levels and robustness-specific metrics.The suite covers KITTI, SemanticKITTI, nuScenes, and Waymo Open datasets.
- Findings: Sensor configurations affect feature learning, causing models trained with different sensor setups and protocols to show inconsistent corruption resilience.This indicates that robustness depends partly on the sensing configuration used during training.
- Findings: Voxel and point-voxel fusion representations show clear superiority over projection-based methods such as range view under corruption.The comparison links robustness differences to the choice of 3D data representation.
- Approach: Out-of-context augmentation and flexible rasterization can improve robustness, motivating density-insensitive training and flexible voxelization strategies.These techniques are proposed as a solution for enhancing the robustness of existing 3D perception models.
- Benchmark: Robo3D benchmarks 34 LiDAR-based perception models across semantic segmentation and 3D object detection under corruptions.The benchmark is designed to evaluate both detectors and segmentors rather than a single task.
2. Related Work
Related work covers LiDAR semantic segmentation and 3D object detection through multiple point-cloud representations. Despite these advances, corruption robustness remains insufficiently characterized.
- LiDAR-based Semantic Segmentation: LiDAR semantic segmentation methods use raw-point, range-view, bird’s-eye-view, voxel, and multi-view fusion representations.Projection-based methods rasterize irregular point clouds into 2D grids and can be more hardware-friendly for deployment.
- LiDAR-based 3D Object Detection: LiDAR 3D detectors include point-based, voxel-based, pillar-based, and point-voxel fusion methods with different geometry, resolution, and feature-integration choices.Voxel methods use sparse convolutions on non-empty voxels, while point-voxel fusion combines representation advantages.
- Open Problem: Robustness of 3D segmentors against corruptions remains obscure, partly because comprehensive robustness evaluation benchmarks have been lacking.The related-work discussion frames corruption robustness as distinct from the representation advances themselves.
3. The Robo3D Benchmark
Robo3D constructs a physically motivated corruption benchmark for LiDAR-based 3D perception, spanning eight corruption types, three severity levels, and multiple existing datasets. It evaluates robustness with corruption error and resilience-rate metrics across detection and segmentation models.
- Corruption types: Robo3D models eight real-world corruption types, including weather effects, motion disturbances, sensor interference, incomplete echoes, and cross-sensor failures.Corruptions are implemented as point-cloud mappings constrained by physical principles or engineering experience.
- Corruption types: The corruption simulations cover physically grounded effects such as attenuation, backscattering, wet-ground echo loss, snow interference, motion offsets, beam dropping, crosstalk, and incomplete echoes.For incomplete echoes, selected points from vehicle, bicycle, and motorcycle classes are removed while ground-truth bounding boxes remain unchanged.
- Corruption sets: The benchmark creates SemanticKITTI-C, KITTI-C, nuScenes-C, and WOD-C from existing validation sets, with three severity levels for each corruption.The resulting collections contain 97,704, 90,456, 144,456, and 143,424 annotated LiDAR point clouds, respectively.
- Benchmark analysis: Figure 2 benchmarks 34 LiDAR-based detection and segmentation models across six robustness sets using task accuracy, mean corruption error, mean resilience rate, and corruption-type sensitivity.The figure organizes these comparisons into three rows, from accuracy versus mCE to mRR and sensitivity analysis.
- Evaluation metrics: Mean corruption error normalizes corrupted-task accuracy against CenterPoint for detection and MinkUNet for segmentation, while mean resilience rate measures retained accuracy relative to clean evaluation.The task-specific accuracy can be mIoU, AP, NDS, or APH(L2), depending on the task and metric.
4. Experimental Analysis
Robo3D evaluates 34 LiDAR-based detection and segmentation models under diverse corruptions, revealing task-, representation-, and configuration-dependent robustness. The analysis finds that corruption-aware augmentation and voxelization choices can substantially affect resilience.
- Benchmark Configuration: 34 LiDAR-based detection and segmentation models are benchmarked using public checkpoints or retraining under conventional settings for fair corruption comparisons.The benchmark includes detectors, segmentors, and recent 3D augmentation methods.
- Benchmark Analysis: Existing 3D detectors and segmentors remain vulnerable to real-world corruptions despite improving performance on clean benchmarks.Recent methods can achieve higher clean-task accuracy while showing worse corruption error than baselines.
- Representation Effects: Voxel and point-voxel fusion representations show clear robustness advantages over projection-based methods such as range view.Voxelization retains 3D structure and can mitigate local point variations during feature learning.
- Task Particularity: Detectors and segmentors respond differently to corruptions: detectors are more robust to motion blur and crosstalk, whereas segmentors are steadier under fog, wet ground, and snow.The difference reflects object-level detection versus point-level semantic labeling.
- Augmentation Effects: Out-of-context augmentations improve robustness across most corruptions, but wet-ground corruption limits scene-mixing effectiveness because ground points are lost.Common in-context augmentations improve clean accuracy but are less effective for robustness.
- Voxelization Effects: Voxel-size preferences vary by corruption for segmentation, while detectors consistently benefit from higher voxelization resolution.Segmentors favor larger voxels for motion blur but finer voxelization for fog, snow, and crosstalk.
5. Boosting Corruption Robustness
The paper improves corruption robustness through flexible voxelization and density-insensitive training, using cross-density consistency between full and partial point-cloud views. These changes reduce corruption error, though they slightly lower clean-set task accuracy.
- Flexible Voxelization: Flexible voxelization replaces fixed voxel sizes with dynamically perturbed dimensions to improve generalizability across corruption types.Different corruptions favor different voxel sizes, motivating dynamic offsets sampled from a continuous uniform distribution.
- Density-Insensitive Training: Density-insensitive training randomly masks points to create high- and low-density views without simulating a specific corruption pattern.The framework uses paired full and partial inputs because real corruption patterns are difficult to predict.
- Density-Insensitive Training: Cross-consistency learning combines completion and confirmation losses to align predictions between dense and sparse views.Completion mimics dense representations from sparse inputs, while confirmation pursues local agreements.
- Experimental Analysis: 2.6% mCE and 1.5% mCE reductions result from the voxelization modification on the two baselines, respectively.The variant using both completion and confirmation achieves the best mCE and mRR results among the evaluated variants.
- Experimental Analysis: A masking ratio between 0.3 and 0.6 tends to yield lower mCE, while both proposed techniques slightly reduce clean-set task accuracy.The paper identifies a trade-off between robustness and the proportion of occluded information.
6. Discussion and Conclusion
Robo3D benchmarks LiDAR-based 3D perception robustness across eight corruption types and three severity levels on four driving datasets. The study analyzes model resilience and proposes cross-density consistency training with flexible voxelization, while noting scope limits involving combined corruptions and multimodal models.
- Benchmark: Robo3D defines eight corruption types with three severity levels across four large-scale autonomous-driving datasets.It benchmarks representative 3D detectors and segmentors under real-world corruptions and sensor failure.
- Analysis: The study analyzes robustness across sensor setups, data representations, task particularity, and augmentation effects.These analyses provide the basis for the proposed robustness techniques.
- Methods: Cross-density consistency training and flexible voxelization are proposed to improve robustness against corruptions.The paper presents these methods as a foundation for future robust and reliable 3D perception research.
- Limitations: The benchmark excludes scenarios with multiple coupled corruptions and models using multimodal inputs.The authors identify both omissions as potential future directions.
Appendix
The appendix supplies supporting material for Robo3D, covering corruption case studies, generation details, complete benchmark analyses, and qualitative results.
- Appendix Contents: Section 7 provides a case study for each of Robo3D’s eight corruption types.
- Appendix Contents: Section 8 details implementation procedures for generating each corruption type.
- Appendix Contents: Section 9 reports additional complete results and discussions for the benchmarked 3D detectors and segmentors.
- Appendix Contents: Section 10 presents qualitative results for benchmarked methods under each corruption type.
- Appendix Contents: Section 11 acknowledges the public resources used in the work.
7. Case Study: 3D Natural Corruption
Robo3D models eight physically motivated LiDAR corruptions arising from weather, motion, and sensor effects. The case studies show how each corruption alters point measurements through attenuation, reflection, missing beams, jitter, or false returns.
- General Corruption Model: Robo3D represents each LiDAR point with coordinates and intensity, then maps it to a corrupted point using physically constrained rules.The corruption mapping is governed by physical principles or engineering experience.
- Weather Corruptions: Fog causes back-scattering and attenuation that shift LiDAR point ranges and intensities.The simulation computes hard and soft responses using attenuation and backscattering coefficients before selecting the corrupted measurement.
- Weather Corruptions: Wet ground attenuates laser echoes according to water height and mirror refraction, retaining ground points only when measured intensity exceeds the noise floor.The method estimates the ground plane before applying the wetness-dependent intensity mapping.
- Weather Corruptions: Snow particles intersect laser beams and modify measurements according to induced geometry and snowfall rate.The simulation samples snow particles in 2D while accounting for potential occlusions.
- Motion Corruption: Motion blur is simulated by adding Gaussian-distributed coordinate jitter to each LiDAR point.The intensity remains unchanged while offsets modify the x, y, and z coordinates.
7.5. Beam Missing
The benchmark models beam missing, crosstalk, incomplete echoes, and cross-sensor effects as point-cloud corruptions arising from occlusion, interference, sensor sensitivity, and differing LiDAR configurations.
- Beam Missing: Beam missing drops points from randomly selected beams to represent zero readings caused by dust or insects occluding LiDAR impulses.The corruption Cbeam(p) retains points outside the selected beams and returns None for points on them.
- Crosstalk: Crosstalk adds large Gaussian jitter to a sampled subset of points, modeling interference between nearby LiDAR sensors that creates noisy mid-range points.The offset ξc is sampled from a Gaussian distribution with standard deviation σc.
- Incomplete Echo: Incomplete echo removes queried points from vehicle, bicycle, and motorcycle classes to simulate dark surfaces causing significant LiDAR point miss detection.Ground-truth bounding boxes remain unchanged because they represent the objects’ original real-world positions.
- Cross-Sensor Effect: Cross-sensor corruption drops selected beams and subsamples points to model distribution discrepancies across LiDAR devices with different beam counts, fields of view, and sampling frequencies.The benchmark generates this corruption by dropping points from certain beams before subsampling remaining points.
8. Additional Implementation Detail
The benchmark provides reproducible parameter settings for weather, motion, beam, crosstalk, incomplete-echo, and cross-sensor corruptions across its datasets.
- Reproducibility: Corruption generation procedures are physically principled and can be applied to other LiDAR point-cloud datasets with minimal modifications.The section documents implementation details intended to enable reproduction.
- Weather Corruptions: Fog, wet ground, and snow severity levels are controlled through dataset-specific attenuation, back-scattering, water-height, snowfall-rate, and ground-plane settings.Fog uses attenuation and back-scattering coefficients, while wet ground and snow use water height and snowfall rate parameters.
- Sensor Corruptions: Motion blur, beam missing, crosstalk, incomplete echo, and cross-sensor corruptions use dataset-specific parameter values for noise, dropped beams, sampled points, and subsampling.The settings include σt, m, kt, ke, and cross-sensor beam and point sampling parameters.
9. Additional Experimental Result
The supplementary experiments report complete corruption-error, resilience-rate, and task-specific accuracy results across six benchmark settings, alongside realizations and analyses of density-insensitive training.
- Benchmark Results: Complete CE, RR, and task-specific accuracy results are reported for SemanticKITTI-C, KITTI-C, nuScenes-C, and WOD-C segmentation and detection settings.The task metrics include IoU, AP, NDS, and APH depending on the dataset and task.
- Density-Insensitive Training: The density-insensitive training framework has separate detection and segmentation realizations using full and partial point clouds in teacher and student branches.The detection and segmentation realizations are illustrated in Figures 16 and 17.
- Density-Insensitive Training: Detection computes completion and confirmation losses on BEV feature maps, whereas segmentation computes them at the logits level.The authors report improvements across a wide range of corruption types and verify effectiveness on both tasks in Table 26.
10. Qualitative Experiment
The qualitative experiments visualize Robo3D’s eight corruption types at three severity levels and compare representative detection and segmentation models.
- Corruption Visualizations: The benchmark visualizes all eight corruption types under light, moderate, and heavy severity levels.These visualizations appear in Figures 19 and 20.
- Model Comparisons: Detection comparisons use SECOND and CenterPoint on WOD-C, while segmentation comparisons use six segmentors on SemanticKITTI-C.The segmentation models are RangeNet++, PolarNet, Cylinder3D, RPVNet, SPVCNN, and WaffleIron.
- Video Demonstrations: Four project-page video demos provide evaluations across hundreds of frames for a more comprehensive benchmark view.The demos supplement the figures shown in the file.
11. Public Resources Used
The work acknowledges public datasets, APIs, models, libraries, and simulation resources used during development, with their associated licenses recorded.
- Datasets: Public dataset resources include SemanticKITTI, nuScenes, and the Waymo Open Dataset, each listed with its applicable license.
- Models and libraries: The resource list includes perception models and implementations such as CENet10, KPConv-PyTorch, PIDS, WaffleIron, PolarSeg, and MinkowskiEngine.
- Models and libraries: The authors also acknowledge OpenPCDet, PointRCNN, PartA2-Net, SemanticKITTI-API, PV-RCNN, CenterPoint, and lidar-camera-robust-benchmark.
- Simulation and supporting resources: Simulation and supporting resources include LiDAR-fog-sim, LiDAR-snow-sim, mmdetection3d, LaserMix, nuScenes-devkit, RangeNet++, SalsaNext, and FIDNet.