Source-linked AI summary
Physically Realizable Adversarial Examples for LiDAR Object Detection
James Tu, Mengye Ren, Siva Manivasagam, Ming Liang, Bin Yang, Richard Du, Frank Cheng, Raquel Urtasun
TL;DR
LiDAR-based autonomous driving perception faces adversarial vulnerabilities, while physically realizable attacks on 3D point clouds remain underexplored. The paper learns universal 3D adversarial meshes placed on vehicle rooftops to hide vehicles, evaluates them across detectors and input representations, and studies data-augmentation defenses.
Problem
Adversarial attacks threaten safety-critical autonomous driving, but physically realizable attacks on LiDAR-based 3D perception have received limited exploration.
Method
The paper learns universal adversarial 3D meshes and places them on vehicle rooftops to make target vehicles invisible to LiDAR detectors.
Results
The attack hides vehicles across multiple LiDAR detectors and input representations, while data augmentation substantially reduces but does not eliminate attack success.
Takeaways & Limitations
Universal physical 3D adversarial objects expose vulnerabilities in LiDAR perception, and random-mesh data augmentation can improve detector robustness.
Takeaways & Limitations
The evaluation targets detectors that process point clouds exclusively without auxiliary inputs, and arbitrary point-cloud attacks are not always physically realizable.
Abstract
from arXiv · showhide
Modern autonomous driving systems rely heavily on deep learning models to process point cloud sensory data; meanwhile, deep models have been shown to be susceptible to adversarial attacks with visually imperceptible perturbations. Despite the fact that this poses a security concern for the self-driving industry, there has been very little exploration in terms of 3D perception, as most adversarial attacks have only been applied to 2D flat images. In this paper, we address this issue and present a method to generate universal 3D adversarial objects to fool LiDAR detectors. In particular, we demonstrate that placing an adversarial object on the rooftop of any target vehicle to hide the vehicle entirely from LiDAR detectors with a success rate of 80%. We report attack results on a suite of detectors using various input representation of point clouds. We also conduct a pilot study on adversarial defense using data augmentation. This is one step closer towards safer self-driving under unseen conditions from limited training data.
1. Introduction
The paper investigates physically realizable adversarial attacks against LiDAR-based autonomous driving, extending prior point-cloud attacks beyond arbitrary, potentially unrealizable modifications. It proposes a universal rooftop adversarial object that can hide target vehicles and evaluates attacks and defenses across detectors.
- Deep neural networks process LiDAR point clouds for autonomous-driving perception but remain vulnerable to adversarial attacks, motivating security research in this setting.
- Point-cloud attacks can add, remove, or modify points, but arbitrary point-cloud changes may be impossible for LiDAR sensors to physically generate.Fixed angular frequencies and light-projection geometry constrain realizable sensor observations.
- Prior LiDAR mesh attacks learned objects for a few specific frames, limiting universality and reuse across scenes, and were evaluated on only a few hundred in-house frames.
- The proposed adversary is a universal, physically realizable 3D object placed on vehicle rooftops to hide entire host vehicles from LiDAR detectors.The rooftop setting is presented as usable across scenes and small vehicle types.
- 80% attack success at IoU 0.7 was reported for hiding vehicles, whereas a random rooftop object caused only a small detection-accuracy decrease.
- The study evaluates multiple detector architectures and point-cloud input representations, reports cross-model transferability, and pilots data-augmentation and adversarial-training defenses.
2. Related Work
Related work spans adversarial attacks on images, point clouds, and physical-world inputs. In LiDAR perception, the paper distinguishes its universal, rooftop-based adversary from frame-specific prior mesh attacks.
- Image attacks: Adversarial examples were first studied for image classification and later extended to image detection and semantic segmentation, with white-box and black-box attack settings.
- Point cloud attacks: Point-cloud attacks modify data by adding, removing, or perturbing points, adding adversarial clusters, removing salient points, or using structured physical-fidelity constraints.
- Physical world attacks: Physical-world attack research addresses the fact that image-pixel or point-location perturbations alone may not be realizable outside digital inputs.
- LiDAR attacks: LidarAdv learns adversarial meshes from one frame and is input-specific, whereas this work trains on all frames and vehicles to create a universal adversary.
- LiDAR attacks: The paper additionally places its adversarial object on a vehicle roof to hide the vehicle, unlike the prior adversarial object described in LidarAdv.
3. Physically Realizable Adversarial Examples
The paper learns physically realizable mesh adversaries that attach to vehicle rooftops and suppress LiDAR detections, targeting an “invisible” host vehicle. The method combines differentiable rendering, rooftop fitting, constrained mesh optimization, and black-box search, with data augmentation proposed as a defense.
- 3. Physically Realizable Adversarial Examples: Rooftop cargo creates an “invisible” car by placing an adversarial object on the vehicle and hiding the host from the detector.The placement is intended to be physically plausible and reproducible in real-world scenes.
- 3.1. Surface Parameterization: The adversary is represented as a mesh whose vertices receive learnable local displacements and global transformations under physical box constraints.The mesh representation is chosen for compactness, while constraints limit geometry and translation.
- 3.2. LiDAR Simulation: LiDAR points are rendered from mesh–ray intersections using a differentiable raycaster, while rooftop regions are fitted from vehicle point clouds.The rooftop is approximated by fitting a CAD model and selecting vertices within the top 0.2m.
- 3.3.1 Objective: The objective minimizes confidence for relevant detection proposals while adding a Laplacian regularizer to preserve mesh smoothness.Relevant proposals exceed both confidence and IoU thresholds, and negative labels are targeted because missed detections are most problematic.
- 3.3.2 Attack Algorithms: White-box attacks optimize mesh parameters through gradients, whereas black-box attacks use a genetic algorithm when preprocessing blocks gradient flow.The white-box procedure re-parameterizes vertices and transformations so box constraints are obeyed inherently.
- Defense: Random data augmentation and adversarial training are considered defenses because rooftop cargo is rare in training data and the attacks are heavily out-of-distribution.The paper frames augmentation as a simple defense and adversarial training as a stronger alternative.
4. Experiments
Experiments evaluate attacks across datasets, detector architectures, point-cloud representations, transfer settings, and defense training. Results show strong attacks on several detectors, representation-dependent robustness, and significant defense benefits.
- Datasets and models: Experiments use KITTI Car samples and evaluate LiDAR-only detectors with occupancy voxels, density voxels, raw points, or pillars.The evaluated models are PIXOR, PIXOR (density), PointRCNN, and PointPillar.
- Experimental setup: Attack quality is measured by attack success rate at IoU 0.7 and recall across a range of IoU thresholds.A successful attack hides a vehicle that was originally detected with output IoU greater than 0.7.
- White-box and black-box attacks: White-box and black-box attacks significantly reduce recall, while the initial icosphere has little detection impact and both attack types achieve similar performance.These comparisons use PIXOR with density voxels.
- Transferability and detector robustness: Transferability is high between PIXOR models with identical architectures, while attack strength varies substantially across detector architectures and input representations.The attack is strong on PIXOR variants and PointPillar but significantly weaker on PointRCNN.
- Transferability and detector robustness: PointRCNN attack success is close to 0 near the sensor but increases with distance as fewer LiDAR points are available.The authors hypothesize that PointRCNN’s point-level bounding-box anchors make nearby vehicles harder to suppress.
- Input representations: PIXOR variants have different vulnerable regions, and PIXOR* is significantly more robust than PIXOR despite identical backbones and training routines.The comparison attributes this difference to input representation, contrasting density or occupancy voxelization with pillar representation.
- Adversarial defense: Defense training significantly reduces attack success rates, while standard KITTI validation performance is similar or better; residual attack success remains 5-20%.The defense evaluation retrains an adversarial mesh on the defended model and reports remaining vulnerability.
5. Conclusion
The paper demonstrates that universal, physically realizable 3D adversarial objects can hide vehicles from LiDAR detectors, while data augmentation improves but does not eliminate vulnerability.
- 3D-printed meshes placed on vehicle roofs can make target vehicles “invisible” to LiDAR detectors without prior scene knowledge.
- Training with data augmentation using random meshes significantly improves robustness, but the attack remains incompletely defeated.