Source-linked AI summary
LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World
Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, Raquel Urtasun
TL;DR
LiDARsim addresses the need for realistic LiDAR simulation to test self-driving perception and autonomy on unseen, long-tail, and safety-critical scenarios. It builds reusable 3D assets from real fleet data and combines ray casting with learned sensor effects, enabling close agreement with real-world results and end-to-end closed-loop evaluation.
Problem
Self-driving testing needs to cover unseen and safety-critical scenarios, but existing real-world, prerecorded-data, and simulation approaches have limits, especially when simulation bypasses sensory perception.
Method
LiDARsim constructs catalogs of real-data static maps and dynamic objects, composes scenes, raycasts the sensor, and uses a neural network to model deviations such as raydrop.
Results
LiDARsim has a very small domain gap and supports perception testing on unknown objects plus closed-loop evaluation in safety-critical scenarios, with results matching closely with the real world.
Takeaways & Limitations
Realistic LiDAR simulation can support more confident full-autonomy testing and provide insights into system behavior on novel scenarios.
Takeaways & Limitations
Prerecorded real-world data cannot test how the system interacts with the environment when its executed plan differs from what occurred.
Abstract
from arXiv · showhide
We tackle the problem of producing realistic simulations of LiDAR point clouds, the sensor of preference for most self-driving vehicles. We argue that, by leveraging real data, we can simulate the complex world more realistically compared to employing virtual worlds built from CAD/procedural models. Towards this goal, we first build a large catalog of 3D static maps and 3D dynamic objects by driving around several cities with our self-driving fleet. We can then generate scenarios by selecting a scene from our catalog and "virtually" placing the self-driving vehicle (SDV) and a set of dynamic objects from the catalog in plausible locations in the scene. To produce realistic simulations, we develop a novel simulator that captures both the power of physics-based and learning-based simulation. We first utilize ray casting over the 3D scene and then use a deep neural network to produce deviations from the physics-based simulation, producing realistic LiDAR point clouds. We showcase LiDARsim's usefulness for perception algorithms-testing on long-tail events and end-to-end closed-loop evaluation on safety-critical scenarios.
1. Introduction
LiDARsim addresses the difficulty of testing self-driving systems on unknown, long-tail, and safety-critical scenarios by realistically simulating LiDAR sensory input. It combines real-data assets with physics-based and learning-based simulation to support end-to-end evaluation.
- Motivation: Unknown objects and dangerous scenarios remain open challenges for verifying whether self-driving vehicles can detect, handle, and maneuver safely.Examples include a goose on the road and a hidden vehicle entering the lane.
- Limitations of Existing Testing: Structured real-world testing is expensive, time consuming, and limited to few test cases, while prerecorded data cannot cover unseen events or interactive behavior.Safety-critical scenarios are also difficult to test safely and ethically.
- Limitations of Existing Testing: Existing simulation can generate long-tail scenarios and support closed-loop reactions, but often bypasses perception by simulating behavior or trajectories instead of sensory input.This limits full-autonomy testing because the perception module is not exercised.
- LiDARsim: LiDARsim builds real-data 3D assets, composes scenes, and combines ray casting with a neural network to simulate realistic LiDAR point clouds.The approach uses real static maps and dynamic objects, followed by physics-based rendering and learned deviations.
- Results: The simulator has a very small domain gap and supports perception testing on unknown objects and closed-loop evaluation in safety-critical scenarios.The authors report these experiments as evidence of realistic sensory simulation’s value for self-driving.
2. Related Work
Prior simulation systems use virtual environments, physics engines, graphics engines, and generative models, but their simplified scenes and sensing models can leave a substantial real-to-simulation gap. LiDARsim instead brings real driving data into asset construction and uses learning to enhance graphics-generated LiDAR.
- Virtual Environments: Virtual environments and graphics-based simulators support robotics, driving, and virtual sensor simulation, but often use simplified scenes and tasks with limited diversity or noise.These limitations can hinder generalization from simulation to the real world.
- Virtual Environments: Artist-generated environments and simplified physics may omit material reflectivity and incidence-angle effects that influence LiDAR returns.For example, LiDAR rays can penetrate window glass at most incidence angles without producing detectable returns.
- Virtual Label Transfer: Virtual labeled data can provide annotations at scale for tasks such as optical flow and semantic segmentation, supporting transfer from simulation to real-world perception.The related work frames simulated labels as especially useful when real labels are difficult to acquire.
- Point Cloud Generation: Point-cloud generation methods directly apply generative models, whereas LiDARsim uses deep learning to enhance graphics-generated LiDAR data.Its design combines graphics-based simulation with learned realism improvements rather than relying solely on either approach.
- Sensor Simulation in the Real World: Recent simulators increasingly incorporate real-world data, but LiDARsim focuses specifically on realistic LiDAR simulation over driving scenes with complex actors.This distinguishes its scope from related work on camera images, indoor RGBD environments, and off-road terrain.
3. Reconstructing the World for Simulation
LiDARsim reconstructs static environments and dynamic objects from fleet-collected LiDAR data to create reusable, realistic simulation assets. Static maps use aligned multi-pass observations, while object meshes are completed and refined from annotated object scans.
- Overview: The simulator’s objective is to represent complex scenes with many actors and realistic point-cloud geometry using real data rather than CAD or procedural models.The authors construct catalogs of static environments and dynamic objects from several cities.
- Static Map Reconstruction: Static maps are built by collecting repeated drives, removing moving objects, aligning LiDAR sweeps, and aggregating them into a common scene representation.A static scene is created from three passes on average, with Graph-SLAM and multisensor fusion used for alignment.
- Static Map Reconstruction: Aggregated LiDAR points are converted into surfel-based 3D meshes through voxel downsampling and normal estimation.Surfel representations support occlusion reasoning and efficient collision checking; one point is sampled per 4 × 4 × 4 cm^3 voxel.
- Static Map Reconstruction: Each surfel stores geometric and sensing metadata, including intensity, sensor distance, and incidence angle, to support later realism enhancement.The reconstructed map can be visualized by recorded intensity.
- Dynamic Object Reconstruction: Dynamic objects are reconstructed from short annotated LiDAR snippets by accumulating points, mirroring vehicle geometry for completion, and refining alignment errors.The method currently targets rigid objects such as vehicles and is intended to expand to deformable cyclists and pedestrians.
- Dynamic Object Reconstruction: The resulting vehicle bank contains 25,000 vehicles and exhibits substantial geometric and intensity diversity.Examples include modified vehicles, vehicles carrying bikes or equipment, and vehicles with readable intensity patterns.
4. Realistic Simulation for Self-driving
LiDARsim composes real-world static and dynamic assets into traffic scenes, raycasts physics-based point clouds, and uses learning to model raydrop for greater realism.
- Traffic scenarios combine dynamic object meshes with cataloged 3D static environments.
- The physics-based simulator takes sensor location, 3D assets, and traffic scenario as inputs to simulate LiDAR geometry and intensity.
- Raycasting accounts for sensor motion, moving-object motion blur, and nearest surfel intersections during a LiDAR sweep.
- The simulator masks rays hitting the self-driving vehicle after generating the physics-based point cloud.
- Real LiDAR produces approximately 10% fewer points than raycasted output because some rays fail to return when signals are undetectable.
- A raydrop network uses real data, a 64 x 2048 polar grid, and an 8-layer U-Net to predict return probabilities and sample realistic point clouds.
5. Experimental Evaluation
LiDARsim is evaluated against real LiDAR and simulation baselines across perception, data augmentation, and safety-testing tasks. Results show close agreement with real data, improvements over CARLA, and usefulness for unknown-object and safety-critical evaluation.
- Evaluation design: LiDARsim is evaluated through high-fidelity comparisons, perception benchmarks, data augmentation, and edge-case and safety testing.The evaluation includes KITTI and SemanticKITTI comparisons, ablations, mixed real-and-simulated training, and closed-loop scenarios.
- Comparison against existing simulation: LiDARsim achieves performance close to SemanticKITTI and real KITTI data while significantly outperforming CARLA on segmentation and BEV vehicle detection.The comparison uses vehicle and background segmentation on SemanticKITTI and vehicle detection on the KITTI hard validation setting.
- Ablation studies: 2% AP: ML Raydrop improves vehicle detection over raycasting and random raydrop, approaching the oracle ground-truth raydrop performance.The ablation compares no raydrop, ground-truth raydrop, random raydrop, and the proposed learned raydrop network.
- Ablation studies: A 3.6% mAP gap with real-data-based objects compares favorably with a 9% gap for LiDARsim using CAD vehicle models.Both variants use the same LiDARsim pipeline; only the dynamic-object bank is replaced.
- Combining real and LiDARsim data: With around 10% real data, adding simulated data reaches performance similar to 100% real data, with less than 1% mIOU difference.With large-scale training data, simulation provides only marginal additional segmentation gains, while augmentation further boosts detection.
- Safety and edge-case testing: At IoU=0.5, almost 95% of true detections and missed detections match between real and LiDARsim evaluation for a model trained only on real data.This ground-truth detection agreement measures whether real and simulated evaluation produce the same detections and missed detections.
- Safety and edge-case testing: LiDARsim supports testing unseen-object perception and closed-loop safety scenarios, including animal, construction, random-object, and occluded-vehicle cases.OSIS closely matches its real-data behavior qualitatively, while the research-prototype planner succeeds in 90% of 110 generated safety-critical scenarios.
6. Conclusion
LiDARsim combines real-world data, physics, and machine learning to simulate realistic LiDAR sensor data. The authors report close agreement with the real world and use the simulator to evaluate perception and autonomy in novel, safety-critical scenarios.
- Conclusion: LiDARsim combines real-world data, physics, and machine learning to simulate realistic LiDAR sensor data.The conclusion describes this combination as the basis of the simulator.
- Conclusion: Perception algorithms trained on real data can be applied directly to LiDARsim without additional training or domain adaptation.The conclusion frames this capability for evaluation in novel and safety-critical scenarios.
- Conclusion: Results match closely with the real world while providing new insights into the autonomy system.The authors also envision extensions for intensity, weather, end-to-end training, reinforcement learning, and imitation learning.