Augmented LiDAR Simulator for Autonomous Driving

Jin Fang; Dingfu Zhou; Feilong Yan; Tongtong Zhao; Feihu Zhang; Yu Ma; Liang Wang; Ruigang Yang

Source-linked AI summary

Augmented LiDAR Simulator for Autonomous Driving

Jin Fang, Dingfu Zhou, Feilong Yan, Tongtong Zhao, Feihu Zhang, Yu Ma, Liang Wang, Ruigang Yang

arXiv:1811.07112v2Nov 17, 2018cs.CV

TL;DR

LiDAR-based autonomous-driving perception needs large annotated datasets, but labeling real 3D point clouds is costly and difficult. The paper proposes a scan-and-simulate framework that combines scanned backgrounds with data-driven synthetic obstacles. Detectors trained on simulated data perform within two percentage points of those trained on real data, while mixing real and simulated data can exceed 95% accuracy.

Problem
LiDAR point clouds are difficult and expensive to annotate at the scale required for deep-learning-based autonomous-driving perception.
Method
The framework scans real environments for backgrounds and uses learned obstacle distributions to synthesize annotated point clouds without manual background modeling.
Results
Performance with simulated data alone is comparable to real-data training within two percentage points, while mixing real and simulated data can exceed 95% accuracy.
Takeaways & Limitations
The simulator provides realistic, flexible, and scalable annotated LiDAR data for autonomous-driving perception training.
Takeaways & Limitations
The current system uses high-quality dense hardware, and simulating foreground LiDAR intensity would require modeling near-infrared material reflectance and updating foreground CAD textures.

Abstract

from arXiv · show

In Autonomous Driving (AD), detection and tracking of obstacles on the roads is a critical task. Deep-learning based methods using annotated LiDAR data have been the most widely adopted approach for this. Unfortunately, annotating 3D point cloud is a very challenging, time- and money-consuming task. In this paper, we propose a novel LiDAR simulator that augments real point cloud with synthetic obstacles (e.g., cars, pedestrians, and other movable objects). Unlike previous simulators that entirely rely on CG models and game engines, our augmented simulator bypasses the requirement to create high-fidelity background CAD models. Instead, we can simply deploy a vehicle with a LiDAR scanner to sweep the street of interests to obtain the background point cloud, based on which annotated point cloud can be automatically generated. This unique "scan-and-simulate" capability makes our approach scalable and practical, ready for large-scale industrial applications. In this paper, we describe our simulator in detail, in particular the placement of obstacles that is critical for performance enhancement. We show that detectors with our simulated LiDAR point cloud alone can perform comparably (within two percentage points) with these trained with real data. Mixing real and simulated data can achieve over 95% accuracy.

1. Introduction

The paper addresses the cost and difficulty of labeling LiDAR point clouds by introducing a hybrid simulator that combines scanned real backgrounds with data-driven synthetic obstacles. Its simulated data achieve comparable perception performance to real data, while mixing both sources can outperform real-only training.

Motivation: 3D LiDAR labeling is difficult and expensive because real-time scanners produce sparse point clouds and autonomous-driving systems require large labeled datasets.
Motivation: Traditional computer-graphics simulators remain limited by manually crafted environments, insufficient scale, and background-creation costs exceeding 10K USD per kilometer.
Contribution: The simulator combines scanned real-world backgrounds with learned obstacle poses, types, and shapes to generate realistic annotated LiDAR data.This avoids manual background modeling while retaining flexibility in traffic-pattern regeneration and sensor-parameter changes.
Results: Synthetic-only training achieves competitive performance for 3D obstacle detection and semantic segmentation, while combining real and simulated data can outperform real-only training.

2. Related Work

Prior simulation work largely uses virtual worlds and computer-graphics assets, whereas this paper focuses on LiDAR augmentation with controllable sensor parameters. The proposed framework combines scanned backgrounds with synthetic obstacles and learned placement strategies.

Existing work: Most prior graphics-based approaches require synthetic urban scenes with computer-generated foreground and background models built in advance.
Existing work: Existing LiDAR simulators such as CARLA and AutonoVi-Sim generate point data from virtual worlds, primarily to support learning and control experiments.
Novelty: The paper positions its method as the first data-augmentation approach focused on LiDAR point clouds rather than primarily synthetic images.
Proposed approach: The proposed method combines real-world scanned backgrounds with learned obstacle placement and allows LiDAR placement and line-count parameters to change arbitrarily.

3. Methodology

The methodology simulates LiDAR acquisition in real traffic environments through background construction, synthetic obstacle generation and placement, point-cloud rendering, and verification.

3. Methodology: The framework models LiDAR data acquisition through four modules: static-background construction, movable-obstacle generation and placement, point-cloud simulation, and final verification.

3.1. Static Background Generation

Static backgrounds are captured with a professional mobile LiDAR scanner to preserve dense, detailed real-world geometry. The scanned scenes are cleaned of movable obstacles and processed into simulation-ready backgrounds.

3.1. Static Background Generation: A RIEGL VMX-1HA scanner provides dense, accurate, feature-rich static backgrounds instead of generating them from artificial virtual worlds.
3.1. Static Background Generation: Repeated scans can increase point-cloud resolution to about 1 cm, enabling simulation of LiDAR types with point distances larger than 1 cm.
3.1. Static Background Generation: Dynamic and static movable obstacles are removed from the scanned background using initial semantic segmentation followed by manual correction.
3.1. Static Background Generation: The scanner captured more than 200 million points over an approximately 600 m × 270 m area, illustrating the scale and detail of the background data.

3.2. Movable Obstacle Generation

The simulator generates movable obstacles using real-scene distributions rather than random placement, while selecting diverse CAD models according to category frequency. A probability map encodes likely obstacle positions and directions for weighted pose sampling.

3.2. Movable Obstacle Generation: Obstacle pose generation uses real traffic distributions because placement strongly influences detection and segmentation results.The method replaces random placement with a data-driven strategy based on labeled real scenes.
Probability Map for Obstacle Placement: The Probability Map spreads observed obstacle evidence with a Gaussian kernel and samples positions and directions by weighted random sampling.Grid weights and direction values are accumulated around observed obstacles before sampling for a scanner pose.
Model Selection: Obstacle categories are selected using prior occurrence frequencies learned from labeled data.This makes simulated category frequencies reflect the real dataset rather than an arbitrary selection rule.
Model Selection: The model library combines high-frequency CAD models for common cases with low-frequency models to preserve diversity.The paper also includes uncommon categories such as traffic cones, baby carriages, and tricycles.

3.3. Sensor Simulation

The sensor simulator models LiDAR returns physically and renders hybrid point-and-mesh scenes through cube maps. It then traces sensor beams through these maps to generate points and obstacle annotations.

3.3.1. Model Design: LiDAR simulation models returned pulse energy using surface reflectivity, incident angle, atmospheric attenuation, and target distance.The formulation defines returned and emitted pulse energy together with reflectivity, angular reflection, attenuation, and distance terms.
3.3.1. Model Design: A simulated HDL-64E emits 64 beams across vertical angles and rotates to provide 360° scene coverage.The beam geometry follows the Velodyne HDL-64E S3 configuration described in the paper.
3.3.2. Point Cloud Rendering: The framework projects the surrounding scene onto six cube faces centered at the LiDAR origin to create depth, normal, and material maps.Environment points are rendered with surface splatting while obstacle models use regular rendering in a shared pipeline.
3.3.2. Point Cloud Rendering: For each beam, the simulator finds a target distance, normal, and material before generating a point, discarding weak returns or sky hits.The resulting obstacle points receive tight oriented bounding boxes by adjusting the original CAD boxes.

4. Experimental Results and Analysis

The experiments evaluate simulation quality indirectly through downstream DNN performance, using models trained on simulated point clouds and comparisons involving real KITTI data. All simulations and experiments use parameters based on the Velodyne HDL-64E.

4. Experimental Results and Analysis: The evaluation compares DNN performance trained with different simulated point clouds rather than comparing simulator systems directly.The study evaluates both public and self-collected point-cloud datasets.
4. Experimental Results and Analysis: All model parameters and subsequent experiments are based on the Velodyne HDL-64E sensor.This choice follows the sensor’s popularity in autonomous driving.
4. Experimental Results and Analysis: Table 2 compares CARLA, Proposed, Real KITTI, and simulation-plus-real training conditions on the KITTI benchmark.The combined conditions train on simulation data first and then fine-tune on KITTI training data.

4.1. Evaluation on Public Dataset

The public-dataset evaluation compares the proposed simulator with CARLA and real data across instance segmentation and 3D object detection. Simulation helps substantially, but purely simulated training has limited generalization without real-data adaptation.

Evaluation setup: The evaluation uses KITTI-derived training and validation data, with AP-50 and AP-70 for detection and mean bounding-box/mask AP for instance segmentation.The study evaluates instance segmentation and 3D object detection using established task-specific metrics.
Experimental results: More than 20 points of improvement in mean AP and Mask AP over CARLA were obtained for instance segmentation, while object detection AP also improved by a large margin.The comparison uses models trained purely on simulation data from the proposed method versus CARLA.
Experimental results: About 5 points of improvement were achieved in segmentation mean AP and detection AP 70 using the proposed simulated dataset, whereas CARLA data produced only slight gains over real-data training.The gains are reported when simulation data supports training relative to models trained only on real data.
Analysis: Simulation-only training can reach only a 29.28% detection rate for 3D object detection, indicating a substantial generalization gap to real application data.The analysis attributes this drawback to the difference between the simulation training domain and the testing data.

4.2. Simulation for Real Application

On a self-collected dataset, the simulator combines scanned backgrounds with generated obstacles and data-driven placement to produce competitive annotated training data. Ablations show that scanned backgrounds, obstacle-pose modeling, and random point dropout materially affect performance.

Results on Self-collected Dataset: A 100K-frame simulation model achieved a comparable result to a 16K-real-data model with only a 2-point gap, while mixing 1.6K real frames raised mean AP to 94.10 and surpassed 16K real data.The self-collected dataset contains 100K training and 20K testing frames with full 360° labels.
Results on Self-collected Dataset: Combining 16K real data with 100K simulation data beat 100K real data and saved more than 80% of annotation cost.Adding simulation data also improved models that already had a large labeled real-data set.
Ablation Studies: Scanned backgrounds performed close to real backgrounds and improved mean bounding-box and mask AP by about 10 points over synthetic backgrounds such as CARLA.Table 4 compares background choices for instance segmentation.
Ablation Studies: The probability-map obstacle-placement method with pose augmentation achieved results comparable to manually labeled real poses while reducing manual labor.Five placement strategies were evaluated, including random, rule-based, probability-map, augmented, and manually labeled poses.
Ablation Studies: Random point dropout steadily improved mean Mask AP by 2 points, addressing the simulated frame’s higher point count relative to real HDL-64E scans.Real frames contain about 102,000 points versus about 117,000 in simulated frames.
Sensor Generalization: The framework supports different LiDAR configurations through parameters such as channel count, range, field of view, and angular resolution, including VLS-128.With the same LiDAR location and obstacle poses, the VLS-128 simulated and real point clouds were visually indistinguishable.

5. Conclusions and Future Works

The paper concludes that scan-based augmented LiDAR simulation can generate realistic, scalable annotated data without manual labeling. It identifies low-fidelity LiDAR and foreground intensity modeling as future directions.

Conclusions: The performance gap between detectors trained with real or simulated data was within two percentage points, supporting the simulator’s reported realism.The system uses scanned backgrounds, statistically realistic obstacle poses and types, and a general LiDAR renderer.
Conclusions: Scanning a scene can rapidly produce abundant labeled data by combining backgrounds and obstacle placements without manual labeling, aside from computation costs.The conclusion presents scalability as a design property of the framework.
Future Works: Future work includes low-fidelity LiDAR simulation and modeling foreground-object intensity through near-infrared material reflectance.Low-fidelity hardware may trade simulation versatility against cost, while intensity modeling requires additional NIR textures.