Source-linked AI summary
A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving
Xiangyu Yue, Bichen Wu, Sanjit A. Seshia, Kurt Keutzer, Alberto L. Sangiovanni-Vincentelli
TL;DR
The paper tackles the manual effort required to create large, point-level-labeled LiDAR datasets. It presents a game-based framework for generating annotated point clouds, paired images, calibrated sensor data, and configurable test scenes; experiments report improved segmentation and repaired model blind spots.
Problem
Creating large 3D LiDAR point-cloud datasets with point-level labels requires substantial manual annotation, while neural-network verification needs broader scene variation.
Method
A video-game framework automatically extracts labeled LiDAR point clouds and corresponding images, calibrates them, and supports user-configured scenarios for network testing.
Results
+9% validation accuracy (IoU) was reported for point-cloud segmentation with synthesized data, and configurable-scene testing identified weaknesses that were fixed through retraining.
Takeaways & Limitations
The framework supports both training-data augmentation and systematic testing, analysis, and improvement of LiDAR-based neural networks.
Abstract
from arXiv · showhide
3D LiDAR scanners are playing an increasingly important role in autonomous driving as they can generate depth information of the environment. However, creating large 3D LiDAR point cloud datasets with point-level labels requires a significant amount of manual annotation. This jeopardizes the efficient development of supervised deep learning algorithms which are often data-hungry. We present a framework to rapidly create point clouds with accurate point-level labels from a computer game. The framework supports data collection from both auto-driving scenes and user-configured scenes. Point clouds from auto-driving scenes can be used as training data for deep learning algorithms, while point clouds from user-configured scenes can be used to systematically test the vulnerability of a neural network, and use the falsifying examples to make the neural network more robust through retraining. In addition, the scene images can be captured simultaneously in order for sensor fusion tasks, with a method proposed to do automatic calibration between the point clouds and captured scene images. We show a significant improvement in accuracy (+9%) in point cloud segmentation by augmenting the training dataset with the generated synthesized data. Our experiments also show by testing and retraining the network using point clouds from user-configured scenes, the weakness/blind spots of the neural network can be fixed.
I. INTRODUCTION
The paper addresses the costly creation and limited diversity of labeled LiDAR data by proposing a game-based framework for generating annotated point clouds, paired images, and configurable test scenes.
- LiDAR provides direct distance measurements and robust environmental sensing, but autonomous-driving perception also requires semantic understanding.
- Existing interactive and game-based annotation approaches still require substantial human effort or primarily target image-based ground truth.
- The proposed framework targets systematic testing of LiDAR neural networks, addressing the limited modification space and manual background annotation in prior verification work.
- It automatically extracts labeled point clouds with corresponding scene images, supports calibration for sensor fusion, and enables user-configured scenarios.
- +9% validation accuracy (IoU) was achieved for point-cloud segmentation using synthesized data, while configurable scenes exposed weaknesses for retraining.
A. In-Game Simulation Setup and Method for Data Collection
The framework uses GTA-V and DeepGTAV to simulate an autonomously driven ego vehicle with a virtual LiDAR scanner, collecting point clouds and images alongside automatic labels.
- GTA-V supplies simulated point clouds and high-fidelity in-game images through a framework based on DeepGTAV and Script Hook V.
- An autonomously driven ego car carries a virtual LiDAR scanner while the system simultaneously collects point clouds and captures the game screen.
- Ray casting returns the first hit point for each laser ray, enabling calculation of distance, object category, and instance ID for automatic annotation.
- Users can configure vertical and horizontal FOVs and resolutions, pitch angle, maximum laser range, and scanning frequency.
B. Automatic Calibration Method
The calibration method maps each LiDAR point to its corresponding camera pixel by using shared camera–LiDAR placement and geometric projection parameters.
- The system automatically finds the corresponding image pixel for each LiDAR point from camera and scanner parameters.
- For a laser ray with azimuth φ and zenith θ, the method computes pixel index (i, j) using camera and LiDAR geometry.
- The calibration uses shared camera and LiDAR centers, camera near-plane geometry, field-of-view parameters, and camera-axis vectors.
- The computed far point Pfar supplies the ray-casting API with a point in the laser direction.
- The mapped car point cloud matches the corresponding car in the image fairly accurately after calibration.
C. Configurable In-game Scene
A configurable mode lets users construct varied or hazardous driving scenes and systematically sample scene changes to test and improve neural networks.
- Configurable scenes can generate training data for driving situations that are dangerous or rare in the real world.
- Users can vary car model, location, orientation, number of cars, background, car color, weather, and time of day.
- Systematic sampling of scene modifications produces data for testing, analyzing, and improving neural-network performance through retraining.
III. EXPERIMENTS AND RESULTS
The experiments evaluate two uses of synthesized data: improving segmentation accuracy during training and testing networks to guide performance improvement through retraining.
- The experiments test whether framework-collected data improves validation accuracy when used during training.
- The experiments also test whether collected data can systematically evaluate a neural network and improve its performance through retraining.
A. Evaluation Metrics
The evaluation focuses on point-wise classification of LiDAR scans, using IoU, Precision, and Recall to measure segmentation accuracy. IoU is the primary metric because it accounts for both false positives and false negatives.
- LiDAR point cloud segmentation assigns a class label to each point in a detected scan.The task supports autonomous vehicles' perception and navigation.
- The evaluation computes Intersection-over-Union, Precision, and Recall for segmentation accuracy.
- IoU is the primary accuracy metric because it accounts for both false positives and false negatives.Precision and Recall measure accuracy with respect to false positives and false negatives, respectively.
B. Experimental Setup
The experiments analyze SqueezeSeg using labeled KITTI LiDAR scans, with the real-world data restricted to a forward-facing 90° field of view. The resulting dataset contains 10,848 scans split into training and validation sets.
- The analysis uses SqueezeSeg, a convolutional neural network for point cloud segmentation.
- KITTI labels are converted from 3D bounding boxes to point-wise labels within a forward-facing 90° horizontal field of view.The restriction reflects the reliable 3D bounding-box labels available for front-view LiDAR scans.
- 10,848 labeled LiDAR scans are split into 8,057 training scans and 2,791 validation scans.Each scan contains x, y, and z coordinates plus an intensity value.
C. Experimental Results
Experiments evaluate synthetic LiDAR data for segmentation training and use user-configured scenes to expose and repair neural-network blind spots. Combined real and synthetic training improves accuracy, while targeted retraining improves most poorly performing scene positions with little KITTI change.
- Synthetic-data training: 8,585 synthetic LiDAR scans were generated in autonomous-driving scenes with (x, y, z) measurements and a 90° horizontal FOV matching KITTI.The generated scans did not contain intensity.
- Synthetic-data training: Combining KITTI and GTA-V data improved segmentation accuracy by almost 9% over training with real-world data alone.Purely synthetic training performed poorly because the synthetic and KITTI distributions differed.
- User-configured testing: 555 sampled car-location scenes revealed low-IoU blind spots both far from the LiDAR scanner and at some closer locations.The experiment varied only the car-location X-Y dimensions rather than the full modification space.
- User-configured testing: mIoU was computed across seven scene backgrounds for each position in the 15 × 10 X-Y modification space.The validation set contained 1,200 scans, while 1,050 scans formed the retraining set.
- User-configured testing: The pretrained network performed poorly at far field-of-view boundaries and unexpectedly at the close position (-3, 5), where errors may be dangerous.Low-mIoU positions were selected to form the retraining set.
- Retraining results: After retraining, most originally low-mIoU positions improved, while KITTI IoU changed only from 60.8% to 60.6%.Performance degraded at only a small fraction of positions.
IV. CONCLUSIONS AND FUTURE WORK
The paper presents a game-based framework for synthesizing annotated LiDAR point clouds and calibrating them with scene images. Synthetic data improves segmentation validation accuracy by 9%, while systematic testing and retraining helps identify and fix neural-network blind spots.
- Conclusions and future work: The framework synthesizes annotated LiDAR point clouds from a virtual game world and automatically calibrates them with corresponding scene images.It supports both training-data generation and systematic neural-network testing.
- Conclusions and future work: Synthetic data improved validation accuracy for point-cloud segmentation by 9%.The paper also reports that the intensity channel was effective in the first experiments.
- Conclusions and future work: Systematic sampling and testing identified potential neural-network weaknesses or blind spots that could be fixed through retraining.Future work includes simulating LiDAR intensity information.