PaperlayerView exact revision on arXiv

Source-linked AI summary

Precise Synthetic Image and LiDAR (PreSIL) Dataset for Autonomous Vehicle Perception

Braden Hurl, Krzysztof Czarnecki, Steven Waslander

arXiv:1905.00160v2cs.CVcs.RO

TL;DR

Public 3D driving data is limited and class-imbalanced, and collecting detailed LiDAR annotations is resource-intensive. PreSIL generates a KITTI-compatible synthetic dataset in GTA V with precise LiDAR and extensive image and point-cloud labels. Pre-training with 40,000 PreSIL instances improves KITTI 3D detection average precision by 1–5% across difficulty levels; the synthetic scan captures all points at one instant rather than over time.

  • Problem

    Public 3D driving data is limited and class-imbalanced, while detailed LiDAR annotation is time-consuming and difficult.

    Source: §I · p. 1

  • Method

    PreSIL generates KITTI-compatible synthetic driving data in GTA V, including precise LiDAR point clouds and detailed image and point-cloud annotations.

    Source: §I · p. 2, §III · p. 3, B. Precise LiDAR point clouds · p. 4

  • Results

    Pre-training with 40,000 PreSIL instances improves KITTI 3D detection average precision by 1–5% at all difficulty levels.

    Source: C. Experimental Results · p. 7

  • Takeaways & Limitations

    PreSIL provides a publicly available synthetic dataset and code for autonomous-driving perception research, with new annotations for training perception networks.

    Source: §V · pp. 7–8

    Synthetic LiDAR points are captured simultaneously, unlike the up-to-100 ms capture-time spread within a rotating Velodyne HDL-64E scan.

    Source: E. Limitations · p. 5

Abstract

from arXiv · show

We introduce the Precise Synthetic Image and LiDAR (PreSIL) dataset for autonomous vehicle perception. Grand Theft Auto V (GTA V), a commercial video game, has a large detailed world with realistic graphics, which provides a diverse data collection environment. Existing works creating synthetic LiDAR data for autonomous driving with GTA V have not released their datasets, rely on an in-game raycasting function which represents people as cylinders, and can fail to capture vehicles past 30 metres. Our work creates a precise LiDAR simulator within GTA V which collides with detailed models for all entities no matter the type or position. The PreSIL dataset consists of over 50,000 frames and includes high-definition images with full resolution depth information, semantic segmentation (images), point-wise segmentation (point clouds), and detailed annotations for all vehicles and people. Collecting additional data with our framework is entirely automatic and requires no human annotation of any kind. We demonstrate the effectiveness of our dataset by showing an improvement of up to 5% average precision on the KITTI 3D Object Detection benchmark challenge when state-of-the-art 3D object detection networks are pre-trained with our data. The data and code are available at https://tinyurl.com/y3tb9sxy

I. INTRODUCTION

PreSIL addresses the limited availability and class imbalance of public 3D driving data, alongside the expense and difficulty of annotating LiDAR point clouds. It provides a large, automatically generated GTA V dataset with precise LiDAR and extensive image and point-cloud annotations.

I. INTRODUCTION

  • Public 3D training data is limited and class-imbalanced, while annotating LiDAR objects is costly and some point-wise labels are difficult for people to produce.KITTI training data contains 28,742 cars, 4,487 pedestrians, and 1,627 cyclists; SUN RGB-D reports an average of 114 seconds to annotate each 3D object instance.Source: §I · p. 1
  • The dataset adds 2D and 3D object labels, augmented object details, image segmentation, point-wise segmentation, and sensor calibration information.Augmented labels include entity IDs, associated 2D point counts, speed, pitch, roll, and model names.Source: §I · p. 2
  • PreSIL provides more than 50,000 KITTI-format frames with extended labels and precise LiDAR data for people, addressing gaps in publicly available training data.The dataset includes 1920 x 1080 color images and depth maps, instance segmentation for vehicles and people, semantic segmentation for other pixels, and point clouds with entity IDs for vehicles and people.Source: §I · p. 2
  • The authors report that PreSIL data can boost a state-of-the-art object detection network on the KITTI 3D Object Detection benchmark.GTA V is presented as a diverse synthetic data source whose established capture process can generate annotated data with less time and cost than manual annotation.Source: §I · pp. 1–2

II. RELATED WORK

Prior work has used GTA V to generate image-based driving data, including object detection, distance estimation, and lane-marking tasks. These approaches demonstrate the range of 2D data generated from the game.

II. RELATED WORK

  • GTA V has been used to generate synthetic data for several autonomous-driving perception tasks.The section introduces examples including object detection and semantic segmentation.Source: §II · p. 2, A. 2D Object Detection · p. 2

A. 2D Object Detection

  • Driving in the Matrix uses depth and stencil buffers to produce tight 2D boxes across more than 200,000 images, and reports better KITTI results than Cityscapes when training only on its synthetic data.Martinez et al. generate over 480,000 instances for tasks including vehicle distance and lane markings; the section summarizes GTA V 2D data as improving detection-network performance.Source: A. 2D Object Detection · p. 2
  • GTA V 2D datasets also support distance estimation and lane-marking tasks using more than 480,000 generated instances.The cited work trains a CNN on 2D images for these tasks.Source: A. 2D Object Detection · p. 2

B. 3D point cloud Segmentation

Earlier GTA V LiDAR work uses ray casting, while other synthetic-data approaches generate image labels or modify real point clouds. PreSIL is presented as a KITTI-compatible dataset whose capture process also accommodates new annotations and sensory data.

B. 3D point cloud Segmentation

  • Earlier GTA V LiDAR work uses ray casting, which can reduce pedestrians to cylinders and miss some vehicles; PreSIL addresses both limitations.Prior GTA V point-cloud segmentation work reports about 9 percent IoU improvement from augmenting KITTI data without reflectance, but its experiments were conducted only on cars.Source: B. 3D point cloud Segmentation · p. 2
  • Fang et al. augment real-world point clouds by removing object points and inserting simulated points, but do not create matching images for sensor fusion.The cited method increases accuracy by adding its augmentations to the training dataset.Source: B. 3D point cloud Segmentation · p. 2

C. 3D Scene Understanding

  • Playing for Data and Playing for Benchmarks generate semantic-segmentation data, but their detouring technique permits only coarse semantic labels.URSA labels encountered textures to improve GTA V semantic-segmentation labels and releases more than 1,000,000 images.Source: C. 3D Scene Understanding · pp. 2–3

D. Simulators

  • Commercial game worlds are described as more realistic than currently available simulators, while CARLA's LiDAR uses ray casting onto basic collision meshes.The passage says CARLA plans to release a LiDAR simulator similar to the authors' GTA V simulator, but it is not yet available.Source: D. Simulators · p. 3

III. SYNTHETIC DATA

  • PreSIL is formatted for compatibility with KITTI and includes new annotations and sensory data when available, while some data elements must be approximated because of GTA V's limitations.The section covers the capture process, tool limitations, and new data formats.Source: §III · p. 3

A. Capturing Process

PreSIL generates point clouds from depth-buffer information and in-game parameters, while collecting object labels and image segmentation. Its point clouds capture object classes and distant vehicles that ray casting can miss.

A. Capturing Process

  • The data-collection process combines DeepGTAV scenario and object retrieval with GTAVisionExport depth and stencil collection through Script Hook V.The camera and LiDAR share a position so the camera-perspective depth buffer can generate LiDAR point clouds; the setup approximates KITTI sensor placement.Source: A. Capturing Process · p. 3
  • Object positions, dimensions, headings, types, and vehicle models come from native game functions; image boxes are tightened using instance-wise segmentation.Segmentation checks projected depth against 3D boxes, then uses stencil classes and depth disparity to separate instances when needed.Source: A. Capturing Process · p. 3

B. Precise LiDAR point clouds

  • The method projects simulated LiDAR directions onto the image plane and obtains depth through interpolation, relying on the camera and LiDAR sharing a position.It uses 0.09° horizontal and 0.42° vertical angular resolution, with a 120-metre maximum range.Source: B. Precise LiDAR point clouds · p. 4
  • When neighboring depth samples differ substantially, the algorithm avoids interpolation to reduce artifacts between objects.It interpolates only when the maximum of four depths is less than 1.08 times their minimum; that threshold was chosen by visual inspection.Source: B. Precise LiDAR point clouds · p. 4
  • PreSIL's point clouds accurately represent object classes, including pedestrians and bicycles, and capture vehicles missed by ray casting.The comparison also shows ray casting may miss vehicles while PreSIL captures them.Source: B. Precise LiDAR point clouds · p. 4

C. Depth Calculation

PreSIL converts GTA V depth-buffer values into pixel depths using a transformation based on near- and far-clipping-plane parameters. The resulting depths support point-cloud generation.

C. Depth Calculation

  • The transformation TD converts the depth-buffer value for a pixel into a depth value, using the near-plane distance and clipping-plane parameters.The near- and far-plane distances come from in-game functions; the horizontal and vertical near-plane distances are measured from the plane center to the pixel.Source: C. Depth Calculation · pp. 4–5
  • Some vehicles beyond 30 metres are captured by PreSIL point clouds but missed by ray casting.The figure compares point clouds produced by the two methods.Source: Figure 4 · p. 5
  • The near-plane width and height are calculated from the screen aspect ratio, near-plane distance, and vertical field of view.These values remain constant unless graphics settings change.Source: C. Depth Calculation · p. 5

D. Synchronization

The data sources in GTA V are not naturally synchronized: DirectX buffers and native functions operate at different time steps. PreSIL pauses the game and flushes graphics buffers to collect aligned data.

D. Synchronization

  • DirectX buffers and native functions operate at different time steps, so their data do not always align without synchronization.Using a buffer from the next time step after native-function calls aligns them most of the time, but not in every instance.Source: D. Synchronization · p. 5
  • PreSIL pauses the game, renders the script cameras twice, collects data from native functions and DirectX, then resumes the game.Source: D. Synchronization · p. 5
  • The synchronization procedure was tested on Nvidia and AMD GPUs and visually inspected along a route with regular turns.Misalignment is easiest to see during rapid camera rotation, which creates greater inter-frame disparity.Source: D. Synchronization · p. 5

E. Limitations

PreSIL omits several real-LiDAR properties because GTA V lacks reflectivity information and limits control over scan timing. Its synthetic scanner also covers only the camera's forward field of view.

E. Limitations

  • PreSIL point clouds omit reflectivity because GTA V does not provide it.Reflectivity affects object detection and the range at which a surface can be detected; the method does not model that representation.Source: E. Limitations · p. 5
  • Unlike KITTI's rotating scanner, PreSIL captures every point in a scan at the same instant.KITTI's Velodyne HDL-64E scan can have up to 100 ms between point-capture times, a behavior difficult to reproduce in-game.Source: E. Limitations · p. 5
  • The synthetic LiDAR captures only the camera's field of view, extending 45 degrees to either side of forward.Source: E. Limitations · p. 5
  • PreSIL does not approximate the noise in which a LiDAR point is missing despite being within the scanner's maximum range.The passage identifies this as a noise source in Velodyne HDL-64E point clouds.Source: E. Limitations · p. 5

F. Extensions

PreSIL extends KITTI-compatible labels with information for relating objects across modalities and representing vehicles on steep slopes. It also supplies image data and annotations that support additional perception tasks.

F. Extensions

  • Pitch and roll annotations help bounding boxes encompass vehicles on steep slopes, where KITTI-format boxes can miss parts of the object.The extended annotations are stored separately to keep labels compliant with KITTI format.Source: F. Extensions · p. 6, Figure 6 · p. 6
  • Each object’s augmented annotations include its entity ID, instance-mask pixel count, and model string identifier.Entity IDs relate point-cloud points to detection boxes; model strings could support experiments treating vehicle models as classes.Source: F. Extensions · p. 6
  • PreSIL includes 1920 x 1080 images with full-resolution depth, stencil, and instance-level segmentation data.These additional files are useful for semantic segmentation and depth completion tasks.Source: F. Extensions · p. 6

G. Dataset Statistics

PreSIL contains many pedestrians and vehicles, fewer cyclists than KITTI, and additional classes. Its pedestrian locations cover a broader area, and configurable scenes can target coverage gaps.

G. Dataset Statistics

  • PreSIL’s pedestrian-location heatmap covers a larger area than KITTI’s, with pedestrians farther to the sides and fewer in the roadway gap.The heatmaps span 40 metres to either side and 100 metres forward.Source: G. Dataset Statistics · p. 6, Figure 7 · p. 6
  • PreSIL has an abundance of pedestrians and vehicles but fewer cyclists than KITTI, and includes bus, motorbike, and trailer classes absent from KITTI.Class counts and average counts per image containing each class are compared in Table I.Source: G. Dataset Statistics · p. 6, Table I · p. 6
  • User-configurable scenes can test model coverage and support training on data that reveals coverage gaps.Future work may use these scenes to improve pedestrian or cyclist coverage.Source: G. Dataset Statistics · p. 6

IV. EXPERIMENTS AND RESULTS

The experiments evaluate AVOD-FPN on KITTI 3D object detection using pedestrian-focused training configurations with synthetic and real data. The setup compares KITTI-only training with PreSIL pre-training and fine-tuning, while the supplied passages do not provide the outcome values.

IV. EXPERIMENTS AND RESULTS

  • The experiments target whether synthetic data improves autonomous-driving perception benchmarks and how synthetic pre-training affects pedestrian detection; AVOD-FPN is selected for evaluation.The authors describe it as a top-five KITTI 3D object detection benchmark performer for pedestrians, with publicly available code and 0.1-second runtime.Source: §IV · p. 6
  • Pedestrians are the evaluation focus because KITTI has fewer pedestrians and the authors’ LiDAR generator improves pedestrian realism; cyclists are excluded because PreSIL has fewer cyclists than KITTI.The authors note that scenarios with abundant cyclists could be constructed.Source: §IV · p. 7

A. Evaluation Metrics

  • KITTI average precision averages precision across recall levels, with easy, moderate, and hard categories based on 2D box height, occlusion, and truncation.The reported experimental table covers average precision for pedestrians on the KITTI 3D object detection benchmark.Source: A. Evaluation Metrics · p. 7, Table II · p. 7

B. Experimental Setup

  • The setup uses a roughly 50/50 KITTI split of 3,712 training and 3,769 test instances, and PreSIL splits of about 40,000 training and 10,000 validation instances.AVOD-FPN is trained on KITTI alone or on a 10,000-instance PreSIL subset or the full 40,000, then PreSIL-trained checkpoints are fine-tuned on KITTI and evaluated on the KITTI test split.Source: B. Experimental Setup · p. 7

C. Experimental Results

Pretraining AVOD-FPN with 40,000 PreSIL instances improves AP across all difficulty levels, with the largest improvement on hard examples. The paper also notes limits in cross-dataset comparison and in GTA V point-cloud reflectance, while concluding that PreSIL provides standard annotations at a fraction of real-world dataset cost.

C. Experimental Results

  • 1–5% AP improvement at all difficulties is reported when AVOD-FPN is pretrained with 40,000 PreSIL instances.The largest improvement is on hard examples; the reported results use a test split of KITTI training data.Source: C. Experimental Results · p. 7
  • No comparison with other datasets’ impact on network performance is possible because those datasets are currently unavailable.The authors propose collecting data from a simulation environment such as CARLA for future comparison.Source: C. Experimental Results · p. 7
  • GTA V point clouds lack reflectance values, which the authors consider potentially important for detection.They suggest estimating reflectance from in-game properties as future work.Source: C. Experimental Results · p. 7

V. CONCLUSIONS

  • PreSIL provides standard annotations for autonomous-driving perception tasks at a fraction of the cost of a real-world dataset.The authors also report a quantitative analysis of synthetic-data improvements in 3D object detection.Source: §V · p. 7
  • The authors identify tracking data and 360-degree image and LiDAR coverage as future additions.They also propose transfer learning and adapting data across distributions as future work.Source: §V · pp. 7–8

Related papers

Loading 1905.00160v2…