Source-linked AI summary
AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms
Wei Li, Chengwei Pan, Rong Zhang, Jiaping Ren, Yuexin Ma, Jin Fang, Feilong Yan, Qichuan Geng, Xinyu Huang, Huajun Gong, Weiwei Xu, Guoping Wang, Dinesh Manocha, Ruigang Yang
TL;DR
CG-based autonomous-driving simulation is costly to build and produces images and traffic that lack the richness and flexibility of real scenes. AADS combines scanned street backgrounds, synthesized traffic, and view synthesis; the authors report its use for AD tasks and for generating annotated training data or simulating the full system. Its RGB viewpoint changes and scene conditions remain bounded by the captured data.
Problem
CG-based simulation requires costly high-fidelity environmental models, while its imagery and scripted traffic lack the richness and flexibility of real-world scenes.
Source: INTRODUCTION · p. 1Method
AADS combines scanned street scenes with synthesized traffic based on real trajectories and synthesizes background views from captured imagery.
Source: INTRODUCTION · pp. 1, 3, Evaluations of augmented background… · p. 4Results
AADS was shown effective for AD tasks and can generate realistic, finely annotated training data or simulate the entire AD system in-line with little human intervention.
Source: CONCLUSIONS · p. 8Takeaways & Limitations
The system is presented as a way to use captured scene data and general traffic trajectories for training or in-line simulation of autonomous-driving systems.
Source: CONCLUSIONS · p. 8RGB viewpoint changes are limited because large deviations from captured views degrade image quality, and lighting or environmental changes such as snow and rain must match the captured images.
Source: CONCLUSIONS · pp. 8–9
Abstract
from arXiv · showhide
Simulation systems have become an essential component in the development and validation of autonomous driving technologies. The prevailing state-of-the-art approach for simulation is to use game engines or high-fidelity computer graphics (CG) models to create driving scenarios. However, creating CG models and vehicle movements (e.g., the assets for simulation) remains a manual task that can be costly and time-consuming. In addition, the fidelity of CG images still lacks the richness and authenticity of real-world images and using these images for training leads to degraded performance. In this paper we present a novel approach to address these issues: Augmented Autonomous Driving Simulation (AADS). Our formulation augments real-world pictures with a simulated traffic flow to create photo-realistic simulation images and renderings. More specifically, we use LiDAR and cameras to scan street scenes. From the acquired trajectory data, we generate highly plausible traffic flows for cars and pedestrians and compose them into the background. The composite images can be re-synthesized with different viewpoints and sensor models. The resulting images are photo-realistic, fully annotated, and ready for end-to-end training and testing of autonomous driving systems from perception to planning. We explain our system design and validate our algorithms with a number of autonomous driving tasks from detection to segmentation and predictions. Compared to traditional approaches, our method offers unmatched scalability and realism. Scalability is particularly important for AD simulation and we believe the complexity and diversity of the real world cannot be realistically captured in a virtual environment. Our augmented approach combines the flexibility in a virtual environment (e.g., vehicle movements) with the richness of the real world to allow effective simulation of anywhere on earth.
INTRODUCTION
Autonomous-driving simulation can support safety validation and labeled training, but CG-based environments are costly to build and lack the richness and traffic flexibility of real scenes. AADS combines scanned street imagery with data-driven traffic and view synthesis to create realistic, scalable simulation inputs.
INTRODUCTION
- Safety validation and labeled training motivate simulation, but building high-fidelity CG environments is costly and their rendered images and scripted traffic lack real-world richness and flexibility.Simulation can test perception, navigation, and control, or generate labeled training data.Source: INTRODUCTION · p. 1
- AADS uses LiDAR and cameras to scan street scenes, then synthesizes backgrounds and traffic to create photorealistic driving scenarios.The pipeline includes data preprocessing, background and trajectory synthesis, moving-object augmentation, and LiDAR simulation, producing RGB images, point clouds, and annotated trajectories.Source: INTRODUCTION · p. 1, Figure 1 · p. 2
- Captured real-world trajectories guide the placement and movement of synthetic traffic, yielding patterns described as natural and reflective of real-world complexity and diversity.The system repositions and renders 3D vehicles, computer-generated pedestrians, and other movable subjects into background images using estimated outdoor illumination.Source: INTRODUCTION · pp. 1, 3
- AADS is presented as enabling large-scale testing without costly CG models or tedious manual traffic-flow programming.The paper also describes a view-synthesis method that supports interpolation and extrapolation from few images with better image quality and fewer artifacts than previous approaches.Source: INTRODUCTION · p. 3
- The contribution includes new datasets of dense metropolitan traffic, including trajectory data and a 3D street-view dataset with pixel- and point-level annotations.The authors say wide availability of 3D scene scans and vehicle trajectories is needed for automatically generating new traffic scenarios.Source: INTRODUCTION · p. 3
Previous methods
Prior simulation approaches rely on CG environments, but their synthetic imagery and environmental models remain limiting, while realistic traffic involving dense, heterogeneous agents is difficult to simulate. AADS contrasts with fixed-view image augmentation by supporting viewpoint changes and traffic simulation for closed-loop applications.
Previous methods
- CG-based simulators support end-to-end, closed-loop training and testing, but their rendered images remain visibly synthetic and may lack details needed by machine-learning methods.The cited examples include CARLA, AirSim, and Drive Constellation.Source: Previous methods · p. 3
- Large point clouds are difficult to turn into detailed rendered images, and conventional traffic algorithms often omit realistic individual vehicle and pedestrian movements.Dense traffic with heterogeneous agents, such as pedestrians crossing at an intersection, is identified as a particular challenge.Source: Previous methods · p. 3
- Prior augmentation rendered 3D vehicles onto captured backgrounds from a fixed viewpoint, whereas AADS supports changing viewpoints and realistic traffic simulation for closed-loop planning and driving decisions.The paper also describes using AADS to generate looped data for reinforcement learning and learning-by-demonstration.Source: Previous methods · p. 3
RESULTS
The results section describes AADS data as useful for perception and prediction tasks, alongside datasets of annotated street scenes and heterogeneous traffic trajectories. The datasets provide image, point-cloud, and trajectory data collected in complex urban settings.
RESULTS
- Simulated RGB and LiDAR images were useful for improving perception performance, while simulated trajectories were useful for improving predictions of obstacle movements.The paper identifies obstacle-motion prediction as relevant to planning and control.Source: RESULTS · p. 3
The datasets
- The labeling pipeline uses easier-to-produce 3D annotations to generate 2D annotations by projecting them into image frames.The authors combine 3D and 2D labeling stages to reduce the time and labor burden of manual 2D labeling.Source: The datasets · pp. 3–4
- The dataset annotations cover 25 classes across five groups and include 35 lane-marking types annotated in both 2D and 3D.A tricycle class was added; lane markings are categorized by color, type, and usage.Source: The datasets · p. 4
- The ApolloScape-RGB release contains 143,906 video frames with pixel-level annotations, while ApolloScape-PC provides 3D point-level annotations.The authors report that their dataset compares favorably with other street-view datasets in scene complexity, pixel-level annotations, and number of classes.Source: The datasets · p. 4
- ApolloScape-TRAJ contains around 100,000 RGB images at 1920 by 1080 resolution and 1000 km of trajectories for heterogeneous moving traffic agents.The dataset includes challenging scenarios with vehicles, bicycles, and pedestrians moving around one another under varied lighting and traffic densities.Source: The datasets · p. 4
Evaluations of augmented background synthesis
AADS synthesizes background images for specified views from fixed-view captures, avoiding the need to model the full environment. Its refined-depth view synthesis is reported to reduce artifacts relative to comparison methods and to support viewpoint changes in closed-loop driving.
Evaluations of augmented background synthesis
- The background-synthesis method uses image-based rendering to create specific views from images captured at fixed viewpoints, avoiding prerequisite modeling of the full environment.The paper focuses on wide-baseline stereo street scenes and refines depth maps initialized from laser-scanned point clouds because those maps can contain outliers and holes.Source: Evaluations of augmented background… · p. 4
- Refined depth maps reduced view-change artifacts: unrefined depth produced fluctuations, while refined depth kept window frames straight in the illustrated regions.The comparison contrasts the results shown in Fig. 3 (F and H) with those using refined depth in Fig. 3 (yellow rectangle).Source: Evaluations of augmented background… · p. 4
- AADS handled both view interpolation and extrapolation, preserving object shapes and reducing misalignment and seam artifacts compared with the Liu and Chaurasia methods in challenging viewpoint changes.The reported comparisons used four reference images; the cited failure cases for the other methods included distortion at larger translations, ghosting, misalignment, and seams.Source: Evaluations of augmented background… · p. 4
- The authors report that synthesized front-camera views support a driving car changing lanes several times because the method interpolates or extrapolates viewpoints.This closed-loop example is shown in the accompanying video.Source: Evaluations of augmented background… · p. 4
Evaluations of trajectories synthesis
The authors evaluate traffic synthesis by comparing simulated traffic with the input data on velocity and minimum-distance distributions. Their method produces distributions closer to the input data on both measures.
Evaluations of trajectories synthesis
- The evaluation compares the method with ground truth and Chao et al. (26) using velocity and minimum-distance probability distributions.Traffic was simulated on a straight four-lane road; the authors divided each metric into 30 intervals.Source: Evaluations of trajectories synthesis · p. 4
- Figure 2 pairs dataset comparisons with RGB images, annotations, a point cloud, and labeled traffic trajectories.The caption identifies these as components of ApolloScape and its extension.Source: Figure 2 · p. 5
- Our simulation results are closer to the input data in both velocity and minimum-distance distributions.The comparison used traffic simulated on a straight four-lane road, with agent counts, positions, and velocities initialized from the dataset.Source: Evaluations of trajectories synthesis · pp. 4–5
AADS evaluations by AD applications
AADS data were evaluated for image detection and point-cloud instance segmentation, as well as for traffic-agent prediction. Results include gains from adding simulated data and performance approaching real-data training in point-cloud segmentation.
AADS evaluations by AD applications
- AADS-RGB images were more visually similar to CityScapes than images from CARLA or VKITTI.The authors describe this as a small domain gap; the evaluation trained detectors on the datasets and tested them on CityScapes.Source: AADS evaluations by AD applications · p. 7
- Adding 800 simulated images to ApolloScape-1600 (AADS-RGB-2400) improved results by roughly 1%.The authors report this as evidence that their simulation data may be closer to real-world data than VR data.Source: AADS evaluations by AD applications · p. 8
- With 100,000 simulated data points, instance-segmentation performance exceeded training on 4,000 real points and approached training on 16,000 or 100,000 real points.The models used PointNet++ and were evaluated on KITTI-PC; the metrics were mAP(Bbox) and mAP(mask).Source: AADS evaluations by AD applications · p. 8
- Simulation pretraining improved validation accuracy by 2 to 4%; 100,000 simulated points plus 1,600 real points outperformed 16,000 real points on average mAP.With 32,000 real points for fine-tuning, the model surpassed one trained on 100,000 real points; the authors say simulation may reduce manually labeled data by 80 to 90%.Source: AADS evaluations by AD applications · p. 8
- AADS object placement outperformed random or rule-based placement by 4 to 7%, with a 1 to 4% gap from real-data training depending on the application.Adding 20,000 simulated frames to 20,000 real frames sharply reduced trajectory-prediction error; cars had the largest reduction because they were well represented in simulated trajectories.Source: AADS evaluations by AD applications · p. 8
CONCLUSIONS
AADS supports autonomous-driving tasks using captured scene data and traffic trajectories, with little human intervention. Its practical use depends on capturing the background, and its RGB viewpoint and environmental variation are constrained.
CONCLUSIONS
- AADS’s demonstrated tasks use captured scene data specific to a location and general traffic trajectory data, with little human intervention.The system may generate annotated training data or simulate the AD system from perception to planning.Source: CONCLUSIONS · p. 8
- AADS can be used in real-world scenarios when the background can be captured.Source: CONCLUSIONS · p. 8
- RGB view changes are limited: large deviations from captured viewpoints degrade image quality, while the authors consider road-plane rotation and 2D translation sufficient for typical driving.Source: CONCLUSIONS · p. 8
- Lighting and environmental conditions such as snow or rain must match the captured images; synthesizing those changes remains future work.The authors cite preliminary seasonal-change results and identify full lighting and environmental synthesis as a direction they are pursuing.Source: CONCLUSIONS · pp. 8–9
Data preprocessing
AADS prepares captured scenes for simulation by clearing moving objects, estimating illumination, and synthesizing backgrounds and object appearances. Its figures also identify the view-synthesis stages and a trajectory-prediction evaluation using added simulated frames.
Data preprocessing
- Semantic labels from ApolloScape are used to remove cars, bicycles, trucks, and pedestrians from scanned images and point clouds; image holes are filled with semantic-guided inpainting.Point-cloud completion is handled in the depth-processing stage for novel background synthesis.Source: Data preprocessing · p. 9
- TrafficPredict evaluation compares trajectory prediction using 20,000 real trajectory frames with prediction using 20,000 additional simulation frames.Source: Figure 7 · p. 9
- Outdoor illumination is estimated so rendered CG models can have consistent shadows on the background and vehicle bodies.Source: Data preprocessing · p. 9
- Novel-view synthesis uses four nearest reference images, warps them into the target view, stitches them, then post-processes with hole filling and color blending.Source: Figure 8 · p. 9
- Real-image textures can optionally be transferred to CG vehicle models after retrieving and aligning corresponding models, then completing their appearance with symmetric priors.Source: Data preprocessing · p. 10
Augmented background synthesis
AADS synthesizes backgrounds for requested views from captured scenes rather than modeling the full environment in advance. It filters and completes depth, warps reference images, stitches them using pixel-labeling costs, and composes simulated traffic into the result.
Augmented background synthesis
- AADS synthesizes a background in the requested view during simulation, avoiding full-scene modeling ahead of time through view synthesis.Source: Augmented background synthesis · p. 10
- Depth maps are filtered and completed because scanned depths are unreliable or incomplete due to stereo baseline limits, removed moving objects, and scanning failures.A small-kernel median filter prunes outliers, a guided filter preserves thin structures and aligns edges, and Poisson completion propagates depth into holes.Source: Augmented background synthesis · p. 10
- For a target view, four nearest references are warped into it using forward mapping, depth inpainting, backward mapping, and occlusion testing.Source: Augmented background synthesis · p. 10
- Instead of directly blending warped images, AADS formulates synthesis as pixel labeling that selects a source color for each target pixel.The pixel-labeling energy includes data and smoothness terms; its data term combines view-angle and camera-position information.Source: Augmented background synthesis · p. 10
- Occluded pixels receive infinite penalty in the labeling energy, while non-occluded pixels receive no occlusion penalty.Depth testing identifies occlusions near depth edges.Source: Augmented background synthesis · p. 10
- Smoothness costs include truncated color and depth differences plus a gradient term to help stitching across illumination differences.The implementation uses RGB threshold 0.5 and depth threshold 5 m; TRW-S solves the labeling problem, followed by modified Poisson blending.Source: Augmented background synthesis · p. 10
- Large view extrapolations can leave disoccluded regions unfilled by stitching, producing artifacts that are filled with plausible blurred color during Poisson blending.Source: Augmented background synthesis · p. 11
- Traffic agents are initialized in HD-map lanes and interact under collision-avoidance, pedestrian-yielding, and other motion constraints; rendered objects yield annotated RGB and LiDAR data.Movement choices are optimized frame by frame using states estimated from captured heterogeneous-agent trajectories.Source: Augmented background synthesis · p. 11
LiDAR synthesis
AADS models LiDAR beams using sensor-specific parameters fitted from real point clouds, then intersects beams with projected scene maps to synthesize point clouds. The approach is described as extendable to other sensor configurations.
LiDAR synthesis
- AADS models LiDAR beam angles and their angular and distance noise to represent the physical process of pulse emission and return.The HDL-64E S3 example emits 64 beams at vertical angles from −24.33° to 2°; fitted parameters account for variation even among sensors of the same type.Source: LiDAR synthesis · p. 11
- Sensor parameters are fitted from real point clouds, with manually marked laser-beam points used to estimate vertical angles and noise.The implementation uses Gaussian noise, with distance noise variance 0.5 cm and azimuth angular noise variance 0.05°.Source: LiDAR synthesis · p. 11
- Point clouds are synthesized by intersecting laser beams with projected scene maps rather than directly with mixed background points and CG meshes.The authors state the method can extend to arbitrary LiDAR sensors and different numbers and poses of sensors.Source: LiDAR synthesis · p. 11
SUPPLEMENTARY MATERIALS
The supplementary materials list a point-cloud simulation visual evaluation and five accompanying movies covering the simulation pipeline and related demonstrations.
SUPPLEMENTARY MATERIALS
- The supplement includes a point-cloud simulation visual evaluation and movies on the scan-and-simulation pipeline, lane-change synthesis, data augmentation, and novel-view synthesis.It also lists a full movie.Source: Figure S1