PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles

Alexander Amini, Tsun-Hsuan Wang, Igor Gilitschenski, Wilko Schwarting, Zhijian Liu, Song Han, Sertac Karaman, Daniela Rus

arXiv:2111.12083v1cs.ROcs.CVcs.LG

TL;DR

Existing simulators lack photorealism and diverse sensor representations, limiting data-driven simulation across modalities. VISTA 2.0 synthesizes novel views for RGB, LiDAR, and event-based sensors and trains perception-to-control policies. Those policies transfer to a full-scale vehicle and show greater robustness than policies trained only on real-world data.

  • Problem

    Existing simulators lack photorealism and diverse sensor representations, and scaling data-driven simulation to multiple sensor types remains a challenge.

    Source: §I · p. 1

  • Method

    VISTA 2.0 processes real-world datasets to synthesize novel views for RGB, LiDAR, and event-based sensors and trains perception-to-control policies using guided policy learning.

    Source: §I · p. 1

  • Results

    VISTA-trained policies transfer directly to a full-scale vehicle and exhibit greater robustness than policies trained solely on real-world data.

    Source: §I · p. 2, §VI · p. 6

  • Takeaways & Limitations

    Closed-loop evaluation in VISTA showed consistent results with real-world tests, supporting its use for policy evaluation before deployment.

    Source: §VI · p. 6, C. Online real-world test · p. 6

Abstract

from arXiv · show

Simulation has the potential to transform the development of robust algorithms for mobile agents deployed in safety-critical scenarios. However, the poor photorealism and lack of diverse sensor modalities of existing simulation engines remain key hurdles towards realizing this potential. Here, we present VISTA, an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras, enabling the rapid generation of novel viewpoints in simulation and thereby enriching the data available for policy learning with corner cases that are difficult to capture in the physical world. Using VISTA, we demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle. The policies learned in VISTA exhibit sim-to-real transfer without modification and greater robustness than those trained exclusively on real-world data.

I. INTRODUCTION

Existing simulators lack photorealism and diverse sensor representations, while scaling data-driven simulation across sensor types remains a challenge. VISTA 2.0 extends data-driven simulation to multiple sensors and supports policy learning, testing, and full-scale vehicle deployment.

I. INTRODUCTION

  • Simulation offers a safer, faster way to expose agents to difficult edge cases, but existing engines lack photorealism and diverse sensor representations, and scaling across sensor types remains a challenge.Exposure to edge cases during training is described as critical to robustness to out-of-distribution events.Source: §I · p. 1
  • VISTA 2.0 is a multi-sensor, data-driven engine for autonomous-vehicle simulation, perception, and learning, synthesizing viewpoints across 2D RGB cameras, 3D LiDARs, and asynchronous event-based sensors.It processes existing real-world datasets using only local viewpoint changes for efficient computational rendering and low memory costs.Source: §I · p. 1
  • VISTA synthesizes novel viewpoints for RGB, LiDAR, and event-based sensors and uses the generated data to train perception-to-control policies with guided policy learning.The policies are directly transferred to a full-scale autonomous vehicle.Source: §I · pp. 1–2
  • VISTA-trained policies transfer directly to a full-scale vehicle and show greater robustness than policies trained solely on real-world data.The paper also reports improved recovery from challenging off-orientation positions and successful closed-loop event-camera control after decoupling the learning signal in VISTA.Source: §I · p. 2

II. RELATED WORK

Prior work includes model-driven simulation, data-driven novel-view synthesis, cross-sensor translation, and flexible policy learning. VISTA builds on data-driven simulation while targeting unified multi-sensor synthesis and closed-loop evaluation.

Cross-Sensor Transfer

  • Prior cross-sensor work predicts or translates between modalities, whereas VISTA targets a unified framework that jointly simulates diverse sensors and supports novel-view synthesis.Examples include monocular depth prediction, depth completion, and event-based translation or prediction –.Source: §II · p. 2

Simulation

  • Model-driven simulators use predefined scenery and physics, while data-driven simulators reconstruct virtual scenes from real data to synthesize novel views; VISTA focuses on local synthesis for scalability and transferable policies.The paper contrasts data-driven photorealism with model-based engines that lack sufficient photorealism for direct policy transfer.Source: §II · p. 2

Driving Policy Learning

  • Simulation supports policy-learning approaches beyond real-world imitation learning, including reinforcement learning and guided policy learning, and enables closed-loop evaluation before deployment.VISTA uses simulation for edge-case data generation and closed-loop evaluation.Source: §II · p. 2

III. MULTI-SENSOR SIMULATION

VISTA 2.0 extends VISTA 1.0's local RGB novel-view synthesis to additional sensor modalities. Its LiDAR synthesis accounts for sensor structure and occlusion when rendering views from a virtual agent's position.

A. Background

  • VISTA 1.0 synthesizes RGB images at novel viewpoints near a dataset trajectory by updating agent state, retrieving the closest frame, projecting it into 3D, and reprojecting it to the agent viewpoint.VISTA 2.0 extends this approach to local LiDAR and event-data synthesis.Source: A. Background · p. 2

B. LiDAR synthesis

  • LiDAR synthesis aims to generate a pointcloud at the virtual agent's position from the nearest human-collected scan, using the agent's relative rotation and translation.Each point is represented by its 3D position and intensity.Source: B. LiDAR synthesis · p. 2
  • A rigidly transformed LiDAR scan can break the sensor's ring pattern and retain points occluded from the new viewpoint, so synthesis must preserve sensor structure and cull occluded points.The paper's culling illustration describes a lightweight, GPU-accelerated algorithm that does not raycast a scene mesh.Source: B. LiDAR synthesis · p. 2, Figure 3 · p. 3
  • The multi-sensor simulation renders observations as virtual-agent trajectories unroll, synthesizing RGB images, LiDAR pointclouds, and continuous differential events.For LiDAR, dense depth and intensity representations are estimated from a sparse transformation and sampled to render sparse pointclouds.Source: Figure 2 · p. 3, Figure 4 · p. 4

C. Event synthesis

VISTA synthesizes events from RGB data by interpolating frames over time and estimating brightness changes, while supporting event generation in either RGB or virtual event-camera pixel space.

C. Event synthesis

  • VISTA generates events by estimating brightness changes over time in interpolated RGB frames, using an event-generation model with a contrast threshold.The threshold is sampled from a Gaussian to simulate noise.Source: C. Event synthesis · p. 4
  • Bidirectional optical flow enables arbitrary-time interpolation between consecutive RGB frames to construct a continuous image stream for event simulation.The interpolation network handles visibility from both directions.Source: C. Event synthesis · p. 4
  • Smaller simulation intervals improve temporal accuracy until optical-flow displacement reaches the subpixel limit, while adaptive sampling balances accuracy and efficiency.The maximum interval bounds the computational resources required.Source: C. Event synthesis · p. 4
  • Events can be generated in RGB image space and reprojected, or rendered directly in virtual event-camera space without pixel reprojection.The authors favor event-camera space because it shifts subpixel interpolation from pixel coordinates to mesh color or intensity.Source: C. Event synthesis · p. 4, Figure 5 · p. 4

A. Data generation and training using VISTA

VISTA pairs simulated sensor measurements with privileged-controller labels for policy learning, using randomized starts and branching to diversify training data.

A. Data generation and training using VISTA

  • Randomized simulator resets and privileged-controller actions generate sensor measurements paired with optimal control labels for policy supervision.The experiments use a pure-pursuit controller, though the framework permits other optimal controllers.Source: A. Data generation and training using… · pp. 4–5
  • A shuffled buffer approximates independent training samples, while rejection sampling balances the control-label distribution before examples are added.The passage identifies label balancing as the purpose of rejection sampling.Source: A. Data generation and training using… · p. 5
  • Arbitrary-control branching creates examples that disentangle ego-motion from scene information in event patterns, addressing a failure mode of closed-loop policies trained only on real data.Open-loop evaluation may not expose this failure because event data conflates vehicle motion with control.Source: A. Data generation and training using… · p. 5

B. Input representation and model architectures

The models share a deterministic curvature estimator and use sensor-specific feature extractors for RGB, event, and LiDAR inputs.

B. Input representation and model architectures

A. Experimental setup and data collection

The experiments use a sensor-equipped full-scale vehicle and roughly three hours of driving data collected across varied environments.

A. Experimental setup and data collection

  • The full-scale test vehicle is a 2019 Lexus RX 450H equipped with RGB, LiDAR, event-camera, odometry, and GPS sensors.The vehicle also carries an NVIDIA 2080Ti GPU and an AMD Ryzen 7 3800X processor.Source: A. Experimental setup and data… · p. 5
  • The roughly three-hour dataset spans daytime and nighttime, sun and rain, and urban and rural roads; sensor data and curvature feedback support simulation and policy evaluation.GPS data are used only for evaluation.Source: A. Experimental setup and data… · p. 5
  • Open-loop evaluation measures curvature-command mean squared error, while closed-loop evaluation measures lane-center deviation, crash rate, and interventions.Simulation crashes are lateral deviations over 2 meters from human trajectories; real-world takeovers occur when the vehicle leaves the road.Source: A. Experimental setup and data… · p. 5

B. Offline evaluation

Open-loop error favors IL, but the paper treats it as a poor proxy for driving performance because it misses compounding feedback errors. In VISTA closed-loop tests, GPL policies show lower lane-center deviation and crash rates than IL across varied environments and sensor modalities.

B. Offline evaluation

  • IL policies can gradually drift off the road and fail to recover when recovery training data are lacking.Closed-loop tests in simulation or reality are presented as a better proxy for policy performance than open-loop evaluation.Source: B. Offline evaluation · pp. 5–6
  • Open-loop IL outperforms GPL on control error, but open-loop error poorly predicts closed-loop success.It measures errors around human trajectories and ignores compounding feedback errors that can move the vehicle into out-of-distribution states.Source: Table I · p. 5, B. Offline evaluation · p. 5
  • GPL policies consistently reduce lane-center deviation and crash rates versus IL across varied environments, lighting conditions, and sensor modalities.Tab. II reports the closed-loop VISTA comparison.Source: Table II · p. 6, B. Offline evaluation · p. 6

C. Online real-world test

In real-world tests, GPL policies outperform IL for RGB and LiDAR, while event-camera results are mixed; a robustness test also reports better edge-case recovery for GPL than real-world IL. The conclusion presents VISTA as an open-source multimodal simulator whose learned policies transfer directly to a full-scale vehicle, with consistent simulation and real-world results.

C. Online real-world test

  • GPL policies perform significantly better than IL for RGB and LiDAR in real-world tests, while event-camera GPL has fewer interventions but greater center deviation and more interventions than RGB and LiDAR GPL.Each policy was run for three trials totaling 45 km; Fig. 6(D) compares the percentage of deviations below center-offset thresholds.Source: C. Online real-world test · p. 6
  • GPL policies significantly outperform real-world IL at recovering from edge cases in the robustness test.The test initializes the car at ±30° rotation and ±2 m translation from lane center and measures recovery success.Source: Table III · p. 6, C. Online real-world test · p. 6
  • Event cameras see only road-boundary components nonparallel to vehicle ego-motion, whereas RGB and LiDAR provide lane-following information at every step.Event policies were 8x more jittery than RGB; branching reduced event-policy interventions from 28.0±3.6 (IL) and 18.0±1.0 (GPL) to 6.5±3.8.Source: C. Online real-world test · p. 6

VI. CONCLUSION

  • VISTA supports RGB, 3D LiDAR, and event-camera synthesis, and policies learned in it transfer directly to a full-scale vehicle with consistent simulation and real-world results.The conclusion identifies VISTA as open source and describes its sensor synthesis as high fidelity and suitable for policy learning and evaluation.Source: §VI · p. 6
Loading 2111.12083v1…