PaperlayerView exact revision on arXiv

Source-linked AI summary

SimNet: Learning Reactive Self-driving Simulations from Real-world Observations

Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blazej Osinski, Hugo Grimmett, Peter Ondruska

arXiv:2105.12332v1cs.ROcs.CVcs.LG

TL;DR

Validating self-driving systems through road testing is costly, motivating realistic simulations that also react to the SDV. SimNet learns to generate driving episodes from historical data; its evaluation reports realism and reactivity and reveals a previously unreported causal-confusion issue in an ML planner.

  • Problem

    Scripted traffic is time-consuming and lacks road-testing fidelity, motivating a way to generate realistic simulations that react to SDV behavior.

    Source: §I · p. 1

  • Method

    SimNet learns to generate realistic, reactive driving episodes from historical observations using a trainable simulation system.

    Source: §I · pp. 1–2

  • Results

    SimNet performs well in realism and reactivity, and evaluating an ML planner uncovered a previously unreported causal confusion that log replay would not reveal.

    Source: C. Forward simulation · p. 5, D. Evaluating SDV with SimNet · p. 6, §VI · p. 6

  • Takeaways & Limitations

    The authors see SimNet as a step toward reducing the need for on-road testing and release its source code to stimulate simulation development.

    Source: §VI · p. 6

Abstract

from arXiv · show

In this work, we present a simple end-to-end trainable machine learning system capable of realistically simulating driving experiences. This can be used for the verification of self-driving system performance without relying on expensive and time-consuming road testing. In particular, we frame the simulation problem as a Markov Process, leveraging deep neural networks to model both state distribution and transition function. These are trainable directly from the existing raw observations without the need for any handcrafting in the form of plant or kinematic models. All that is needed is a dataset of historical traffic episodes. Our formulation allows the system to construct never seen scenes that unfold realistically reacting to the self-driving car's behaviour. We train our system directly from 1,000 hours of driving logs and measure both realism, reactivity of the simulation as the two key properties of the simulation. At the same time, we apply the method to evaluate the performance of a recently proposed state-of-the-art ML planning system trained from human driving logs. We discover this planning system is prone to previously unreported causal confusion issues that are difficult to test by non-reactive simulation. To the best of our knowledge, this is the first work that directly merges highly realistic data-driven simulations with a closed-loop evaluation for self-driving vehicles. We make the data, code, and pre-trained models publicly available to further stimulate simulation development.

I. INTRODUCTION

Validating self-driving vehicles through road testing is expensive, time-consuming, and difficult to reproduce, while log replay and scripted traffic do not provide realistic reactive behavior. The paper proposes learning realistic, reactive driving episodes from historical observations and uses the system to evaluate an ML planner.

I. INTRODUCTION

  • Road testing can require thousands or millions of miles and makes direct comparisons between system versions difficult because the same situation cannot be repeated.Industry testing uses safety drivers who take over in unusual or failure situations.Source: §I · p. 1
  • Log replay does not react when the SDV acts differently from the recorded run, while scripted traffic is time-consuming and lacks road-testing fidelity.A slight braking change during replay can produce an unrealistic collision with a trailing car.Source: §I · p. 1
  • The proposed system learns to generate realistic, reactive driving episodes from historical observations, including traffic responses such as slowing or overtaking when the SDV slows.The intended behaviors also capture stochastic multimodality observed on the road.Source: §I · pp. 1–2
  • The paper evaluates simulation realism and reactivity, examines training-data scale and usefulness for assessing an ML planner, and releases code and pretrained models.The authors describe the work as connecting simulation and planning with large-scale datasets in a realistic self-driving setting.Source: §I · pp. 1–2

II. RELATED WORKS

Prior work includes trajectory prediction, learned planning, and rule-based driving simulators. This paper instead uses simulation to evaluate an existing ML planner and learns traffic behavior from observations rather than relying on hand-coded actors.

II. RELATED WORKS

  • Motion simulation differs from one-shot motion prediction because it generates traffic-agent motion one timestep at a time while reflecting agents’ intentions and the motion of other participants.The passage situates this distinction alongside prediction methods including,, and [27].Source: §II · p. 2
  • Learned planning methods frame planning as learning from demonstrations, while earlier approaches also include expert-cost optimization, [29], and inverse reinforcement learning.The passage notes that engineering an expert cost function is complex and time-consuming.Source: §II · p. 2
  • The paper evaluates an existing ML planning system [1], reporting that it is prone to imitation-learning causal confusion that is difficult to observe without reactive simulation.The issue involves incorrectly associating other cars’ motion with the SDV’s desired action.Source: §II · p. 2
  • Rule-based simulators such as SUMO and CARLA use hand-coded actors that may lack realistic, varied behavior; this work learns traffic behavior from observed data.The authors state that accuracy improves with more training data and that collecting data is easier than engineering a realistic simulation.Source: §II · p. 2

III. SELF-DRIVING SIMULATION AS A LEARNING PROBLEM

The paper formulates self-driving simulation as generating realistic driving episodes that react to a given SDV control policy. Episodes comprise states of nearby traffic participants, whose behavior is stochastic while the SDV follows its known controller.

III. SELF-DRIVING SIMULATION AS A LEARNING PROBLEM

  • The simulation is intended to generate realistic episodes that react to the SDV’s control policy, using historical driving scenes over a semantic map.The scenes capture observed behavior across diverse driving scenarios.Source: §III · p. 2
  • Each episode is a sequence of states, with each state recording nearby traffic participants’ positions, rotations, sizes, and speeds.This representation matches the perception system’s vectorized detections that can be fed to the SDV control algorithm.Source: §III · p. 2
  • The SDV is modeled as a traffic participant, but its dynamics are controlled by the known function implementing the self-driving algorithm.Source: §III · p. 2
  • The simulation samples the joint behavior of stochastic traffic participants and deterministic SDV behavior to create new driving experiences.For a given initial state, multiple futures are possible as participants act and react to new information such as merging, nudging, or slowing.Source: §III · p. 3
  • Performance measures include collisions and traffic-rule violations, with attention to metric accuracy and false-positive versus false-negative events.Source: §III · p. 3

IV. GENERATING REALISTIC AND REACTIVE DRIVING EPISODES

The method samples driving episodes as a Markov process, using an initial-state distribution and learned participant transitions conditioned on prior states. It supports full, journey, scenario, and behavior simulations through different choices of initial conditions or participant paths.

IV. GENERATING REALISTIC AND REACTIVE DRIVING EPISODES

  • Episode sampling is formalized as a Markov process, with transition probabilities conditioned on the previous state.Source: §IV · p. 3
  • Participant actions are assumed locally independent given the previous state, reflecting participants controlling their own behavior while observing and reacting to others.Source: §IV · p. 3
  • The participant transition policy is controlled by a neural network that produces steering and velocity used to update participant position.Source: §IV · p. 3
  • Sampling selects an SDV map location, draws a feasible initial state with traffic-participant count and poses, then forward-simulates the episode using participant policy and SDV control.The process is summarized in Figure 2.Source: §IV · p. 3
  • Full simulation generates episodes from locations, journey simulation fixes the SDV’s initial location, and scenario simulation starts from a selected historical state.Source: §IV · p. 3
  • Behavior simulation can prescribe a traffic participant’s path while retaining some reactivity during execution, such as modeling another car cutting off the SDV.Source: §IV · p. 3

A. Initial state sampling

SimNet represents nearby traffic as a bird’s-eye-view state over a semantic map and uses a conditional generative adversarial network to sample initial scenes. Vehicle positions and rotations are extracted from the generated scenes.

A. Initial state sampling

  • The bird’s-eye-view state renders nearby traffic participants over a semantic map, capturing local context and a variable number of participants in one image.Source: A. Initial state sampling · pp. 3–4
  • A conditional generative adversarial network generates initial states from semantic-map scenes and can synthesize new states for any map location.It is trained on pairs of semantic maps and real states; the discriminator distinguishes synthetic states from real ones.Source: A. Initial state sampling · p. 4
  • A connected-components algorithm extracts vehicle positions and rotations by computing each component’s centroid and minimum bounding box.Source: A. Initial state sampling · p. 4

B. Forward simulation

SimNet advances a driving episode one step at a time, predicting each traffic participant’s motion while the SDV follows its control policy. The process repeats until the episode ends or simulation is interrupted.

B. Forward simulation

  • The generated sequence advances one step at a time by executing the participant policy for each agent and the SDV control policy.Source: B. Forward simulation · p. 4
  • Each step independently predicts motion for every traffic participant, while the SDV’s position is obtained by applying its control policy.The participant model uses a ResNet-50 backbone on bird’s-eye-view rasterized states and is trained to predict steering and velocity from past trajectories.Source: B. Forward simulation · p. 4
  • Figure 4 presents separate exemplar episodes, moving from sampled initial scenes to participant positions and then each vehicle’s trajectory.Source: Figure 4 · p. 4
  • The updated positions form the next state, and generation continues until the episode is complete or simulation is interrupted, such as by an SDV collision.Source: B. Forward simulation · p. 4

V. EXPERIMENTS

The experiments assess simulated episodes over five seconds, measuring realism and reactivity on disjoint sets of 960 scenes. SimNet is described as both realistic and reactive, with realism improving as training data increases.

V. EXPERIMENTS

  • The evaluation forward-simulates five seconds across many scenes, using two metrics evaluated on disjoint sets of 960 scenes.The horizon is chosen to capture maneuvers while retaining ground-truth annotations from tracked agents.Source: §V · p. 4

SIMULATION TRAINED ON VARIOUS AMOUNTS OF DATA FOR VARIOUS

  • More training data leads to more realistic simulations, with model-predicted paths compared against ground-truth paths in Figure 5.Source: Figure 5 · p. 5
  • Log replay is perfectly realistic but not reactive, whereas SimNet is described as both realistic and reactive.Table I presents realism and reactiveness for non-reactive and reactive simulation.Source: SIMULATION TRAINED ON VARIOUS AMOUNTS… · p. 5, Table I · p. 5
  • Realism is measured by average L2 displacement from ground-truth agent positions, while reactivity is measured by the share of synthetic scenes without a collision.The reactivity test places a static car ahead of a moving car; trailing cars should respond by stopping.Source: SIMULATION TRAINED ON VARIOUS AMOUNTS… · p. 5

A. Implementation Details

The experiments use the Lyft Motion Prediction Dataset and bird’s-eye-view raster representations centered on agents of interest.

A. Implementation Details

  • The Lyft Motion Prediction Dataset contains more than 1,000 hours of dense traffic episodes captured from 20 self-driving vehicles, and the authors follow its train, validation, and test split.Source: A. Implementation Details · p. 5
  • The rasterized state includes lanes, crosswalks, and traffic-light information, and the experiments focus on vehicles, pedestrians, and cyclists.Rasters are 224x224 and centered around each agent of interest.Source: A. Implementation Details · p. 5
  • The experiments use batch size 64 and Adam; the full pipeline takes around 400 ms per frame on a modern GPU.The authors describe this runtime as acceptable for a system without real-time constraints.Source: A. Implementation Details · p. 5

B. Initial state sampling

SimNet samples initial states and intended trajectories for traffic participants, with intersections used to showcase realistic scenes and diverse behaviors.

B. Initial state sampling

C. Forward simulation

SimNet performs well on realism and reactivity, with performance improving as training data grows. In planner evaluation, reactive simulation distinguishes rear-collision and passiveness errors and exposes a causal-confusion failure that log replay misses.

C. Forward simulation

  • SimNet performs very well on realism and reactivity, and performance improves with more training data.Table I reports quantitative results, while Figures 5 and 6 show qualitative comparisons of realism and reactivity, respectively.Source: C. Forward simulation · p. 5

D. Evaluating SDV with SimNet

  • The planner evaluation compares log replay with SimNet, and the reported rear-collision and passiveness errors differ substantially between the methods.The discrepancy raises whether reported errors reflect planner mistakes or artifacts of the evaluation method.Source: D. Evaluating SDV with SimNet · p. 5, Table II · p. 5
  • In a qualitative example, SimNet-controlled traffic stops behind a static vehicle while log replay crashes into it; the planner also waits when the chase-car cue is absent.Source: Figure 6 · p. 6
  • Reactive simulation eliminates both false-positive and true-positive log-replay rear-collision cases because chasing vehicles respond to a static or slower ego.This response can partially explain why SimNet reports higher passiveness errors than log replay.Source: D. Evaluating SDV with SimNet · pp. 5–6
  • Passiveness errors rise from 32 to 124, exceeding the 60 total rear collisions, as SimNet reveals the planner may wait at intersections when neighboring agents are static.The authors identify this as previously unreported causal confusion that log replay cannot uncover because neighboring agents move as recorded.Source: D. Evaluating SDV with SimNet · p. 6

VI. CONCLUSIONS

SimNet generates realistic, reactive driving episodes from historical logs and performs well on both realism and reactivity. Its use to evaluate a planner exposed causal confusion, a failure mode SimNet itself also exhibits.

VI. CONCLUSIONS

  • SimNet generates realistic and reactive driving episodes from historical logs to validate self-driving vehicle performance.Source: §VI · p. 6
  • Evaluation uncovered previously unreported causal confusion in ChaufferNet [1], and SimNet exhibits the same failure mode despite also using imitation learning.The authors identify addressing causal confusion as further work to improve both ML planners and simulators.Source: §VI · p. 6
  • The authors view the work as a step toward reducing the need for on-road testing in self-driving development.Source: §VI · p. 6

Related papers

Loading 2105.12332v1…