SceneGen: Learning to Generate Realistic Traffic Scenes

Shuhan Tan; Kelvin Wong; Shenlong Wang; Sivabalan Manivasagam; Mengye Ren; Raquel Urtasun

Source-linked AI summary

SceneGen: Learning to Generate Realistic Traffic Scenes

Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, Raquel Urtasun

arXiv:2101.06541v1Jan 16, 2021cs.CVcs.AIcs.LGcs.RO

TL;DR

Existing traffic simulators rely on heuristics that limit realism and leave a content gap between synthesized and real scenes, weakening their usefulness for self-driving development. SceneGen uses a neural autoregressive model conditioned on the ego-vehicle state and HD map to generate traffic scenes. Evaluations on two large-scale datasets show stronger modeling of real traffic scenes, while sensor simulation enables perception models that generalize to the real world.

Problem
Hand-crafted traffic-scene heuristics cannot represent the complexity and diversity of real scenes, limiting simulator fidelity for training and testing self-driving vehicles.
Method
SceneGen is a neural autoregressive model that conditions on the ego SDV state and HD map while sequentially inserting actors into a scene.
Results
On two large-scale self-driving datasets, SceneGen better estimates real traffic-scene distributions and generates more realistic samples than competing baselines.
Takeaways & Limitations
Coupling SceneGen with sensor simulation can produce realistic labeled data for training perception models that generalize to the real world.
Takeaways & Limitations
SceneGen assumes a fixed canonical ordering over the sequence of actors.

Abstract

from arXiv · show

We consider the problem of generating realistic traffic scenes automatically. Existing methods typically insert actors into the scene according to a set of hand-crafted heuristics and are limited in their ability to model the true complexity and diversity of real traffic scenes, thus inducing a content gap between synthesized traffic scenes versus real ones. As a result, existing simulators lack the fidelity necessary to train and test self-driving vehicles. To address this limitation, we present SceneGen, a neural autoregressive model of traffic scenes that eschews the need for rules and heuristics. In particular, given the ego-vehicle state and a high definition map of surrounding area, SceneGen inserts actors of various classes into the scene and synthesizes their sizes, orientations, and velocities. We demonstrate on two large-scale datasets SceneGen's ability to faithfully model distributions of real traffic scenes. Moreover, we show that SceneGen coupled with sensor simulation can be used to train perception models that generalize to the real world.

1. Introduction

Realistic traffic-scene generation is needed for scalable self-driving simulation, but heuristic-based methods cannot capture real-world complexity. SceneGen replaces those rules with a neural autoregressive model that generates realistic scenes and supports perception training.

SceneGen addresses the scalability challenge of creating realistic traffic scenarios for improving and validating self-driving vehicles.
Heuristic-based actor placement cannot fully capture the complexity and diversity of real traffic, creating a content gap between synthesized and real scenes.
SceneGen conditions on the SDV state and HD map, then sequentially inserts actors without hand-crafted rules or heuristics.
Across two large-scale self-driving datasets, SceneGen better models real traffic-scene distributions and generates more realistic samples than competing baselines.
Coupled with sensor simulation, SceneGen generates realistic labeled data that can train perception models generalizing to the real world.

2. Related Work

Traffic simulation has evolved from rule-based and behavior-focused models toward probabilistic and deep-learning approaches for generating realistic scene snapshots. SceneGen builds on autoregressive modeling to avoid the heuristics that limit existing methods.

Existing simulators often use predetermined actor locations and lane-following behavior, producing insufficient realism for testing self-driving vehicles.
Traffic-scene generation complements microscopic behavior simulation by producing realistic snapshots for initializing simulations and creating labeled perception-training data.
Probabilistic traffic-scene models learn distributions from real data but have commonly been limited to restricted settings such as highways or intersections.
Deep-learning methods such as MetaSim reduce manual tuning but retain scene-graph heuristics and assumptions that constrain scene complexity and diversity.
Autoregressive models factorize complex joint distributions into conditional distributions, enabling SceneGen to insert actors sequentially.

3. Traffic Scene Generation

SceneGen models traffic scenes as an autoregressive conditional distribution over actors, using the SDV state and HD map to sequentially generate realistic scene layouts and actor attributes.

3.1. The Autoregressive Generation Process: The model conditions generation on HD-map semantics, including lane boundaries, drivable areas, and traffic-light states, to produce actors consistent with road topology.The map is represented as polygons and polylines that provide contextual priors around the SDV.
3. Traffic Scene Generation: Each actor is represented by a class, bird’s-eye-view location, oriented bounding-box attributes, and velocity, with pedestrians represented by a center point.The actor parameterization includes vehicles, pedestrians, and bicyclists, while the stopping token has no spatial attributes.
3.1. The Autoregressive Generation Process: SceneGen autoregressively factorizes the conditional distribution over scene actors, simplifying dependencies among actors and with the map while supporting sequential generation.The process uses a canonical actor ordering and a stopping token to handle the random number of actors.
3.1. The Autoregressive Generation Process: A recurrent ConvLSTM processes a bird’s-eye-view encoding of the SDV, HD map, and previously generated actors to predict the next conditional distribution.The recurrent features summarize the generated scene so far and feed subsequent actor predictions.
3.2. A Probabilistic Model of Actors: The actor distribution factorizes into class, location, bounding box, and velocity terms, conditioning later attributes on previously sampled attributes.The stated factorization makes location class-dependent, bounding-box attributes dependent on class and location, and velocity dependent on class, location, and box.
3.2. A Probabilistic Model of Actors: SceneGen models locations with class-specific quantized categorical distributions and uses mixture distributions to capture multimodal actor sizes.Uniform quantization supports multimodal position distributions, while bivariate log-normal mixtures model width and length variation such as sedans versus trucks.

4. Experiments

SceneGen is evaluated on two self-driving datasets using likelihood, distributional, qualitative, ablation, and sim2real analyses. It achieves the strongest reported modeling results, generates diverse scenes, and produces perception-training data with the lowest sim2real gap.

4.1. Datasets: The evaluation covers Argoverse and ATG4D, whose maps provide different subsets of lanes, drivable areas, crosswalks, and traffic-light information.ATG4D contains 5,500 logs, while Argoverse provides 13,122 training and 5,015 validation scenes.
4.3. Results: SceneGen achieves the best NLL and MMD results across ATG4D and Argoverse, while learning-based methods outperform hand-tuned baselines on deep-feature MMD.These results indicate stronger modeling of real traffic-scene distributions than the competing methods.
4.3. Results: SceneGen samples better reflect real-scene complexity than baselines, although it occasionally produces plausible but unlikely near-collision scenes.MetaSim and Lane Graph are constrained by heuristic lane-following behavior, while LayoutVAE struggles to position actors accurately on the map.
4.5. Ablations: Increasing mixture components consistently lowers NLL, while adding lanes, drivable areas, and crosswalks generally improves NLL; traffic lights slightly degrade performance.The ablations test distributional flexibility and progressively richer map conditioning in ATG4D.
4.6. Discovering interesting scenes: SceneGen searches for high normalized NLL scenes to discover unusual cases, including an ATG4D traffic violation with violating-actor NLL 21.28.The procedure uses likelihood as a mechanism for identifying unlikely scenes.
4.4. Sim2Real Evaluation: SceneGen scenes yield the lowest sim2real gap when used with LiDAR simulation to train detectors evaluated on real ATG4D scenes.The comparison uses 250,000 generated scenes per method and simulated LiDAR for perception training.

5. Conclusion

SceneGen is a neural autoregressive traffic-scene model that supports both sampling new scenes and evaluating existing-scene likelihoods. By avoiding hand-crafted rules and heuristics, it models real-world traffic complexity and diversity for safer, more scalable self-driving simulation.

5. Conclusion: SceneGen samples new traffic scenes and evaluates the likelihood of existing ones with a neural autoregressive model.The model is presented as a flexible alternative to rule-based traffic-scene generation.
5. Conclusion: By eschewing rules and heuristics, SceneGen generates realistic traffic scenes and takes a step toward safe and scalable self-driving.The conclusion links the model’s realism to its intended simulation objective.

Supplementary Materials SceneGen: Learning to Generate Realistic Traffic Scenes

The supplied supplementary-materials extract lists the paper’s authors and identifies its arXiv posting date.

The listed authors are Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun.
The paper is identified as an arXiv computer-vision submission.
The arXiv version shown is dated 16 January 2021.

Abstract

The abstract identifies a realism gap in heuristic traffic-scene simulators and presents SceneGen as a neural autoregressive alternative. It reports stronger real-scene distribution modeling and real-world-generalizing perception training with sensor simulation.

Abstract: SceneGen addresses the limited complexity and diversity of heuristic traffic-scene generation with a neural autoregressive model.It conditions on ego-vehicle state and an HD map while inserting actors and synthesizing their sizes, orientations, and velocities.

1. Additional Model Details

SceneGen represents traffic scenes as rasterized map and actor features, then autoregressively predicts actor attributes with specialized probabilistic modules. It is trained by maximizing the likelihood of real scenes through the generation process.

Model architecture: SceneGen encodes the HD map, SDV, and previously generated actors in a bird’s-eye-view multi-channel image.The input covers an 80m × 80m region at 0.25m-per-pixel resolution, producing a 320 × 320 image.
Model architecture: A ConvLSTM backbone summarizes the partially generated scene before downstream actor-prediction modules.The backbone uses two ConvLSTM layers, while its features feed subsequent actor modules.
Actor modules: SceneGen predicts actor class, quantized location, bounding-box size, heading, and velocity with categorical and mixture distributions.Locations use 0.25m quantization; sizes use bivariate log-normal mixtures, while headings and velocity directions use Von-Mises mixtures.
Training: The training objective decomposes scene likelihood into per-actor class, location, bounding-box, and velocity likelihoods.The location term uses cross-entropy on quantized positions, with online negative hard-mining to address positive-negative imbalance.
Training: Training uses teacher forcing and backpropagation through time over up to 25 generation steps, distributed across 16 GPUs.Scenes are also randomly rotated during training.

2. Additional Experiment Details

The experiments compare SceneGen with probabilistic, procedural, graph-based, and variational baselines using likelihood and distributional metrics. Evaluation measures similarity between generated and real scene statistics, including learned motion features.

Baselines: The baseline suite includes Prob. Grammar, MetaSim, Procedural, Lane Graph, and LayoutVAE, spanning hand-crafted and learned scene-generation approaches.Prob. Grammar and Procedural use rule-based placement, while MetaSim and Lane Graph transform actor attributes with graph networks.
Baselines: Prob. Grammar places actors along lane segments using hand-tuned distributions for class, geometry, heading, velocity, and speed.Its parameters are tuned by hand, with actor placement and attributes sampled from lane-based priors.
Sampling analysis: The experiments also vary SceneGen’s number of sample proposals, evaluating M = 1, 10, and 20 on ATG4D.The analysis reports both motion-feature MMD and scene-statistics MMD.
Evaluation metrics: MMD compares generated and real distributions of actor classes, bounding-box sizes, speeds, and headings, while excluding empty scenes.ATG4D uses approximately 5,000 sampled scenes, whereas Argoverse uses all 5,015 validation scenes.
Evaluation metrics: A second MMD evaluates features extracted by a pretrained motion-forecasting model that predicts actor locations over the next three seconds.The feature extractor uses a bird’s-eye-view actor image and predicts locations at 0.5-second increments.

3. Additional Experiment Results

SceneGen remains strongest under vehicle-only evaluation, and increasing the number of sample proposals improves deep-feature realism. The sampling analysis also links higher proposal counts to more plausible traffic behavior.

Vehicle-only evaluation: SceneGen achieves the best vehicle-only MMD results on both ATG4D and Argoverse.This controlled comparison evaluates the class most easily handled by heuristics while testing whether SceneGen’s advantage persists.
Sampling strategy: Using more than one sample proposal decreases deep-feature MMD, indicating improved scene-level realism.The evaluated proposal counts are M = 1, 10, and 20.
Sampling strategy: With M = 1, generated vehicles can disregard traffic rules, whereas additional proposals produce more plausible qualitative scenes.The authors expect finer tuning of M could also improve actor-level statistics such as class, size, and speed.

4. Additional Qualitative Results

SceneGen produces diverse, rare, and spatially structured traffic scenes across ATG4D and Argoverse. Its samples include unusual but plausible maneuvers and multiple outcomes from identical inputs.

Qualitative comparisons: SceneGen generates rare but plausible scenes, including a vehicle three-point turn and bicyclists turning left in a car lane.These qualitative comparisons are shown against MetaSim, Lane Graph, and LayoutVAE on ATG4D and Argoverse.
Sample diversity: Given the same SDV state and HD map, SceneGen generates diverse samples reflecting multiple possible traffic outcomes.Examples include pedestrians crossing, an unprotected vehicle left turn, and a bus proceeding straight through an intersection.
Location distributions: The generation heatmaps show concentrated vehicle-location probabilities near lane centerlines and more diffuse pedestrian probabilities over crosswalks and sidewalks.Heatmaps are visualized at generation steps t = 0, 5, 10, 15, and 20.