Implicit Latent Variable Model for Scene-Consistent Motion Forecasting

Sergio Casas; Cole Gulino; Simon Suo; Katie Luo; Renjie Liao; Raquel Urtasun

Source-linked AI summary

Implicit Latent Variable Model for Scene-Consistent Motion Forecasting

Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, Raquel Urtasun

arXiv:2007.12036v1Jul 23, 2020cs.CVcs.LGcs.ROstat.ML

TL;DR

Autonomous vehicles need motion forecasts that represent uncertain, interacting futures consistently across traffic participants for safe planning. This paper introduces an interaction-graph implicit latent variable model with a distributed scene representation and deterministic decoder, achieving improvements across two datasets and safer, more comfortable planning trajectories.

Problem
Existing motion forecasts struggle to represent multiple plausible futures consistently across interacting actors from imperfect sensor data.
Method
The model represents unobserved scene dynamics in a distributed interaction-graph latent space and deterministically decodes latent samples into joint actor trajectories.
Results
The ILVM significantly outperforms state-of-the-art methods on ATG4D and nuScenes, including a 75% collision-rate reduction versus ESP on ATG4D.
Takeaways & Limitations
The forecasts produce scene-consistent samples and improve the safety and comfort of motion-planning trajectories.

Abstract

from arXiv · show

In order to plan a safe maneuver an autonomous vehicle must accurately perceive its environment, and understand the interactions among traffic participants. In this paper, we aim to learn scene-consistent motion forecasts of complex urban traffic directly from sensor data. In particular, we propose to characterize the joint distribution over future trajectories via an implicit latent variable model. We model the scene as an interaction graph and employ powerful graph neural networks to learn a distributed latent representation of the scene. Coupled with a deterministic decoder, we obtain trajectory samples that are consistent across traffic participants, achieving state-of-the-art results in motion forecasting and interaction understanding. Last but not least, we demonstrate that our motion forecasts result in safer and more comfortable motion planning.

1 Introduction

Safe autonomous driving requires forecasts that capture multiple plausible futures and interactions among all traffic participants. The proposed implicit latent variable model represents scene uncertainty jointly and generates scene-consistent trajectory samples for efficient planning.

Motivation: Multi-modal forecasts must capture actors’ goals, interactions, geometry, topology, and unobserved traffic rules.These factors jointly influence how an urban scene may evolve.
Motivation: Motion planners need trajectory distributions that cover ground truth while maintaining low enough entropy for comfortable progress.The system should reason about multiple futures and plan proactively around their interactions.
Challenge: Independent actor forecasts fail to produce scene-consistent futures, while autoregressive models require sequential sampling and can accumulate errors.Both alternatives address joint forecasting only with important limitations.
Approach: The ILVM models the scene as an interaction graph and distributes a latent representation across traffic participants using graph neural networks.The latent space summarizes unobserved scene dynamics from sensor data and supports decoding for all actors.
Approach: A deterministic decoder maps inputs and scene latent samples to trajectories, so stochasticity is captured by the latent variables and samples can be generated in parallel.The model produces scene-consistent trajectory samples while enabling efficient inference.
Results: The model significantly outperforms motion-forecasting state of the art on ATG4D and nuScenes while yielding less entropic distributions that better cover ground truth.These forecasts also support safer and more comfortable motion-planning trajectories.

2 Related Work

Related motion-forecasting methods differ in how they represent uncertainty, multi-agent interactions, and the joint trajectory distribution. The paper contrasts their limitations with a latent-space formulation that uses deterministic joint decoding and parallel sampling.

Perception and forecasting: Unified perception-and-forecasting backbones predict current and future states from LiDAR and map inputs but remain limited by unimodal and marginal actor distributions.Such distributions cannot provide trajectory samples consistent across actors.
Multi-agent forecasting: Perfect-perception multi-agent methods assume ground-truth histories, unlike self-driving systems affected by noisy detections, false positives, false negatives, and identity switches.Their input assumptions do not match imperfect onboard perception.
Joint-distribution approximations: Independent-factorization approaches offer efficient sampling but can have low expressivity, unstable optimization, limited coverage, or high memory demands.These trade-offs arise across Gaussian-mixture and non-parametric trajectory representations.
Autoregressive methods: Autoregressive methods model multi-agent behavior sequentially, but inference can suffer from compounding errors and is unsuitable for real-time self-driving applications.Training uses ground-truth previous states, whereas inference relies on approximate learned samples.
Proposed formulation: The proposed approach models interaction in a scene latent space and deterministically decodes all actors’ futures without output-level independence assumptions.This design provides efficient parallel sampling and more consistent trajectory samples across actors.

3 Scene Level Reasoning for Motion Forecasting

The model represents scene uncertainty with actor-anchored latent variables and a deterministic decoder, using graph-based message passing to generate socially consistent trajectories from sensor-derived features. Its architecture jointly extracts actor features, reasons over interactions, and predicts future trajectories through parallel sampling.

3.1 Implicit Latent Variable Model with Deterministic Decoder: The model defines future trajectories through latent scene variables Z that capture unobserved dynamics, including actor goals, interactions, and future traffic-light states.
3.1 Implicit Latent Variable Model with Deterministic Decoder: A deterministic mapping Y = f(X, Z) decodes sensor inputs and latent samples into trajectories, avoiding explicit factorization and enabling one-stage parallel sampling.Generation draws Z from p(Z|X) and then applies the deterministic decoder.
3.1 Implicit Latent Variable Model with Deterministic Decoder: The continuous latent Z is designed to capture all generative stochasticity, supporting high expressivity without separately modeling continuous uncertainty in P(Y|X,Z).
3.1 Implicit Latent Variable Model with Deterministic Decoder: The latent representation is distributed across actor-anchored variables z_n in an interaction graph, allowing capacity to scale with actor count while encoding scene-level information.Each z_n is anchored to actor n, but message propagation makes it dependent on the full scene input X.
3.2 Joint Perception and Motion Forecasting Architecture: The system extracts per-actor features from LiDAR and HD maps, then uses encoder, prior, and decoder modules built around a scene interaction module.The perception backbone detects traffic participants and applies RRoI Align to obtain individual actor representations.
3.2 Joint Perception and Motion Forecasting Architecture: The scene interaction module models traffic participants as nodes in a fully connected graph and uses spatially aware message passing to update their representations.Its message function uses terminal-node states and bounding-box coordinates, with feature-wise max-pooling for aggregation.
3.2 Joint Perception and Motion Forecasting Architecture: The encoder estimates q_φ(Z|X,Y) during training, while the prior estimates p_γ(Z|X) at inference time when future trajectories are unavailable.The encoder uses observed actor features and target futures; the prior initializes node representations from actor features alone.
3.2 Joint Perception and Motion Forecasting Architecture: The deterministic decoder is identified as the key contribution toward socially consistent trajectories, and the model achieves superior scene-level consistency in its joint forecasts.

4 Experimental Evaluation

The evaluation introduces scene-level metrics and compares ILVM with forecasting baselines on ATG4D and nuScenes, followed by qualitative, planning, and ablation analyses. ILVM improves scene consistency, precision, coverage, and downstream motion-planning safety and comfort.

Evaluation setup: The evaluation measures five-second trajectories on ATG4D and nuScenes, using scene-level metrics alongside established motion-forecasting measures.The study also evaluates downstream motion planning and performs ablations of the model components.
Scene-level metrics: Scene samples contain one future trajectory for every actor, while scene collision rate measures samples whose actor trajectories overlap during the forecast.Collision is determined from future bounding-box overlap exceeding a small IOU threshold at any timestep.
Motion forecasting results: 75% reduction in collision rate, 20% reduction in meanSFDE, and 19% reduction in minSFDE are reported on ATG4D versus the strongest collision-rate baseline, ESP.These results are reported with S = 15 samples and correspond to improved collision performance, precision, and ground-truth coverage.
Qualitative analysis: Qualitative samples show that ILVM captures yielding interactions at intersections and complex road-induced multimodal behavior that baselines miss.The visualizations include selected distinct samples and Monte Carlo marginal distributions from 50 samples.
Motion forecasting results: ILVM improves all reported metrics on nuScenes, with particularly significant gains in scene consistency and the precision metrics meanSADE and meanSFDE.The dataset provides a different distribution of motions and road topologies for testing generalization.
Motion planning results: ILVM forecasts enable the planner to produce safer and more comfortable trajectories, with similar progress but fewer collisions and lower 5-second distance to expert demonstrations.The planner uses 50 sampled future trajectories for each detected vehicle as obstacles when estimating plan costs.
Ablation study: A learned prior improves the precision-diversity trade-off, while graph-based encoders and decoders provide scene-level understanding absent from independent ablations.The ablation findings attribute large scene-level collision differences to the proposed graph-based components.

5 Conclusion and Future Work

The paper uses a latent variable model to represent dependencies among actors’ future trajectories, enabling fast parallel sampling and scene-consistent forecasts. It reports improvements on two challenging datasets and in motion planning, while leaving better use of joint trajectory distributions to future work.

The model represents dependencies among actor trajectories through an implicit joint distribution.
Deterministic decoding enables fast parallel sampling while assigning scene stochasticity to the latent variables.
The method improves scene-level sample quality and produces more precise, socially consistent predictions on two challenging datasets.
The forecasts significantly improve motion planning even though the planner does not explicitly use their strong scene consistency.
Future work should design a planner that better exploits joint trajectory distributions.

A.3 Analysis of Our Method

The method uses a deterministic decoder and replaces the likelihood-based reconstruction objective with a Huber loss on trajectory waypoints. Its β setting corresponds to a very small fixed variance, supporting the goal that the latent variable captures nearly all scene-level uncertainty.

The deterministic decoder makes trajectories functions of inputs and latent samples, so the latent variable captures the generative stochasticity.
The Huber waypoint loss corresponds to assuming a Gaussian or Laplacian conditional distribution with fixed diagonal covariance.
The constant covariance term does not affect optimization gradients with respect to the predicted mean.
β = 0.05 yields a fixed variance of 2.5cm, which is extremely small relative to typical five-second vehicle motion.

B Datasets

The evaluation uses ATG4D and nuScenes, which differ in vehicle platforms, LiDAR sensors, maps, cities, and driving environments. ATG4D provides large-scale North American data, while nuScenes covers Boston and Singapore with sparser LiDAR.

The benchmarks cover two datasets with different vehicle platforms, LiDAR sensors, maps, and cities.
ATG4D contains more than one million frames, 6,500 25-second snippets, precise 3D tracks, 64-beam LiDAR, and maps from several North American cities.
nuScenes contains 1,000 20-second snippets from Boston and Singapore, using 32-beam LiDAR and high-definition maps.
The nuScenes LiDAR is sparser than ATG4D’s, making object detection more challenging.

C Implementation Details

The implementation section describes component-level implementation details for the joint perception and motion forecasting model, followed by adaptations required for the baselines.

The section covers implementation details for each model component and adaptations needed for the baselines.

C.1 ILVM Details

The model combines sensor-derived LiDAR and map features with interaction-graph modules to infer latent scene dynamics and generate trajectory samples in parallel. Baseline adaptations use shared perception and actor-feature extraction, while autoregressive and explicit marginal-likelihood models require additional sampling or training adjustments.

Input representation: Voxelized LiDAR and rasterized HD-map streams are processed separately, concatenated, and fused into shared spatial features.The LiDAR representation uses BEV voxelization and compensated historical sweeps; map semantics are encoded in separate raster channels.
Scene interaction: The scene interaction module propagates edge messages from paired actor states and bounding-box coordinates, then aggregates them with feature-wise max-pooling.The module is used in the Prior, Encoder, and Decoder networks and is designed to tolerate graph-topology changes.
Latent motion forecasting: Prior and Encoder interaction modules predict latent means and sigmas, while the Decoder deterministically maps sampled latent representations to trajectory samples.The Prior and Encoder use two SIMs each, with 64-dimensional inputs, hidden states, and outputs.
Latent motion forecasting: Algorithm 2 samples S latent scene representations from the Prior and decodes each into a bird’s-eye-view scene trajectory sample.The output is a set of scene trajectory samples Y^1, Y^2, ···, Y^S for detected actors.
Baseline setup: For explicit marginal models, constant bivariate-normal noise across time produces temporally consistent waypoint offsets relative to predicted means and covariances.The reparameterized sample is y^s_n,t = μ_n,t + A_n,t · ε^s_n, where A is the covariance Cholesky factor.
Baseline setup: Baseline comparisons use the shared detection and actor-feature pipeline, with autoregressive models trained using noisy conditioning and marginal models adapted with mixture-of-trajectory outputs.A heuristic sampler is added for marginal models because their waypoint Gaussians are independent across time.

D Additional Evaluation Details and Results

The evaluation section reports additional detection and motion-forecasting results, including precision–recall analysis at IoU thresholds 0.5 and 0.7. The associated figure summarizes each model using mean Average Precision.

Evaluation overview: Figure 7 evaluates precision–recall at IoU 0.5 and 0.7, with the legend reporting mAP for each model.The horizontal axis begins at 0.7 recall.

D.1 Detection

ILVM achieves the best detection performance at both evaluated IoU thresholds. Because the compared models share the same backbone and detection head, the authors conjecture that the joint objective eases optimization of detection and motion forecasting.

Detection results: ILVM achieves the best detection performance at both IoU thresholds.The models use the same backbone and detection header, so the authors conjecture that the joint learning objective helps optimize both tasks.

D.2 Motion Forecasting

The motion-forecasting evaluation examines sample quality, scene consistency, detection–forecasting hit rate, error components, and diversity–precision tradeoffs. ILVM produces socially consistent samples, strong cumulative hit rates, and advantages concentrated in along-track forecasting while balancing diversity against precision.

Metrics and sample quality: Scene-level metrics address whether one predicted scene jointly explains multiple actors, unlike actor-level minFDE, which can select different trajectories per actor.minSFDE selects trajectories from the single predicted scene with the lowest average error across vehicles.
Scene consistency: ILVM’s scene collision rate barely increases as detector recall rises, whereas social-feature sharing alone is insufficient and joint-sampling models perform better.Higher recall introduces more low-probability actors and therefore more opportunities for predicted collisions.
Cumulative hit rate: ILVM significantly outperforms all baselines in cumulative hit rate across every time step of the prediction horizon.The metric combines true-positive detection with an L2 trajectory error threshold swept from 0.0m to 5.0m.
Error breakdown: ILVM’s primary error advantage is along-track forecasting, while cross-track accuracy is equal to or better than ESP and significantly better than other baselines.The along-track/cross-track breakdown distinguishes longitudinal from lateral error after projection into the ground-truth trajectory coordinate.
Precision–diversity tradeoff: During training, meanSFDE continues improving beyond 50k iterations while minSFDE reaches its optimum, illustrating a precision–diversity tradeoff.The evaluation emphasizes that a single ground-truth realization represents only one of multiple plausible futures.
KL ablation: Increasing the KL-term weight reduces multimodality and recall, whereas decreasing it produces high-entropy distributions with unrealistic samples and lower precision.The ablation links high beta to single-future predictions and low beta to broad coverage at the expense of realism.

E Additional Visualizations

The visualizations show that ILVM generates scene-consistent, diverse, and map-aware futures while supporting meaningful latent-space interpolation. Qualitative planning examples further associate its forecasts with safer ego-motion outcomes, though evaluation is limited to open-loop scenarios.

Scene Consistency: Each scene-visualization row compares models using two representative scene-level samples selected for maximum average pairwise Euclidean distance.This selection is intended to reveal how well models learn scene-level social interaction between agents.
Latent Space Interpolation: Linear interpolation between distinct latent samples produces semantically meaningful trajectories and smooth multi-agent variations, including speed changes and left-turn-to-U-turn behavior.The interpolations also preserve consistency among interacting vehicles.
Overall Sample Quality: Across additional urban scenarios, ILVM generally produces more accurate, less entropic distributions that better capture map topology and multi-agent interactions.The visualizations blend 50 sampled trajectories, with time encoded from 0s to 5s by a rainbow color map.
Ego-motion Planning: Qualitative open-loop planning examples show fewer ego-vehicle harmful events with ILVM forecasts, while baseline forecasts can induce aggressive maneuvers or collisions.The open-loop SDV relies entirely on forecasts for a 5-second horizon, and closed-loop experiments are out of scope.
Scene Consistency: ILVM samples capture conditional interaction outcomes, realistic lane changes, map geometry, and dynamically feasible futures across urban scenarios.Examples include yielding decisions, traffic-light-dependent outcomes, bus-versus-left-turn priority, and rare parking maneuvers.