PaperlayerView exact revision on arXiv

Source-linked AI summary

LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving

Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, Raquel Urtasun

arXiv:2101.06547v3cs.ROcs.AIcs.CVcs.LG

TL;DR

LookOut addresses limited mode coverage and sample inefficiency in forecasting uncertain, interacting traffic futures. It predicts diverse joint scene futures, estimates their probabilities, and plans contingencies, achieving safer, less-conservative driving with improved forecast diversity and reconstruction trade-offs.

  • Problem

    Traffic forecasting must handle rule violations, erratic behavior, and many interaction-dependent future scenes, while existing motion forecasting has limited mode coverage and sample efficiency.

  • Method

    LookOut generates diverse joint multi-agent trajectory samples, scores their probabilities, and optimizes contingency plans whose diversity reflects distinct ego-vehicle reactions.

  • Results

    LookOut drives safer and less conservatively than previous state-of-the-art models, while achieving better reconstruction at similar diversity or higher diversity at similar reconstruction.

  • Takeaways & Limitations

    Planning over diverse, probability-weighted future realizations supports defensive reactions to hazards without overreacting to low-probability dangers.

  • Takeaways & Limitations

    The model assumes a fixed prior over latent scene dynamics and Gaussian latent variables for its forecasting and diverse sampling distributions.

Abstract

from arXiv · show

In this paper, we present LookOut, a novel autonomy system that perceives the environment, predicts a diverse set of futures of how the scene might unroll and estimates the trajectory of the SDV by optimizing a set of contingency plans over these future realizations. In particular, we learn a diverse joint distribution over multi-agent future trajectories in a traffic scene that covers a wide range of future modes with high sample efficiency while leveraging the expressive power of generative models. Unlike previous work in diverse motion forecasting, our diversity objective explicitly rewards sampling future scenarios that require distinct reactions from the self-driving vehicle for improved safety. Our contingency planner then finds comfortable and non-conservative trajectories that ensure safe reactions to a wide range of future scenarios. Through extensive evaluations, we show that our model demonstrates significantly more diverse and sample-efficient motion forecasting in a large-scale self-driving dataset as well as safer and less-conservative motion plans in long-term closed-loop simulations when compared to current state-of-the-art models.

1. Introduction

LOOKOUT addresses the challenge of uncertain, multi-modal, and safety-critical traffic futures by predicting diverse scene-level outcomes and planning shared short-term actions with contingent responses. Experiments report safer, less-conservative driving and improved diversity–reconstruction trade-offs.

  • Motivation: Traffic futures are difficult to forecast because human behavior can be erratic and actor interactions create many possible scene-level outcomes.At a merge, for example, either actor may yield, producing distinct joint futures.
  • Motivation: Independent actor predictions miss scene consistency, while joint methods require many samples to cover long-tail futures that may be critical for safety.Existing diversity objectives can spend samples on actors or actions irrelevant to the SDV’s behavior.
  • Planning gap: Existing planners often optimize independently sampled actor marginals or one long-term trajectory, ignoring mutually incompatible futures and the value of later replanning.A planner may otherwise optimize an expected trajectory even though only one consistent scene future will occur.
  • Approach: LOOKOUT predicts diverse, scene-consistent futures while directly optimizing diversity of downstream ego plans, then scores scenarios so rare hazards inform planning without excessive conservatism.Its contingency planner shares an initial short-term action while preparing separate long-term responses for different futures.
  • Results: LOOKOUT achieves significantly safer and less-conservative driving, while also offering better reconstruction at similar diversity and higher diversity at similar reconstruction quality.The evaluation spans large-scale open-loop and closed-loop experiments across complex scenarios.

2. Related Work

The related work spans modular autonomy pipelines, independent and joint motion forecasting, and optimization-based motion planning. LOOKOUT builds on these directions by combining scene-consistent forecasting with planning that handles unlikely but safety-critical futures.

  • Autonomy pipelines: Modular autonomy pipelines offer safety, interpretability, error tracing, and data-efficiency advantages over black-box end-to-end models.The paper focuses its review on cascading detection, forecasting, and planning systems.
  • Motion forecasting: Motion forecasting methods range from independent actor models to joint stochastic models that sample scene-level futures through latent variables and trajectory decoders.Joint approaches include autoregressive and implicit latent-variable models.
  • Motion planning: Motion planners generate safe, comfortable, goal-progressing trajectories by minimizing costs through continuous optimization, sampling, or search.Collision cost is one component used to promote safety.
  • LOOKOUT positioning: LOOKOUT’s inference pipeline trains the backbone, actor, and prediction decoder first, followed by the diverse sampler and scenario scorer.The staged design connects joint forecasting with planning-centric diversity and scenario probability estimation.
  • Risk-aware planning: Planning methods for rare safety-critical scenarios seek trajectories that can react safely while also preserving progress and comfort, rather than always braking preemptively.This motivates planning responses that account for rare futures without treating every one as certain.

3. Diverse Prediction and Planning

LookOut combines joint scene-level prediction with a planning-centric diverse sampler and contingency planning. The system predicts multiple consistent futures, scores their probabilities, and selects an immediate action that remains safe while preserving future-specific reactions.

  • 3.1. Joint Perception and Motion Forecasting: The joint perception-and-forecasting model represents all actors’ future trajectories with an implicit latent-variable decoder driven by a spatially aware GNN.Actor contexts are extracted from LiDAR and map inputs, while latent variables capture scene dynamics and the decoder produces scene-consistent trajectories.
  • 3.1. Joint Perception and Motion Forecasting: The baseline inference procedure samples K latent variables independently from the prior and decodes the resulting futures in parallel, but remains sample-inefficient.The decoder is expressive and supports parallel decoding, yet broad coverage requires many prior samples.
  • 3.2. Planning-Centric Diverse Sampler: The planning-centric sampler replaces prior Monte Carlo sampling with a learned mapping from scene contexts to a compact set of latent samples whose futures provide diverse planning-relevant coverage.Its diversity objective favors distinct scenarios requiring different SDV reactions, such as a vehicle cutting in versus staying in its lane.
  • 3.3. Scenario Probability Estimation: The model assigns probabilities to the diverse futures by scoring each scenario with a GNN and renormalizing the scores to match an approximate categorical distribution.The scoring function is trained against the approximate scenario distribution using KL(pψ∥q), despite observing only one ground-truth realization per training example.
  • 3.4. Contingency Planner: Contingency planning selects a common short-term action followed by future-specific trajectories, avoiding both expectation-based commitment and worst-case over-conservatism.The immediate action must be safe across predicted realizations, while later branches are optimized for individual futures and executed through model predictive control.

4. Experiments

Experiments evaluate LOOKOUT on large-scale open-loop forecasting and closed-loop planning, including ablations of its sampler, planning energy, scenario scorer, and contingency planner. LOOKOUT achieves safer, less-conservative behavior while maintaining strong reconstruction and diversity.

  • Experimental Setup: Experiments use ATG4D for open-loop forecasting and a separate closed-loop LiDAR simulator for end-to-end driving evaluation.The simulator uses challenging scenarios from real logs that do not overlap with ATG4D scenes, and unrolls approximately 18 seconds.
  • Experimental Setup: Closed-loop baselines use the PLT planner because their forecasting models do not provide motion planners.Supplementary results pair the baselines with LOOKOUT’s contingency planner.
  • Comparison against state-of-the-art: 21% higher progress per collision than CVAE + PLT combines 8% fewer collisions with 12% greater progress.This result indicates safer driving without reducing route progress.
  • Comparison against state-of-the-art: LOOKOUT achieves the safest plans, most progress, and best prediction reconstruction while remaining among the most diverse methods.Baselines often trade greater diversity for less safety, progress, or reconstruction quality, whereas LOOKOUT avoids this diversity trade-off.
  • Ablation Study: The diverse sampler preserves nearly the same comfort and progress as independent Monte Carlo sampling while avoiding significantly more collisions.The ablation compares LOOKOUT’s diverse sampler with independent sampling from the prior.
  • Ablation Study: The planning diversity energy lowers collision rate by 28%, while improving jerk but regressing lateral acceleration.The authors hypothesize that it favors early preventive lateral displacement over late hard braking.
  • Ablation Study: Scenario scoring improves safety and progress by preventing unnecessary premature braking for low-probability risks.The ablation removes scenario scoring and assigns equal probabilities to diverse scenarios.
  • Ablation Study: Replacing the PLT planner with the contingency planner reduces collisions by 38% and improves almost every metric.Closed-loop examples show contingent trajectories responding safely to different predicted futures while permitting non-conservative immediate actions.

5. Conclusion

LOOKOUT combines joint multimodal forecasting with contingency planning to produce diverse predictions and safer SDV trajectories. Experiments show safer and less-conservative driving than previous state-of-the-art models.

  • Conclusion: LOOKOUT jointly models actor trajectories, targets rare SDV-relevant behavior, estimates scenario probabilities, and plans contingently across diverse futures.The system generates multimodal samples from a joint distribution and optimizes decision-making over those samples.
  • Conclusion: Experiments on a large-scale dataset and closed-loop simulations demonstrate safer, less-conservative driving than previous state-of-the-art models.

A. Implementation Details

The implementation section introduces the autonomy model’s submodules and training details.

  • Implementation Details: The section covers implementation details for the submodules of the end-to-end driving model and their training.

A.1. Joint Perception and Motion Forecasting Details

The joint perception and forecasting model processes LiDAR and map features, extracts actor contexts, models scene interactions, and decodes future waypoints.

  • Data Input Parameterization: The model uses voxelized LiDAR in bird’s-eye view, with past sweeps aligned to the current SDV frame to provide motion information.
  • Shared Perception Backbone: Separate LiDAR and HD-map convolutional backbones are fused into a shared multiresolution feature map for detection and forecasting.The backbone output is downsampled relative to the input and processed by a convolutional header.
  • Object Detection Header: Object detection predicts confidence scores and bounding boxes, then filters detections with non-maximal suppression and a probability threshold.
  • Actor Feature Extraction: Rotated ROI Align extracts actor-centered local context, which is combined with global centroid and rotation features.The local crop covers a fixed spatial region around each actor and is encoded into a 512-dimensional vector.
  • Scene Interaction Module: The scene interaction module applies a three-layer MLP to actor-node states and projected inter-actor distances within a graph.SIM is used throughout the prior, encoder, decoder, diverse sampler, and scenario scorer.
  • Scene Interaction Module: Gaussian latent samples are concatenated with actor features and decoded into 20 outputs representing x,y waypoints over 10 time steps.

A.2. Planning-Centric Diverse Sampler Details

The implementation combines a diverse sampler with Frenet-frame trajectory sampling and a planner cost function covering safety, traffic rules, progress, and comfort.

  • Diverse sampler network: The diverse sampler replaces the encoder and prior networks while using two identically structured SIMs for scene-level latent vectors.One SIM processes the A vector and the other processes the B vector.
  • Diverse sampler network: The model predicts only diagonal A matrices because full matrices would exceed memory for joint scene latent representations.The higher dimensionality arises because latent vectors represent all actors jointly rather than one actor marginally.
  • Scenario probability estimation: Scenario probabilities are estimated from the S predicted scenarios using one message-passing round, average pooling, and an MLP producing S scores.Each actor node is initialized with its S predicted trajectories.
  • Trajectory sampling: The planner samples lateral and longitudinal trajectories in the Frenet frame of the route’s goal motion-path using quintic-polynomial lateral profiles.Lane-center motion-paths provide strong priors for potential SDV paths.
  • Planner costs: Planner costs encode collision and headway safety, lane and traffic-rule compliance, route progress, and vehicle comfort constraints.Comfort penalties include curvature, acceleration, deceleration, lateral acceleration, jerk, and dynamic or kinematic violations.
  • Training: Training uses Adam at learning rate 1.25e-5 for 50,000 iterations on the forecaster and 40,000 iterations on the diverse sampler.The forecaster uses batch size 4 on 16 Nvidia RTX 5000 GPUs, while the diverse sampler uses batch size 1 because of memory constraints.

B.1. Operating point for evaluation

Evaluation fixes detector operating points for forecasting and planning, then measures scene-level reconstruction, diversity, and baseline configurations using shared perception components.

  • Operating points: Motion forecasting is evaluated at detector recalls of 90% for vehicles, 60% for bicyclists, and 70% for pedestrians.The operating points are selected on true-positive detections using common recall targets across models.
  • Operating points: Planning evaluation operates each baseline at its maximum F1 point on the detector precision-recall curve.This operating point is chosen to minimize false positives and false negatives.
  • Forecasting metrics: minSADE measures closest-scene ground-truth recall, while meanSADE measures the average scene-level difference between predicted scenes and ground truth.The metrics evaluate reconstruction quality at the scene level.
  • Forecasting metrics: The scene metrics use N actors, T timesteps, S scene samples, predicted trajectory y_n,s, and ground-truth trajectory y_n,GT.These quantities define the actors, temporal horizon, samples, and trajectories used in the metric equations.
  • Diversity metrics: meanPlanASD measures pairwise distance between contingent plans, while meanSASD and minSASD measure pairwise diversity among scene samples.The planning-centric diversity metric evaluates whether distinct predicted futures induce distinct SDV plans.
  • Baselines: Baselines include actor-independent and scene-consistent models, including MultiPath, CVAE, DPP, DLow, ESP, and ILVM.All baselines share the perception backbone, object detection, and actor feature extraction models.

C. Additional Evaluation Results

Additional evaluations show that LOOKOUT retains strong safety and progress when baselines use the contingency planner, achieves a favorable accuracy-diversity tradeoff, and has a pedestrian and bicyclist limitation.

  • Closed-loop planning: The contingency planner increases progress and decreases acceleration and deceleration for all baselines, while increasing lateral acceleration.LOOKOUT retains the best safety and progress per collision, with similar values on other metrics to ILVM with the contingency planner.
  • Motion forecasting: LOOKOUT has the lowest vehicle minSADE and meanSADE while achieving 61% greater meanSASD than ILVM.Vehicles constitute the vast majority of actors in the evaluated road scenes.
  • Motion forecasting: For pedestrians and bicyclists, LOOKOUT maintains a competitive accuracy-diversity tradeoff but does not improve accuracy while preserving diversity without vehicle regression.Improving this tradeoff remains future work.
  • Planning versus diversity: LOOKOUT achieves the safest closed-loop plans and a 3% absolute safe-rate improvement over MultiPath, equivalent to a 24% lower collision rate.MultiPath is the only baseline with greater planning diversity in this comparison.
  • Planning versus diversity: LOOKOUT trades some progress for caution under uncertainty, whereas unrealistic or insufficiently realistic trajectory diversity can reduce progress.The latter issue is exemplified by MultiPath, whose SDV struggles to avoid all predicted actor trajectories.

D. Additional Visualizations

The visualizations illustrate how LOOKOUT represents interaction-dependent futures and converts them into cautious, comfortable contingency plans while preserving relevant prediction diversity.

  • Contingency planning: In an intersection where a left-turning vehicle may merge into the SDV lane, LOOKOUT predicts speed variation and plans safe alternatives for going before or after it.The SDV maintains a comfortable velocity while postponing commitment until the other vehicle’s timing is clearer.
  • Contingency planning: For uncertain unprotected left turns, the SDV takes a cautious immediate action that permits either proceeding or braking after the other actor’s behavior becomes clear.Scenarios 2 and 6 illustrate this delayed-decision strategy.
  • Contingency planning: The planner slows when following a vehicle that might turn right, considers multiple speeds during an unprotected left turn, and avoids immediate hard braking in a narrow passage.These actions preserve comfortable progress while allowing later decisions as evidence arrives.
  • Sample diversity: LOOKOUT shows diverse future modalities for turning vehicles and strongly predicts forward acceleration for the SDV in one illustrated scene.The samples characterize possible actor trajectories over the forecast horizon.
  • Motion forecasting visualizations: LOOKOUT represents multiple modalities at both intersections in Scenario 1, while ESP and ILVM predominantly fit one mode per intersection.MultiPath also predicts multiple modalities in this scenario.
  • Motion forecasting visualizations: In Scenario 2, LOOKOUT varies bus speed and trajectory because the bus lies in the SDV’s path, whereas MultiPath produces many curb-entering predictions.The visualization emphasizes diversity in actor behaviors relevant to the SDV’s decision.
  • Figure guides: Figure 8 distinguishes the SDV, its immediate one-second action, and colored contingent trajectories for each possible future scenario.Figure 9 blends 15 future scenarios, with rainbow colors encoding time from 0 to 5 seconds.
Loading 2101.06547v3…