PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory Forecasts in Unknown Environments Conditioned on Grid-Based Plans

Nachiket Deo, Mohan M. Trivedi

arXiv:2001.00735v2cs.CVcs.RO

TL;DR

The paper addresses trajectory forecasting in unknown environments, where scene variability and multimodal futures make goal and path inference difficult. It conditions continuous trajectory generation on plans sampled from a reformulated grid-based MaxEnt IRL policy that jointly infers goals and paths. Evaluations on Stanford drone and NuScenes show diverse, scene-conforming forecasts over long horizons.

  • Problem

    Unknown-environment forecasting must handle varied scene structure and multimodal future trajectories while inferring goals and paths from available context.

  • Method

    P2T jointly infers goals and grid paths with MaxEnt IRL, then generates continuous trajectories conditioned on sampled plans.

  • Results

    Evaluations on Stanford drone and NuScenes show diverse trajectories that conform to the underlying scene over long prediction horizons.

  • Takeaways & Limitations

    Grid-based plans provide interpretable, multimodal structure for generating continuous forecasts aligned with plausible scene paths.

Abstract

from arXiv · show

We address the problem of forecasting pedestrian and vehicle trajectories in unknown environments, conditioned on their past motion and scene structure. Trajectory forecasting is a challenging problem due to the large variation in scene structure and the multimodal distribution of future trajectories. Unlike prior approaches that directly learn one-to-many mappings from observed context to multiple future trajectories, we propose to condition trajectory forecasts on plans sampled from a grid based policy learned using maximum entropy inverse reinforcement learning (MaxEnt IRL). We reformulate MaxEnt IRL to allow the policy to jointly infer plausible agent goals, and paths to those goals on a coarse 2-D grid defined over the scene. We propose an attention based trajectory generator that generates continuous valued future trajectories conditioned on state sequences sampled from the MaxEnt policy. Quantitative and qualitative evaluation on the publicly available Stanford drone and NuScenes datasets shows that our model generates trajectories that are diverse, representing the multimodal predictive distribution, and precise, conforming to the underlying scene structure over long prediction horizons.

I. INTRODUCTION

Trajectory forecasting in unknown environments must infer goals and paths from scene structure while representing nonlinear, multimodal futures. P2T addresses this by sampling grid-based plans with MaxEnt IRL and generating continuous trajectories conditioned on them.

  • Unknown-environment forecasting requires inferring agent goals and path preferences from scene layout without prior trajectory observations.
  • Scene variability, nonlinear motion, and multimodality make future trajectory prediction difficult over long horizons.
  • Regression-based forecasts can average distinct modes, producing trajectories that may violate the underlying scene.
  • P2T reformulates MaxEnt IRL to jointly infer terminal goals and paths on a coarse 2-D scene grid.
  • An attention-based generator produces continuous trajectories conditioned on sampled grid plans, and clustering yields K forecasts for downstream planning.
  • Qualitative examples show multimodal grid plans and continuous forecasts conforming to scene structure and past motion.

II. PRELIMINARIES

The preliminaries formulate grid-based trajectory forecasting as a finite-horizon MaxEnt MDP with known initial and goal states. The policy samples stochastic paths, while visitation frequencies support reward learning and expose the limitation that conventional formulations require a predefined goal.

  • The MDP uses scene-grid cells as states, four adjacent-cell actions, deterministic transitions, and a finite horizon.
  • Standard goal-conditioned MaxEnt IRL assumes known initial and goal states and learns rewards from demonstrated state-action sequences.
  • The MaxEnt policy assigns action probabilities through soft value iteration using state rewards and a fixed terminal goal.
  • Expected state visitation frequencies are propagated from the initial-state distribution and compared with demonstration visitation frequencies during reward learning.
  • Because real forecasting often has unknown goals, the goal-conditioned formulation cannot directly represent the required inference problem.
  • Conventional policy samples provide future paths but not timestamps, so MDP steps need not correspond to fixed time intervals or agent speeds.

III. PROPOSED APPROACH

P2T combines learned grid rewards, a goal-independent MaxEnt policy, and an attention-based trajectory generator. The resulting plans capture plausible multimodal goals and paths before continuous trajectories are decoded.

  • P2T’s reward model maps local scene context and track-history motion features to transient path rewards and terminal goal rewards on a coarse grid.
  • A goal-independent MaxEnt policy samples state sequences leading toward different plausible goals; each sampled sequence is treated as a plan.
  • The policy can generate multimodal grid plans without requiring predefined goal states.
  • An attention-based generator outputs continuous-valued trajectories conditioned on sampled plans.

A. Inferring goals and paths by learning rewards

The method reformulates MaxEnt IRL so a policy can infer both plausible goals and paths on a 2-D grid, without requiring a predefined goal state. The resulting policy can be sampled to produce plans toward different goals.

  • Potentially any grid cell can serve as either a path state or a goal state, and the policy terminates upon reaching a goal state.
  • The action space connects path states to adjacent cells through up, down, left, and right moves, with an end action transitioning to the goal state at the same location.
  • Transitions map path-state and action pairs to path or terminal goal states, with no transitions leaving a goal state.
  • The model learns separate path and goal reward functions, rpθ and rgθ, rather than relying on a single predefined terminal reward.
  • Modified value iteration uses learned goal rewards to induce a multimodal distribution over potential goals while propagating path rewards toward them.
  • The policy is sampled from the inferred-goal MDP to generate state sequences representing plans toward different plausible goals.

B. Reward model

The reward model uses convolutional and pooling layers to combine static bird’s-eye-view scene features with agent-motion and grid-location features. It outputs separate path and goal rewards at each grid cell and is designed to transfer across novel scene configurations.

  • Convolutional and pooling layers map local scene patches to path and goal rewards, supporting transfer to novel scenes with different scene-element configurations.
  • CNNfeat extracts features from the static bird’s-eye-view scene representation, while CNNp and CNNg map combined features to path and goal rewards.
  • Scene features have spatial dimensions matching the 2-D grid used for the state space.
  • Motion features include the agent’s speed and each grid cell’s agent-centric x and y locations, with the x-axis aligned to the current motion direction.
  • The scene is represented as a 200 × 200 bird’s-eye-view image, and CNNfeat uses the first two ImageNet-pretrained ResNet34 blocks.
  • After downsampling and convolution, the model produces a 25 × 25 grid with one path or goal reward value per cell from CNNp and CNNg.

C. Trajectories conditioned on plans

The model generates continuous trajectories conditioned on sampled grid-based plans and recent motion, using plan encoding and attention to align predictions with inferred paths and goals. It samples and clusters trajectories to represent multimodal futures with a configurable number of outputs.

  • Trajectory generation: Sampled plans are combined with track history to generate continuous-valued future trajectories over a prediction horizon.The trajectory generator is a recurrent encoder-decoder with soft attention.
  • Plan encoder: Plans encode scene features, nearby agent states, and grid-cell coordinates, then a bidirectional GRU aggregates the entire state sequence.The resulting plan representation supplies scene-structured information to the decoder.
  • Attention-based decoder: The attention-based decoder attends to selected plan states while generating trajectories, allowing alignment with different motion speeds.It can focus on earlier states for slow-moving agents and later states for fast-moving agents.
  • Sampling and clustering: Sampling trajectories from the MaxEnt policy represents multimodal path-and-goal distributions, while K-means clustering provides a concise set for downstream planners.The number of output clusters can vary without retraining the model.
  • Training: Training samples 200 plans and trajectories, clusters them into K outputs, and minimizes MinADEK, with K set to 10 for NuScenes and 20 for SDD.The model is also pretrained using the ground-truth plan and optimized with Adam at learning rate 0.0001.

A. Datasets

Evaluation uses two public trajectory datasets spanning diverse pedestrian, bicyclist, skateboarder, and vehicle behavior in campus scenes and complex urban traffic. Both datasets provide scene representations alongside tracked-agent trajectories.

  • Stanford drone dataset: The Stanford drone dataset contains pedestrian, bicyclist, skateboarder, and vehicle trajectories across 60 Stanford campus scenes.Its bird’s-eye-view scenes include roads, sidewalks, walkways, buildings, parking lots, terrain, foliage, roundabouts, and four-way intersections.
  • NuScenes dataset: NuScenes comprises 1000 twenty-second urban-traffic scenes recorded in Boston and Singapore with vehicle tracks and high-definition scene data.The benchmark split focuses on vehicle trajectories in varied road layouts and lane structures.

B. Metrics

The evaluation combines multimodal displacement metrics with scene-compliance metrics. MinADEK, MinFDEK, and miss rate reward at least one accurate prediction, while off-road and off-yaw metrics expose diverse but scene-inconsistent forecasts.

  • Metric comparison: Figure 5 shows that off-road and off-yaw rates penalize diverse but imprecise forecasts that displacement and miss-rate metrics fail to penalize.Warm colors indicate higher errors in the figure.
  • Metric limitations: MinADEK, MinFDEK, and MRK,2 can overlook implausible forecasts when one prediction is close to ground truth.A randomly diverse set can therefore achieve low values without conforming to the scene.
  • Scene-compliance metrics: Off-road rate measures the fraction of predicted points outside drivable or labeled path areas.NuScenes uses a drivable-area mask, while SDD uses manually labeled paths and obstacles.
  • Scene-compliance metrics: Off-yaw rate measures predicted yaw deviation from the nearest lane direction and ignores deviations up to 45° to preserve lane-change behavior.It addresses predictions that fall onto oncoming traffic or illegal lanes despite remaining on-road.

C. Comparison with the state of the art

On SDD, P2T achieves the lowest MinFDEK and is closely outperformed by HBA-Flow on MinADEK, while on NuScenes it reports state-of-the-art results on almost all metrics, including lower off-road and off-yaw rates.

  • Evaluation setup: The comparison covers prior and concurrently developed state-of-the-art models on both the Stanford drone and NuScenes datasets.NuScenes comparisons include benchmark baselines, challenge winners, MTP, and Multipath.
  • SDD: P2T achieves the lowest MinFDEK values on SDD, while HBA-Flow closely outperforms it on MinADEK.The SDD results use the split from [11] and report errors in bird’s-eye-view pixel coordinates.
  • SDD: P2T outperforms PECNet on the second SDD split, whereas Y-Net achieves lower MinADEK and MinFDEK values.Y-Net also conditions trajectories on goals and intermediate waypoints, highlighting the relevance of static-scene modeling.
  • NuScenes: P2T achieves state-of-the-art results on almost all NuScenes metrics, with significantly lower off-road and off-yaw values than previous methods.It maintains low MinADEK and miss-rate values, which the authors associate with diverse predictions and scene compliance.

D. Ablations

Ablations show that grid-based plans improve scene compliance, the attention-based generator improves trajectory accuracy, and MaxEnt IRL provides a smaller additional benefit over behavior cloning.

  • Grid-based plans: Removing grid-based plans causes the decoder to attend to all grid features rather than only features along sampled state sequences.The ablation removes both the MaxEnt policy and plan encoder and uses 2×2 max pooling to keep attention memory tractable.
  • Trajectory generator: P2TCS matches P2TIRL on off-road and off-yaw metrics but performs worse on MinADE, MinFDE, and miss rate.Its smoothing-spline and constant-speed construction remains scene compliant but deviates more from ground truth than the attention-based GRU decoder.
  • Reward layers: P2TIRL slightly outperforms behavior cloning P2TBC on most metrics, with the difference more prominent on SDD than NuScenes.P2TBC replaces learned reward layers with a policy that directly predicts action probabilities at grid cells.
  • Grid-based plans: P2TIRL outperforms the LVM across all ablation metrics on both SDD and NuScenes, while LVM has higher off-road and off-yaw rates.The authors interpret this pattern as evidence that grid-based plans provide an inductive bias toward scene-compliant trajectories.
  • Qualitative evidence: Qualitative NuScenes examples visualize inputs, goal state-visitation frequencies, path state-visitation frequencies, and predictions.These examples complement the quantitative ablations by showing how plans and predictions are represented.

E. Runtime

The complete model runs in 79 ms per prediction instance while sampling 1,000 state sequences and trajectories before clustering them into 10 outputs; qualitative examples show scene-aligned diversity.

  • Runtime: The complete model performs inference in 79 ms, or 12 Hz, for K=10 on an NVIDIA GeForce GTX 1080 Ti GPU.The implementation samples 1,000 state sequences from the MaxEnt policy, generates 1,000 trajectories, and clusters them into K outputs.
  • Qualitative behavior: Qualitative examples show MaxEnt policy exploration of plausible goals and paths across NuScenes and Stanford drone scenes.For NuScenes, explored states correspond to reachable lanes and distinguish travel-direction lanes from oncoming traffic.
  • Qualitative behavior: The predicted trajectories closely follow explored policy states, producing diverse and scene-compliant predictions across scene configurations.The examples show multimodal goal and path state-visitation frequencies, with SDD paths favoring roads while avoiding terrain or obstacles.
  • Runtime: The authors report that the proposed model can be deployed with onboard detectors, trackers, and HD maps for autonomous driving.This deployment statement accompanies the reported 79 ms inference time.
Loading 2001.00735v2…