Source-linked AI summary
TNT: Target-driveN Trajectory Prediction
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, Congcong Li, Dragomir Anguelov
TL;DR
Predicting moving agents’ futures is difficult because their intents and behaviors are unknown and multimodal. TNT represents moderate-horizon modes with target states, generates and scores target-conditioned trajectories, and achieves state-of-the-art performance on four real-world benchmarks.
Problem
Moving-agent prediction is challenging because future behavior is intrinsically multimodal and agent intent is unknown.
Method
TNT uses three end-to-end stages: target prediction, target-conditioned motion estimation, and trajectory scoring and selection.
Results
TNT achieves state-of-the-art performance on four challenging real-world vehicle and pedestrian trajectory-prediction datasets.
Takeaways & Limitations
Explicit target states provide an interpretable representation for multimodal trajectory prediction within the paper’s evaluated settings.
Takeaways & Limitations
The authors identify long-term future prediction as future work, proposing iterative prediction of intermediate targets and trajectories.
Abstract
from arXiv · showhide
Predicting the future behavior of moving agents is essential for real world applications. It is challenging as the intent of the agent and the corresponding behavior is unknown and intrinsically multimodal. Our key insight is that for prediction within a moderate time horizon, the future modes can be effectively captured by a set of target states. This leads to our target-driven trajectory prediction (TNT) framework. TNT has three stages which are trained end-to-end. It first predicts an agent's potential target states $T$ steps into the future, by encoding its interactions with the environment and the other agents. TNT then generates trajectory state sequences conditioned on targets. A final stage estimates trajectory likelihoods and a final compact set of trajectory predictions is selected. This is in contrast to previous work which models agent intents as latent variables, and relies on test-time sampling to generate diverse trajectories. We benchmark TNT on trajectory prediction of vehicles and pedestrians, where we outperform state-of-the-art on Argoverse Forecasting, INTERACTION, Stanford Drone and an in-house Pedestrian-at-Intersection dataset.
1 Introduction
TNT addresses multimodal trajectory prediction by representing future modes with explicit target states and then generating, scoring, and selecting target-conditioned trajectories. It is trained end-to-end and achieves state-of-the-art performance across four vehicle and pedestrian benchmarks.
- Framework: The framework encodes interactions with the environment and other agents before producing target-driven trajectory hypotheses.Its context includes scene elements and other moving agents relevant to future behavior.
- Motivation: Future prediction is difficult because agents’ intents and resulting behaviors are unknown, producing intrinsically multimodal futures.Possible vehicle behaviors include turning, continuing straight, slowing, speeding up, yielding, and changing lanes.
- Key insight: TNT captures moderate-horizon uncertainty with physically interpretable target states that correlate with agent intent.Targets can represent locations and behaviors such as lane changes or right turns.
- Framework: TNT predicts a distribution over discretized candidate targets, estimates target-conditioned trajectories, and scores and selects a compact diverse prediction set.The three stages are trained end-to-end and use likelihood scores for final selection.
- Results: TNT achieves state-of-the-art performance on Argoverse Forecasting, INTERACTION, Stanford Drone, and an in-house Pedestrian-at-Intersection dataset.The benchmarks cover both driving and pedestrian trajectory prediction.
2 Related Work
Prior trajectory-prediction methods model multimodal futures with latent-variable generative models or decomposed, interpretable subtasks. TNT is most closely related to approaches that discretize intents or trajectories, while differing in its use of explicit targets rather than latent endpoint modes.
- Implicit multimodality: Latent-variable methods model multimodal futures using conditional VAEs, flow-based generative models, or adversarial discriminators.Examples include DESIRE, PRECOG, and SocialGAN.
- Decomposed prediction: Other approaches decompose prediction into destination or goal estimation followed by planning or goal-directed trajectory generation.These methods include Bayesian destination prediction with inverse reinforcement learning and Gaussian goal representations.
- Endpoint-guided methods: Mangalam et al. generate endpoints to guide trajectories but retain CVAE latent variables to model endpoint modes.This distinguishes the concurrent endpoint-guided approach from TNT.
- Discrete representations: IntentNet manually defines motion categories, whereas MultiPath and CoverNet quantize trajectories or use anchors to represent output alternatives.Manual intent categories are task- and dataset-dependent and may miss within-category multimodality.
3 Formulation
TNT formulates future prediction by conditioning on plausible targets, separating intent uncertainty from control uncertainty. Its stages predict discrete targets, generate target-conditioned motions, and select a compact diversified set for downstream use.
- Problem formulation: The task predicts future states sF from observed agent states sP and contextual states cP over a fixed horizon T.The context contains interactions with other agents and scene elements.
- Uncertainty decomposition: Future uncertainty is decomposed into target or intent uncertainty and control uncertainty in the fine-grained motion toward a target.Turning left versus right illustrates intent uncertainty, while executing a turn illustrates control uncertainty.
- Probabilistic formulation: The target-conditioned formulation marginalizes over a context-dependent space of plausible targets T(cP).This expresses the overall future-state distribution as a mixture over target-conditioned trajectory distributions.
- Target representation: Discretizing target locations turns target-distribution estimation primarily into classification while allowing explicit target distributions and incorporation of road topology.Properly designed targets are intended to capture intent uncertainty, after which control uncertainty can use simple unimodal distributions.
- Framework stages: The three stages predict discrete target states, estimate unimodal motions to each target, and score a diversified final set of representative predictions.The scoring stage is designed for applications requiring a small set rather than the full future distribution.
4 Target-driveN Trajectory Prediction
TNT predicts multimodal futures by discretizing likely target locations, generating target-conditioned trajectories, and scoring and selecting a compact diverse set. Its target space is adaptable to vehicles and pedestrians, while end-to-end training connects context encoding with all three stages.
- Framework overview: TNT encodes scene context and uses three stages: target prediction, target-conditioned motion estimation, and trajectory scoring and selection.The final output contains K trajectories selected from M target-conditioned hypotheses.
- Target prediction: Targets are fixed-horizon locations represented through discrete candidate locations with continuous spatial offsets.The target predictor estimates a distribution over candidates and their likely offsets using scene context and target coordinates.
- Target prediction: Vehicle candidates are sampled from lane centerlines, whereas pedestrian candidates are sampled from a virtual grid around the agent.The choice of target space is application-dependent; vehicle sampling assumes vehicles remain near lanes.
- Target prediction: Discrete target prediction avoids mode averaging by representing alternative future locations separately rather than directly regressing a single future.TNT can over-sample candidates and retain the top M targets to balance target recall and efficiency.
- Target-conditioned motion estimation: Motion estimation predicts one future trajectory per target under conditional independence across time steps and a unimodal trajectory distribution given the target.The stage takes context and a target as input and outputs a most likely trajectory, trained with per-step Huber regression.
- Trajectory scoring and selection: Trajectory scoring evaluates complete trajectories rather than only targets or independently decomposed motion steps, then greedy selection rejects near-duplicates.The scoring stage uses a maximum entropy model, and selection retains trajectories sufficiently distant from those already chosen.
5 Experiments
Experiments evaluate TNT across vehicle and pedestrian datasets, examine implementation and ablations, and compare its performance with prior methods. TNT achieves state-of-the-art results across the reported benchmarks while retaining diverse predictions through target-based selection.
- Datasets: TNT is evaluated on Argoverse, INTERACTION, PAID, and Stanford Drone datasets covering vehicle and pedestrian trajectory prediction.Argoverse and INTERACTION provide driving scenarios, while PAID and Stanford Drone evaluate pedestrian prediction.
- Implementation Details: Context is encoded from vectorized map elements and trajectories, while vehicle and pedestrian targets are sampled from lanes or grids, respectively.Vehicle candidates come from lane centerlines or boundaries; pedestrian candidates come from cells in a surrounding 2D grid.
- Implementation Details: TNT uses three 2-layer MLP stages and is trained end-to-end with Adam for approximately 50 epochs.The hidden size is 64, with learning rate 0.001 and batch size 128.
- Performance Breakdown: S3 reduces the prediction set from M = 50 to K = 6 with little performance loss, while S1 provides target recall and S2 generates target-conditioned trajectories.The near-equal minFDE between S1 and S2 supports trajectory generation ending at the conditioned targets.
- Ablation Study: Denser target sampling improves performance before saturation, and target offset regression improves performance by 0.16m.For pedestrians, grid targets outperform map-based targets; for vehicles, targets are sampled along lane polylines.
- Ablation Study: Increasing CVAE samples by 10× only marginally improves minADE, supporting the assumption that motion is unimodal given a target.This comparison contrasts sampled multimodal motion estimation with TNT’s target-conditioned motion model.
- Comparison with State-of-the-Art: TNT outperforms prior methods and achieves state-of-the-art performance on vehicle and pedestrian benchmarks, with results on par with or better than the Argoverse Challenge 2020 winner.The comparisons include Argoverse, INTERACTION, PAID, and Stanford Drone evaluations.
6 Conclusion
TNT is presented as a novel, interpretable framework for multimodal trajectory prediction, with state-of-the-art performance across four challenging real-world datasets. The authors identify extending the framework to long-term prediction as future work.
- TNT comprises three interpretable stages: target prediction, target-conditioned motion estimation, and trajectory scoring.
- TNT achieves state-of-the-art performance on four challenging real-world prediction datasets.
- The SDD comparison table reports results in pixel units.
- Future work will extend TNT to long-term prediction by iteratively predicting intermediate targets and trajectories.