PaperlayerView exact revision on arXiv

Source-linked AI summary

TPNet: Trajectory Proposal Network for Motion Prediction

Liangji Fang, Qinhong Jiang, Jianping Shi, Bolei Zhou

arXiv:2004.12255v2cs.CV

TL;DR

Motion prediction must capture multiple plausible traffic-agent futures while respecting physical constraints, a combination that remains difficult for direct-regression methods. TPNet generates trajectory proposals, then classifies and refines them; across four datasets, it reports state-of-the-art results, though predictions may still leave drivable areas.

  • Problem

    Direct-regression motion predictors struggle to provide reasonable multimodal predictions while integrating traffic rules and movable-area constraints.

  • Method

    TPNet generates candidate trajectories for different intentions, then classifies and refines them while filtering proposals using movable-area constraints.

  • Results

    TPNet achieves state-of-the-art results on the ETH, UCY, ApolloScape, and Argoverse trajectory-prediction datasets.

  • Takeaways & Limitations

    The proposal-based pipeline supports multimodal and safer prediction while allowing prior knowledge to be incorporated into the deep learning method.

  • Takeaways & Limitations

    Predictions may still fall outside the drivable area, leaving room for improvement in the DAC metric.

Abstract

from arXiv · show

Making accurate motion prediction of the surrounding traffic agents such as pedestrians, vehicles, and cyclists is crucial for autonomous driving. Recent data-driven motion prediction methods have attempted to learn to directly regress the exact future position or its distribution from massive amount of trajectory data. However, it remains difficult for these methods to provide multimodal predictions as well as integrate physical constraints such as traffic rules and movable areas. In this work we propose a novel two-stage motion prediction framework, Trajectory Proposal Network (TPNet). TPNet first generates a candidate set of future trajectories as hypothesis proposals, then makes the final predictions by classifying and refining the proposals which meets the physical constraints. By steering the proposal generation process, safe and multimodal predictions are realized. Thus this framework effectively mitigates the complexity of motion prediction problem while ensuring the multimodal output. Experiments on four large-scale trajectory prediction datasets, i.e. the ETH, UCY, Apollo and Argoverse datasets, show that TPNet achieves the state-of-the-art results both quantitatively and qualitatively.

1. Introduction

Motion prediction for autonomous driving must model multiple plausible agent intentions while respecting traffic rules and movable areas. TPNet addresses this with a two-stage proposal-based framework that generates, classifies, and refines constrained trajectories.

  • Motivation: Traffic agents can have multiple plausible future paths, so prediction must represent multimodal intentions.Vehicle behavior may differ even with limited observations, such as turning right versus going straight.
  • Motivation: Predictions must incorporate physical constraints such as roads for vehicles and sidewalks or crosswalks for pedestrians.Reliable prediction therefore models both the agent’s past trajectory and traffic constraints.
  • Research gap: Earlier dynamic and statistical models become unreliable for long-term prediction because they are sensitive to observation noise and fail to model agent intentions.Deep models commonly regress future positions or distributions, while sampling-based multimodality remains difficult to make reasonable.
  • Proposed approach: TPNet first generates future trajectory hypotheses, then classifies and refines them to produce final predictions under traffic constraints.Different intentions are represented through diverse proposals, while movable-area knowledge filters unsuitable results.
  • Contributions: TPNet is presented as a unified framework for vehicles and pedestrians and reports state-of-the-art results on ETH, UCY, ApolloScape, and Argoverse.The framework is designed to incorporate prior knowledge while producing multimodal predictions that comply with traffic rules and conditions.

2. Related work

Prior motion-prediction research includes classic kinematic and statistical models as well as deep learning approaches. These methods model future locations, maneuvers, or environmental context through diverse parametric and learned representations.

  • Classic methods: Classic motion-prediction methods use kinematics equations, polynomial fitting, Gaussian processes, Gaussian mixtures, and Kalman Filters.These approaches target future location prediction and, in some cases, vehicle maneuver recognition.
  • Classic methods: Maneuver-recognition methods include Bayesian networks, Hidden Markov models, support-vector machines, and random-forest approaches.These methods complement statistical location-prediction models by representing vehicle maneuvers.
  • Deep learning methods: Deep learning approaches extract information from agents’ past positions, directions, vehicle context, and surrounding environments before directly regressing future positions.CNN encoder-decoder and perception-RNN architectures are among the described approaches.

3. Trajectory Proposal Network

TPNet uses a two-stage pipeline to reduce trajectory search complexity and produce interpretable, constrained predictions. It first generates proposals around a predicted endpoint, then selects and refines plausible trajectories.

  • Stage 1: Proposal generation: The first stage predicts a rough future endpoint to reduce the proposal search space and then generates trajectory proposals.The framework’s proposal-generation process supplies candidate future paths before final prediction.
  • Physical constraints: Proposals outside the movable area are further punished so physical constraints can influence the prediction process.The framework uses constrained proposal generation to support safe predictions.
  • Stage 2: Classification and refinement: The second stage classifies proposals to identify plausible future trajectories and refines them to preserve diversity in the final predictions.The highest-scoring proposal is used as the final prediction, while proposal quality is further improved through refinement.
  • Interpretability: Monitoring generated proposals makes the prediction process more interpretable and flexible than directly regressing an entire trajectory.Examining proposals also provides a way to inspect and explain possible erroneous predictions.

3.1. Base Feature Encoding Module

The Base Feature Encoding Module encodes the target agent’s observed positions together with optional surrounding-road information. Road semantics are represented as an image and processed using a ResNet-18 backbone.

  • Inputs: The module takes the target agent’s past positions and optional surrounding road information as inputs.Road information is related to the agent’s position and may include lane lines and crosswalks.
  • Input representation: Road information and the target’s past positions are encoded into an image representation.The target trajectory is drawn onto the semantic road image before feature extraction.
  • Feature extraction: A ResNet-18 backbone extracts features from the road semantic image within the encoder-decoder module.The encoder-decoder uses convolutional and deconvolutional blocks and is designed to accommodate different input features.

3.2. Proposal Generation

TPNet generates trajectory proposals using continuous polynomial curves, first from predicted endpoints and, when available, from road-based reference lines. Varying endpoints and curvature parameters produces compact, flexible hypotheses for different possible intentions.

  • Curve Representation: TPNet models trajectories as continuous curves and uses cubic polynomials to balance fitting accuracy with complexity.The average fitting error is 0.048 m for pedestrians on ApolloScape and 0.068 m for vehicles on Argoverse.
  • Base Proposal Generation: For each possible endpoint, varying γ generates curvature points and therefore curves with different shapes.γ controls the curve’s crook degree and is used with the endpoint and past points to form proposals.
  • Base Proposal Generation: Base Proposal Generation uses past positions and an N × N grid of possible endpoints centered on the first-stage predicted endpoint.The grid is defined by its interval and size N.
  • Proposal Construction: The resulting proposals are generated through a cubic polynomial fitting function from past points, possible endpoints, and curvature parameters.The base formulation uses possible endpoints selected from the endpoint set and γ values in [−2, −1, 0, 1, 2].
  • Multimodal Proposal Generation: Road-aware multimodal generation samples relative one-dimensional endpoint displacements along reference lines representing possible vehicle center lanes.Proposals are then generated for each sampled endpoint, reducing dependence on a single regressed endpoint.

3.3. Proposal Classification and Refinement

Given trajectory proposals, TPNet classifies their quality and refines their endpoint and curvature parameters. Training labels proposals by average distance to the ground-truth trajectory, while shared encoder-decoder features support both outputs.

  • Module Roles: The Classification Module selects the highest-scoring proposal, while the Refinement Module adjusts its endpoint and γ parameters.Together, the modules convert a proposal set into the final predicted trajectory.
  • Classification Module: Proposal quality is measured by the average distance between uniformly sampled ground-truth and proposal trajectory points.A proposal receives a positive label when its average distance is below a threshold such as 1 m.
  • Classification Module: Negative proposals are uniformly sampled to maintain a 3:1 negative-to-positive ratio and reduce the impact of overwhelming negative samples.Remaining proposals after positive assignment are potential negative samples.
  • Refinement Module: TPNet refines proposals using two coordinates and one variate, corresponding to endpoint coordinates and γ-related supervision.The supervised refinement variables are t_x, t_y, and t_γ.
  • Model Design: For each proposal, base and proposal features are concatenated before separate fully connected layers classify and refine it.The encoder-decoder extracts features for every proposal using the same module as the earlier framework stage.

3.4. Prior Knowledge

TPNet incorporates prior knowledge by defining a movable area from historical trajectories and high-definition maps, then penalizing proposals that extend outside it. Score decay preserves prediction diversity compared with discarding such proposals.

  • Motivation: Traffic and roadway priors constrain where agents can move, but DNN-based solutions cannot guarantee these constraints because of model complexity and limited explainability.Examples include vehicles tending to drive on roads and pedestrians using sidewalks or crosswalks.
  • Movable Area: The movable area is a polygonal region derived from the agent’s historical trajectory and high-definition maps.It represents the area where the agent can travel in the future.
  • Score Constraint: During inference, TPNet decays classification scores for proposals whose trajectory points lie outside the movable area.The decay uses the ratio of proposal points outside the area and a decaying factor σ.
  • Score Constraint: Score decay preserves prediction diversity better than abandoning predictions outside the movable area.The method penalizes such proposals rather than removing them outright.

3.5. Objective Function

TPNet is trained with a multi-task objective combining endpoint prediction, proposal refinement, and proposal classification. The loss also accounts for multimodality by pairing positive samples with randomly sampled negatives during refinement.

  • The objective uses predicted endpoints and proposal-level confidence and trajectory parameters against corresponding ground-truth targets.The loss includes predicted end point pe, ground-truth end point p∗e, proposal confidences ci, trajectory parameters ti, and ground-truth labels c∗i.
  • The multi-task loss combines endpoint prediction, trajectory refinement, and proposal classification objectives.Euclidean loss is used for endpoint prediction and refinement, while binary cross entropy is used for classification.
  • Refinement training uses positive samples together with part of randomly sampled negative samples to reflect multimodal future trajectories.A β term controls the ratio of sampled negatives.

4. Experiments

TPNet is evaluated across pedestrian, vehicle, and cyclist trajectory benchmarks using standard displacement, compliance, and multimodal metrics. Results show strong baseline comparisons, improved multimodal accuracy, and safer predictions after map-based proposal constraints, while ablations examine stage design and proposal-grid choices.

  • Evaluation Setup: TPNet is evaluated on ETH, UCY, ApolloScape, and Argoverse using ADE, FDE, weighted displacement, minimum displacement, and drivable-area compliance metrics.ETH and UCY evaluate pedestrian prediction; ApolloScape includes vehicles, pedestrians, and cyclists; Argoverse includes multimodal and safety metrics.
  • Baseline Comparison: On ETH and UCY, TPNet-1 already outperforms Social LSTM and Social GAN’s multimodal results, while TPNet-20 is competitive with all baselines.TPNet-1 uses the highest classification score; TPNet-20 selects the best prediction among the top-20 classification scores.
  • Baseline Comparison: TPNet outperforms baseline methods for every agent type on ApolloScape, with particularly strong vehicle trajectory prediction.The authors attribute the vehicle advantage to the curve representation being more compatible with vehicle trajectories.
  • Multimodal Prediction: Adding proposals with different intentions improves minADE by 60cm and minFDE by 1m on Argoverse.TPNet can also generate multimodal predictions without reference lines, and top-20 predictions outperform top-1 predictions on ETH and UCY.
  • Safety Guarantee: Map input improves TPNet’s results, but predictions can still leave the drivable area before safety-aware score decay is applied.The safety-constrained TPNet-map-safe variant reaches DAC 0.99.
  • Ablation Study: Ablations show that the complete two-stage pipeline improves FDE beyond curve sampling and endpoint refinement, while a 6m × 6m proposal grid performs best.Removing classification and regression yields 4.01m FDE; endpoint refinement improves FDE by 5cm, and the complete pipeline improves it by a further 8cm.

5. Conclusion

TPNet uses a two-stage pipeline that generates future trajectory proposals, then classifies and refines them. Proposals can represent different intentions and be filtered outside movable areas to support multimodal and safe motion prediction.

  • TPNet first generates possible future trajectories as proposals, then uses a DNN-based model to classify and refine them.
  • Generating proposals for different intentions realizes multimodal predictions.
  • Filtering proposals outside the movable area can ensure safe prediction.
Loading 2004.12255v2…