PaperlayerView exact revision on arXiv

Source-linked AI summary

HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attention

Xiaolong Tang, Meina Kan, Shiguang Shan, Zhilong Ji, Jinfeng Bai, Xilin Chen

arXiv:2404.06351v2cs.CV

TL;DR

Static trajectory forecasting independently predicts adjacent steps despite their overlapping histories, potentially producing instability and temporal inconsistency. HPNet uses historical predictions to model relationships between successive forecasts and extend attention beyond the visible history, achieving state-of-the-art performance on Argoverse and INTERACTION with accurate, stable trajectories.

  • Problem

    Static methods independently forecast adjacent steps despite overlapping histories, which may lead to instability and temporal inconsistency.

    Source: §1 · p. 2

  • Method

    HPNet combines historical prediction attention with agent and mode attention to use prior prediction embeddings when refining current multimodal forecasts.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    HPNet achieves state-of-the-art performance on Argoverse and INTERACTION and produces accurate, stable future trajectories.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Historical prediction embeddings let HPNet extend its effective observation window beyond the current visible range without increasing computational overhead.

    Source: §1 · p. 2, §3.3 · p. 5

Abstract

from arXiv · show

Predicting the trajectories of road agents is essential for autonomous driving systems. The recent mainstream methods follow a static paradigm, which predicts the future trajectory by using a fixed duration of historical frames. These methods make the predictions independently even at adjacent time steps, which leads to potential instability and temporal inconsistency. As successive time steps have largely overlapping historical frames, their forecasting should have intrinsic correlation, such as overlapping predicted trajectories should be consistent, or be different but share the same motion goal depending on the road situation. Motivated by this, in this work, we introduce HPNet, a novel dynamic trajectory forecasting method. Aiming for stable and accurate trajectory forecasting, our method leverages not only historical frames including maps and agent states, but also historical predictions. Specifically, we newly design a Historical Prediction Attention module to automatically encode the dynamic relationship between successive predictions. Besides, it also extends the attention range beyond the currently visible window benefitting from the use of historical predictions. The proposed Historical Prediction Attention together with the Agent Attention and Mode Attention is further formulated as the Triple Factorized Attention module, serving as the core design of HPNet.Experiments on the Argoverse and INTERACTION datasets show that HPNet achieves state-of-the-art performance, and generates accurate and stable future trajectories. Our code are available at https://github.com/XiaolongTang23/HPNet.

1. Introduction

Static methods independently forecast adjacent steps despite overlapping histories, a pattern that may cause instability and temporal inconsistency. HPNet models relationships between successive predictions using historical prediction embeddings, aiming for more accurate, stable forecasts and a longer attention range without added computational overhead.

1. Introduction

  • Independent forecasts from overlapping historical inputs may produce instability and temporal inconsistency, complicating safe and reliable autonomous-driving decisions.HPNet instead treats trajectory prediction as dynamic, incorporating historical predictions alongside historical frames.Source: Figure 1 · p. 1, §1 · p. 2
  • HPNet’s Historical Prediction Attention models a general relationship between successive predictions, allowing them to remain consistent, change slightly, or share a motion goal in complex situations.It is combined with Agent Attention and Mode Attention in Triple Factorized Attention.Source: §1 · p. 2
  • Historical predictions serve as references to improve stability and accuracy, while extending the visible attention range without increasing computational overhead in online inference.The longer range is presented as beneficial for accuracy in practical applications.Source: §1 · p. 2

2. Related Work

Related work considers attention across trajectory-prediction dimensions and approaches for producing multimodal futures. HPNet combines factorized attention with learnable mode queries in a three-part prediction pipeline.

2. Related Work

  • Applying attention separately across spatial and temporal axes can support semantic consistency and lower computational complexity than joint self-attention.HPNet’s Triple Factorized Attention models three dimensions through Historical Prediction Attention, Agent Attention, and Mode Attention.Source: §2 · p. 2
  • Deterministic methods produce multimodal trajectories in one shot, while anchor-based methods depend on anchor quality and learnable queries offer a more flexible alternative.Hybrid approaches combine sample-specific anchors with learnable mode queries or refine anchor-free proposals using anchor-based queries.Source: §2 · pp. 2–3
  • HPNet’s overview combines agent and lane features with mode queries, refines prediction embeddings through three attention modules, and decodes trajectories with an MLP.The predicted trajectories are fed into the pipeline again to enhance prediction precision.Source: Figure 2 · p. 3

3. Method

HPNet encodes historical agent and map context, then refines prediction embeddings through Triple Factorized Attention before decoding multimodal trajectories. Its distinctive Historical Prediction Attention relates current and historical predictions, extending the temporal information available to forecasting.

3. Method

  • HPNet aggregates spatio-temporal agent and lane features with learnable mode queries, refines prediction embeddings through three attention modules, then decodes trajectories and feeds them back for refinement.Agent Attention models interactions between agents, Historical Prediction Attention links current and historical predictions, and Mode Attention models interactions across predicted paths.Source: §3 · p. 3

3.1. Spatio-Temporal Context Encoding

  • Agent and lane features are encoded with relative spatio-temporal positions, while parallel temporal and spatial attention aggregate agent history and agent-lane interactions.Temporal attention uses historical agent embeddings; spatial attention attends to lane nodes within a radius, and the two results are summed into prediction embeddings.Source: §3.1 · pp. 3–4

3.2. Agent Attention

  • Agent Attention applies self-attention across nearby agents for each mode and time step, modeling agent interactions and potentially mitigating collisions.The attended agents lie within radius R2 of the target agent.Source: §3.2 · p. 4

3.3. Historical Prediction Attention

  • Historical Prediction Attention attends to earlier prediction embeddings for each agent and mode, modeling dynamic correlations between successive forecasts.Using prediction embeddings rather than final historical trajectories keeps training parallel; the attention also extends the effective observation window to [t − I2 − I1, t] without additional computational cost.Source: §3.3 · pp. 4–5

3.4. Mode Attention

  • Mode Attention applies self-attention across an agent’s different prediction modes, modeling interactions between possible future trajectories.Triple Factorized Attention is repeated twice so prediction embeddings can interact more fully.Source: §3.4 · p. 5

3.5. Multimodal Output

  • A two-layer MLP decodes prediction embeddings into multiple future locations, which are used as trajectory proposals and re-encoded to refine the predictions.The refinement process produces trajectory refinements and probability scores, and the final trajectory sums each proposal and its refinement.Source: §3.5 · p. 5

3.6. Training Objective

  • Training uses winner-takes-all mode selection by minimum endpoint displacement, Huber losses for proposals and refined trajectories, and cross-entropy for probability scores.For joint prediction, agents’ predictions in the same mode form a predicted future, and joint endpoint displacement selects the mode to optimize.Source: §3.6 · p. 5

4. Experiments

HPNet is evaluated on Argoverse marginal prediction and INTERACTION joint prediction, where it outperforms prior methods. Ablations find gains in accuracy and stability from Historical Prediction Attention without delayed reaction to abrupt turns.

4.1. Experimental Setup

  • Experiments use Argoverse for marginal trajectory prediction and INTERACTION for joint prediction in real-world driving scenarios with high-definition maps and 10 Hz motion data.Evaluation uses minADE, minFDE, miss rate, and b-minFDE on Argoverse, and minJointADE and minJointFDE on INTERACTION, with K = 6 modes.Source: §4.1 · pp. 5–6

4.2. Comparison with State-of-the-art

  • HPNet achieves the best single-model results on Argoverse, improving over GANet by up to 0.052 in b-minFDE, 0.062 in minFDE, and 0.045 in minADE; on INTERACTION, it improves over FJMP by 0.099 in minJointFDE and 0.020 in minJointADE.HPNet also performs best in Argoverse’s official ranking metric when ensembled.Source: Table 1 · p. 6, §4.2 · pp. 6–7, Table 2 · p. 6

4.3. Ablation Study

  • On Argoverse validation, the full Triple Factorized Attention model reaches 1.506 b-minFDE; removing the module worsens b-minFDE by 0.326, while removing Agent, Historical Prediction, or Mode Attention worsens it by 0.205, 0.021, or 0.025.Agent Attention has the largest stated effect; Historical Prediction Attention also improves four metrics.Source: §4.3 · p. 7, Table 3 · p. 7
  • Across predicted time steps, HPNet beats its baseline in b-minFDE, with a widening relative advantage as its visible history extends beyond the baseline’s fixed 20 steps.The study spans 10 prediction times from 20 to 30, using a 20-step visible history and an equally long historical prediction window.Source: §4.3 · p. 7, Figure 3 · p. 7
  • HPNet’s successive predictions have about 2.25 summed ADE versus 2.90 for the baseline, and it consistently predicts the same turn-left goal where the baseline alternates goals.Unlike DCMS [40], HPNet’s successive predictions share a motion goal in complex road conditions without forced overlapping waypoints.Source: §4.3 · p. 8, Figure 4 · p. 8
  • At a sudden right turn, HPNet quickly switches to right-turn predictions with no delay versus the baseline, while retaining stable left-or-right possibilities before the turn.The paper attributes reduced influence from past predictions to lower similarity between current and historical prediction embeddings, which reduces their attention weights.Source: §4.3 · p. 8, Figure 5 · p. 8

5. Conclusion

HPNet introduces Historical Prediction Attention to model relationships between successive predictions, and experiments on Argoverse and INTERACTION report state-of-the-art performance with accurate, stable trajectories.

  • HPNet is a dynamic trajectory prediction method evaluated on Argoverse and INTERACTION.Source: §5 · p. 8
  • Historical Prediction Attention uses historical prediction embeddings to guide current forecasts.The module models the dynamic relationship between successive predictions.Source: §5 · p. 8
  • Experiments report state-of-the-art performance and accurate, stable future trajectories.The paper also reports that Historical Prediction Attention can effectively improve accuracy and stability.Source: §5 · p. 8

1. Training Objective for Joint Prediction

The joint-prediction objective uses a simple adjustment to select a mode based on joint endpoint displacement; training includes regression and probability losses.

  • Joint prediction uses a simple adjustment to the training objective and treats all agents’ predictions in the same mode as one predicted future.The optimized mode is determined by minimum joint endpoint displacement between predicted futures and ground truth.Source: §1 · p. 12
  • The regression loss combines Huber losses for trajectory proposals and refined final trajectories.Source: §1 · p. 12
  • Future probability scores are optimized with cross-entropy loss.Source: §1 · p. 12

2. Implementation Details

Implementation uses different attention-layer counts, local-area radii, and history spans for Argoverse and INTERACTION; the models contain 4.1M and 5.3M parameters, respectively.

  • Argoverse uses one Spatio-Temporal Attention layer and two Triple Factorized Attention layers, with radius 50 and a 20-time span.Horizontal flipping, agent occlusion, and lane occlusion are used at ratios 0.5, 0.05, and 0.2.Source: §2 · p. 12
  • Training uses 8 RTX 4090 GPUs for 64 epochs, AdamW, batch size 16, dropout 0.1, and weight decay 1 × 10^-4.Initial learning rates are 5 × 10^-4 for Argoverse and 3 × 10^-4 for INTERACTION, with cosine annealing.Source: §2 · p. 12
  • INTERACTION uses one Spatio-Temporal Attention layer and three Triple Factorized Attention layers, with radius 80 and a 10-time span.Horizontal flipping and lane occlusion are used; agent occlusion is not used.Source: §2 · p. 12
  • The model sizes are 4.1M for Argoverse and 5.3M for INTERACTION.Source: §2 · p. 12

3. Inference Latency

On Argoverse validation with an NVIDIA V100, HPNet adds modest inference latency relative to the baseline without Historical Prediction Attention; the authors suggest serial processing.

  • Serial inference takes 27.62 ms for HPNet and 22.76 ms for the baseline without Historical Prediction Attention per single time step.The measurement covers all agents at that time step.Source: §3 · p. 12
  • Parallel inference takes 92.02 ms for HPNet and 81.08 ms for the baseline across all agents at all 20 time steps.Source: §3 · p. 12
  • The authors characterize the added latency as small and suggest serial processing for practical use.They state that Historical Prediction Attention will not impede real-time operational capabilities while enhancing prediction performance.Source: §3 · p. 12

4. Quality Results

Two Argoverse validation examples show HPNet producing stable, accurate motion goals across consecutive moments where the baseline changes its predictions. Figure 6 marks lanes, histories, ground truth, and predictions in distinct colors.

4. Quality Results

  • Across two examples, HPNet maintains stable, accurate motion goals while the baseline varies its predicted goals across consecutive moments.The examples are presented as qualitative evidence of improved prediction accuracy and stability.Source: §4 · p. 12
  • For an agent moving forward, HPNet consistently predicts going straight, while the baseline presents three goal types at four consecutive moments.Source: §4 · p. 12
  • For an agent intending to turn right, HPNet consistently predicts a right turn; Figure 6 colors lanes grey, history green, ground truth red, and predictions blue.Source: §4 · p. 12, Figure 6 · p. 13

Related papers

Loading 2404.06351v2…