PaperlayerView exact revision on arXiv

Source-linked AI summary

ProphNet: Efficient Agent-Centric Motion Forecasting with Anchor-Informed Proposals

Xishun Wang, Tong Su, Fang Da, Xiaodong Yang

arXiv:2303.12071v3cs.CVcs.RO

TL;DR

Motion forecasting must handle heterogeneous inputs and multimodal future behavior while meeting tight runtime constraints for autonomous driving. ProphNet unifies input encoding and combines diverse proposals with goal-oriented anchors. It reports state-of-the-art accuracy on popular benchmarks with scene-centric-level low inference latency, including peak latency as low as 28ms.

  • Problem

    Motion forecasting must reconcile heterogeneous multi-source inputs and multimodal future behavior with tight runtime and computation limits for onboard driving.

    Source: §1 · pp. 1–2

  • Method

    ProphNet uses a unified agent-centric representation and end-to-end anchor-informed proposals that fuse diverse history-based proposals with goal-oriented anchors, with hydra heads producing multimodal predictions.

    Source: §1 · p. 2

  • Results

    ProphNet achieves state-of-the-art accuracy on popular benchmarks while maintaining scene-centric-level low inference latency, with peak latency as low as 28ms.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present ProphNet's uniform encoding and succinct architecture as amenable to real-world driving deployment.

    Source: §1 · p. 2

Abstract

from arXiv · show

Motion forecasting is a key module in an autonomous driving system. Due to the heterogeneous nature of multi-sourced input, multimodality in agent behavior, and low latency required by onboard deployment, this task is notoriously challenging. To cope with these difficulties, this paper proposes a novel agent-centric model with anchor-informed proposals for efficient multimodal motion prediction. We design a modality-agnostic strategy to concisely encode the complex input in a unified manner. We generate diverse proposals, fused with anchors bearing goal-oriented scene context, to induce multimodal prediction that covers a wide range of future trajectories. Our network architecture is highly uniform and succinct, leading to an efficient model amenable for real-world driving deployment. Experiments reveal that our agent-centric network compares favorably with the state-of-the-art methods in prediction accuracy, while achieving scene-centric level inference latency.

1. Introduction

Motion forecasting must reconcile heterogeneous inputs, multimodal futures, and tight runtime budgets. ProphNet unifies input encoding and combines diverse proposals with goal-oriented anchors, reporting strong accuracy and low inference latency.

1. Introduction

  • Motion forecasting must handle heterogeneous, interrelated inputs, multimodal future behavior, and tight runtime requirements under limited computation budgets.The task supports safe and comfortable autonomous driving in dynamic environments.Source: §1 · p. 1
  • ProphNet outperforms state-of-the-art methods in Argoverse-1 prediction accuracy while considerably speeding agent-centric inference, achieving the best accuracy–latency balance.The paper also reports peak latency as low as 28ms.Source: Figure 1 · p. 1, §1 · p. 2
  • Earlier methods often used intricate modality-specific designs, while agent-centric models increased latency by computing scene representations separately for each agent.The larger feature volume can make agent-centric processing computationally prohibitive for onboard deployment.Source: §1 · pp. 1–2
  • ProphNet uniformly encodes agent history, agent relations, and road graphs into AcSR, where self-attention learns intra- and inter-modality interactions in one compact space.This modality-agnostic design contrasts with separate self-attention encoders for each input modality.Source: §1 · p. 2
  • AiP fuses diverse proposals generated from agent history with goal embeddings learned from AcSR, and hydra heads use random AiP combinations to produce complementary predictions.Proposals are generated for diversity before anchors add goal-oriented context; the heads also support ensembling.Source: §1 · p. 2
  • The paper presents the agent-centric model as achieving state-of-the-art accuracy on popular benchmarks while maintaining scene-centric-level low inference latency.The authors describe the uniform encoding and succinct architecture as amenable to real-world driving deployment.Source: §1 · p. 2

2. Related Work

Related work spans rasterized and vectorized representations, scene-centric and agent-centric reference frames, and methods for heterogeneous interactions and multimodal outputs. ProphNet is positioned among approaches that seek to simplify input fusion while learning multimodality end to end.

2. Related Work

  • Motion forecasting representations are rasterized or vectorized, and use scene-centric or agent-centric reference frames; scene-centric models are generally more computationally efficient, while agent-centric models deliver better accuracy.Rasterized methods encode rendered scene context with image-oriented networks; vectorized methods represent agent dynamics and map elements in vector form.Source: §2 · p. 2
  • Prior heterogeneous-input methods use input-specific fusion or separate interaction modules, increasing model complexity through explicit processing and combination of sources.Examples include LaneGCN's four interaction sub-networks and separate agent-to-agent and agent-to-road modules in SceneTransformer.Source: §2 · pp. 2–3
  • Multimodal prediction approaches use planning candidates, manually defined regions, clustered trajectories, learned anchors, or sampled goals; ProphNet instead seeks end-to-end learned anchor-informed proposals.The cited approaches include mmTransformer, MultiPath, MultiPath++, TNT, and DenseTNT.Source: §2 · p. 3

3. Method

ProphNet encodes target-agent history, neighboring agents, and road geometry in a unified agent-centric representation, then combines diverse history-based proposals with learned goal-context anchors. Hydra prediction heads produce multiple trajectory distributions for selection and final prediction.

3.1. Input Representation; 3.2. Agent-Centric Scene Representation (AcSR)

  • ProphNet combines agent history, neighboring-agent relations, and road-graph inputs into a single agent-centric representation using uniform gMLP encoders.The road graph includes lane centerlines, boundaries, and traffic controls; the unified representation supports interaction learning across input sources.Source: §3.1 · p. 3, §3.2 · p. 4

3.3. Proposal Generation

  • K proposals are generated from the target agent’s full motion history, preserving spatial and temporal cues while avoiding constraints from specific driving scenarios.The proposal queries attend to the full history encoding; orthogonal initialization further encourages proposal diversity.Source: §3.3 · p. 4

3.4. Anchor Learning

  • K learned anchors correspond to proposals and use endpoint-supervised embeddings to inject goal-oriented scene context into those proposals.The embeddings, rather than predicted endpoints, serve as anchors and carry sample-specific information.Source: §3.4 · p. 4

3.5. Anchor-Informed Proposals (AiP)

  • AiP directly sums proposal and anchor embeddings, combining history-derived trajectory candidates with goal-based contextual information for subsequent prediction heads.The paper reports that attention-based fusion led to some multimodality collapse, motivating direct summation.Source: §3.5 · pp. 4–5

3.6. Hydra Prediction Heads

  • Hydra heads use repeated random subsets of M embeddings from K=2M AiP embeddings to produce N groups of multimodal trajectories for ensembling.Non-maximum suppression merges the resulting N×M trajectories into M outputs; each trajectory is parameterized as a Gaussian mixture model and trained using ground-truth likelihood.Source: §3.6 · p. 5

4. Experiments

Experiments evaluate ProphNet on Argoverse-1 and Argoverse-2, report favorable benchmark comparisons and roughly 28 ms inference latency, and examine its components and qualitative outputs.

4. Experiments; 4.2. Implementation Details

  • The experiments compare benchmark methods, conduct component ablations, and report results on both datasets.The implementation uses 16 NVIDIA V100 GPUs, with M = 6 output modes, K = 12 AiP elements, and N = 3 hydra heads.Source: §4 · p. 5, §4.2 · p. 5

4.1. Datasets and Metrics

  • The evaluation covers Argoverse-1 and Argoverse-2, using each benchmark’s official minADE, minFDE, brier-minFDE, and miss-rate metrics.Argoverse-1 sequences cover 2 seconds of history and 3 seconds of future; Argoverse-2 extends the prediction horizon to 6 seconds and includes five agent types.Source: §4.1 · p. 5

4.3. Results on Argoverse-1; 4.4. Results on Argoverse-2

  • ProphNet outperforms published methods on Argoverse-2 and shows strong Argoverse-1 test performance against scene-centric methods, significantly outperforming MultiPath++ and overall performing better than Wayformer.The Argoverse-1 test comparison uses a single model without ensembling; the Argoverse-2 comparison highlights performance over TENET.Source: Table 3 · p. 6, §4.4 · p. 6, Table 2 · p. 6

4.5. Latency Analysis

  • 28 ms inference latency places this agent-centric model at the scene-centric level; hydra heads increase latency only from 27.4ms to 28.0ms.The latency comparison uses a single NVIDIA V100 GPU and 64 agents; the authors attribute efficiency to the uniform design and exclusion of computationally heavy operations.Source: §4.5 · pp. 6–7, Table 4 · p. 7

4.6. Ablation Study

  • Ablations associate improved performance with full history encoding, unified feature space, learned anchors, and hydra prediction heads.The authors report that adding anchors particularly improves minFDE, while the full hydra-head model further improves over a single-head variant.Source: §4.6 · p. 7, Table 5 · p. 7

4.7. Qualitative Results

  • After training, off-diagonal similarities between proposal queries remain close to zero, while anchor-informed proposals produce accurate multimodal trajectories from diverse proposals.Direct proposal trajectories can be diverse but off-road; visualized anchor endpoints lie in road areas and mostly follow lane centerlines, though they are not used during inference.Source: §4.7 · pp. 7–8, Figure 4 · p. 7, Figure 5 · p. 8
  • Three hydra heads produce complementary diverse trajectories, and ensembling them further improves the final output’s multimodal coverage.The qualitative example is a challenging scenario in which the target agent is static.Source: §4.7 · p. 8, Figure 6 · p. 8

5. Conclusion

ProphNet combines unified heterogeneous-input encoding, anchor-informed proposals, and hydra prediction heads for multimodal forecasting, with experiments demonstrating accuracy and latency effectiveness on two benchmarks.

  • ProphNet constructs AcSR through uniform encoding, forms AiP from proposals and anchors, and feeds it to hydra prediction heads for multimodal trajectories.Source: §5 · p. 8
  • Experiments on two popular benchmarks demonstrate the effectiveness of ProphNet in prediction accuracy and inference latency.Source: §5 · p. 8
  • The paper advocates practical driving models that combine high accuracy with succinct architecture and efficient inference.Source: §5 · p. 8

Appendix

The appendix collects additional benchmark comparisons, attention visualizations, latency-evaluation details, qualitative prediction results, and ablations.

A. Leaderboard comparisons

  • Appendix A adds leaderboard comparisons for Argoverse-1 and Argoverse-2.Source: Appendix · p. 10

B–C. Attention and latency

  • Appendix B visualizes learned attention across multi-sourced input, while Appendix C details latency evaluation.Source: Appendix · p. 10

D–E. Qualitative results and ablations

  • Appendices D and E provide additional qualitative predictions and ablation studies.Source: Appendix · p. 10

A. ProphNet on Leaderboards

Leaderboard comparisons rank ProphNet first on Argoverse-1 and second on Argoverse-2; only ProphNet and QCNet are reported to perform stably and strongly on both leaderboards.

A. ProphNet on Leaderboards

  • The leaderboard comparison collects leading published and unpublished methods to examine generalizability across the two benchmarks.Source: A. ProphNet on Leaderboards · p. 10
  • 1st on Argoverse-1 and 2nd on Argoverse-2: ProphNet ranks among the top leaderboard submissions on both benchmarks.The comparison ensembles six models for Argoverse-1 and three for Argoverse-2; unpublished methods cannot be analyzed qualitatively because their details are undisclosed.Source: A. ProphNet on Leaderboards · p. 10
  • Only ProphNet and unpublished QCNet yield stable and superior performance across both leaderboards.Source: A. ProphNet on Leaderboards · p. 10

B. Visualization of Learned Attentions

Learned attention focuses on nearby potentially interacting agents and frontal road polylines, including routes consistent with possible future trajectories.

B. Visualization of Learned Attentions

C. Details on Latency Evaluation

The latency evaluation compares open-sourced models under a common hardware and agent-count setting, with Wayformer implemented from its paper specifications because open-source materials were unavailable.

C. Details on Latency Evaluation

D. More Qualitative Results

ProphNet produces multiple plausible trajectories in four representative challenging scenes, including complex road layouts and crowded intersections.

D. More Qualitative Results

  • In complex road topologies, ProphNet produces multiple accurate trajectories that reasonably align with lane centerlines.The illustrated cases are scenes (a-b).Source: D. More Qualitative Results · p. 10
  • At crowded intersections, ProphNet predicts rich and rational multimodal future trajectories.The illustrated cases are scenes (c-d).Source: D. More Qualitative Results · p. 10

E. More Ablation Study

Additional ablations favor orthogonal query initialization, simple proposal-anchor summation, anchor embeddings, and gMLP encoding; long-horizon predictions can occasionally go off-road in challenging scenes.

E. More Ablation Study

Related papers

Loading 2303.12071v3…