PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Motion Prediction via Neural Motion Message Passing

Yue Hu, Siheng Chen, Ya Zhang, Xiao Gu

arXiv:2003.06594v1cs.CV

TL;DR

Motion prediction needs to account for interactions among traffic actors, but explicitly representing those interactions remains a challenge. The paper introduces NMMP to learn actor and directed-interaction representations, then uses it in pedestrian and joint pedestrian–vehicle systems. Both systems outperform prior state-of-the-art methods across several benchmarks, and the paper reports interaction interpretability.

  • Problem

    Motion prediction must model hidden interactions among traffic actors, which matter in applications such as collision avoidance and planning.

    Source: §1 · p. 1

  • Method

    NMMP infers an interaction graph from actor trajectories and learns representations for actors and their directed interactions, supporting systems with individual and interactive branches for pedestrian and joint pedestrian–vehicle prediction.

    Source: §1 · pp. 1–2

  • Results

    Both proposed systems outperform prior state-of-the-art methods across several widely used benchmarks.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The evaluations and visualizations show that NMMP captures interaction patterns and the systems predict precise trajectories in different scenarios.

    Source: §6 · p. 8

Abstract

from arXiv · show

Motion prediction is essential and challenging for autonomous vehicles and social robots. One challenge of motion prediction is to model the interaction among traffic actors, which could cooperate with each other to avoid collisions or form groups. To address this challenge, we propose neural motion message passing (NMMP) to explicitly model the interaction and learn representations for directed interactions between actors. Based on the proposed NMMP, we design the motion prediction systems for two settings: the pedestrian setting and the joint pedestrian and vehicle setting. Both systems share a common pattern: we use an individual branch to model the behavior of a single actor and an interactive branch to model the interaction between actors, while with different wrappers to handle the varied input formats and characteristics. The experimental results show that both systems outperform the previous state-of-the-art methods on several existing benchmarks. Besides, we provide interpretability for interaction learning.

1. Introduction

Motion prediction must account for hidden interactions among actors, but prior approaches either use static spatial relationships or omit explicit representations of the interactions themselves. The paper proposes NMMP and applies it in pedestrian and joint pedestrian–vehicle systems, reporting improvements over prior state-of-the-art methods in both settings.

1. Introduction

  • Prior approaches model interactions through spatial, social, or graph-based mechanisms, but the first two use static actor locations and previous graph methods do not explicitly represent in-between interactions.The paper identifies directed interactions and interaction interpretability as extensions to graph-based methods.Source: §1 · pp. 1–2
  • NMMP infers an interaction graph from actor trajectories and learns representations for both actors and their corresponding interactions, including directed interactions.The paper also presents interaction representations as interpretable: similar interaction patterns have similar impacts on the acting actors.Source: §1 · pp. 1–2
  • The proposed systems cover open-area pedestrian motion and urban driving with pedestrians and vehicles, using a shared NMMP module alongside individual and interactive branches.Distinct wrappers handle the settings’ input formats and characteristics; the pedestrian system adds a GAN, while the joint system adds map-processing CNNs and coordinate transformation for vehicle headings.Source: §1 · p. 2
  • Both systems outperform previous state-of-the-art methods, and the paper reports interpretability through quantitative benchmarks and visualization analysis.The paper presents these results across the pedestrian and joint pedestrian–vehicle settings.Source: §1 · p. 2

2. Related Works

Prior motion-prediction work ranges from individual trajectory models to spatial, social, and graph-based approaches to representing interactions. The paper situates NMMP among graph methods and emphasizes representations for interactions as well as actors.

2. Related Works

  • Sequence-to-sequence models encode prior trajectories but consider each actor individually and ignore interactions among actors in a scene.The paper notes that traditional hand-crafted rules and energy potentials fail to generalize to complex scenes.Source: §2 · p. 2
  • Spatial-centric methods use spatial relationships to model interactions implicitly, while social methods aggregate neighboring actors’ information into a shared representation.Social approaches include pooling and attention; the paper notes that max pooling can ignore actor uniqueness and attention increases computational complexity.Source: §2 · p. 2
  • Graph-based methods explicitly model pairwise actor interactions, but the paper identifies directed interactions and interpretable interaction representations as extensions to prior work.The paper characterizes prior graph methods as focusing on actor features rather than explicitly learning features for in-between interactions.Source: §2 · p. 2
  • NMMP uses neural message passing with actors as nodes and interactions as edges, treating edges as an integral part of the model.Figure 2 describes trajectory embeddings learned through LSTM, with interacted-actor and in-between interaction embeddings learned through message passing.Source: §2 · p. 2, Figure 2 · p. 3

3. Neural Motion Message Passing

NMMP builds an interaction graph from observed actor trajectories and learns representations for actors and their interactions. Repeated node-to-edge and edge-to-node message passing updates both representations, with concatenation used to preserve interaction direction.

3. Neural Motion Message Passing

  • NMMP takes actor trajectories as input and provides feature representations for actors and their corresponding interactions, enabling actors to share history information for collaborative future-trajectory prediction.The interaction graph uses actors as nodes and pairwise interactions as edges, and is initialized as fully connected.Source: §3 · p. 3
  • The initial actor representation uses an LSTM trajectory embedding, while the initial interaction representation combines actor embeddings with relative spatial information.The method encodes displacement between consecutive timestamps and the difference between actors’ current positions.Source: §3 · p. 3
  • In node-to-edge message passing, each interaction receives motion messages from its actor pair; in edge-to-node passing, each actor embedding is updated from its corresponding interactions.The method repeats these two phases K times to obtain final interacted actor embeddings.Source: §3 · p. 3
  • The edge-to-node phase concatenates interaction information rather than summing or averaging it, distinguishing the direction of interactions.The paper’s example is asymmetric: a front actor slowing may affect a back actor, while the reverse need not affect the front actor equally.Source: §3 · p. 3
  • The method normalizes interaction embeddings using the number of incoming and outgoing edges, and interaction density is positively related to the number of iterations K.The paper identifies the incoming and outgoing edge counts as the respective normalization quantities.Source: §3 · p. 3

4. Motion Prediction Systems

The section presents NMMP-based systems for flexible pedestrian settings and urban scenes with both pedestrians and vehicles. Both combine individual behavior modeling with interaction modeling, using architectures suited to each setting.

4.1. Pedestrian motion prediction

  • The pedestrian setting represents flexible walking in open areas, where trajectories are highly nonsmooth and may fit surveillance scenarios..Source: §4.1 · pp. 3–4

4.1.1 System architecture

  • PMP-NMMP predicts trajectories with an individual branch and an interaction branch that refines it, while a discriminator classifies complete trajectories and adversarial training encourages realism.The predicted coordinate is the previous position plus individual and interaction components; Gaussian noise encourages diversity.Source: Figure 3 · p. 4, §4.1.1 · p. 4

4.1.2 Loss function

  • The pedestrian system trains its generator against a discriminator in a min-max game to encourage stochastic, realistic predictions.The discriminator estimates the real probability of a complete trajectory.Source: §4.1.2 · p. 4

4.2. Joint pedestrian and vehicle motion prediction

  • The joint setting targets urban driving scenes with pedestrians and vehicles, whose distinct motion patterns and environmental information such as drivable areas must be considered..Source: §4.2 · p. 4

4.2.1 System architecture

  • JMP-NMMP combines type-specific individual branches with an interactive branch that predicts an interaction-based compensation for each actor; the branches are combined for the final trajectory.The system also uses an ego-coordinate frame for individual predictions and a global frame for collaborative interaction modeling.Source: §4.2.1 · pp. 4–5, Figure 4 · p. 5
  • The joint system’s individual branch predicts each actor’s trajectory component in its ego-coordinate system, while the interactive branch combines NMMP actor embeddings with a scene-map embedding.The RGB scene map supplies prior traffic-condition and rule information; the combined branch outputs are transformed into global coordinates.Source: §4.2.1 · p. 5

4.2.2 Loss function

  • The joint system combines an individual-branch ℓ2 loss and a final-prediction ℓ2 loss, weighted equally with λ = 0.5.The losses compare their respective predictions with the ground-truth future trajectory.Source: §4.2.2 · p. 5

5. Experiments

The experiments evaluate pedestrian and joint pedestrian–vehicle forecasting, comparing the proposed systems with prior methods and testing NMMP design choices. The authors also examine how interaction embeddings and crowd size relate to prediction behavior.

5.1.1 Pedestrian motion prediction

  • Pedestrian evaluation uses ETH-UCY and Stanford Drone; autonomous-driving evaluation uses a dataset of 3,148 samples, split into 1,888 training, 629 validation, and 631 test samples.ETH-UCY testing uses a leave-one-out approach across five sets.Source: §5.1.1 · p. 5, §5.1.2 · p. 6
  • PMP-NMMP improves ETH-UCY state-of-the-art average ADE/FDE to 0.41m/0.82m and significantly outperforms competitive methods on Stanford Drone.Table 1 reports ETH-UCY results in meters; Table 2 reports Stanford Drone results in pixels.Source: §5.1.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6

5.1.2 Joint motion prediction

  • JMP-NMMP performs best for both vehicles and pedestrians on NuScenes; GAN adds time cost without improving performance.Table 3 compares ADE/FDE in meters for forecasts of 3 seconds from 0.5 seconds of observed trajectories.Source: §5.1.2 · pp. 6–7, Table 3 · p. 6

5.2. Ablation study

  • Replacing social pooling with NMMP in the generator and discriminator yields stable gains on all five ETH-UCY sets.The ablation compares the pooling baseline, NMMP in the generator, and NMMP in both the generator and discriminator.Source: §5.2 · p. 7, Table 4 · p. 7
  • On UNIV, performance improves and then sharply worsens as NMMP iterations increase, with excessive iterations linked to oversmoothing.The authors describe oversmoothing as mixing local information with global context; the set contains scenes with 2 to 57 pedestrians.Source: §5.2 · p. 7, Table 5 · p. 7
  • The double-decoder structure outperforms the single decoder, and its final output outperforms the individual-branch output on UNIV.The double structure uses individual and interactive decoders whose outputs sum to the final trajectory.Source: Table 6 · p. 7, §5.2 · p. 8

5.3. Interpretability exploration

  • In the embedding visualization, nearby interaction embeddings correspond to similar actor-pair trajectory relations, while different relations occupy different positions.The interactions are mapped to two dimensions with t-SNE, and selected trajectory pairs are shown alongside them.Source: §5.3 · p. 8, Figure 7 · p. 8
  • PMP-NMMP’s ADE/FDE normalization ratio over SGAN is positive across pedestrian counts and increases in more crowded scenes.The authors attribute the larger advantage in crowded scenes to NMMP’s capacity to handle abundant social interactions.Source: §5.3 · p. 8, Figure 8 · p. 8

6. Conclusions

The paper proposes NMMP-based systems for pedestrian and joint pedestrian–vehicle motion prediction, reports that both outperform prior state-of-the-art methods across several benchmarks, and uses evaluations and visualizations to examine interaction patterns and trajectory precision.

  • Both NMMP-based systems outperform prior state-of-the-art methods across several widely used benchmarks.Source: §6 · p. 8
  • NMMP explicitly models interactions between traffic actors in a scene.Source: §6 · p. 8
  • Evaluations and visualizations show that NMMP captures interaction patterns and the systems predict precise trajectories across different scenarios.Source: §6 · p. 8

Further experimental analysis

The further analysis visualizes interaction embeddings alongside actor trajectories in joint motion prediction and describes how traffic scenes are rasterized into images.

Loading 2003.06594v1…