PaperlayerView exact revision on arXiv

Source-linked AI summary

EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning

Chenxin Xu, Robby T. Tan, Yuhong Tan, Siheng Chen, Yu Guang Wang, Xinchao Wang, Yanfeng Wang

arXiv:2303.10876v2cs.CVcs.MA

TL;DR

Existing motion prediction methods often overlook guaranteed equivariance and invariant interaction reasoning under Euclidean transformations. EqMotion addresses this with equivariant geometric features, invariant pattern features, and invariant interaction reasoning, achieving state-of-the-art performance across four scenarios.

  • Problem

    Most existing motion prediction methods do not guarantee equivariance under Euclidean transformations because geometric relationships can become irretrievable in abstract features.

  • Method

    EqMotion combines equivariant geometric feature learning, invariant pattern feature learning, and invariant interaction reasoning in a parametric motion prediction network.

  • Results

    EqMotion achieves state-of-the-art prediction performance across four different scenarios.

  • Takeaways & Limitations

    The model is lightweight, with a size below 30% of many other models, and using 5% of the data can achieve comparable performance to methods using full data.

  • Takeaways & Limitations

    For most scenarios, EqMotion uses a fully connected graph because interacted neighbors are not explicitly defined; massive-node cases instead use distance-threshold neighborhoods.

Abstract

from arXiv · show

Learning to predict agent motions with relationship reasoning is important for many applications. In motion prediction tasks, maintaining motion equivariance under Euclidean geometric transformations and invariance of agent interaction is a critical and fundamental principle. However, such equivariance and invariance properties are overlooked by most existing methods. To fill this gap, we propose EqMotion, an efficient equivariant motion prediction model with invariant interaction reasoning. To achieve motion equivariance, we propose an equivariant geometric feature learning module to learn a Euclidean transformable feature through dedicated designs of equivariant operations. To reason agent's interactions, we propose an invariant interaction reasoning module to achieve a more stable interaction modeling. To further promote more comprehensive motion features, we propose an invariant pattern feature learning module to learn an invariant pattern feature, which cooperates with the equivariant geometric feature to enhance network expressiveness. We conduct experiments for the proposed model on four distinct scenarios: particle dynamics, molecule dynamics, human skeleton motion prediction and pedestrian trajectory prediction. Experimental results show that our method is not only generally applicable, but also achieves state-of-the-art prediction performances on all the four tasks, improving by 24.0/30.1/8.6/9.2%. Code is available at https://github.com/MediaBrain-SJTU/EqMotion.

1. Introduction

EqMotion addresses the frequent neglect of Euclidean motion equivariance and interaction invariance in multi-agent motion prediction. It combines equivariant and invariant representations with invariant interaction reasoning and achieves state-of-the-art results across four scenarios.

  • Motion prediction forecasts future trajectories of multiple interacting agents from historical observations across applications including physics, molecule dynamics, autonomous driving, and human-robot interaction.
  • Euclidean motion equivariance requires transformed inputs to produce equally transformed outputs, while interaction invariance preserves agent relationships under translation, rotation, and reflection.
  • Most existing methods overlook or fail to guarantee these properties because abstract feature transformations make geometric relationships irretrievable; augmentation still cannot ensure equivariance.
  • EqMotion combines equivariant geometric feature learning, invariant pattern feature learning, and invariant interaction reasoning in a parametric network.
  • 24.0/30.1/8.6/9.2% prediction-error reductions are reported across particle dynamics, molecule dynamics, human skeleton motion, and pedestrian trajectories, respectively.
  • EqMotion is reported as generally applicable and superior to existing state-of-the-art methods across all four evaluated tasks.

2. Related Work

Prior work studies equivariant networks and motion prediction through diverse architectures and application-specific methods. These approaches span symmetry-aware convolutions and graph methods, physical-system interaction modeling, and multimodal trajectory prediction.

  • Equivariant-network research includes rotation-aware image filters and graph neural networks modeling rotation and translation symmetries.
  • Physical-system motion prediction methods use graph neural networks to simulate complex systems and explicitly infer interaction relationships.
  • Human and vehicle trajectory prediction has used social-force, Markov-process, and recurrent-neural-network approaches.
  • Later trajectory methods model multiple futures with generator-discriminator structures, multi-head outputs, conditional variational autoencoders, memory mechanisms, or Gaussian mixtures.
  • Autonomous-driving prediction methods additionally incorporate high-definition map information.

3. Background and Problem Formulation

The paper formulates multi-agent motion prediction as mapping historical agent trajectories to future trajectories. It defines Euclidean equivariance and invariance and uses these properties to characterize prediction, interaction reasoning, geometric features, and pattern features.

  • 3.1. Motion Prediction: For each agent, X_i stores T_p historical motion steps and Y_i stores T_f future motion steps in an n-dimensional space, usually n=2 or 3.
  • 3.1. Motion Prediction: The full system stacks the M agents’ past and future motions as X ∈ R^(M×T_p×n) and Y ∈ R^(M×T_f×n).
  • 3.1. Motion Prediction: The prediction network F_pred maps historical motions X to predicted future motions Ŷ that should approximate ground-truth future motions Y.
  • 3.2. Equivariance and Invariance: Euclidean transformations comprise translation, rotation, and reflection, represented by a translation vector and an orthogonal matrix.
  • 3.2. Equivariance and Invariance: An operation is equivariant when transforming its input transforms its output correspondingly, whereas invariance leaves the output unchanged under that transformation.
  • 3.2. Equivariance and Invariance: The motion predictor is called motion equivariant when it satisfies the equivariance definition, while the interaction model is interaction invariant when it satisfies the invariance definition.
  • 3.2. Equivariance and Invariance: EqMotion learns geometric features that are equivariant and pattern features plus interaction reasoning that are invariant under Euclidean transformations.

4. Methodology

EqMotion alternates equivariant geometric and invariant pattern feature learning, while an invariant reasoning module supplies interaction categories and an equivariant output layer produces predictions. The design explicitly preserves the corresponding transformation properties throughout the network.

  • Architecture: EqMotion initializes geometric features and pattern features, then repeatedly updates them through equivariant geometric and invariant pattern feature learning layers.The final prediction is obtained from the learned geometric feature through an equivariant output layer.
  • Equivariant Geometric Feature Learning: Equivariant geometric learning models spatial interactions by using interaction categories to weight aggregation of neighboring agents’ geometric features.The geometric update aggregates learned weights multiplied by pairwise geometric-feature differences.
  • Invariant Interaction Reasoning: The interaction reasoning module infers an invariant interaction graph whose categorical edge weights represent interaction types between agents.It uses initial pattern features and geometric relationships to infer interaction categories when relationships are unavailable.
  • Equivariant Geometric Feature Learning: The equivariant nonlinear function increases representation ability by splitting conditions with an invariant criterion and applying an equivariant equation to each condition.This design adds nonlinearity while preserving equivariance.
  • Feature Initialization: The initialization layer gives geometric features equivariance and pattern features invariance, assigning them complementary geometric and transformation-independent motion information.Geometric features preserve transform-sensitive motion attributes, while pattern features capture attributes independent of Euclidean transformations.
  • Theoretical Analysis: Theoretical analysis establishes invariance or equivariance for initialization, interaction reasoning, geometric learning, pattern learning, and the output layer, yielding an equivariant whole network.The stated result applies to Euclidean transformations represented by rotation matrix R and translation vector t.

5. Experiment

EqMotion is evaluated across particle, molecule, human-skeleton, and pedestrian-trajectory prediction scenarios using displacement, joint-position, and interaction-recognition metrics. It achieves strong prediction results across tasks, while ablations show contributions from its proposed modules and equivariant operations.

  • Experimental scenarios: EqMotion is evaluated on particle dynamics, molecule dynamics, 3D human skeleton motion, and pedestrian trajectory prediction.The experiments use physical simulations, MD17, H3.6M, and ETH-UCY settings.
  • Particle dynamics: Recognition consistency measures the ratio of identical interaction predictions under 20 random Euclidean transformations.Interaction recognition is evaluated for spring connections and electrostatic forces in physical simulations.
  • Particle dynamics: EqMotion achieves state-of-the-art future prediction on the charged physical simulation across all future timestamps.Figure 3 compares displacement error over time and identifies EqMotion as the best-performing method.
  • Molecule dynamics: 34.2%/30.1%: average ADE/FDE decreases across four MD17 molecules, where EqMotion achieves state-of-the-art prediction on every molecule.The evaluated molecules are Aspirin, Benzene, Ethanol, and Malonaldehyde.
  • Pedestrian trajectory prediction: 10.4%/7.9%: FDE reduction under deterministic/multi-prediction pedestrian settings, with the lowest average ADE and FDE among compared methods.EqMotion also achieves the best or second-best ADE/FDE on most ETH-UCY subsets.
  • Ablation studies: All three proposed modules contribute to accurate H3.6M prediction, with equivariant geometric feature learning identified as most important.Inner-agent attention, inter-agent aggregation, and the nonlinear operation also each contribute to prediction accuracy.
  • Ablation studies: EqMotion has the smallest model size, below 30% of other models’ sizes, while achieving the lowest H3.6M short-term MPJPE.It also achieves the best prediction performance across all tested training-data ratios and can outperform some full-data baselines using 5% of the training data.

6. Conclusion

EqMotion is a theoretically Euclidean-equivariant motion prediction network with three complementary feature and reasoning designs. Evaluated across four scenarios, it achieves state-of-the-art prediction performance.

  • EqMotion is theoretically equivariant under Euclidean transformations.
  • The model combines equivariant geometric feature learning, invariant pattern feature learning, and invariant interaction reasoning.
  • EqMotion achieves state-of-the-art prediction performance across four different scenarios.

A. Theoretical Proofs

The theoretical proofs establish that EqMotion preserves equivariance for geometric features while maintaining invariance for pattern features and interaction reasoning under Euclidean transformations.

  • Theorem 1 establishes EqMotion’s equivariance and the interaction reasoning module’s invariance properties.
  • The initialization layer produces an equivariant geometric feature and an invariant pattern feature.
  • The invariant pattern features use motion quantities such as velocity norms and inter-timestep angles, whose values are preserved by rotations.
  • The reasoning module produces invariant interaction categorical vectors under Euclidean transformations.
  • Geometric feature learning remains equivariant because inner-agent attention, inter-agent aggregation, and nonlinear operations are equivariant.
  • Pattern feature learning remains invariant under Euclidean transformations.

D.1. Dataset Description

The experiments cover particle, molecule, human skeleton, and pedestrian trajectory datasets, spanning physical interactions, molecular motion, human actions, and social pedestrian scenarios.

  • Particle dynamics use five interacting particles in Springs and Charged simulations with physical force-based interactions.
  • MD17 experiments predict 10 future timestamps from 10 observed timestamps for Aspirin, Benzene, Ethanol, and Malonaldehyde.
  • Human3.6M contains seven subjects performing 15 action classes with 22 body joints, using subject-held-out testing.
  • ETH-UCY contains five pedestrian subsets and uses 3.2 seconds of observations to predict 4.8 seconds under leave-one-out evaluation.

D.2. Implementation Details

Implementation uses a four-layer network trained with Adam on a single RTX-3090 GPU, with scenario-specific feature dimensions, category counts, schedules, and learning rates.

  • All experiments use four feature learning layers, Adam optimization, a single NVIDIA RTX-3090 GPU, and two-layer ReLU MLPs.
  • Particle Dynamics: Particle dynamics use 64-dimensional geometric and pattern features, two categories, batch size 50, learning rate 5e-4, and 200 epochs.
  • Molecule Dynamics: Molecule dynamics use the same 64-dimensional features, two categories, batch size 50, learning rate 5e-4, and 300 epochs.
  • Human Skeleton Motion: Human skeleton experiments use 72 or 96 geometric coordinates for short- or long-term prediction, with 64-dimensional pattern features and four categories.
  • Pedestrian Trajectories: Pedestrian experiments use 64-dimensional geometric and pattern features, four categories, subset-specific learning rates and decay schedules, and 50 epochs.

E. Further Experiment Results

On Human3.6M, increasing the number of feature-learning layers initially improves performance, after which performance stabilizes once the layer count is sufficient.

  • Increasing the feature-learning layers L initially improves Human3.6M performance by learning more comprehensive geometric and pattern features.
  • Performance tends to stabilize when the number of feature-learning layers is sufficient.

F. Limitation and Future Work

The work focuses on a generally applicable motion prediction method, while future work will add task-specific designs and incorporate additional data such as images and videos with map information.

  • The current work focuses on a generally applicable motion prediction method.
  • Future work will add task-specific designs for different tasks to further improve model performance.
  • Future work will explore using images and videos containing map information to assist prediction.
Loading 2303.10876v2…