Source-linked AI summary
GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning
Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, Siheng Chen
TL;DR
Trajectory prediction needs richer relational reasoning than pair-wise interaction modeling provides. GroupNet learns multiscale pair-wise and group-wise interactions with explicit strength and category representations, improving prediction systems across synthetic and real-world evaluations.
Problem
Prior trajectory-prediction methods provide limited relational reasoning because they primarily model pair-wise interactions, despite the importance of understanding social influence among agents.
Method
GroupNet is a trainable multiscale hypergraph neural network with three-element interaction embeddings for neural strength, category, and per-category functions.
Results
GroupNet captures group behaviors and interaction factors in unsupervised synthetic simulations, improves a CVAE-based system on three real-world benchmarks, and further improves prior state-of-the-art systems.
Takeaways & Limitations
GroupNet functions as a general social-modeling module that supports relational reasoning and can be plugged into different trajectory-prediction systems.
Takeaways & Limitations
The study uses relatively small agent counts and short time windows because of dataset attributes.
Abstract
from arXiv · showhide
Demystifying the interactions among multiple agents from their past trajectories is fundamental to precise and interpretable trajectory prediction. However, previous works only consider pair-wise interactions with limited relational reasoning. To promote more comprehensive interaction modeling for relational reasoning, we propose GroupNet, a multiscale hypergraph neural network, which is novel in terms of both interaction capturing and representation learning. From the aspect of interaction capturing, we propose a trainable multiscale hypergraph to capture both pair-wise and group-wise interactions at multiple group sizes. From the aspect of interaction representation learning, we propose a three-element format that can be learnt end-to-end and explicitly reason some relational factors including the interaction strength and category. We apply GroupNet into both CVAE-based prediction system and previous state-of-the-art prediction systems for predicting socially plausible trajectories with relational reasoning. To validate the ability of relational reasoning, we experiment with synthetic physics simulations to reflect the ability to capture group behaviors, reason interaction strength and interaction category. To validate the effectiveness of prediction, we conduct extensive experiments on three real-world trajectory prediction datasets, including NBA, SDD and ETH-UCY; and we show that with GroupNet, the CVAE-based prediction system outperforms state-of-the-art methods. We also show that adding GroupNet will further improve the performance of previous state-of-the-art prediction systems.
1. Introduction
GroupNet addresses limited relational reasoning in trajectory prediction by learning multiscale pair-wise and group-wise interactions with explicit interaction representations. It is evaluated for relational reasoning and prediction performance across synthetic simulations and three real-world benchmarks.
- Trajectory prediction must model social influence from agents’ past spatio-temporal states for precise and interpretable predictions.
- Existing interaction methods improve empirical performance but often underestimate relational reasoning and model only pair-wise interactions.
- GroupNet learns a trainable multiscale hypergraph that captures pair-wise and group-wise interactions across multiple group sizes.
- Its three-element interaction embedding represents neural interaction strength, neural interaction category, and a per-category function end-to-end.
- Synthetic physics experiments show unsupervised relational reasoning of group behaviors, interaction strength, and interaction category.
- GroupNet improves a CVAE-based system on three real-world benchmarks and further improves previous state-of-the-art prediction systems.
2. Related Works
Related work models trajectory interactions using social aggregation, graphs, attention, and transformers, while relational-reasoning methods infer latent interaction structures from trajectories.
- Traditional and sequence-to-sequence approaches encode trajectories individually, while newer methods model interactions for socially plausible predictions.
- Spatial-centric and social mechanisms represent trajectories, aggregate neighboring actors, and diffuse neighboring information.
- Attention and transformer methods capture agents’ spatial and temporal dependencies.
- Relational-reasoning methods infer latent interaction graphs from individual-agent trajectories without direct knowledge of underlying relations.
3. Problem Formulation
The task predicts multiple agents’ future trajectories from their past trajectories while inferring their interaction relationships. The model may perform unsupervised relational reasoning when it infers social influences without relation labels.
- The problem formulation represents past and future trajectories for N agents as X−∈R^N×T_p×2 and X+∈R^N×T_f×2.
- Each trajectory coordinate x_t^i is a two-dimensional position at time t.
- The prediction model learns g(·) so predicted future trajectories g(X−) are as close as possible to ground-truth future trajectories X+.
- Unsupervised relational reasoning means inferring social influences without ground-truth supervision for interaction category or strength.
4. GroupNet
GroupNet learns multiscale hypergraph topologies from trajectory affinities and performs message passing with interaction embeddings that explicitly represent strength and category.
- Multiscale hypergraph topology inference: GroupNet learns a multiscale hypergraph whose nodes are agents and whose hyperedges represent interactions across multiple group sizes.The finest scale models pair-wise connections, while larger scales model group-wise relations.
- Multiscale hypergraph topology inference: Trajectory embeddings produce an affinity matrix whose values measure pairwise agent correlation for topology inference.The embeddings are normalized before computing cosine-like relational weights.
- Multiscale hypergraph topology inference: At each scale, hyperedges group highly correlated agents subject to a fixed group-size constraint and mandatory participation of the selected node.Small-agent scenes can use enumeration, while larger scenes use a greedy approximation.
- Multiscale hypergraph topology inference: Unlike prior topology approaches, GroupNet learns structures that jointly model pair-wise and group-wise connections at multiple scales.Its multiscale hypergraph keeps node numbers fixed while varying hyperedge sizes.
- Multiscale hypergraph neural message passing: Message passing alternates node-to-hyperedge and hyperedge-to-node updates to learn agent and interaction embeddings over the inferred hypergraphs.The interaction embedding contains neural strength, neural category, and a per-category function, all trained end-to-end.
- Multiscale hypergraph neural message passing: GroupNet uses hypergraph message passing and treats interaction strength and category as intermediate features rather than only final relational outputs.This design is presented as promoting information propagation, relational reasoning, and prediction performance.
5. Prediction System with GroupNet
The prediction system integrates GroupNet into a CVAE-based encoder–decoder that models stochastic future trajectories while reconstructing the past. Its decoder uses residual blocks to produce both outputs from encoded representations and past trajectories.
- CVAE-based prediction system: The CVAE system models future trajectories conditioned on past trajectories using a latent code for all agents.Its evidence lower bound combines expected future-trajectory likelihood with a KL regularization term.
- Encoding process: The encoding process generates approximate-posterior Gaussian parameters from GroupNet embeddings of future and past trajectories.During testing, latent codes are sampled from a conditional prior and concatenated with past-trajectory embeddings.
- Decoding process: The residual decoder takes the encoded representation and past trajectories as input to predict future trajectories and reconstruct past trajectories.This design is intended to avoid input information loss.
- Decoding process: Two decoding blocks produce future and reconstructed-past trajectories whose final outputs are sums across the blocks.Each block contains a GRU encoder and two MLP output heads.
- Training objective: Training minimizes a combined loss containing ELBO, past-trajectory reconstruction, and Social-GAN variety terms.The variety term selects the closest prediction among sampled futures.
6. Experiments and Analysis
Experiments evaluate GroupNet’s relational reasoning on synthetic physics simulations and its trajectory-prediction effectiveness on NBA, SDD, and ETH-UCY datasets. Results show that GroupNet captures multiscale group interactions and consistently improves prediction systems, with performance gains in challenging interaction settings.
- Experimental setup: GroupNet is evaluated on physical simulations and NBA, ETH-UCY, and SDD real-world datasets, using relational-reasoning and trajectory-prediction measures.The real-world evaluations use minADEK and minFDEK, measuring best-of-K trajectory and endpoint distances.
- Relational reasoning: GroupNet infers spring edges and light-bar hyperedges across multiscale hypergraphs, assigning distinct interaction categories to the two groups.The affinity matrix highlights particles connected by the spring or light bar, while the inferred topology identifies their group structures.
- Relational reasoning: GroupNet significantly outperforms pair-wise baselines on unsupervised recognition of free, spring, and light-bar interaction categories.Recognition is evaluated on 2-type and 3-type tasks using trajectories alone, without prior interaction-category information.
- Trajectory prediction: 15.0%/17.6% reductions in NBA minADE20/minFDE20 at 4.0s are achieved by CVAE with GroupNet versus the best baseline, NMMP.Replacing NMMP’s encoder with GroupNet further reduces minADE20/minFDE20 by 6.0%/12.2%.
- Trajectory prediction: 6.5% lower SDD minADE20 is achieved by CVAE with GroupNet versus PECNet, while GroupNet+PECNet reduces minADE20/minFDE20 by 3.1%/3.4%.CVAE with GroupNet achieves state-of-the-art performance on SDD and the best reported ADE in the cited table.
- Analysis: GroupNet improves prediction consistently across datasets and methods, while multiple-scale performance initially increases and then stabilizes once sufficient scales are used.Qualitative NBA results show larger gains in scenes featuring fierce confrontation between two teams.
- Limitations and future work: The paper identifies a scope boundary: the agent number is relatively small and the time window relatively short because of dataset attributes.Future work targets more complex interacting systems and long-term prediction with time-varying agent numbers.