PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Tracking

Jiachen Li, Hengbo Ma, Zhihao Zhang, Jinning Li, Masayoshi Tomizuka

arXiv:2102.09117v1cs.CVcs.AIcs.LGcs.MAcs.RO

TL;DR

Multi-agent prediction must account for interactions among agents in crowded, uncertain scenes, where accurate forecasting supports safe navigation. STG-DAT combines graph-based attention, trajectory and scene context, and vehicle kinematic constraints; it reports state-of-the-art prediction performance and improved tracking accuracy.

  • Problem

    Many multi-agent methods blend agent features without explicit relational reasoning or modeling indirect interactions, despite the need to forecast behavior in crowded, interactive settings.

    Source: §I · p. 1

  • Method

    STG-DAT uses a spatio-temporal graph dual-attention network for heterogeneous-agent forecasting, alongside trajectory and scene context and a vehicle kinematic constraint layer.

    Source: §I · p. 2, C. Encoder with Graph Dual-Attention… · p. 5, F. Qualitative and Ablative Analysis · p. 10

  • Results

    STG-DAT achieves state-of-the-art prediction performance on multiple benchmark datasets and improves multi-target tracking performance.

    Source: §I · p. 2, §VII · p. 12

  • Takeaways & Limitations

    The vehicle kinematic constraint layer supports physically feasible, smoother trajectories and improves robustness to noisy data.

    Source: §I · p. 2

Abstract

from arXiv · show

An effective understanding of the environment and accurate trajectory prediction of surrounding dynamic obstacles are indispensable for intelligent mobile systems (e.g. autonomous vehicles and social robots) to achieve safe and high-quality planning when they navigate in highly interactive and crowded scenarios. Due to the existence of frequent interactions and uncertainty in the scene evolution, it is desired for the prediction system to enable relational reasoning on different entities and provide a distribution of future trajectories for each agent. In this paper, we propose a generic generative neural system (called STG-DAT) for multi-agent trajectory prediction involving heterogeneous agents. The system takes a step forward to explicit interaction modeling by incorporating relational inductive biases with a dynamic graph representation and leverages both trajectory and scene context information. We also employ an efficient kinematic constraint layer applied to vehicle trajectory prediction. The constraint not only ensures physical feasibility but also enhances model performance. Moreover, the proposed prediction model can be easily adopted by multi-target tracking frameworks. The tracking accuracy proves to be improved by empirical results. The proposed system is evaluated on three public benchmark datasets for trajectory prediction, where the agents cover pedestrians, cyclists and on-road vehicles. The experimental results demonstrate that our model achieves better performance than various baseline approaches in terms of prediction and tracking accuracy.

I. INTRODUCTION

STG-DAT targets multi-agent trajectory prediction in uncertain, interactive settings where long-term forecasts must account for agent relations, scene context, and vehicle feasibility. It combines generative prediction with spatio-temporal graph attention and is presented as useful for both forecasting and multi-target tracking.

I. INTRODUCTION

  • Long-term prediction in crowded settings must address uncertain behavior, interactions, road-layout context, and traffic rules; Fig. 1 pairs interactive scenarios with training-derived occupancy and velocity context.Source: §I · p. 1, Figure 1 · p. 2
  • The prediction model can serve as the process model in recursive Bayesian tracking, with long-term prediction supporting handling of occlusion and missing observations.Source: §I · p. 1
  • The paper extends prior work’s modified Wasserstein generative modeling from two vehicles in one scenario to multiple heterogeneous agents across scenarios.Source: §I · p. 1
  • STG-DAT is a generative system for heterogeneous interactive agents, using spatio-temporal graph attention to model relational effects and weigh information across agents and time steps.Source: §I · pp. 1–2
  • The vehicle kinematic constraint layer is designed to ensure feasible, smooth trajectories, reduce prediction error, and improve robustness to noisy data.Source: §I · p. 2
  • The approach achieves state-of-the-art prediction accuracy on multiple benchmark datasets and improves multi-target tracking performance.Source: §I · p. 2

A. Interaction-Aware Trajectory Prediction

  • Earlier physics- or rule-based methods are described as limited to short-term prediction, while learning-based methods may ignore physical feasibility and produce implausible forecasts.Source: A. Interaction-Aware Trajectory… · p. 2

B. Multi-Target Tracking and State Estimation

The section frames the proposed predictor as a more versatile alternative for tracking and describes its relational-graph and Wasserstein generative foundations. It also reviews graph neural network operations and the assumptions behind the tracking and generative approaches.

B. Multi-Target Tracking and State Estimation

  • The proposed predictor is a more versatile, higher-capacity alternative to a tracking behavior model with assumptions that restrict its applicability.The prior model assumes specific agent numbers, roles, and scenarios.Source: B. Multi-Target Tracking and State… · pp. 2–3

C. Relational Reasoning and Graph Networks

  • STG-DAT represents agents as graph nodes and their relationships as edges, using topological and temporal attention to capture interaction patterns and jointly predict behavior.Source: C. Relational Reasoning and Graph… · p. 3

D. Deep Generative Models

  • The paper uses a Wasserstein generative method as its training basis, extending prior pair-wise prediction to multi-agent prediction.The prior method is cited as.Source: D. Deep Generative Models · p. 3

III. PRELIMINARIES

  • The preliminaries introduce graph neural network basics and the Wasserstein generative modeling approach from prior work.Source: §III · p. 3

A. Graph Neural Network

  • Graph neural networks directly operate on graph structures and incorporate relational inductive bias; this setup updates agent nodes and relations without global attributes.The section describes edge and node updates as the operations used in this setup.Source: A. Graph Neural Network · p. 3

B. Wasserstein Generative Modeling

  • The cited comparison reports that WAE captures the true data distribution and learns latent representations better than vanilla VAE, which tends toward smaller variance and mode collapse.Source: B. Wasserstein Generative Modeling · p. 4

IV. PROBLEM FORMULATION

The task is to predict future trajectories for multiple interactive agents from historical states and context. The formulation allows varying agent counts and is intended for iterative use in multi-target tracking.

IV. PROBLEM FORMULATION

  • The objective is to predict future trajectories for multiple interactive agents from their historical states and context information.Source: §IV · p. 4
  • The formulation covers vehicles, pedestrians, and cyclists, and context may include raw images, semantic maps, or other relevant information.Source: §IV · p. 4
  • The model approximates a conditional future-trajectory distribution, supports a flexible number of agents, and is iteratively applied in multi-target tracking.The history and forecasting horizons are denoted Th and Tf.Source: §IV · p. 4

V. METHOD: STG-DAT

STG-DAT uses an encoder-decoder architecture to predict feasible future trajectories from agent histories and scene context. Its feature extractor embeds agent state, pairwise relations, and local context before graph-based processing.

A. System Overview

  • STG-DAT combines a feature extractor, a spatio-temporal graph encoder with dual attention, and a decoder with a kinematic constraint layer.The decoder generates feasible future trajectories, and graph weight sharing and permutation invariance support varying agent counts.Source: A. System Overview · p. 4

B. Feature Extraction

  • The feature extractor uses separate state, relation, and context embedding components to produce features at each time step.State and relation features use MLPs, while context features use a CNN.Source: B. Feature Extraction · p. 4
  • Pairwise relation embeddings encode relative agent information, while local occupancy maps and velocity fields provide context features for each agent.The relation representation uses Cartesian differences in this work; occupancy maps represent data-driven drivable regions and the velocity field supplies heuristic features.Source: B. Feature Extraction · p. 5

C. Encoder with Graph Dual-Attention Network

The encoder builds history and future spatio-temporal graphs and processes them with dual attention. Its topological layer updates node attributes using neighbor information, while graph connectivity can vary over time.

C. Encoder with Graph Dual-Attention Network

  • History and future graphs concatenate state and context features as node attributes and use relation features as edge attributes; edges are removed beyond a distance threshold.The described fixed-node case assumes the same agents appear throughout the horizon, while graph connectivity may vary by time step.Source: C. Encoder with Graph Dual-Attention… · p. 5
  • The dual-attention network first updates node attributes from the topological perspective, then produces a high-level embedding for each node through temporal attention.Temporal attention summarizes topological and temporal information and identifies the relative significance of information at each time step.Source: C. Encoder with Graph Dual-Attention… · p. 5

1) Topological Attention Layer:

  • Topological attention weights neighboring nodes using node and edge attributes, then updates node attributes through message passing at each time step.The implementation uses no prior attention bias, shares weights across time steps, and permits repeated message passing for higher-order interactions.Source: 1) Topological Attention Layer · pp. 5–6

2) Temporal Attention Layer:

The temporal-attention layer follows topological attention, while feature encoding supplies a sampled latent representation at test time. The vehicle decoder applies kinematic constraints and offers alternative ways to propagate uncertainty.

2) Temporal Attention Layer:

  • The temporal-attention layer takes topological-layer node attributes as input and outputs abstract node attributes for downstream modules.Its attention coefficients are denoted β; the supplied passages do not further describe their computation.Source: 2) Temporal Attention Layer · p. 6

3) Feature Encoding:

  • Feature encoding maps concatenated history and future node attributes to a latent variable that encodes future information conditioned on history during training.At testing, the unavailable future information is represented implicitly by sampling the latent variable from the prior distribution.Source: 3) Feature Encoding · p. 6

D. Decoder with Kinematic Constraint

  • The decoder uses a vehicle kinematic constraint after the GRU, with separate GRUs for each agent type to accommodate differing behavior patterns, speeds, and traversable areas.The recurrent decoder uses history ground truth during burn-in, then feeds each predicted position into the next step until the horizon is reached.Source: D. Decoder with Kinematic Constraint · p. 6, Figure 4 · p. 7
  • The vehicle model uses state s(k) = [x(k), y(k), φ(k), v(k), β(k)] and control input u(k) = [a(k), β̇(k)] to produce the next state.The discretized bicycle model approximates vehicle kinematics; x and y are center-of-mass coordinates, ψ is inertial heading, and v is speed.Source: D. Decoder with Kinematic Constraint · p. 6
  • The decoder can propagate uncertainty through nonlinear Monte Carlo sampling or a linearized Gaussian approximation, trading flexibility and accuracy against computational simplicity.Monte Carlo particle counts may be adjusted for real-time requirements; the Gaussian approximation has limited flexibility and generally lower accuracy, especially for multimodal distributions.Source: D. Decoder with Kinematic Constraint · p. 7

E. Loss Function and Training

The loss follows the paper’s Wasserstein generative-modeling formulation and trains the full system end to end. Its objective combines reconstruction and distribution-matching terms, with γ controlling the reconstruction term’s relative weight.

E. Loss Function and Training

  • The loss formulation uses predicted trajectories x, latent variable z, and condition variable y.The passages identify the benchmark tables as ADE/FDE comparisons for ETH & UCY and SDD, but provide no table values.Source: E. Loss Function and Training · p. 7, Table I · p. 8, Table II · p. 8
  • The loss function is based on the Wasserstein generative-modeling optimization formulation, and the system is trained end to end.The optimization problem follows, and training uses the Adam optimizer.Source: E. Loss Function and Training · p. 7
  • The objective includes reconstruction, KL-divergence, and MMD terms, with γ weighting reconstruction relative to the other loss weights.When γ ≫ α, β, the loss reduces to mean squared error.Source: E. Loss Function and Training · p. 7

VI. EXPERIMENTS

The experiments evaluate the proposed method on three public trajectory-forecasting benchmarks and compare results with state-of-the-art baselines.

VI. EXPERIMENTS

  • The method is evaluated on three publicly available trajectory-forecasting benchmarks, with results compared against state-of-the-art baselines.Source: §VI · p. 8

A. Datasets

The evaluation covers pedestrian, cyclist, and vehicle trajectory settings across several public datasets, using ADE and FDE and a range of forecasting baselines. The paper also reports training, timing, and model-architecture details.

A. Datasets

  • ETH and UCY provide pedestrian trajectories from indoor and outdoor scenes, SDD includes campus pedestrians, cyclists, and vehicles, and ID covers interactive driving scenarios.ETH/UCY and ID trajectories are in world-space meters; SDD trajectories are in image pixels.Source: A. Datasets · p. 8

B. Evaluation Metrics

  • Evaluation uses ADE over the prediction horizon and FDE at the final predicted step, with different history and prediction horizons for ID than for ETH, UCY, and SDD.For ID, the horizon is 10 steps (5.0 s) after 4 historical steps (2.0 s); the ETH, UCY, and SDD sentence specifies a 12-step (4.8 s) future horizon and continues in the next passage.Source: B. Evaluation Metrics · p. 8

C. Baseline Methods

  • The baseline set includes probabilistic and social-recurrent approaches, including P-LSTM, S-LSTM, and S-GAN.P-LSTM adds sampled normal noise to an encoder-decoder input; S-LSTM uses social pooling, and S-GAN adds adversarial learning to S-LSTM.Source: C. Baseline Methods • · p. 8
  • Other baselines include S-ATT, DESIRE, Social-BiGAT, and Trajectron, spanning graph/recurrent, stochastic-control, and generative approaches.The passages describe DESIRE as using a ranking module for sampled trajectories and Social-BiGAT as a graph-based adversarial model.Source: C. Baseline Methods • · p. 8

D. Implementation Details

  • Models used batch size 64 and trained for 100 epochs with early stopping, using Adam with an initial learning rate of 0.001 on a single NVIDIA TITAN X GPU.Source: D. Implementation Details · p. 8
  • The architecture uses a three-layer 128-unit feature MLP and ResNet18 context CNN, 64-dimensional node attributes, and a 32-dimensional latent variable.The decoder is a GRU recurrent layer with 128 hidden units.Source: D. Implementation Details · p. 9

E. Quantitative Analysis

STG-DAT outperforms the reported baselines on pedestrian and vehicle prediction benchmarks and improves tracking accuracy. Ablations associate gains with scene context, dual attention, and kinematic constraints.

E. Quantitative Analysis

  • STG-DAT achieves the smallest ADE and FDE across ETH/UCY scenes, reducing average ADE / FDE by 34.8% / 37.2% versus Trajectron.On SDD, it reduces ADE / FDE by 15.1% / 22.4% versus CGNS; on INTERACTION, the full model improves 5.0s ADE / FDE over Trajectron across RA, UI, and HR scenarios.Source: E. Quantitative Analysis • · pp. 9–10
  • STG-DAT achieves significantly higher tracking accuracy than the constant-velocity and constant-acceleration models.The paper attributes the broader advantage of learning-based tracking models to interaction-aware prediction.Source: E. Quantitative Analysis • · p. 10

F. Qualitative and Ablative Analysis

  • Qualitative predictions cover diverse vehicle and pedestrian behaviors, with groundtruth trajectories generally near the predicted-distribution mean and uncertainty represented.The examples include intersections and roundabouts, as well as going straight, turning, waiting, and stopping.Source: F. Qualitative and Ablative Analysis · p. 10, Figure 5 · p. 11, Figure 6 · p. 11
  • Adding scene context reduces 5.0s ADE / FDE versus trajectory-only prediction by 5.6% / 6.4% in RA, 9.3% / 7.7% in UI, and 2.1% / 2.4% in HR.The gap grows at longer horizons; context helps more in RA and UI, where road geometry matters, while trajectory-only samples can leave feasible areas or violate traffic rules.Source: F. Qualitative and Ablative Analysis · pp. 10–11
  • Dual attention reduces 5.0s ADE / FDE versus equal attention by 21.3% / 25.0% in RA, 27.9% / 21.5% in UI, and 7.8% / 8.0% in HR.The reported gains are larger in RA and UI, where interactions are frequent.Source: F. Qualitative and Ablative Analysis · p. 11
  • Adding the kinematic constraint layer reduces 5.0s ADE / FDE by 20.0% / 13.7% in RA, 12.5% / 16.0% in UI, and 9.5% / 9.3% in HR.The GRU outputs control actions aggregated by a bicycle model; the authors report that the constraint filters unfeasible movements and can reduce overfitting to noisy data or outliers.Source: F. Qualitative and Ablative Analysis · p. 12

VII. CONCLUSION

STG-DAT combines context, heterogeneous-agent trajectories, and physical-feasibility constraints in a generative multi-agent prediction system. Across benchmark prediction tasks it achieves state-of-the-art performance against multiple baselines, and its use in multi-target tracking empirically improves tracking accuracy.

VII. CONCLUSION

  • STG-DAT combines context, heterogeneous interactive-agent trajectories, and physical-feasibility constraints, using a graph dual-attention network and Wasserstein generative modeling.The paper validates it on pedestrian and vehicle prediction tasks across multiple benchmark datasets and reports state-of-the-art prediction performance against multiple baselines.Source: §VII · p. 12

APPENDIX A DATA PREPROCESSING

The appendix describes preprocessing and the construction of global and agent-centered local context information used in training and testing.

APPENDIX A DATA PREPROCESSING

A. Global Context Information

  • Global context consists of a normalized occupancy-density map and a mean-velocity field built from 1m × 1m bins.The density map records normalized agent-location frequency; each velocity-field bin stores average vertical and horizontal speed.Source: A. Global Context Information · p. 12

B. Localization for Local Context Information

  • For each agent, local context is extracted from the global context around its current location and aligned with its moving direction.The same local-context map size is used for all agents during training and testing.Source: B. Localization for Local Context… · p. 12

Related papers

Loading 2102.09117v1…