Source-linked AI summary
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting
Ye Yuan, Xinshuo Weng, Yanglan Ou, Kris Kitani
TL;DR
Accurate multi-agent trajectory forecasting must handle complex social interactions and uncertainty in each agent’s future behavior. AgentFormer jointly models time and social dimensions with agent-aware attention and jointly modeled latent intents. It substantially improves state-of-the-art performance on pedestrian and autonomous-driving datasets.
Problem
Multi-agent forecasting must model complex social interactions and uncertain latent intentions that shape agents’ future trajectories and influence one another.
Method
AgentFormer jointly models temporal and social dimensions using flattened trajectory sequences, agent-aware attention, and jointly inferred latent intents for all agents.
Results
AgentFormer substantially improves state-of-the-art performance on well-established pedestrian and autonomous-driving datasets.
Takeaways & Limitations
Joint modeling preserves agent identities while enabling diverse, socially aware multi-agent future trajectories.
Takeaways & Limitations
Agent-index-based agent encoding is unsuitable because agents have no innate ordering and such encodings break permutation invariance.
Abstract
from arXiv · showhide
Predicting accurate future trajectories of multiple agents is essential for autonomous systems, but is challenging due to the complex agent interaction and the uncertainty in each agent's future behavior. Forecasting multi-agent trajectories requires modeling two key dimensions: (1) time dimension, where we model the influence of past agent states over future states; (2) social dimension, where we model how the state of each agent affects others. Most prior methods model these two dimensions separately, e.g., first using a temporal model to summarize features over time for each agent independently and then modeling the interaction of the summarized features with a social model. This approach is suboptimal since independent feature encoding over either the time or social dimension can result in a loss of information. Instead, we would prefer a method that allows an agent's state at one time to directly affect another agent's state at a future time. To this end, we propose a new Transformer, AgentFormer, that jointly models the time and social dimensions. The model leverages a sequence representation of multi-agent trajectories by flattening trajectory features across time and agents. Since standard attention operations disregard the agent identity of each element in the sequence, AgentFormer uses a novel agent-aware attention mechanism that preserves agent identities by attending to elements of the same agent differently than elements of other agents. Based on AgentFormer, we propose a stochastic multi-agent trajectory prediction model that can attend to features of any agent at any previous timestep when inferring an agent's future position. The latent intent of all agents is also jointly modeled, allowing the stochasticity in one agent's behavior to affect other agents. Our method substantially improves the state of the art on well-established pedestrian and autonomous driving datasets.
1. Introduction
Multi-agent forecasting must capture complex social interactions and uncertain latent intentions across time. AgentFormer jointly models time and social dimensions, preserving agent identities and jointly modeling agents’ latent intent for socially plausible predictions.
- Motivation: Accurate multi-agent forecasting matters for autonomous-system safety but must handle complex interactions and uncertain future behavior.Each agent’s latent intent can govern its trajectory and affect other agents.
- Motivation: Time modeling captures how past states influence future states, while social modeling captures how one agent affects another.These are the two key dimensions of multi-agent interaction modeling.
- Motivation: Most prior methods encode time and social dimensions separately, limiting direct interaction between features from different agents and timesteps.Typical pipelines summarize each agent temporally before applying a social model, or reverse the order.
- AgentFormer: AgentFormer flattens trajectory features across time and agents, enabling joint representation learning across both dimensions.Its sequence representation allows an agent’s state at one time to affect another agent’s future state directly.
- AgentFormer: AgentFormer uses agent-aware attention to preserve agent identity while modeling temporal and social dependencies.The stochastic framework also jointly infers latent codes for all agents so future behaviors can exert social influence on one another.
- Results: AgentFormer outperforms state-of-the-art multi-agent prediction methods on ETH, UCY, and nuScenes datasets.The paper reports substantial improvement and supports the method with ablation studies and attention comparisons.
2. Related Work
Prior trajectory research spans sequence models, generative predictors, and social-interaction architectures. A central limitation of common pipelines is that separate temporal and social encoding prevents direct cross-time, cross-agent feature interaction.
- Sequence Modeling: RNNs historically modeled sequences, while Transformers later gained adoption for capturing long-range dependencies.These models have been applied across language, vision, pose, and trajectory-related tasks.
- Trajectory Prediction: Trajectory prediction evolved from deterministic models toward deep generative models because future trajectories are uncertain and often multi-modal.Examples include social-force models, Gaussian processes, RNNs, and conditional variational autoencoders.
- Social Interaction Modeling: Social-interaction methods commonly use RNNs or Transformers for temporal modeling and graph neural networks for agent interactions.Many approaches process temporal and social dimensions in separate stages.
- Social Interaction Modeling: Separate temporal and social encoding prevents an agent feature at one time from directly interacting with another agent’s feature at a different time.The paper identifies this as limiting the capture of long-range dependencies.
3. Approach
AgentFormer jointly models temporal and social relations in flattened multi-agent trajectory sequences, addressing the loss of time and agent-identity information in standard Transformers. Its agent-aware attention distinguishes same-agent from other-agent elements, while the stochastic framework jointly models agents’ latent intents and optional semantic-map context.
- AgentFormer: AgentFormer jointly learns temporal and social representations instead of encoding the two dimensions in separate stages.An agent’s state at one time can directly affect another agent’s state at a future time through the joint representation.
- Sequence Representation: The model represents past and future multi-agent trajectories as flattened sequences across agents and timesteps for Transformer processing.The past sequence has length N×(H+1), while the future sequence has length N×T.
- Motivation: Standard Transformers can lose timestep information and do not distinguish elements by agent identity when computing attention.These limitations make direct sequence modeling inadequate for socio-temporal trajectory relations.
- Time Encoder: A time encoder assigns timestamp features based on each element’s timestep rather than its position in the flattened sequence.The timestamp uses a sinusoidal design similar to the original Transformer’s positional encoding.
- Agent-Aware Attention: Agent-aware attention uses a mask to compute attention differently for same-agent and other-agent query-key pairs.The mechanism preserves agent identity while retaining multi-head attention for distributed representations.
- Stochastic Prediction Framework: The stochastic prediction framework jointly infers all agents’ latent intents so one agent’s future behavior can affect other agents’ predicted trajectories.The model can optionally incorporate semantic maps by rotating and cropping agent-centered patches.
4. Experiments
Experiments evaluate AgentFormer on pedestrian and autonomous-driving benchmarks, compare it with prior methods, and test its joint socio-temporal design and agent-aware attention through ablations and visualizations.
- Datasets: ETH/UCY and nuScenes provide pedestrian and autonomous-driving benchmarks for evaluating multi-agent trajectory forecasting.ETH/UCY contains pedestrian social scenarios, while nuScenes contains large-scale driving scenes with semantic maps.
- Baseline comparisons: AgentFormer achieves the best FDE on ETH/UCY and significantly outperforms prior Transformer-based methods.The authors attribute its strong FDE performance to directly attending to any agent’s features at any previous timestep when predicting future positions.
- Ablation studies: All four separate social-temporal model combinations perform worse than the jointly modeled AgentFormer.The ablation replaces AgentFormer with combinations of GCN or Transformer social models and LSTM or Transformer temporal models.
- Ablation studies: Removing joint latent intent modeling, agent-aware attention, or semantic maps worsens performance, with the largest drops from removing agent-aware attention or using agent encoding.These results support the contribution of agent-aware attention and joint latent modeling within the full forecasting framework.
- Trajectory visualization: Visualized forecasts exhibit following and collision avoidance, while attention emphasizes adjacent agents’ turning points and the target’s past timesteps.The attention pattern is described as supporting reasoning about trajectory dynamics and curvature.
5. Conclusion
AgentFormer jointly models temporal and social dimensions while preserving agent identities, and its stochastic framework jointly models agents’ latent intent. Experiments show substantial state-of-the-art gains on pedestrian and autonomous-driving datasets.
- AgentFormer simultaneously models time and social dimensions using a multi-agent trajectory sequence representation.
- Its stochastic forecasting framework uses agent-aware attention and jointly models all agents’ latent intent to produce diverse, socially-aware future trajectories.
- Substantial improvements over state-of-the-art performance were demonstrated on challenging pedestrian and autonomous-driving datasets.
A. Handling a Time-Varying Number of Agents
AgentFormer can handle changing agent counts over time by removing missing agents’ trajectory features rather than filling them. Its agent-aware attention mask is computed from query-key agent identities.
- AgentFormer generalizes to time-varying agent counts when agents leave the scene or are missed by detection.
- Missing agents’ trajectory features can simply be removed from the sequence at each timestep.
- The attention mask is computed from agreement between the agent identities of each query and key.
B. Additional Implementation Details
The supplementary details describe semantic-map encoding, trajectory-sampler training, separate-model ablations, and variants isolating joint latent modeling and agent-aware attention.
- Encoding Semantic Maps: For nuScenes, each agent’s semantic map has 100×100 spatial dimensions, three channels, and 3-meter pixel spacing.
- Training Trajectory Sampler: The trajectory sampler uses dataset-specific scaling factors, KL clipping, Adam optimization, 50 epochs, and scheduled learning-rate halving.
- Ablation Study Details: Separate-model ablations combine temporal LSTM or Transformer modules with social GCN or Transformer modules before trajectory decoding.
- Ablation Study Details: Component ablations remove joint latent modeling or replace agent-aware attention with standard attention, with an additional agent-encoding variant.
C. Additional Attention Visualization
Attention visualizations show that AgentFormer can use any agent’s past features when predicting a future position. Attention is strongest for the target agent’s recent history and nearby agents.
- The model attends to past and future trajectory features of all agents when inferring a target future position.
- It pays more attention to the target agent’s own trajectories and recent timesteps than to other temporal features.
- It attends more to nearby agents than distant agents.
D. Trajectory Sample Visualization
On nuScenes, AgentFormer’s agent-aware attention produces trajectory samples that cover ground-truth futures better and generate fewer implausible paths than the version without it.
- Agent-aware attention covers the ground-truth future trajectories significantly better than the variant without agent-aware attention.
- The full method produces much fewer implausible trajectories, including paths that go out of the road.
- Figure 7 compares trajectory samples from the method and its variant without agent-aware attention on nuScenes.