Source-linked AI summary
Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction
Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D'Souza, Samira Ebrahimi Kahou, Felix Heide, Christopher Pal
TL;DR
Robust multi-agent trajectory prediction requires modeling contextual, social, and temporal information while capturing diverse futures. The paper proposes latent-variable sequential set Transformers called AutoBots, using temporal and social attention with learnable seeds for scene-consistent prediction. AutoBot achieves competitive or strong benchmark results across motion-forecasting tasks, with computationally efficient training, while requiring structured perceptual inputs and facing memory limits.
Problem
Motion prediction requires models that are insensitive to agent ordering while capturing high-order interactions and multi-modal future trajectories.
Method
AutoBot is a latent-variable encoder-decoder for sequences of sets that uses multi-head attention for temporal and social processing and predicts future trajectories through discrete modes.
Results
AutoBot achieves competitive ego-centric nuScenes performance, strong Argoverse performance, and socially consistent multi-agent predictions, while training on a single desktop GPU.
Takeaways & Limitations
The model supports scene-consistent multi-agent forecasting and diverse sequence-of-sets generation across driving, pedestrian, and handwritten-stroke tasks.
Takeaways & Limitations
AutoBot depends on a perceptual pipeline supplying structured entity sets and does not model errors or unknown entities omitted from those inputs; memory may limit set count.
Abstract
from arXiv · showhide
Robust multi-agent trajectory prediction is essential for the safe control of robotic systems. A major challenge is to efficiently learn a representation that approximates the true joint distribution of contextual, social, and temporal information to enable planning. We propose Latent Variable Sequential Set Transformers which are encoder-decoder architectures that generate scene-consistent multi-agent trajectories. We refer to these architectures as "AutoBots". The encoder is a stack of interleaved temporal and social multi-head self-attention (MHSA) modules which alternately perform equivariant processing across the temporal and social dimensions. The decoder employs learnable seed parameters in combination with temporal and social MHSA modules allowing it to perform inference over the entire future scene in a single forward pass efficiently. AutoBots can produce either the trajectory of one ego-agent or a distribution over the future trajectories for all agents in the scene. For the single-agent prediction case, our model achieves top results on the global nuScenes vehicle motion prediction leaderboard, and produces strong results on the Argoverse vehicle prediction challenge. In the multi-agent setting, we evaluate on the synthetic partition of TrajNet++ dataset to showcase the model's socially-consistent predictions. We also demonstrate our model on general sequences of sets and provide illustrative experiments modelling the sequential structure of the multiple strokes that make up symbols in the Omniglot data. A distinguishing feature of AutoBots is that all models are trainable on a single desktop GPU (1080 Ti) in under 48h.
1 INTRODUCTION
The paper addresses generative modeling for sequences of sets, where models must respect unordered agent structure while capturing temporal interactions and diverse futures. It proposes AutoBot, a latent-variable conditional mixture model evaluated across motion forecasting and sequential-stroke tasks.
- Problem: Motion prediction models should be insensitive to agent ordering while processing trajectories that evolve over time.The setting combines temporal sequences with unordered sets of agents.
- Problem: High-order interactions and diverse futures motivate models that represent multi-modal distributions beyond standard autoregressive sequence models.The formulation introduces a discrete latent variable Z for sequential models of sets.
- Architecture: The model generates multiple possible futures by encoding input trajectories into context and decoding all future steps at once using learnable seed parameters.Each possible future is treated as a trajectory mode.
- Method: AutoBot is a parameterized conditional mixture model for sequences of sets with a discrete latent-variable formulation.The model is called a Latent Variable Sequential Set Transformer.
- Contributions: The approach proposes modeling set-structured continuous variables, analyzes permutation sensitivity, and evaluates diverse tasks.Evaluations include autonomous-driving, pedestrian-trajectory, handwritten-character, and stroke-prediction settings.
2 BACKGROUND
The background reviews Transformer and Set Transformer attention components used to process sequence and set structure. AutoBot’s architecture applies these components to temporal and social dimensions for efficient encoding and decoding.
- Architectural background: Transformers and Set Transformers provide the architectural background for processing sequences and sets.The paper reviews these components before describing AutoBot.
- Attention blocks: Multi-Head Self Attention retrieves values from key-value representations based on query-key matching and includes a residual connection.For a set-valued input X, MHSA performs intra-set attention.
- Attention blocks: Multi-Head Attention Blocks combine MHSA with a row-wise feed-forward network, residual connections, and layer normalization.The block omits positional encoding and dropout compared with the reviewed Transformer encoder.
- Decoder blocks: Multi-head Attention Block Decoders are used to produce decoded sequences from sequence inputs.AutoBot uses these decoder blocks in its architecture.
- AutoBot architecture: AutoBot’s architecture alternates attention over time and agents in the encoder, then applies temporal decoder attention followed by agent attention.The decoder uses encoder context and learnable seeds to predict each mode.
3 LATENT VARIABLE SEQUENTIAL SET TRANSFORMERS
AutoBot is a latent-variable encoder-decoder architecture for processing sequences of sets and generating multimodal, temporally and socially consistent predictions. Its alternating temporal and social processing, learnable decoder seeds, and exact discrete-latent training objective support efficient multi-agent forecasting.
- 3.1 ENCODER: INPUT SEQUENCE REPRESENTATION: AutoBot encodes sequences of sets by repeatedly processing temporal relationships and then temporal slices across agents.The encoder applies temporal MAB processing to each agent’s evolution and social MAB processing to each timestep, repeating both operations Lenc times to form context C.
- 3.2 DECODER: MULTIMODAL SEQUENCE GENERATION: The decoder uses learnable seed matrices, each corresponding to a discrete latent-variable setting, to generate c predictions for one scene.Each seed is repeated across agents, combined with encoded environmental features, and decoded separately for each mode.
- 3.2 DECODER: MULTIMODAL SEQUENCE GENERATION: Temporal decoding is followed by per-timestep attention across agents to enforce socially consistent future scenes.The decoder first encodes each element’s future evolution independently, then applies a MAB to each temporal slice; these operations repeat Ldec times.
- 3.2 DECODER: MULTIMODAL SEQUENCE GENERATION: The decoder outputs a tensor over hidden features, agents, future timesteps, and modes, which φ transforms into the desired prediction representation.In trajectory experiments, φ produces parameters for distributions in (x, y) space.
- 3.3 TRAINING OBJECTIVE: Training maximizes the likelihood of future trajectories conditioned on observed sequence-of-sets inputs using discrete latent variables with exactly computable log-likelihoods.The objective uses an exact posterior over discrete latent variables and minimizes the KL divergence between the old posterior and the model prior.
- 3.3 TRAINING OBJECTIVE: Mode entropy regularization penalizes only the output mode with maximum entropy to reduce high-variance mode sequences and improve coverage of data modes.The model uses a bivariate Gaussian for each agent at every timestep, and experiments report the importance of this regularizer for learning to cover modes.
4 EXPERIMENTS
AutoBots are evaluated across autonomous-driving, multi-agent forecasting, and sequential stroke-generation tasks. Results show strong trajectory prediction, socially consistent joint forecasts, and qualitative advantages on Omniglot, with efficient computation.
- nuScenes: AutoBot-Ego achieves the best Min ADE (10) and Off Road Rate performance on nuScenes, with strong results on the other metrics.The model produces road-consistent trajectories and covers multiple future directions, while assigning different speeds to modes.
- Argoverse: AutoBot-Ego performs strongly on Argoverse relative to the 2020 competition winner while using less computation, less augmentation, and no ensembling.The authors note that many leaderboard methods use additional data or ensembling, complicating direct comparisons.
- Omniglot: AutoBot produces more consistent and realistic Omniglot characters than an LSTM baseline in stroke completion, while AutoBot-Ego improves character completion across latent modes.The tasks provide partial strokes or complete strokes and require parallel completion or generation of a final stroke.
- TrajNet++: Social attention in the decoder significantly reduces collisions and improves scene-level minADE and minFDE versus ablated multi-agent variants.AutoBot forecasts all agents jointly rather than decomposing the scene rollout into independent individual futures.
5 RELATED WORK
Prior work addresses set processing, diverse generation, and sequential state modelling, while AutoBots jointly models heterogeneous structures composed of sequences and sets.
- Processing set structured data: Set-valued models should be permutation-invariant and support arbitrary cardinality, but standard feed-forward and recurrent networks do not inherently provide these properties.Pooling, graph-based models, and related approaches address set processing and high-order interactions.
- Processing set structured data: AutoBots uses a joint single-pass attention-based approach to model complex structures composed of sets and sequences while generating diverse, heterogeneous data.This contrasts with approaches focused primarily on set processing.
- Diverse set generation: Diverse-generation methods include variational models, Gaussian mixtures, and continuous or discrete latent-variable approaches for producing multiple futures.Prior motion-prediction methods encode individual or joint intentions, sample timestep-wise mixtures, or use latent variables.
- Diverse set generation: AutoBots builds on joint latent-variable modelling by using one transformer conditioned on latent variables and adding a maximum-entropy likelihood loss to improve outcome variability.The method targets diverse samples while jointly modelling multiple actors.
- Sequential state modelling: Prior motion-prediction work largely uses recurrent networks or separately encodes agents, whereas AutoBots jointly represents temporal and social dimensions as a sequence of sets.The paper describes this as reusing model capacity in a lightweight approach.
6 CONCLUSION
The paper proposes AutoBots for modelling time-evolving sets with discrete latent variables and evaluates its ability to produce diverse, scene-consistent predictions. It also identifies structured-input and memory boundaries for the approach.
- 6 CONCLUSION: AutoBots models time-evolution in sequential sets using discrete latent variables and multi-head attention for intra-set, temporal, and latent-mode processing.The architecture targets joint representations of social and temporal information.
- 6 CONCLUSION: AutoBot-Ego achieves competitive nuScenes ego-centric motion-forecasting results, while AutoBot produces diverse sequences of sets and scene-consistent TrajNet++ predictions.Scene consistency is attributed to attending to all agents’ hidden states during generation.
- 6 CONCLUSION: The approach requires a perceptual pipeline to provide structured sets of entities, so errors or unknown entities omitted from those sets are not modelled.Memory limitations may also constrain the number of sets processed at a time.
- B.2 AUTOBOT ENCODER PROPERTIES: The model’s permutation-equivariance analysis applies to the unordered agent dimension M of tensors representing agents with ordered attributes and time.The appendix motivates this structure for objects that may not be uniquely identifiable.
B.2 AUTOBOT ENCODER PROPERTIES
The appendix establishes permutation equivariance for AutoBot’s encoder and decoder across the unordered agent dimension, and describes ego-only forecasting and latent-mode conditioning mechanisms.
- B.2 AUTOBOT ENCODER PROPERTIES: For an input tensor with dimensions (K, M, t), the M dimension is unordered while K attributes and t time are ordered.Selecting a timestep yields a multiset of M K-dimensional vectors.
- B.2 AUTOBOT ENCODER PROPERTIES: AutoBot’s encoder is permutation equivariant on M because its embedding, intra-set attention, and temporally encoded attention functions are each equivariant.The proof uses uniform application across sets and composition of equivariant functions.
- B.2 AUTOBOT ENCODER PROPERTIES: AutoBot’s decoder is permutation equivariant on M because its feed-forward, temporal, and intra-set attention components preserve equivariance.The decoder is initialized with a seed set and iterated autoregressively in the appendix proof.
- AutoBot-Ego: AutoBot-Ego selects the ego element after intra-set attention and encodes its history conditioned on the past sequence of all elements.Its decoder omits intra-set attention because it generates only one agent’s future.
- Latent variables: The model uses learnable mode-specific vectors to compute a prior distribution over discrete latent variables before observing the future.A row-wise linear projection maps the mode representations to a vector of size c.
- Context encoder: For contextual maps, a four-layer CNN encodes birds-eye-view inputs into a 7×7×(7∗c) volume before further mode-based processing.The nuScenes and TrajNet inputs provide birds-eye-view scene maps.
C.4 IMPLEMENTATION AND TRAINING DETAILS
The implementation uses standard optimization and regularization choices, reports short convergence time on nuScenes, and compares one-shot seed decoding with autoregressive decoding across scaling variables.
- Implementation and training: AutoBots is implemented in PyTorch, uses dropout in encoder and decoder blocks, and is trained with Adam and scheduled learning-rate annealing.The reported hyperparameters vary across the four datasets.
- Implementation and training: Approximately 3 hours of compute on one Nvidia Geforce GTX 1080Ti correspond to about 80 epochs for the nuScenes experiments.The run uses approximately 2 GB of VRAM.
- Inference-rate ablation: Seed parameters enable one-shot prediction of the entire future trajectory, contrasting with autoregressive rollout timestep by timestep.The paper evaluates an otherwise identical autoregressive ablation across numbers of agents, input steps, prediction horizon, and modes.
D.1 ABLATION STUDY ON NUSCENES
The nuScenes ablation study evaluates AutoBot-Ego by removing map information and the latent-variable component. These ablations test the contributions of map conditioning and non-autoregressive latent decoding.
- Ablation findings: Removing map information makes AutoBot-Ego perform significantly worse across all reported metrics.The ablation indicates that the model attends effectively to map information.
- Ablation findings: The No Latent Variable variant removes both the discrete latent variable and decoder seed parameters.Without these components, the model becomes autoregressive and samples from the output distribution.
- Experimental setup: Table 5 compares AutoBot-Ego against ablated counterparts on the nuScenes dataset.The study focuses on measuring the performance benefits of AutoBot-Ego.
D.2 ANALYSIS OF ENTROPY REGULARIZATION
The entropy-regularization analysis examines how the entropy loss affects multimodal trajectory prediction. Moderate regularization improves concentration around distinct modes, while increasing regularization improves most nuScenes metrics up to a limit.
- Particle experiment: Moderate entropy regularization learns low-variance modes without overlap, unlike training without entropy loss.With λe = 0.0, all modes are covered, but some short trajectories use one mode with uncertainty growing over time.
- NuScenes ablation: Increasing λe improves all evaluated nuScenes metrics except min FDE, which remains poor regardless of the model.The analysis reports that enforcing low entropy on predicted bivariate distributions positively affects multimodal predictions.
- NuScenes ablation: Increased entropy regularization improves AutoBot-Ego performance, particularly Min ADE 10 and Off Road Rate, up to a limit.Table 6 varies the entropy regularization term on nuScenes.
D.3 TRAJNET++ ABLATION AND QUALITATIVE RESULTS
The TrajNet++ ablation varies the numbers of social and temporal layers in AutoBot’s encoder and decoder. Increasing decoder depth improves joint metrics, consistent with stronger modeling of high-order scene interactions.
- Layer ablation: Increasing the number of decoder layers improves performance on the joint metrics.The comparison is reported for the general AutoBot architecture.
- Layer ablation: More social decoder layers allow the model to use high-order interactions when modeling the scene.The reported relationship is presented as an expected consequence of increasing decoder depth.
- Qualitative comparison: Figure 9 provides an example scene generated by each model variant.The figure accompanies the encoder-and-decoder layer ablation study.
D.4 TRAJNET, A PEDESTRIAN MOTION PREDICTION DATASET
The TrajNet experiments evaluate AutoBot for multi-agent forecasting over 12 future timesteps and compare variants with different social-attention components. Social attention during decoding produces fewer collisions and more scene-consistent multimodal futures.
- Dataset and task: AutoBot predicts the next 12 timesteps from all agents’ states over the past eight timesteps.The experiment applies the general architecture to multi-agent scene forecasting.
- Model variants: AutoBot-AntiSocial removes intra-set attention from the encoder and decoder, while AutoBot-Ego removes social attention during trajectory generation.AutoBot retains the general architecture with social attention.
- Quantitative results: Using social data in the decoder greatly reduces collisions and improves scene-level minADE and minFDE relative to the ablated variants.The models use c = 5 predictions and report scene-level error metrics.
- Qualitative results: AutoBot produces scene-consistent predictions across all shown modes, whereas the ablated variants include collisions or socially inconsistent trajectories.The qualitative comparisons include Figure 10 and additional TrajNet results.
- Qualitative results: AutoBot’s modes condition the entire scene future evolution, supporting alternative realistic multi-agent futures rather than independent one-agent predictions.The results emphasize sequential set attention in decoding and parallel prediction of all agents.
D.5 OMNIGLOT ADDITIONAL DETAILS AND RESULTS
Additional Omniglot experiments show that AutoBot generally generates plausible, diverse, and consistent character completions, while the LSTM baseline often struggles with legibility or valid discrete latent variations. Some failure cases remain, including incorrect character completions by both models.
- Additional task 1 results highlight AutoBot’s effectiveness for generating consistent, diverse futures from input stroke sequences.
- In task 2, AutoBot-Ego produces realistic character variations, although not every mode predicts the correct character.
- Qualitative TrajNet examples show AutoBot maintaining scene consistency, social distance, and safe-looking paths across multi-agent modes.
- AutoBot generates plausible characters on Omniglot test examples, whereas the LSTM struggles to create legible ones.
- Some task 1 characters remain difficult: AutoBot and the LSTM both fail to complete certain characters correctly, with the cause unresolved.