Source-linked AI summary
THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling
Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, Fabien Moutarde
TL;DR
Multi-agent forecasting must represent multiple possible futures while keeping agents’ scene-level modalities coherent and collision-free. THOMAS combines hierarchical sparse heatmap prediction with learned trajectory recombination, achieving state-of-the-art scene-consistent prediction and first place on the Interaction online leaderboard. Its central assumption is that good proposals already exist in marginal predictions and mainly require coherent alignment.
Problem
Predicting multiple agents requires diverse futures whose modalities remain coherent across agents rather than producing mutually colliding independent predictions.
Method
THOMAS uses a graph-based encoder, hierarchical sparse heatmap decoding, deterministic endpoint sampling, and learned recombination of marginal trajectories into joint predictions.
Results
THOMAS achieves state-of-the-art scene-consistent prediction, significantly improves over the vanilla model, and ranks 1st on the Interaction online test leaderboard.
Takeaways & Limitations
The recombination module can be added after multimodal trajectory predictors to generate scene-consistent modalities across agents.
Takeaways & Limitations
THOMAS assumes good trajectory proposals already exist in marginal predictions and need coherent alignment among agents.
Abstract
from arXiv · showhide
In this paper, we propose THOMAS, a joint multi-agent trajectory prediction framework allowing for an efficient and consistent prediction of multi-agent multi-modal trajectories. We present a unified model architecture for simultaneous agent future heatmap estimation, in which we leverage hierarchical and sparse image generation for fast and memory-efficient inference. We propose a learnable trajectory recombination model that takes as input a set of predicted trajectories for each agent and outputs its consistent reordered recombination. This recombination module is able to realign the initially independent modalities so that they do no collide and are coherent with each other. We report our results on the Interaction multi-agent prediction challenge and rank $1^{st}$ on the online test leaderboard.
1 INTRODUCTION
THOMAS addresses the challenge of predicting fast, diverse, and mutually coherent trajectories for all neighboring agents. It combines efficient heatmap-based multimodal prediction with learned recombination of agent trajectories.
- Motivation: Autonomous-driving systems need fast, coherent future trajectories for all neighboring agents to plan vehicle actions.Motion forecasting transforms perception data into future predictions used for planning.
- Motivation: Agent interaction matters because vehicles must adapt speed, yield right of way, and merge around neighbors.Prior interaction mechanisms include social pooling, graphs, and attention.
- Motivation: Multimodality is necessary because agents may follow discrete alternatives such as turning left or right, which Gaussian variance cannot represent.Modern predictors therefore generate K possible trajectories for each agent.
- Problem: Independent per-agent modalities can produce mutually inconsistent scenes, including collisions between the most probable trajectories of different agents.Earlier prediction datasets did not account for cross-agent modality coherence.
- THOMAS: THOMAS predicts sparse future probability heatmaps per agent, samples diverse endpoints, and recombines them to enforce scene consistency.The model encodes agent histories and the HD-Map lanelet graph, while hierarchical refinement supports efficient decoding.
- Contributions: The hierarchical heatmap decoder enables unconstrained, computationally efficient simultaneous multi-agent prediction.Its design targets optimized computational costs for heatmap generation.
- Contributions: The recombination model realigns sampled endpoints into scene-consistent trajectories across agents.It addresses collisions and coherence among independently predicted modalities.
2 RELATED WORK
Related work develops learning-based, multimodal, and multi-agent trajectory predictors using recurrent, convolutional, graph, attention, and candidate-based approaches. THOMAS builds on sparse heatmap prediction while targeting the unresolved problem of scene consistency across agents.
- Learning-based prediction: Learning-based trajectory predictors use recurrent, convolutional, graph-neural, and point-based architectures to model histories, maps, and ordered trajectory points.These approaches have rapidly overtaken physics-based methods in trajectory prediction.
- Multimodality: Multimodal prediction is produced through multiple heads or candidate-based methods using anchors, generators, context graphs, or dense spatial grids.The passage lists several alternative ways to obtain endpoint candidates.
- Multi-agent prediction: Multi-agent predictors include simultaneous-agent models, autoregressive roll-outs, and architectures that repeat or combine agent features.Much prior work still focuses on more single-agent-oriented frameworks.
- Closest approaches: GOHOME uses a graph encoder and sparse lane-raster heatmaps, while DenseTNT uses attention over a dense point grid.The cited approaches constrain predictions by drivable areas or their neighborhoods.
- Open problem: Previous methods did not consider scene consistency for multi-agent prediction.This leaves cross-agent consistency as the problem most directly distinguishing THOMAS from the related approaches described here.
3 METHOD
THOMAS encodes agent histories, HD-map context, and interactions, then hierarchically decodes sparse future heatmaps and recombines independently sampled modalities into scene-consistent predictions.
- Graph encoder: The shared graph encoder summarizes each agent’s history, HD-map context, and interactions for subsequent decoding and modality recombination.TrajEncoder processes trajectories, GraphEncoder processes lanelet graphs, and attention layers incorporate lane and agent interactions.
- Hierarchical heatmap decoder: Hierarchical heatmap decoding evaluates the full spatial grid coarsely, then refines only the highest-ranking cells until reaching the desired resolution.At each level, selected grid points are upsampled and processed using agent encodings and coordinate features.
- Hierarchical heatmap decoder: The sparse hierarchical process computes 1,856 grid points instead of 147,456 for a 192-meter range at 0.5-meter resolution, with no performance loss.The example uses initial resolution R0 = 8m, two refinement iterations with (N1, N2) = (16, 64), and upscaling factor f = 4.
- Trajectory generation: Each agent heatmap is sampled for K endpoints to maximize coverage, after which a trajectory generator reconstructs full trajectories from agent history and endpoints.The full trajectories are generated with a fully connected MLP, while the endpoint sampler follows the MissRate optimization algorithm.
- Modality recombination: Scene modalities must align agents’ maneuvers and avoid collisions so that joint predictions represent realistic scenarios.The recombination module is trained with a scene-level minimum displacement error using winner-takes-all selection over scene modalities.
- Modality recombination: The recombination module converts independently sampled marginal modalities into scene modalities by selecting coherent endpoint associations across agents.Its learned scene embeddings query agent modality vectors through cross-attention and compute matching scores for candidate assignments.
4 EXPERIMENTS
Experiments evaluate THOMAS on joint multi-agent consistency, recombination, qualitative behavior, and decoding efficiency using the Interaction dataset and related baselines. THOMAS improves joint consistency while preserving comparable marginal distance performance and reorganizing modalities to avoid collisions.
- 47,584 training, 11,794 validation, and 2,644 testing cases evaluate scenes containing 1–40 agents predicted simultaneously.
- Comparison with state-of-the-art: THOMAS significantly outperforms ILVM and SceneTransformer on every joint metric while maintaining comparable marginal distance performance.The comparison uses a GOHOME encoder and adapts each method to predict endpoints for a fair evaluation.
- Recombination module: THOMAS reduces joint SMR from 14.8% to 11.8% relative to joint algorithmic sampling, which reduces collision rate from 7.2% to 2.6% without improving joint SMR similarly.The authors characterize THOMAS as a hybrid sampler that learns to recombine deterministic samples into more coherent solutions.
- Recombination module: Scalar joint models struggle to learn multimodality because the same selected modality is trained across all agents, producing poor marginal error and high cross-collision rates.SceneTransformer avoids the same issue by sharing weights across modalities and differentiating them through feature initialization.
- Qualitative examples: The recombination model selects and reorders modalities already present in the vanilla predictions so non-colliding choices align across agents.Qualitative examples compare the initial best modality with the reordered best modality after recombination.
- Qualitative examples: After recombination, modality spread decreases for most agents while multimodality remains when agents have substantially different possible intentions, such as different exits or turns.
5 CONCLUSION
THOMAS is a recombination module that can follow any multi-modal trajectory predictor to produce scene-consistent modalities across agents. The authors report significant gains over the vanilla model and state-of-the-art results for scene-consistent prediction.
- THOMAS adds a recombination module after multi-modal trajectory prediction to generate scene-consistent modalities across agents.It selects coherent agent modalities and restricts the modality budget to agents that need it.
- THOMAS achieves significant performance increases over the vanilla model and state-of-the-art results against existing scene-consistent prediction methods.
- The reported experiments use the publicly available Interaction 1.2 dataset.Dataset preprocessing, training, and architecture details are provided in the appendices.
A.1 METRIC DETAILS
The evaluation adopts each benchmark’s specified miss definition. Argoverse and NuScenes use a fixed 2-meter threshold, while Waymo and Interaction use lateral and speed-dependent longitudinal thresholds.
- Argoverse and NuScenes define a miss as a prediction farther than 2 meters from the ground truth.
- Waymo and Interaction define a miss using a 1-meter lateral threshold and a longitudinal threshold that depends on speed.
- The Interaction longitudinal threshold is 1 meter below 1.4m/s, increases with speed from 1.4m/s to 11m/s, and is 2 meters at or above 11m/s.
A.2 DATASET PROCESSING
Interaction 1.2 scenes are centered on a reference agent and use available-agent histories together with lane-graph context. Training includes agents with ground truth at the 3-second prediction horizon.
- Training uses the provided Interaction 1.2 training/validation split and includes agents with ground truth at prediction horizon T=3s.
- Scenes are centered and oriented using a randomly chosen reference agent during training and the agent nearest the all-agent barycenter during validation and testing.
- HD-Maps are converted into context graphs with predecessor, successor, left, and right lanelet relations.Long lanelets are upsampled to a maximum of 10 points each.
- Each agent history contains position, yaw, and speed from the past second sampled at 10Hz, plus a presence mask and zero padding for untracked frames.
A.3 TRAINING DETAILS
Models are trained with Adam under a scheduled learning-rate decay and a 16-epoch stopping point. Training limits each scene to eight randomly sampled predicted agents, while heatmaps use multi-resolution focal-loss supervision.
- Training uses Adam with batch size 32, an initial learning rate of 1e−3, halving at epochs 3, 6, 9, and 13, and stopping at epoch 16.
- Because scenes can contain up to 40 agents, training randomly samples eight available agents for prediction in each batch.No other data augmentation is used.
- Final heatmaps cover 192 meters at 0.5m resolution, producing 384 × 384 images.
A.4.1 IMPLICIT LATENT VARIABLE MODEL
THOMAS combines heatmap-based endpoint generation with collision-aware deterministic sampling and hierarchical refinement, then evaluates speed, coverage, and leaderboard performance on multi-agent prediction tasks.
- Model and sampling: THOMAS uses GOHOME-based Scene Interaction Modules for the prior, posterior, and decoder, with β = 1 weighting the KL term.
- Model and sampling: The deterministic sampler selects endpoints sequentially across agents and modalities while suppressing nearby heatmap mass to discourage collisions.It zeros corresponding regions on the sampled agent’s other modalities and on other agents’ heatmaps within the same modality.
- Model and sampling: Collision-aware sampling substantially reduces collision rate but barely improves multi-agent consistency without an additional learned model.
- Evaluation: The online leaderboard comparison notes that DenseTNT achieves 0% SCR because it explicitly checks collisions during prediction search, directly improving its cSMR.
- Evaluation: Hierarchical refinement trades inference speed against marginal MissRate6 coverage by varying the final-iteration upsampling count from 2 to 128.
- Evaluation: Below N = 16, coverage diminishes while speed gains are small; the model therefore retains N = 64, with inference time increasing from 41 ms to 46 ms.