PaperlayerView exact revision on arXiv

Source-linked AI summary

HOME: Heatmap Output for future Motion Estimation

Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, Fabien Moutarde

arXiv:2105.10968v2cs.CVcs.RO

TL;DR

Motion forecasting needs multimodal predictions that cover diverse future behaviors without being restricted to a fixed set of trajectories. HOME predicts an unconstrained 2D future-location probability heatmap, samples final points with metric-specific procedures, and decodes trajectories from them. On Argoverse, it ranks first on the benchmark leaderboard and provides a tunable miss-rate/FDE trade-off without retraining.

  • Problem

    Motion forecasting must cover diverse plausible futures and avoid large misses, while fixed trajectory outputs can suffer mode collapse or restricted prediction spaces.

  • Method

    HOME predicts an unconstrained 2D heatmap of future location probabilities, samples finite final points with two metric-specific algorithms, and builds trajectories conditioned on them.

  • Results

    HOME ranks first on the Argoverse Motion Forecasting benchmark leaderboard and achieves state-of-the-art performance through optimized heatmap sampling.

  • Takeaways & Limitations

    The sampling parameter controls the miss-rate/FDE trade-off for multimodal predictions without retraining the model.

Abstract

from arXiv · show

In this paper, we propose HOME, a framework tackling the motion forecasting problem with an image output representing the probability distribution of the agent's future location. This method allows for a simple architecture with classic convolution networks coupled with attention mechanism for agent interactions, and outputs an unconstrained 2D top-view representation of the agent's possible future. Based on this output, we design two methods to sample a finite set of agent's future locations. These methods allow us to control the optimization trade-off between miss rate and final displacement error for multiple modalities without having to retrain any part of the model. We apply our method to the Argoverse Motion Forecasting Benchmark and achieve 1st place on the online leaderboard.

I. INTRODUCTION

HOME reframes multimodal motion forecasting as an unconstrained 2D probability heatmap over future agent locations, then samples final points and decodes trajectories. Its sampling methods expose a tunable trade-off between miss rate and final displacement error.

  • Motivation: Motion forecasting must capture map context, agent history, and interactions while covering multiple plausible driving choices.The motivation includes behaviors such as turning, slowing, overtaking, and going straight.
  • Motivation: Coverage is critical because a single unconsidered possibility can cause a large miss, even when other predictions are close.The paper frames misses as more consequential than small offsets across most predictions.
  • Motivation: Fixed multi-trajectory regression can suffer mode collapse, while training only the closest prediction reduces data allocated to other modalities.These drawbacks motivate alternatives to directly predicting a fixed set of trajectories.
  • Approach: HOME represents future-location uncertainty with an unconstrained 2D heatmap whose pixels encode possible final positions.The heatmap avoids choosing modes or means and accommodates multimodal predictions.
  • Contributions: The framework uses a CNN, RNN, and Attention module, with two sampling algorithms that optimize MRk or minFDEk.The architecture produces the heatmap, while sampling selects finite future locations for the desired metric.
  • Contributions: A simple sampling parameter controls the trade-off between miss rate and final displacement error without retraining.This directly separates model training from the metric-specific sampling choice.

II. RELATED WORK

Prior motion-forecasting methods use CNNs, recurrent and attention-based interaction models, anchors, generative sampling, or HD-map graphs. These approaches improve structure or diversity but can constrain predictions or limit metric-aware sampling.

  • Neural architectures: CNNs process rasterized maps for 2D coordinate prediction, while attention modules model interactions among agents, lanes, and joint representations.Examples combine 1D CNNs, LSTMs, multi-head attention, and CNN-encoded maps in different configurations.
  • Trajectory proposals: Anchor-trajectory methods avoid mode collapse and encourage realistic trajectories but reduce the ability to tune trajectories to the current situation.Anchors may be predefined or generated by a model.
  • Generative methods: Generative approaches model future probability distributions but require multiple independent inference samples without optimizing coverage or average distance.Their sampling procedure does not directly control those objectives.
  • HD-map graph methods: HD-map graph methods encode lane connectivity and structure multimodal outputs around lanes, nodes, goals, or candidate polylines.VectorNet and LaneGCN encode map and actor structure, while TNT, GoalNet, WIMP, and LaneRCNN organize outputs using graph-derived candidates.
  • Grid-based outputs: Grid-based outputs have been used for pedestrian prediction, while vehicle occupancy grids may lack instance-level individual prediction.The paper distinguishes these prior grid representations from its architecture, training, and sampling strategies.

III. METHOD

HOME takes rasterized context and agent histories as input, predicts a future-position heatmap, samples final locations, and builds trajectories conditioned on those endpoints. The method focuses prediction on final points while retaining full trajectory outputs.

  • Problem formulation: The pipeline predicts target-agent positions over a future horizon T from past target and neighbor trajectories plus HD-map context.The formulation focuses on final points and then regresses the full trajectory conditioned on the endpoint.
  • Pipeline: The model receives a rasterized environment image and outputs a probability heatmap for the agent’s location at a fixed future time.A finite set of locations is extracted from the heatmap for coverage-oriented prediction.
  • Map and past trajectory encoding: The input raster combines HD-map semantics, target history, and neighboring-agent histories in a (224, 224, 45) image.Each pixel represents 0.5 x 0.5 m², and the CNN encodes the image to (14, 14, 512).
  • Map and past trajectory encoding: Agent histories are represented as 2D coordinates with padding masks and timestamps, then processed by a 1D convolution and UGRU recurrent layer.Weights are shared across agents except for the target agent.

2) Inter-agent attention for interaction:

The model uses attention to encode interactions between the target agent and other actors, then decodes the combined representation into a spatial probability heatmap.

  • Inter-agent attention for interaction:: Attention models interactions by using the target agent as query and other actors as keys and values.The resulting context vector is added to the target vehicle feature through a residual connection and LayerNormalization.
  • Inter-agent attention for interaction:: The trajectory encoding is repeated to match the context encoding before both representations are concatenated.
  • Inter-agent attention for interaction:: Transpose convolutions increase output size while retaining spatial correspondences through the network.They use stride 1 and kernel size 3.
  • Inter-agent attention for interaction:: The decoder alternates transpose and classic convolutions, ending with a sigmoid to produce a heatmap at 0.5 x 0.5 m² per pixel.The target heatmap places a Gaussian around the ground-truth position and is trained with pixel-wise focal loss.
  • Inter-agent attention for interaction:: The Gaussian uses a standard deviation of 4 pixels, while its surrounding non-null pixels reduce penalties during training.A squared error factor focuses gradients on poorly predicted pixels.

C. Modality sampling

HOME samples multiple future locations from its probability heatmap using a miss-rate objective based on coverage within threshold-radius circles.

  • Modality sampling: Miss Rate counts predictions farther than a specified threshold from the ground truth, while FDE measures mean l2 distance.For multimodal predictions, the corresponding metrics use the minimum over k predictions.
  • Modality sampling: The miss-rate sampler maximizes the heatmap probability integrated over radius-2m circles around the k predicted locations.This follows from defining a miss when the ground truth lies farther than 2m from every prediction.
  • Modality sampling: A greedy algorithm repeatedly selects the location with the highest integrated probability inside its threshold circle.After selection, heatmap values within that circle are set to zero before choosing the next point.
  • Modality sampling: The sampled points are nearly equidistant because zeroing previous circles imposes a strict minimum separation.Their radius-2m circles barely overlap in the illustrated result.
  • Modality sampling: Implementation uses a convolutional approximation of the coverage circle and upsamples the heatmap to 0.25 x 0.25 m² per pixel.Although the miss threshold is 2m, a 1.8m radius was found to perform better in practice.

2) Optimizing Final Displacement Error:

The FDE sampler adapts weighted centroid updates to the heatmap distribution, with iteration count controlling the coverage–distance trade-off.

  • Optimizing Final Displacement Error:: The FDE objective represents the heatmap as weighted points and seeks k centroids minimizing expected displacement.Pixel centers provide the locations and heatmap values provide their associated probabilities.
  • Optimizing Final Displacement Error:: The method replaces classic squared-distance weighting with a distance-based formulation to optimize the l2 norm and improve robustness to outliers.
  • Optimizing Final Displacement Error:: The algorithm updates each centroid from a weighted local neighborhood within a radius of 3m.Distance-based coefficients make partition boundaries flexible and reduce the influence of points near another prediction.
  • Optimizing Final Displacement Error:: Centroids are initialized from miss-rate sampling, and iteration count L tunes the trade-off between Miss Rate and FDE.At L = 0, Miss Rate is optimized; increasing L moves centroids toward high-probability areas.
  • Optimizing Final Displacement Error:: Increasing L sacrifices MR for better FDE by concentrating centroids near high-probability regions and reducing total coverage.Each iteration decreases minFDE6 and increases MR6.

D. Full trajectory generation

HOME separately generates full trajectories by connecting the initial position to sampled future endpoints, assigning each trajectory a probability from endpoint heatmap mass.

  • Full trajectory generation: A separate model generates full trajectories connecting the initial agent position to every sampled future location.
  • Full trajectory generation: The model encodes the target history, concatenates it with endpoint coordinates, and predicts intermediate positions across the time frame.It uses fully connected layers to produce the trajectory locations.
  • Full trajectory generation: A trajectory’s probability is the heatmap probability integrated inside a radius-2m circle around its endpoint.

A. Experimental settings

HOME is evaluated on Argoverse using MR, minFDE, minADE, and probability-aware metrics, with the benchmark ranking based on MR6. The model ranks first on MR6 and also outperforms other methods on p-minFDE6 and p-minADE6.

  • Dataset: 205942 training, 39472 validation, and 78143 test samples comprise the Argoverse Motion Forecasting Dataset.Each sample includes 2 seconds of agent histories and local-map context, with 3 seconds of future target positions as labels.
  • Metrics: MRk and minFDEk are reported for k=1,6, alongside minADEk and probability-aware p-minFDE6 and p-minADE6.The probability-aware metrics add −log(p), where p is the probability assigned to the closest predicted trajectory.
  • Training: Models are trained for 16 epochs with Adam, batch size 32, initial learning rate 0.001, scheduled halving, and raster-frame augmentation.Augmentation drops raster channels with probability 0.1 and rotates half of samples by a random angle in [−π/4, π/4].
  • Results: HOME ranks first on the Argoverse test-set leaderboard by MR6 and significantly improves on previous results.The authors attribute this result to heatmap output enabling strong coverage relative to prior methods.
  • Results: HOME outperforms other methods on p-minFDE6 and p-minADE6, indicating stronger probability-distribution modeling between predictions.Using FDE sampling with L = 4 sacrifices 1.1 points of MR6 for 9 cm of minFDE6, yielding second-best minFDE6 while retaining first-place MR6.

C. Ablation studies

The ablations compare heatmap and regression outputs and examine how training with more modalities affects lower fixed-modality metrics. Heatmaps preserve lower-k performance and reach low miss rates faster as k increases.

  • Output representation: The output-representation ablation replaces the image decoder with global pooling and six-coordinate regression, using winner-takes-all l1 and classification losses.A scalar-bottleneck variant is also included to separate regression effects from spatial-information loss.
  • Heatmap output: Increasing regression-model modalities improves MRk for the total k but worsens fixed-k performance such as MR1 and MR6.The passage reports a similar trend in prior methods when more modalities are trained.
  • Heatmap output: Heatmap outputs allow any number of modalities to be extracted from one training run without impacting lower-k MR1, MR3, and MR6.The figure represents regression results with full lines and heatmap results with dashed lines, including fixed modality counts in orange, green, and red.
  • Heatmap output: The heatmap model scales better with k and converges to 0% MR faster than the regression-output model.This comparison concerns the number of extracted modalities and the resulting miss-rate behavior.

2) Trajectory sampling:

HOME samples its probability heatmap to control the trade-off between miss rate and final displacement error. Increasing Algorithm 2 iterations shifts performance toward lower minFDE6 at the cost of higher MR6.

  • Trajectory sampling: Figure 6 plots FDE6 against MR6, with lower-left performance preferred and blue points generated by increasing Algorithm 2 iterations L from 0 to 7.Orange points represent other top-10 leaderboard methods.
  • Trajectory sampling: Each additional iteration of Algorithm 2 diminishes minFDE6 and increases MR6.The sampling centroids move closer together, reducing total coverage while favoring high-probability regions.
  • Trajectory sampling: The sampling study compares probability-ranked non-maximum suppression and KMeans with HOME's alternative sampling strategies.The non-maximum suppression procedure removes overlapping pixels within a 1.8m radius.
  • Qualitative results: Qualitative examples include straight motion, overtaking, curved roads, leaving the map, and intersections.The heatmap usually follows the context-map prior while also diverting from it based on interactions and observed cues.
  • Conclusion: HOME predicts a future final point on a 2D top-view grid, decodes it into a full trajectory, and samples the resulting future distribution.The conclusion reports state-of-the-art performance on the Argoverse Motion Forecasting benchmark.
Loading 2105.10968v2…