PaperlayerView exact revision on arXiv

Source-linked AI summary

MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction

Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, Alberto Del Bimbo

arXiv:2006.03340v2cs.CV

TL;DR

Autonomous driving requires predicting multiple plausible future trajectories of surrounding agents beyond their currently observed motion. MANTRA uses recurrent trajectory encoders, persistent associative memory, scene-map refinement, and a learned writing controller; it reports state-of-the-art results on three datasets and improves with novel samples.

  • Problem

    Autonomous vehicles need multimodal future trajectory predictions because surrounding agents can have several similarly likely outcomes, while standard recurrent models lack selectively addressable memory.

  • Method

    MANTRA stores separate past and future trajectory embeddings in an external associative memory, decodes retrieved futures conditioned on the observed past, incorporates semantic-map context, and controls memory writing using reconstruction error.

  • Results

    MANTRA obtains state-of-the-art trajectory-prediction results on three traffic datasets and natively produces multimodal predictions.

  • Takeaways & Limitations

    After training, MANTRA can ingest novel trajectory samples and lower error on unseen data through its persistent non-parametric memory.

  • Takeaways & Limitations

    The controller formulation assumes a bounded error function with values in [0, 1].

Abstract

from arXiv · show

Autonomous vehicles are expected to drive in complex scenarios with several independent non cooperating agents. Path planning for safely navigating in such environments can not just rely on perceiving present location and motion of other agents. It requires instead to predict such variables in a far enough future. In this paper we address the problem of multimodal trajectory prediction exploiting a Memory Augmented Neural Network. Our method learns past and future trajectory embeddings using recurrent neural networks and exploits an associative external memory to store and retrieve such embeddings. Trajectory prediction is then performed by decoding in-memory future encodings conditioned with the observed past. We incorporate scene knowledge in the decoding state by learning a CNN on top of semantic scene maps. Memory growth is limited by learning a writing controller based on the predictive capability of existing embeddings. We show that our method is able to natively perform multi-modal trajectory prediction obtaining state-of-the art results on three datasets. Moreover, thanks to the non-parametric nature of the memory module, we show how once trained our system can continuously improve by ingesting novel patterns.

1. Introduction

MANTRA addresses multimodal vehicle trajectory prediction by using an external associative memory to retrieve likely futures from observed past trajectories. Its persistent knowledge base stores meaningful samples, supports scene-conditioned decoding, and can grow online after training.

  • Vehicle futures are inherently multimodal, so a single prediction can average across several similarly likely outcomes.
  • Its external memory retains meaningful, non-redundant trajectory samples and forms an incremental knowledge base for prediction.
  • The model encodes past and future trajectories separately, using observed-past encodings as memory keys to retrieve future encodings for conditioned decoding.
  • MANTRA introduces a Memory Augmented Neural Network architecture for multiple vehicle trajectory prediction.
  • MANTRA reports state-of-the-art results on three traffic datasets and can improve incrementally by ingesting new examples after training.

2. Related Work

Related work contrasts MANTRA with recurrent, episodic-memory, multimodal, and intention-based approaches. MANTRA instead combines persistent associative memory, coordinate-based trajectory prediction, semantic-map refinement, and online growth.

  • MANTRA directly estimates a small ranked set of diverse vehicle trajectories in coordinates, rather than relying on heatmaps or large unranked samples.
  • Unlike intention-based methods, its memory entries encode futures without conditioning on a reference agent or restricting predictions to constrained scenarios.
  • The model refines memory-based trajectory proposals with semantic maps to incorporate surrounding-environment knowledge and physical constraints.
  • Compared with recurrent networks, external element-wise addressable memory selectively stores and retrieves explicit information instead of compressing all temporal knowledge into one hidden state.
  • Unlike episodic Memory Augmented Neural Networks, MANTRA uses persistent memory to retain relevant experience for predictions beyond a restricted observation episode.
  • MANTRA's disjoint past and future embeddings create multiple outputs from one input, yielding multimodal predictions rather than merely improving data fit.

3. Model

MANTRA predicts vehicle futures by retrieving encoded future patterns from an associative memory keyed by observed-past encodings, then decoding them conditionally and refining them with scene context. Its controller limits memory redundancy by writing samples according to reconstruction error, while multiple retrieved futures support multimodal prediction.

  • 3.1. Memory Based Trajectory Prediction: Memory addresses are computed from cosine similarity between the observed-past encoding and stored past encodings, retrieving the most relevant future representations.The memory contains past-future encoding pairs, and the top-K future encodings are read according to their similarity scores.
  • 3.1. Memory Based Trajectory Prediction: Top-K future encodings are combined with the observed-past encoding and decoded separately, producing multiple plausible futures when trajectories are uncertain.The observed-past encoding remains fixed while retrieved future encodings vary; this supports multimodal predictions such as at road bifurcations.
  • 3.1. Memory Based Trajectory Prediction: Past and future trajectories use separate encoders, while a decoder reconstructs only future coordinates conditioned on both representations.This encoder-decoder design provides distinct memory keys and values while conditioning future reconstruction on the observed past.
  • 3.3. Memory controller: The controller writes samples into a permanent memory using reconstruction error, storing high-error cases while minimizing writes when predictions are already accurate.This learned adaptive threshold retains useful, nonredundant samples and limits memory growth.
  • 3.2. Prediction Refinement: Prediction refinement pools CNN features from the scene context along each trajectory and uses a GRU with fully connected layers to output coordinate offsets.The procedure iteratively adjusts predictions to improve compatibility with the environment.
  • 3.4. Training: Training first fits the encoders and decoder as an autoencoder, then trains the memory controller using the learned trajectory representations and resets memory after each epoch.Decoder and refinement use mean squared error, while the controller is optimized with its reconstruction-error-based loss.

4. Experiments

MANTRA is evaluated on multiple vehicle-trajectory datasets using ADE/FDE and best-of-K predictions, with experiments covering multimodality, transfer, and online memory growth.

  • 4.2. Metrics and baselines: Kalman filtering is insufficient for long-horizon KITTI prediction, with FDE@4s higher than 7m, while learning-based baselines perform better.The Multi-Layer Perceptron performs slightly better than the linear regressor.
  • 4.3. Results: Top-K prediction sharply reduces error because MANTRA covers diverse future paths instead of averaging multiple plausible outcomes.The authors report that MANTRA’s error continues decreasing as K increases, unlike DESIRE’s quickly saturating performance.
  • 4.3. Results: MANTRA largely outperforms INFER across timesteps on the INFER KITTI split when generating multiple predictions.With K = 1, it is better or on par at low timesteps but loses precision at 4s.
  • 4.3. Results: MANTRA consistently outperforms INFER on Oxford RobotCar and obtains lower error than other methods on Cityscapes.Cityscapes evaluation extends only to 1s because that is the dataset’s maximum prediction horizon.
  • 4.4. Incremental setting: In the online setting, memory grows slowly while test error decreases, with the controller storing only 16% of newly observed examples.Results use K=5 predictions and average over 100 runs.

5. Model Analysis

Model analysis examines memory contents, component ablations, and decoder conditioning, showing that rotation invariance and past embeddings materially shape compact, multimodal predictions.

  • Ablation Studies: The ablations evaluate removing refinement, decoder decoding, rotation invariance, the trained controller, and the encoder-decoder pipeline.The study reports results on KITTI at K=5 and includes memory size as well as prediction errors.
  • Memory inspection: Memory inspection decodes every stored sample’s future trajectory, providing a snapshot of the patterns retained for K=5 predictions.The analysis uses decoded trajectories to examine what the controller stores.
  • Memory inspection: T-SNE projections show similar trajectory patterns clustered together, with embeddings organized largely by speed and multiple future directions emerging at lower speeds.Higher-speed trajectories vary more in length than curvature, whereas lower-speed trajectories span more possible directions.
  • Decoder Analysis: Past embeddings condition future reconstruction: changing past speed accelerates or decelerates the decoded future, while zeroing or randomizing the past alters reconstruction behavior.The decoder reconstructs only the future, using the past as conditioning information for unseen examples.

6. Conclusions

MANTRA uses associative memory to model multimodal trajectory futures, achieves state-of-the-art results on three traffic datasets, and lowers error by ingesting novel samples.

  • 6. Conclusions: MANTRA natively handles multimodal future trajectories through an associative memory framework and achieves state-of-the-art results on three traffic datasets.The memory can ingest novel samples and lower error on unseen data.

Appendix: Pedestrian Trajectory Prediction

The appendix evaluates MANTRA for pedestrian trajectories on ETH/UCY and SDD, including socially interacting agents and multiple prediction counts.

  • Appendix: Pedestrian Trajectory Prediction: ETH/UCY contains 1,536 pedestrians across five scenarios with group actions, collision avoidance, and crossing trajectories.Evaluation follows a leave-one-out strategy.
  • Appendix: Pedestrian Trajectory Prediction: SDD contains 14k university-campus scenarios with multiple pedestrians, recorded from a bird’s-eye drone view at 2.5 Hz.Trajectories are expressed in pixel coordinates and evaluated using the Trajnet split.
  • Appendix: Pedestrian Trajectory Prediction: The appendix reports SDD results for both K=5 and K=20 predictions and ETH/UCY results for K=20.Several baselines are included for both datasets.
  • Appendix: Pedestrian Trajectory Prediction: MANTRA competes with methods that explicitly model social interaction and outperforms Social-GAN on pedestrian trajectory prediction.It obtains comparable results to more recent goal-based approaches.
Loading 2006.03340v2…