Source-linked AI summary
Remember Intentions: Retrospective-Memory-based Trajectory Prediction
Chenxin Xu, Weibo Mao, Wenjun Zhang, Siheng Chen
TL;DR
Trajectory prediction methods often store all training experience implicitly in model parameters, providing limited links to individual relevant instances. MemoNet stores past-intention pairs in memory, retrieves similar cases to predict destinations, and then completes trajectories; it reports FDE reductions of 20.3%/10.2%/28.3% on SDD/ETH-UCY/NBA and traceable predictions.
Problem
Parameter-based methods implicitly combine all seen instances in model parameters, limiting explicit access to individual experiences during prediction.
Method
MemoNet stores past and intention features in memory banks, uses a trainable addresser to retrieve similar instances, clusters intentions, and completes trajectories after destination prediction.
Results
20.3%/10.2%/28.3% FDE reductions are reported on SDD/ETH-UCY/NBA, with predictions traceable to specific memorized instances.
Takeaways & Limitations
MemoNet provides an instance-based trajectory-prediction framework with a more explicit link between current situations and seen data, promoting interpretability.
Takeaways & Limitations
The method focuses on memorizing past-intention pairs, while special actions such as sharp turns remain challenging from past trajectories alone.
Abstract
from arXiv · showhide
To realize trajectory prediction, most previous methods adopt the parameter-based approach, which encodes all the seen past-future instance pairs into model parameters. However, in this way, the model parameters come from all seen instances, which means a huge amount of irrelevant seen instances might also involve in predicting the current situation, disturbing the performance. To provide a more explicit link between the current situation and the seen instances, we imitate the mechanism of retrospective memory in neuropsychology and propose MemoNet, an instance-based approach that predicts the movement intentions of agents by looking for similar scenarios in the training data. In MemoNet, we design a pair of memory banks to explicitly store representative instances in the training set, acting as prefrontal cortex in the neural system, and a trainable memory addresser to adaptively search a current situation with similar instances in the memory bank, acting like basal ganglia. During prediction, MemoNet recalls previous memory by using the memory addresser to index related instances in the memory bank. We further propose a two-step trajectory prediction system, where the first step is to leverage MemoNet to predict the destination and the second step is to fulfill the whole trajectory according to the predicted destinations. Experiments show that the proposed MemoNet improves the FDE by 20.3%/10.2%/28.3% from the previous best method on SDD/ETH-UCY/NBA datasets. Experiments also show that our MemoNet has the ability to trace back to specific instances during prediction, promoting more interpretability.
1. Introduction
Trajectory prediction must connect past observations to possible future intentions, but parameter-based methods implicitly mix all training experiences. MemoNet instead recalls similar stored instances through retrospective-memory-inspired components and predicts trajectories in two steps.
- Motivation: Parameter-based methods implicitly encode all seen past-future experiences in model parameters, weakening the explicit link to individual instances.The paper identifies interpretability and relevance concerns with storing experience as a whole.
- MemoNet: MemoNet uses past and intention memory banks to store training instances and a trainable memory addresser to retrieve relevant matches.The memory banks correspond to prefrontal-cortex storage, while the addresser corresponds to basal-ganglia-like retrieval.
- MemoNet: MemoNet combines joint reconstruction, memory filtering, a trainable memory addresser, and intention clustering.These designs initialize compatible features, reduce redundancy, search similar instances, and promote diverse predictions.
- Prediction system: The two-step system first predicts destinations as intentions, then fulfills the complete trajectory from those predicted destinations.The destination is used because it carries most of the trajectory’s modality information.
- Results: 20.3%/10.2%/28.3% FDE reductions are reported on SDD/ETH-UCY/NBA, alongside the ability to trace predictions to specific memorized instances.The reported improvements are presented as state-of-the-art results on the evaluated datasets.
2. Related Work
Prior trajectory-prediction methods are predominantly parameter-based, storing learned experience implicitly in model parameters. MemoNet introduces an instance-based retrospective-memory framework that recalls similar trajectories and intentions for prediction.
- Trajectory prediction: Early trajectory-prediction methods used deterministic models, whereas recent methods mainly adopt regression or generation frameworks.The related work lists social-force, Markov, and recurrent models alongside newer encode-decode and generative approaches.
- Trajectory prediction: Regression and generation frameworks optimize model parameters, storing learned experience implicitly rather than addressing individual training instances.This parameter-based organization is the central distinction motivating MemoNet.
- MemoNet: MemoNet memorizes past trajectories and corresponding intentions, then recalls similar previous instances to guide future prediction.The framework is explicitly based on retrospective memory and links current predictions to seen data.
- MemoNet: Unlike a closely related memory-based method, MemoNet handles multi-agent prediction with social influence and focuses its memory on intentions rather than entire trajectories.The cited comparison identifies these as two differences from the prior single-agent approach.
3. Problem Formulation
The task predicts an agent’s future trajectory from its past trajectory and neighboring agents’ past trajectories. The paper decomposes this into intention prediction followed by trajectory completion conditioned on the predicted destination.
- Inputs and target: The observed input consists of the target agent’s past trajectory and the past trajectories of neighboring agents.The formulation represents the target history over Tp timestamps and neighbors as a set of trajectories.
- Inputs and target: The model learns to produce a future trajectory bY = g(X, XN) that is close to the ground-truth trajectory Y.X denotes the target history and XN denotes neighboring-agent histories.
- Two-step strategy: The prediction strategy first estimates the agent’s intention and then generates the complete trajectory conditioned on that intention.This decomposition treats the complex prediction task as two relatively simple problems.
- Two-step strategy: The destination represents intention because it reflects most movement patterns, and the trajectory fulfiller generates bY = gfull(X, XN, byTf).MemoNet performs the intention-prediction stage before the trajectory-fulfillment model.
4. MemoNet: Intention Prediction
MemoNet predicts movement intentions by recalling similar training instances through paired memory banks and a trainable memory addresser. Joint reconstruction creates compatible features, filtering removes redundancy, and clustering produces diverse intentions.
- Memory bank: MemoNet stores paired past-trajectory and future-intention features so past situations are explicitly associated with destinations.The paired banks share addresses, linking each past feature to its corresponding intention feature.
- Memory bank: Joint reconstruction learns compatible past and intention features by reconstructing both the past trajectory and future destination.The decoder receives concatenated features and jointly reconstructs the two targets.
- Memory bank: Memory filtering removes redundant instances when their past starting positions and future intentions are both sufficiently close.The thresholds θpast and θint control the final memory-bank size.
- Memory bank: The training-free filter is permutation-invariant and more efficient than a learned controller for reducing memory redundancy.Unlike the previous controller-based method, its result does not depend on training-sample ordering.
- Memory addresser: A trainable attention addresser learns a similarity metric, transforms query and memory features with separate MLPs, and returns the highest-scoring memory addresses.This replaces reliance on a predefined similarity function such as cosine distance.
- Intention diversity: Intention clustering addresses same-modality recalls by decoding many top matches into coarse anchors and clustering them into diverse predictions.Using L much larger than K helps capture more movement patterns while retaining the memory bank’s enrichment.
5. Trajectory prediction system
The trajectory system separates destination prediction from whole-trajectory fulfillment. It conditions a fulfillment encoder-decoder on MemoNet’s predicted intentions while reconstructing past motion during training.
- Trajectory prediction system: The system first predicts an agent’s destination intention and then fulfills the complete trajectory conditioned on that prediction.MemoNet represents intention with the destination because it carries most trajectory modality information.
- Trajectory fulfilling: The fulfillment encoder combines past and neighboring trajectories with an encoded predicted intention before decoding future and reconstructed past trajectories.The intention feature is transformed by an MLP and concatenated with the trajectory feature.
- Trajectory fulfilling: The trajectory loss combines squared future-trajectory error with squared reconstructed-past error, weighted by β.The decoder reconstructs past motion to preserve most past information.
- Overall training pipeline: Training proceeds by learning feature encoders and a decoder, filtering frozen memory banks, training the addresser, and finally training the fulfillment encoder-decoder.Each later stage freezes the components learned earlier.
6. Experiments
Experiments evaluate MemoNet on SDD, ETH-UCY, and NBA using minADEK/minFDEK, showing improved quantitative performance, diverse predictions, instance traceability, and real-time inference. Ablations examine memory components, anchor counts, filtering thresholds, and speed.
- Quantitative results: MemoNet reduces FDE from 15.88 to 12.66 on SDD versus PECNet, achieving a 20.3% improvement.It also outperforms PECNet by 14.1% in ADE.
- Quantitative results: MemoNet reduces average FDE from 0.39 to 0.35 on ETH-UCY versus AgentFormer, achieving a 10.2% improvement.It achieves the best or close to the best ADE across all five subsets.
- Quantitative results: MemoNet reduces FDE from 2.05 to 1.47 on NBA versus NMMP, achieving a 28.3% improvement.
- Qualitative results: Intention clustering produces diverse and accurate predictions from coarse intention anchors, while trajectory visualizations show improved direction-turning predictions.The method’s predictions are closer to ground truth than those of PECNet and MANTRA in the illustrated best-of-20 cases.
- Qualitative results: Addressed memory instances trace predictions back to seen similar scenarios, providing instance-level experience for multimodal intentions.This gives the model an explicit link to specific memorized samples during prediction.
- Ablation studies: Ablations show that the memory bank, memory filtering, learnable addresser, and intention clustering each contribute to accurate prediction.Too few or too many coarse anchors degrade performance, while filtering thresholds must balance redundant and useful information.
- Inference speed: MemoNet runs at 18.03 ms per sample, corresponding to 55.5 FPS on one RTX-3090 GPU.The reported speed is faster than the common data-collection sampling rate.
7. Conclusion
MemoNet is an instance-based trajectory-prediction approach that stores seen instances for relevant movement-pattern matching and improves state-of-the-art performance while enabling traceability to specific instances. The paper identifies predicting special actions from past trajectories alone as a limitation and proposes environment-conditioned prediction using map information as future work.
- MemoNet stores seen instances in paired memory banks for relevant movement-pattern matching during inference.Its four designs include joint-reconstruction feature learning, memory filtering, a learnable addresser, and intention clustering.
- MemoNet significantly improves state-of-the-art performance on trajectory-prediction datasets and can trace predictions back to specific instances.The authors associate this traceability with improved interpretability.
- Predicting special actions such as sharp turns remains challenging when using past trajectories alone.The proposed future direction is to use map information for environment-conditioned prediction.