Source-linked AI summary
MamMA: A Mamba-Based Pedestrian Trajectory Prediction Algorithm Considering Occupancy Map and Pedestrian Awareness States
Juncen Long, Xiaofeng Jin, Gianluca Bardaro, Simone Mentasti, Matteo Matteucci
TL;DR
Pedestrian trajectory prediction often uses top-down data, although mobile robots typically collect LiDAR and egocentric images that provide different environmental and pedestrian information. MamMA uses Mamba-based models with occupancy maps and awareness states; experiments report better ADE and FDE than state-of-the-art algorithms across five datasets.
Problem
Many prediction algorithms rely on top-down data that differ from the LiDAR and egocentric-image inputs typically collected by mobile robots.
Source: §I · p. 1Method
MamMA uses Mamba-based models to extract occupancy-map and pedestrian features, including awareness states, for trajectory prediction.
Source: §I · pp. 1–2Results
MamMA achieves better ADE and FDE than state-of-the-art algorithms on STCrowd, SiT, JRDB, ETH, and UCY.
Source: §I · p. 1Takeaways & Limitations
The authors conclude that using data from on-board sensors gives MamMA better adaptation to real-world robot deployments and improves prediction accuracy.
Source: §V · p. 7
Abstract
from arXiv · showhide
Many pedestrian trajectory prediction algorithms have been proposed to improve the safety of navigation for mobile robots working in human-robot coexistence environments. Some pedestrian trajectory prediction algorithms extract information about obstacles near pedestrians from top-down view images to improve the accuracy of trajectory prediction. However, mobile robots typically create local occupancy maps using LiDAR, rather than top-down view images. Meanwhile, the vision sensors on board robots provide egocentric view images, which contain fine-grained behavioral information about the pedestrians near the robot. To better use the information collected by LiDAR and on-board vision sensors, we propose MamMA, a Mamba-based pedestrian trajectory prediction algorithm considering occupancy maps and pedestrian awareness states. MamMA divides the occupancy map by patches and extracts obstacle features from each patch to create map features. Pedestrian awareness states are divided and considered, as some studies show that awareness states affect the perception and speed of pedestrians. Furthermore, a Mamba-based model is proposed to predict the future trajectories of pedestrians based on different types of features. Experiments on the STCrowd, SiT, JRDB, ETH, and UCY datasets show that MamMA achieves better average displacement error and final displacement error than the state-of-the-art algorithms.
I. INTRODUCTION
Pedestrian prediction methods often rely on top-down data that do not match mobile robots’ LiDAR and egocentric camera inputs. MamMA combines occupancy-map and awareness-state information with Mamba-based trajectory modeling.
I. INTRODUCTION
- Top-down datasets and image features are common, but robot sensors typically provide LiDAR data and egocentric images instead.Egocentric images can show pedestrians more clearly, providing information such as awareness states.Source: §I · p. 1
- MamMA combines LiDAR-derived occupancy maps and pedestrian awareness states, using Mamba-based models to predict trajectories.The paper reports better ADE and FDE than state-of-the-art algorithms on STCrowd, SiT, JRDB, ETH, and UCY.Source: §I · p. 1
- The paper identifies Mamba-based occupancy-map scanning, awareness-and-position embedding, and temporal and spatial pedestrian feature extraction as contributions.The prediction network also considers obstacles.Source: §I · pp. 1–2
A. Spatio-temporal Graph
- Spatio-temporal graphs represent pedestrians as nodes and their correlations as edges, and can support extraction of temporal and spatial trajectory features.MamMA also uses spatio-temporal graphs to describe historical trajectories and pedestrian interactions, extracting graph features with Mamba models.Source: A. Spatio-temporal Graph · p. 2
B. Pedestrian Awareness and Trajectory
Awareness-related differences in perception and speed can affect pedestrian trajectories, while existing prediction methods rarely account for awareness states. The paper distinguishes top-down datasets from multimodal datasets with LiDAR and egocentric images, and formulates prediction using pedestrian history and occupancy-map data.
B. Pedestrian Awareness and Trajectory
- Mobile-phone use is associated with reduced perception, slower walking, and longer reaction times, which can distinguish future trajectories despite similar histories.The paper says almost all existing prediction algorithms do not consider awareness-state effects.Source: B. Pedestrian Awareness and Trajectory · p. 2
C. Dataset
- ETH and UCY provide top-down pedestrian positions but lack fine-grained pedestrian information and LiDAR data, unlike STCrowd, SiT, and JRDB.STCrowd, SiT, and JRDB provide LiDAR, egocentric images, and 2D and 3D pedestrian position annotations; SiT and JRDB use robot-carried sensors.Source: C. Dataset · p. 2
- The paper states that few existing algorithms use both pedestrian awareness states and occupancy maps, and that its approach uses both to improve prediction accuracy.Some algorithms instead use convolutional neural networks to extract information from top-down images.Source: C. Dataset · p. 2
A. Problem Formulation
- The prediction setup represents pedestrian history and future positions alongside occupied-cell locations in the occupancy map.The supplied formulation identifies awareness state as part of pedestrian information.Source: A. Problem Formulation · p. 2
B. Algorithm Structure
MamMA has pedestrian encoding, map scanning, and trajectory prediction modules. The prediction module uses pedestrian history, interactions, and occupancy-map features across temporal and spatial dimensions.
B. Algorithm Structure
- MamMA’s three modules encode pedestrians, scan occupancy-map patches for obstacles, and predict future trajectories by combining features from the first two modules.The pedestrian module encodes position and awareness state; map scanning extracts obstacle information from each patch.Source: B. Algorithm Structure · p. 2
- The trajectory prediction module uses pedestrian historical trajectories, interactions, and occupancy-map features in both temporal and spatial dimensions.Source: B. Algorithm Structure · p. 2
- The map scanning module extracts information about obstacles in occupancy-map patches, while pedestrian encoding embeds position and awareness state.Source: B. Algorithm Structure · p. 2
C. Pedestrian Encoding Module
The pedestrian encoding module embeds individual pedestrian information and pairwise interactions. Its inputs include displacement, awareness state, position and velocity differences, and interaction type.
C. Pedestrian Encoding Module
- A fully connected layer embeds each pedestrian’s displacement and awareness-state information.The described layer uses PReLU and dimensions [6, n_en].Source: C. Pedestrian Encoding Module · p. 2
- The pedestrian encoding module embeds awareness state and position, while the trajectory prediction module fuses features from pedestrian encoding and map scanning.Source: Figure 3 · p. 3
- Pedestrian-pair interaction features include position and velocity differences and an interaction type.Interaction types are based on awareness states and include normal-to-normal, normal-to-distracted, and distracted-to-all-state pairs.Source: C. Pedestrian Encoding Module · p. 3
D. Map Scanning Module
The map scanning module represents the occupancy map as patches and builds obstacle-informed features for each pedestrian. A Mamba model scans the patch features to produce map features.
D. Map Scanning Module
- The occupancy map is divided into patches, each containing a set of obstacle positions for feature extraction.The passage denotes the number of patches as np and their edge length as dp.Source: D. Map Scanning Module · p. 3
- For each obstacle, relative position and velocity to the pedestrian are embedded and combined with a Hadamard product as an influence feature.A fully connected layer then combines obstacle features within each patch into a patch feature.Source: D. Map Scanning Module · p. 3
- A Mamba model scans features from all map patches to generate map features for pedestrians at each time step.The generated features cover all patches and pedestrians at the specified time.Source: D. Map Scanning Module · p. 3
E. Trajectory Prediction Module
The trajectory prediction module combines pedestrian, social, and static-obstacle information in a spatio-temporal graph. It processes and fuses these features before decoding predicted displacements.
E. Trajectory Prediction Module
- A spatio-temporal graph represents pedestrian information with a node matrix and an adjacency matrix over the observation period.The node matrix has dimensions [Tobs, m, nen], and the adjacency matrix has dimensions [Tobs, m, m, nen].Source: E. Trajectory Prediction Module · pp. 3–4
- Mamba models extract node features over time and adjacency features over spatial and then temporal dimensions.The extracted matrices retain the dimensions of their respective inputs.Source: E. Trajectory Prediction Module · p. 4
- The module fuses the processed node and adjacency features, producing features that account for other pedestrians and static obstacles.It batch-normalizes the fused features and concatenates them with the map features before decoding.Source: E. Trajectory Prediction Module · p. 4
- An MLP processes the concatenated features for decoding, and a Mamba model plus an MLP outputs each pedestrian’s predicted displacement.The MLP maps the decoded representation to displacement predictions.Source: E. Trajectory Prediction Module · p. 4
A. Experiment Settings
Experiments compare MamMA with prior algorithms across five datasets using dataset-specific evaluation protocols. They measure trajectory error and, on three datasets, examine distracted pedestrians and obstacle-proximate trajectories.
A. Experiment Settings
- MamMA is compared with seven named SOTA algorithms across STCrowd, SiT, JRDB, ETH, and UCY using validation or test evaluations as specified for each dataset.STCrowd, SiT, and JRDB use validation sets because test labels are unavailable; ETH and UCY results use MamMA’s best-validation weights and the SOTA authors’ provided weights.Source: A. Experiment Settings · p. 4
- For STCrowd, SiT, and JRDB, samples are divided into All, distracted-state Sdis, and obstacle-proximate Sobs groups.Sobs denotes cases where the ground-truth future trajectory is less than 1 meter from the nearest obstacle.Source: A. Experiment Settings · p. 4
- Evaluation uses minADE3 and minFDE3 from three predicted trajectories, plus pcoll, the proportion of predictions colliding with static obstacles.The loss minimizes ADE; pcoll assesses prediction rationality relative to non-colliding ground-truth trajectories.Source: A. Experiment Settings · p. 4
- Occupancy maps use 0.1-meter resolution over a 48-by-30-meter area, with 3-meter patches; observation and prediction windows are 3.2 and 4.8 seconds.At 2.5 FPS, these windows correspond to Tobs = 8 and Tpred = 12.Source: A. Experiment Settings · p. 4
B. Quantitative Experiments and Analysis
MamMA has the best minADE3, minFDE3, and pcoll on STCrowd, SiT, and JRDB. Its prediction accuracy degrades less under obstacle proximity and distracted-pedestrian conditions than that of the SOTA algorithms.
B. Quantitative Experiments and Analysis
- MamMA has the best minADE3, minFDE3, and pcoll on STCrowd, SiT, and JRDB, indicating adaptation to data collected by real-world robots.Table I reports results for these three datasets.Source: B. Quantitative Experiments and Analysis · p. 4, Table I · p. 5
- On STCrowd, compared with all samples, MamMA’s minADE3 and minFDE3 for Sdis rise by only 0.02 meters, versus over 0.04 and 0.1 meters for SOTA algorithms except Social-STGCNN.For SiT and JRDB, MamMA’s performance degradation for Sdis and Sobs is less than that of almost all SOTA algorithms.Source: B. Quantitative Experiments and Analysis · pp. 4–5
- For STCrowd Sobs, MamMA’s minADE3 and minFDE3 rise 0.05 and 0.1 meters, versus over 0.1 and 0.18 meters for SOTA algorithms.The paper reports the same relative pattern for MamMA on SiT and JRDB.Source: B. Quantitative Experiments and Analysis · p. 5
- When obstacle distance falls from 3 to 0.5 meters, MamMA’s minADE3 rises about 0.04 meters and accuracy decays about 13.5%, versus SGCN’s 0.08 meters and 18.3%.These are the smallest obstacle-related decay amount and ratio among the compared algorithms.Source: B. Quantitative Experiments and Analysis · p. 5, Figure 4 · p. 5
- MamMA has a relatively large model size but an acceptable inference time compared with SOTA algorithms on STCrowd.Table II covers model parameters, model size, and inference time.Source: B. Quantitative Experiments and Analysis · p. 5, Table II · p. 5
C. Qualitative Analysis
Qualitative examples compare MamMA’s predictions with SOTA algorithms in scenes involving obstacles, pedestrian interactions, and awareness states. The reported examples show MamMA handling these situations more accurately.
C. Qualitative Analysis
- In scenes A, B, and C, some SOTA-predicted trajectories collide with static obstacles, while MamMA’s predictions are described as more accurate and less likely to hit them.The authors attribute this to MamMA scanning maps with Mamba models.Source: C. Qualitative Analysis · p. 5, Figure 5 · p. 6
- At an intersection and in a head-on encounter, MamMA predicts pedestrians’ turns and straight paths, while SGCN and Social-Implicit produce colliding or incorrect trajectories.The examples are scenes G and H.Source: C. Qualitative Analysis · p. 5
- In scenes D–F, MamMA predicts distracted pedestrians’ paths as closer to one another or to static obstacles, respectively.The authors describe these predictions as accurate and credit encoding pedestrian awareness.Source: C. Qualitative Analysis · p. 6
- The qualitative analysis concludes that MamMA accounts for static obstacles and awareness states in future-trajectory predictions.The authors also report higher prediction accuracy than the SOTA algorithm in this analysis.Source: C. Qualitative Analysis · p. 6
D. Ablation Study
MamMA combines occupancy maps and awareness states with Mamba-based pedestrian feature extraction. Ablations report gains from maps, awareness states, and Mamba models, while the conclusion reports better performance and similar inference time to state-of-the-art algorithms.
D. Ablation Study
- Maps and awareness states improve MamMA: removing maps worsens minADE3, minFDE3, and pcoll, while omitting awareness states worsens minADE3 and minFDE3 for Sdis.The comparison is between MamMA, MamM (maps only), and Mam (neither maps nor awareness states).Source: D. Ablation Study · p. 6
- Removing any one Mamba model raises minADE3 by at least 0.02 and minFDE3 by at least 0.05 meters; removing Mamap and Mamat makes MamMA worse than almost all state-of-the-art algorithms.The tested models include Mamvt, Mamap, Mamat, and Mampd.Source: D. Ablation Study · pp. 6–7
- On ETH and UCY, Mam uses no occupancy maps or awareness states because these datasets lack LiDAR and make awareness states difficult to annotate, yet it achieves the best performance.The authors relate this result to Mam also outperforming state-of-the-art algorithms on STCrowd.Source: D. Ablation Study · p. 7
V. CONCLUSION
- Across STCrowd, SiT, JRDB, ETH, and UCY, MamMA has better performance and similar inference time to state-of-the-art algorithms, supporting improved real-world deployment adaptation.The conclusion attributes improved accuracy and adaptation to using data collected by on-board sensors.Source: §V · p. 7