PaperlayerView exact revision on arXiv

Source-linked AI summary

Social Graph Mamba: Forecasting Pedestrian Movements Based on Social Context

Hong-Son Nguyen, Yen-Chen Liu

arXiv:2608.21411v1cs.RO

TL;DR

Pedestrian forecasting must model social interactions while remaining scalable and capturing group behavior. SGM replaces attention with SSMs on dynamic interaction graphs, adds social triplet factorization and community-aware conditioning, and achieves competitive benchmark results with linear sequence complexity while supporting physical robot validation.

  • Problem

    Existing attention-based methods have O(N^2) social-interaction complexity, while benchmark-focused approaches may not fully capture natural crowd movement and group structures.

  • Method

    SGM uses Selective SSMs on dynamic interaction graphs, sequential social triplet factorization, and differentiable community-aware conditioning of embeddings and multimodal decoding.

  • Results

    SGM achieves competitive ETH/UCY performance with average ADE/FDE of 0.23/0.37, best SDD minADE20/minFDE20 scores of 5.98/10.05, and physical robot validation.

  • Takeaways & Limitations

    SGM combines competitive prediction with linear sequence complexity and group-aware modeling for practical robot-navigation deployment.

Abstract

from arXiv · show

Forecasting pedestrian motion has always been fundamental for autonomous navigation in crowded environments. While attention-based methods achieve strong performance, they suffer from quadratic computational complexity in modeling social interactions, limiting scalability. Additionally, the existing methods often achieve high accuracy on prediction benchmarks at the individual level, but fail to fully capture the natural movement behaviors of crowds in real-world scenarios, particularly group structures. In this study, we propose Social Graph Mamba (SGM), a novel architecture that replaces attention-based social reasoning with Selective State Space Models (SSMs) operating on dynamically constructed interaction graphs. SGM introduces a dynamic interaction graph with social triplet factorization to decompose crowd interactions sequentially, and a community-aware module to effectively discover group structures via differentiable MinCut optimization and conditions both the embedding space and multi-modal decoder on group membership. Our experiments on standard benchmarks (ETH/UCY, SDD) demonstrate competitive performance with linear sequence complexity compared to quadratic attention-based methods. We further validate SGM in physical robot experiments by integrating predicted trajectories into a Social Force Model (SFM) for real-world implementation.

I. INTRODUCTION

Pedestrian trajectory prediction is essential for autonomous navigation, but attention-based social models scale quadratically and often overlook graph-structured group behavior. SGM addresses these limitations with attention-free SSM reasoning, dynamic social graphs, and community-aware conditioning.

  • Motivation: Accurate pedestrian forecasting supports mobile robots, autonomous vehicles, and intelligent surveillance in crowded environments shaped by social interactions.Relevant interactions include collision avoidance, group cohesion, lane formation, and goal-directed navigation.
  • Limitations of Existing Methods: Attention-based models incur O(N^2) complexity and can become bottlenecks in dense crowds, while ego-centric distance prioritization neglects social graph structure.The scaling affects both attention-weight computation and message passing.
  • Motivation: SSMs offer O(N) sequence processing with input-dependent gating, but applying them to social data requires converting unordered graph interactions into ordered sequences.The paper identifies this mismatch between sequential SSMs and unordered, graph-structured interactions as a central challenge.
  • Proposed Approach: SGM proposes an attention-free graph architecture that uses backbone SSMs to model sequential information while balancing prediction accuracy and computational efficiency.The architecture is designed for prediction in social contexts without self-attention.
  • Proposed Approach: SGM discovers groups without explicit annotations and conditions both contrastive embedding learning and multimodal decoding on inferred community membership.This community-aware mechanism targets collective crowd behavior rather than only local pairwise interactions.
  • Practical Motivation: The framework is intended to bridge accurate pedestrian forecasting with computational constraints for real-time mobile-robot deployment.This contribution connects trajectory prediction with practical robot navigation requirements.

B. Group-Aware Trajectory Prediction

SGM combines prior graph-to-sequence SSM ideas with pedestrian-specific interaction modeling and unsupervised group discovery. Its inputs are robot-centric observed trajectories, and its decoder produces multiple future hypotheses with probabilities.

  • Group-Aware Trajectory Prediction: Existing group-aware methods model collective dynamics, but most require expensive ground-truth group annotations during training.SGM instead discovers groups through differentiable optimization without explicit group labels.
  • State Space Models and Mamba: SSMs provide linear-time sequential processing, while Graph-Mamba converts unordered graphs into sequences through node prioritization and permutation-based training.These ideas motivate SGM’s graph-aware ordering strategy.
  • Group-Aware Trajectory Prediction: The model decomposes crowd dynamics into temporal, ego-centric, and goal-centric SSM branches for social trajectory forecasting.This perspective-based decomposition forms the conceptual basis of Social Triplet Factorization.
  • State Space Models and Mamba: SGM extends Graph-Mamba with dynamic physics-aware graphs, edge-weighted social triplet factorization, direction-aware goals, adaptive directional balance, and community-aware centrality.The edge weights encode proximity, velocity alignment, and time-to-collision.
  • Problem Definition: Observed trajectories are transformed into a robot-centric frame, and the model predicts K future trajectory hypotheses with corresponding probabilities over Tpred steps.The coordinate transformation uses the robot’s last observed position as reference.

B. Overview

SGM builds a dynamic interaction graph whose edge weights represent physics-motivated social interaction strength, then embeds raw position-velocity features for downstream processing.

  • C. Dynamic Interaction Graph: SGM represents agents as graph nodes and assigns interaction-strength edges using a dynamic graph rather than only distance-based adjacency.The graph is defined as G = (V, E), with edge weights wij encoding interaction strength.
  • C. Dynamic Interaction Graph: Distance weighting uses a Gaussian kernel over relative position, with α = 1.0 and σp = 2.0.This component captures spatial proximity.
  • C. Dynamic Interaction Graph: Velocity weighting uses positive cosine similarity between velocity vectors, scaled by β = 0.5, so only aligned motion contributes positively.The max(0, ·) operation suppresses negatively aligned velocities.
  • C. Dynamic Interaction Graph: The time-to-collision component captures collision risk using TTCij = ∥pi − pj∥/∥vi − vj∥ and γ = 2.0 for approaching agents.TTC weights are capped at 10.0 for numerical stability.
  • C. Dynamic Interaction Graph: For efficiency, the graph retains agents within dmax = 10m and keeps the top k = 20 neighbors per agent.This limits the number of retained interactions.
  • B. Overview: Raw position-velocity features in R^4 are embedded by a 3-layer MLP into node features Z(0) in RB×N×Tobs×de.The embedding produces the representation used by later graph-processing modules.

D. Topology-Guided Node Prioritization

SGM orders graph nodes for sequential Mamba processing using dynamic interaction centrality, so influential agents appear later and receive richer accumulated context.

  • D. Topology-Guided Node Prioritization: SGM adapts Graph-Mamba’s node prioritization and permutation training to dynamic, physics-aware edge weights that change with real-time interaction strength.The resulting centrality is recalculated at every timestep rather than inherited from a static topology.
  • D. Topology-Guided Node Prioritization: Agents are sorted in ascending centrality, placing the most influential agents later in the sequence where Mamba has accumulated more context.Centrality is based on weighted degree derived from interaction weights wij.
  • D. Topology-Guided Node Prioritization: Community-aware centrality further raises the priority of group-bridge nodes and members of large communities.This augmentation is active when the community-aware module is used.
  • D. Topology-Guided Node Prioritization: After sorting, features are projected from de to hidden dimension d, producing Z(1) ∈ RB×N×Tobs×d.The projection prepares ordered features for the factorization branches.
  • D. Topology-Guided Node Prioritization: The social triplet factorization includes temporal, egocentric, and goalcentric branches, with edge weights scaling ego and goal tokens for each neighbor.The branch outputs are ZT, ZE, and ZG.

E. Social Triplet Factorization

The model extends social triplet factorization with graph-aware conditioning, direction-aware goal prediction, and learnable directional balance, while adding community-aware group discovery for joint multi-agent forecasting.

  • E. Social Triplet Factorization: Edge-weighted token conditioning scales ego and goal signals by each neighbor’s dynamic interaction intensity.
  • E. Social Triplet Factorization: A direction-aware goal predictor uses the ego agent’s last k=3 observed feature vectors to encode travel direction and velocity.
  • E. Social Triplet Factorization: Cycle Mamba replaces fixed equal forward–backward averaging with a learnable scalar θ that adapts directional balance during training.The parameter is initialized with σ(0)=0.5, and typically converges above 0.5 because trajectory prediction benefits from recency.
  • E. Social Triplet Factorization: Cross-directional hidden-state initialization propagates context between scans using half the parameters of a traditional bidirectional model.
  • E. Social Triplet Factorization: Unlike Social-Mamba, the model jointly forecasts trajectories for all P pedestrians rather than only the ego agent.Naively running egocentric and goalcentric branches P times would be prohibitive.
  • E. Social Triplet Factorization: A community-aware module discovers groups without supervision and conditions feature enrichment and multimodal decoding on inferred membership.Soft assignments and prototypes enrich node features before social triplet factorization, while community-aware centrality augments prioritization.

2) MinCut Optimization:

The community module uses differentiable MinCut optimization and confidence regularization to infer soft group assignments, enrich node features, and reorder agents by community.

  • 2) MinCut Optimization:: Differentiable MinCut optimization maximizes intra-community edge density without imposing balanced group sizes.
  • 2) MinCut Optimization:: The degree matrix is defined as D = diag(d), supporting the normalized MinCut formulation.
  • 2) MinCut Optimization:: An assignment confidence loss is added to encourage more confident community assignments.
  • 2) MinCut Optimization:: The combined community objective is Lcomm = Lcut + 0.5 Lconf.
  • 2) MinCut Optimization:: When community information is available, agents are reordered by community ID so same-community members form contiguous chunks during the Mamba scan.Agents are restored to their original ordering after scanning.
  • 2) MinCut Optimization:: Community-aware centrality augments node prioritization using group size and assignment entropy, with high-entropy assignments acting as bridge scores.

G. Dynamic Fusion and Global Scan

The model dynamically fuses three complementary branches using learned per-agent weights, then removes the prepended token while retaining socially enriched observation states.

  • G. Dynamic Fusion and Global Scan: A dynamic gating mechanism learns to fuse three branches according to scene-dependent relevance.
  • G. Dynamic Fusion and Global Scan: The MLP produces per-agent, per-branch weights through softmax normalization from the last-timestep concatenation.
  • G. Dynamic Fusion and Global Scan: After scanning, the prepended position-0 token is discarded, retaining positions 1 through T as socially enriched observation states.

2) Global Interaction Scan:

A final Mamba scan aggregates population-level crowd patterns across agents, after which dedicated multimodal decoders generate trajectory hypotheses and probabilities.

  • 2) Global Interaction Scan:: The Global Interaction Scan aggregates population-level patterns such as crowd-flow direction and density gradients across all agents.
  • 2) Global Interaction Scan:: The fused representation is reshaped so each timestep becomes an independent batch element processed along the N-agent sequence.
  • 2) Global Interaction Scan:: The decoder generates K trajectory hypotheses from each agent’s final representation.
  • 2) Global Interaction Scan:: Each mode uses a dedicated MLP decoder, providing architectural diversity among hypotheses.
  • 2) Global Interaction Scan:: Mode probabilities indicate the model’s confidence in each trajectory hypothesis.
  • 2) Global Interaction Scan:: Training combines multiple objectives, including a best-of-K trajectory loss, probability weighting λprob = 0.1, and community weighting λcomm = 0.05.

3) Optimization:

The evaluation uses ETH/UCY, SDD, and a configurable simulated crowd dataset, reporting displacement-based metrics under standard trajectory-prediction protocols.

  • Datasets: ETH/UCY uses leave-one-out training across five scenes, with 8 observed frames predicting 12 future frames at 2.5 FPS.Each scene contains diverse crowd densities and interaction patterns.
  • Datasets: SDD follows the standard TrajNet split and predicts 12 future frames from 8 observed frames across eight scenes with diverse agent types.The dataset includes pedestrians, bicyclists, skateboarders, and cars.
  • Datasets: The simulated crowd dataset supports controlled ablations with one robot, 15–29 pedestrians, and 3–5 social groups in configurable SFM scenarios.Simulation uses 10 FPS with Tobs = 10 and Tpred = 20 frames.
  • Evaluation Metrics: ADE and FDE measure average and final ℓ2 displacement errors, while minADE20 and minFDE20 select the best among 20 trajectory hypotheses.The evaluation reports four metrics with K = 20.

B. Implementation Details

SGM uses a compact Mamba-based configuration and is evaluated against graph, Transformer, and generative trajectory predictors on benchmark and physical-navigation tasks. It achieves competitive ETH/UCY results, state-of-the-art SDD scores, and improved ablation performance when its proposed modules are retained.

  • Implementation Details: The base SGM configuration has 1.55M parameters, hidden dimension d = 256, SSM state dimension ds = 16, and K = 20 trajectory hypotheses.Training uses batch size 64, 100 epochs, AdamW, and an NVIDIA RTX 4000 Ada GPU.
  • Physical Experiment: The physical system uses an Agilex Scout Mini with ROS 2 Humble, Zedx tracking, SFM control, and onboard Jetson-Orin inference at approximately 10 Hz.Experiments involve 2–4 human participants walking in predefined group patterns.
  • Benchmark Comparisons: SGM is compared with graph convolutional, Transformer-based, graph-generative, and group-structure methods on ETH/UCY and SDD benchmarks.The comparison includes Social-STGCNN, SGCN, STAR, AgentFormer, Trajectron++, PECNet, and MART.
  • Benchmark Comparisons: 0.23/0.37 average ADE/FDE makes SGM second-best on ETH/UCY, while SDD results establish a new state-of-the-art.The ETH/UCY result remains close to MART while avoiding self-attention.
  • Ablation Study: Removing all proposed enhancements increases ADE and FDE by 49.2% and 36.1%, respectively, relative to the full model.Social triplet factorization contributes most significantly, while the Community-Aware module provides consistent improvements.

2) Number of Modes:

The study reports that moderate multimodal prediction offers a favorable quality–efficiency trade-off, while SGM maintains linear computational complexity and supports plausible trajectories, group discovery, and safer robot navigation.

  • Performance gains from multimodal prediction saturate beyond a moderate number of modes, balancing prediction quality and computational efficiency.
  • SGM achieves linear sequence complexity O(N) and the lowest inference latency among compared methods, requiring 4.27 ms per forward pass with N = 50 agents.
  • SGM generates plausible future trajectories across ETH/UCY scenes with varied crowd densities and interaction patterns.
  • The community-aware module discovers coherent pedestrian groups without explicit group annotations and produces a learned hyperedge incidence representation.
  • 83.3% navigation success with the SGM-enhanced controller exceeds the baseline SFM controller’s 66.7% across ten physical trials.
  • Integrating predicted trajectories into SFM enables safer, more reliable real-time robot navigation in dynamic human-populated environments.
Loading 2608.21411v1…