Source-linked AI summary
Social Attention: Modeling Attention in Human Crowds
Anirudh Vemula, Katharina Muelling, Jean Oh
TL;DR
Robots need human-trajectory prediction that supports safe, efficient, and human-predictable crowd navigation, but proximity alone does not determine which agents matter. Social Attention learns soft attention over all agents within a recurrent trajectory model, outperforming Social LSTM across evaluated crowd sets and metrics while revealing nonlocal attention patterns.
Problem
Proximity-based interaction models can overlook distant pedestrians whose velocity, heading, acceleration, or time-to-collision makes them relevant to future motion.
Method
Social Attention uses a jointly trained recurrent neural-network mixture with soft attention over all humans to model spatial-temporal interactions and predict trajectories.
Results
Social Attention performs better than Social LSTM consistently across all crowd sets in both reported metrics and identifies dynamic or nonlocal pedestrians as important in qualitative cases.
Takeaways & Limitations
The learned attention model provides a data-driven way to represent relative pedestrian influence beyond spatial proximity and analyze human-human interactions in crowds.
Abstract
from arXiv · showhide
Robots that navigate through human crowds need to be able to plan safe, efficient, and human predictable trajectories. This is a particularly challenging problem as it requires the robot to predict future human trajectories within a crowd where everyone implicitly cooperates with each other to avoid collisions. Previous approaches to human trajectory prediction have modeled the interactions between humans as a function of proximity. However, that is not necessarily true as some people in our immediate vicinity moving in the same direction might not be as important as other people that are further away, but that might collide with us in the future. In this work, we propose Social Attention, a novel trajectory prediction model that captures the relative importance of each person when navigating in the crowd, irrespective of their proximity. We demonstrate the performance of our method against a state-of-the-art approach on two publicly available crowd datasets and analyze the trained attention model to gain a better understanding of which surrounding agents humans attend to, when navigating in a crowd.
I. INTRODUCTION
Crowd navigation requires accurate human-trajectory prediction because robots must move safely, efficiently, and socially compliantly. Social Attention addresses the limits of proximity-based interaction models by learning which agents matter for each pedestrian.
- Robots need accurate future human trajectories to plan safe, efficient, and human-predictable paths through crowds.
- Independent motion models miss complex human-human interactions, producing highly suboptimal robot paths.
- Spatially local joint models capture interacting trajectories but assume that only nearby humans influence one another.
- Velocity, time-to-collision, acceleration, and heading can make distant pedestrians relevant to one another.A runner may trigger collision-avoidance adaptation before entering another pedestrian’s local neighborhood.
- Social Attention uses a jointly trained recurrent neural-network mixture with soft attention over all crowd members to predict trajectories without a local-neighborhood restriction.The model represents both spatial and temporal aspects of crowd motion.
III. RELATED WORK
Prior work models crowd motion with social forces, proximity relationships, Gaussian processes, and inverse reinforcement learning. These approaches capture selected interaction patterns but remain limited by missing cooperation, handcrafted features, local assumptions, or restricted evaluation settings.
- Social-force models represent pedestrian motion through attractive destination forces and repulsive collision-avoidance forces.
- Proximity-based methods capture reactive collision avoidance but do not model human-human and human-robot cooperation.The paper identifies cooperation as essential for safe and efficient navigation in dense crowds.
- Interacting Gaussian Processes model joint crowd trajectories with handcrafted interaction potentials based on relative distances.
- Maximum-entropy inverse reinforcement learning jointly predicts trajectories using designed features, but prior tests used scripted environments with no more than four agents.
B. Human Trajectory Prediction
The paper formulates crowd trajectory prediction as a spatio-temporal graph problem and learns interactions across all agents rather than assuming spatial locality. A jointly trained recurrent model predicts future locations while capturing human-human interactions.
- The model asks which surrounding agents humans attend to, because velocity, acceleration, and heading can make nonlocal agents influential.
- B. Human Trajectory Prediction: A feedforward, fully differentiable, jointly trained RNN mixture predicts future locations while coupling agents’ predictions to capture interactions.
- A. Spatio-Temporal Graph Representation: A spatio-temporal graph represents humans as nodes, same-time interactions as spatial edges, and consecutive self-motion as temporal edges.Spatial edges encode relative orientation and distance, while temporal edges encode each human’s trajectory dynamics.
- Shared node and edge factors allow the factor-graph formulation to scale to crowds with more nodes without increasing parameter counts.
B. Model Architecture
The architecture models individual motion and human-human interactions with shared recurrent factors, then adds soft attention over spatial interaction states. This attention lets each node weight neighboring interaction representations rather than treating them uniformly.
- Node, spatial-edge, and temporal-edge RNNs share parameters, modeling individual motion and human-human interaction dynamics.
- B. Model Architecture: An attention module computes soft attention over neighboring spatial edge-RNN hidden states for each node.
- 1) EdgeRNN:: Each spatial edge-RNN processes its edge features at every time-step before contributing to node-level prediction.
- 1) EdgeRNN:: The edge-RNN embeds edge features into fixed-length vectors and supplies them as inputs to recurrent cells.
- 1) EdgeRNN:: Spatial edge-RNN states represent interactions, while temporal edge-RNNs use separate parameters for temporal dynamics.
2) Attention Module:
The attention module scores each node’s temporal state against neighboring spatial-edge states, then aggregates those states into an attention output for the nodeRNN.
- Each node compares its temporal edgeRNN hidden state with the hidden states of its neighboring spatial edgeRNNs.
- The module uses scaled dot-product attention to compute interaction scores between temporal and spatial-edge hidden states.The score is scaled according to the number of spatial edges and the attention dimension.
- The scaling uses m and d_e because the number of spatial edges varies with the number of agents and large attention dimensions can degrade dot-product attention.
- The attention output is a softmax-weighted sum of neighboring spatial-edge hidden states.
- The attention module’s trainable parameters are the projection weights W1 and W2.
- The resulting attention output is passed to the nodeRNN together with the node’s features and temporal-edge state.
3) NodeRNN:
The nodeRNN combines the node’s embedded features with temporal context and the attention output, then predicts parameters of a bivariate Gaussian for the next position.
- The node’s features are embedded into a fixed-length vector before recurrent processing.
- The nodeRNN concatenates the temporal edgeRNN state with the attention output and embeds that context.
- The node-feature and interaction-context embeddings are concatenated and supplied to the recurrent cell.
- A linear output layer maps the recurrent hidden state to a 5D vector representing the mean, standard deviation, and correlation of a bivariate Gaussian.
- The nodeRNN’s trainable parameters include the embedding and recurrent-model weights collected as W_node.
C. Training the model
The model is jointly trained by minimizing the negative log-likelihood of observed future positions under the predicted bivariate Gaussian, with gradients propagated through all recurrent components.
- Training minimizes the negative log-likelihood of each node’s true position at every predicted time-step under the predicted bivariate Gaussian.
- The loss is evaluated over trajectories of all nodes in the training dataset and backpropagated.
- Backpropagation jointly updates the nodeRNN, spatial edgeRNN, and temporal edgeRNN parameters.
D. Inference for path prediction
At inference, the trained model conditions on observed trajectories, samples future locations from predicted bivariate Gaussians, and recursively feeds predictions back into the model.
- The trained model is fitted to observed trajectories and samples forecasted locations from the predicted bivariate Gaussian for all pedestrians.
- For later prediction steps, the model uses the previous predicted location instead of the true coordinates as node features.
- Predicted locations are also used to compute spatial edge features at later time-steps.
A. Datasets and Metrics
Social Attention is evaluated on two public crowd datasets using displacement-error metrics and a leave-one-out protocol. It consistently outperforms Social LSTM across crowd sets, while accounting for late-entering pedestrians and maintaining 10Hz inference.
- Datasets: ETH and UCY contain five crowd sets with 1,536 pedestrians exhibiting complex interactions across four scenes.The datasets are recorded at 25 frames per second and annotated every 0.4 second.
- Metrics: Average Displacement Error measures mean Euclidean distance across predicted time steps, while Final Displacement Error measures distance at the prediction horizon.Evaluation uses a leave-one-out protocol, training and validating on four sets and testing on the remaining set.
- Results: Social Attention performs better than Social LSTM consistently across all crowd sets on both displacement metrics.The independent LSTM performs slightly better in some sparse-crowd cases where interactions are scarce.
- Results: In ETH-Hotel, Social Attention significantly outperforms competing methods, especially when pedestrians are stationary or approach one another with varied velocities and headings.The result supports modeling non-local interactions using features beyond proximity.
- Evaluation scope: Including pedestrians observed for fewer than Tobs time steps lowers prediction accuracy but reflects late arrivals that frequently occur in robot navigation.The authors include these cases, unlike the original Social LSTM evaluation.
- Deployment: Accounting for all crowd agents increases computational complexity, but GPU-parallelized inference achieves real-time performance at 10Hz.
D. Qualitative Results
Qualitative analysis visualizes Social Attention’s learned relative importance assignments and shows both successful non-local attention patterns and systematic failures. The qualitative results complement the quantitative evaluation by illustrating when motion and distance shape attention.
- Visualization: Figure 4 visualizes predicted attention weights around surrounding pedestrians, with larger circles indicating greater relative attention.The red trajectory is the target pedestrian; solid dots show past locations and blue diamonds show current locations.
- Successful cases: The model gives higher weight to a nearby dynamic pedestrian than to distant pedestrians in one scenario.
- Successful cases: The model assigns lower importance to stationary nearby pedestrians than to a dynamic pedestrian farther away.This illustrates attention that is not determined solely by spatial proximity.
- Successful cases: The model assigns equal importance to dynamic pedestrians when they are all too far away to influence the target pedestrian.
- Failure cases: In failure cases, the model can over-attend to distant, non-influential pedestrians while ignoring nearby pedestrians, or assign equal weights when one pedestrian matters more.The authors leave investigation of these prediction failures to future work.
- Conclusion: The paper concludes that Social Attention learns relative pedestrian influence and improves trajectory prediction while providing qualitative insight into crowd attention.The model is evaluated on two publicly available datasets.