PaperlayerView exact revision on arXiv

Source-linked AI summary

Social Attention: Modeling Attention in Human Crowds

Anirudh Vemula, Katharina Muelling, Jean Oh

arXiv:1710.04689v2cs.ROcs.LG

TL;DR

Robots need human-trajectory prediction that supports safe, efficient, and human-predictable crowd navigation, but proximity alone does not determine which agents matter. Social Attention learns soft attention over all agents within a recurrent trajectory model, outperforming Social LSTM across evaluated crowd sets and metrics while revealing nonlocal attention patterns.

  • Problem

    Proximity-based interaction models can overlook distant pedestrians whose velocity, heading, acceleration, or time-to-collision makes them relevant to future motion.

    Source: §I · p. 1, §IV · p. 3

  • Method

    Social Attention uses a jointly trained recurrent neural-network mixture with soft attention over all humans to model spatial-temporal interactions and predict trajectories.

    Source: §I · p. 1

  • Results

    Social Attention performs better than Social LSTM consistently across all crowd sets in both reported metrics and identifies dynamic or nonlocal pedestrians as important in qualitative cases.

    Source: C. Quantitative Results · p. 5, D. Qualitative Results · p. 6

  • Takeaways & Limitations

    The learned attention model provides a data-driven way to represent relative pedestrian influence beyond spatial proximity and analyze human-human interactions in crowds.

    Source: §I · p. 1, §IV · p. 3, §VI · p. 6

Abstract

from arXiv · show

Robots that navigate through human crowds need to be able to plan safe, efficient, and human predictable trajectories. This is a particularly challenging problem as it requires the robot to predict future human trajectories within a crowd where everyone implicitly cooperates with each other to avoid collisions. Previous approaches to human trajectory prediction have modeled the interactions between humans as a function of proximity. However, that is not necessarily true as some people in our immediate vicinity moving in the same direction might not be as important as other people that are further away, but that might collide with us in the future. In this work, we propose Social Attention, a novel trajectory prediction model that captures the relative importance of each person when navigating in the crowd, irrespective of their proximity. We demonstrate the performance of our method against a state-of-the-art approach on two publicly available crowd datasets and analyze the trained attention model to gain a better understanding of which surrounding agents humans attend to, when navigating in a crowd.

I. INTRODUCTION

Crowd navigation requires accurate human-trajectory prediction because robots must move safely, efficiently, and socially compliantly. Social Attention addresses the limits of proximity-based interaction models by learning which agents matter for each pedestrian.

  • Robots need accurate future human trajectories to plan safe, efficient, and human-predictable paths through crowds.Source: §I · p. 1
  • Independent motion models miss complex human-human interactions, producing highly suboptimal robot paths.Source: §I · p. 1
  • Spatially local joint models capture interacting trajectories but assume that only nearby humans influence one another.Source: §I · p. 1
  • Velocity, time-to-collision, acceleration, and heading can make distant pedestrians relevant to one another.A runner may trigger collision-avoidance adaptation before entering another pedestrian’s local neighborhood.Source: §I · p. 1
  • Social Attention uses a jointly trained recurrent neural-network mixture with soft attention over all crowd members to predict trajectories without a local-neighborhood restriction.The model represents both spatial and temporal aspects of crowd motion.Source: §I · p. 1

III. RELATED WORK

Prior work models crowd motion with social forces, proximity relationships, Gaussian processes, and inverse reinforcement learning. These approaches capture selected interaction patterns but remain limited by missing cooperation, handcrafted features, local assumptions, or restricted evaluation settings.

B. Human Trajectory Prediction

The paper formulates crowd trajectory prediction as a spatio-temporal graph problem and learns interactions across all agents rather than assuming spatial locality. A jointly trained recurrent model predicts future locations while capturing human-human interactions.

  • The model asks which surrounding agents humans attend to, because velocity, acceleration, and heading can make nonlocal agents influential.Source: §IV · p. 3

B. Human Trajectory Prediction

  • A feedforward, fully differentiable, jointly trained RNN mixture predicts future locations while coupling agents’ predictions to capture interactions.Source: §IV · p. 3

A. Spatio-Temporal Graph Representation

  • A spatio-temporal graph represents humans as nodes, same-time interactions as spatial edges, and consecutive self-motion as temporal edges.Spatial edges encode relative orientation and distance, while temporal edges encode each human’s trajectory dynamics.Source: A. Spatio-Temporal Graph Representation · p. 3

B. Model Architecture

The architecture models individual motion and human-human interactions with shared recurrent factors, then adds soft attention over spatial interaction states. This attention lets each node weight neighboring interaction representations rather than treating them uniformly.

  • Node, spatial-edge, and temporal-edge RNNs share parameters, modeling individual motion and human-human interaction dynamics.Source: B. Model Architecture · p. 3

B. Model Architecture

1) EdgeRNN:

  • Each spatial edge-RNN processes its edge features at every time-step before contributing to node-level prediction.Source: 1) EdgeRNN · p. 3
  • The edge-RNN embeds edge features into fixed-length vectors and supplies them as inputs to recurrent cells.Source: 1) EdgeRNN · p. 3
  • Spatial edge-RNN states represent interactions, while temporal edge-RNNs use separate parameters for temporal dynamics.Source: 1) EdgeRNN · p. 3

2) Attention Module:

The attention module scores each node’s temporal state against neighboring spatial-edge states, then aggregates those states into an attention output for the nodeRNN.

  • Each node compares its temporal edgeRNN hidden state with the hidden states of its neighboring spatial edgeRNNs.Source: 2) Attention Module · p. 4
  • The module uses scaled dot-product attention to compute interaction scores between temporal and spatial-edge hidden states.The score is scaled according to the number of spatial edges and the attention dimension.Source: 2) Attention Module · p. 4
  • The scaling uses m and d_e because the number of spatial edges varies with the number of agents and large attention dimensions can degrade dot-product attention.Source: 2) Attention Module · p. 4
  • The attention output is a softmax-weighted sum of neighboring spatial-edge hidden states.Source: 2) Attention Module · p. 4
  • The attention module’s trainable parameters are the projection weights W1 and W2.Source: 2) Attention Module · p. 4
  • The resulting attention output is passed to the nodeRNN together with the node’s features and temporal-edge state.Source: 3) NodeRNN · p. 4

3) NodeRNN:

The nodeRNN combines the node’s embedded features with temporal context and the attention output, then predicts parameters of a bivariate Gaussian for the next position.

  • The node’s features are embedded into a fixed-length vector before recurrent processing.Source: 3) NodeRNN · p. 4
  • The nodeRNN concatenates the temporal edgeRNN state with the attention output and embeds that context.Source: 3) NodeRNN · p. 4
  • The node-feature and interaction-context embeddings are concatenated and supplied to the recurrent cell.Source: 3) NodeRNN · p. 4
  • A linear output layer maps the recurrent hidden state to a 5D vector representing the mean, standard deviation, and correlation of a bivariate Gaussian.Source: 3) NodeRNN · p. 4
  • The nodeRNN’s trainable parameters include the embedding and recurrent-model weights collected as W_node.Source: 3) NodeRNN · p. 4

C. Training the model

The model is jointly trained by minimizing the negative log-likelihood of observed future positions under the predicted bivariate Gaussian, with gradients propagated through all recurrent components.

  • Training minimizes the negative log-likelihood of each node’s true position at every predicted time-step under the predicted bivariate Gaussian.Source: C. Training the model · p. 4
  • The loss is evaluated over trajectories of all nodes in the training dataset and backpropagated.Source: C. Training the model · p. 4
  • Backpropagation jointly updates the nodeRNN, spatial edgeRNN, and temporal edgeRNN parameters.Source: C. Training the model · p. 4

D. Inference for path prediction

At inference, the trained model conditions on observed trajectories, samples future locations from predicted bivariate Gaussians, and recursively feeds predictions back into the model.

A. Datasets and Metrics

Social Attention is evaluated on two public crowd datasets using displacement-error metrics and a leave-one-out protocol. It consistently outperforms Social LSTM across crowd sets, while accounting for late-entering pedestrians and maintaining 10Hz inference.

Datasets

  • ETH and UCY contain five crowd sets with 1,536 pedestrians exhibiting complex interactions across four scenes.The datasets are recorded at 25 frames per second and annotated every 0.4 second.Source: A. Datasets and Metrics · p. 4

Metrics

  • Average Displacement Error measures mean Euclidean distance across predicted time steps, while Final Displacement Error measures distance at the prediction horizon.Evaluation uses a leave-one-out protocol, training and validating on four sets and testing on the remaining set.Source: A. Datasets and Metrics · p. 5

Results

  • Social Attention performs better than Social LSTM consistently across all crowd sets on both displacement metrics.The independent LSTM performs slightly better in some sparse-crowd cases where interactions are scarce.Source: C. Quantitative Results · p. 5
  • In ETH-Hotel, Social Attention significantly outperforms competing methods, especially when pedestrians are stationary or approach one another with varied velocities and headings.The result supports modeling non-local interactions using features beyond proximity.Source: C. Quantitative Results · p. 5

Evaluation scope

  • Including pedestrians observed for fewer than Tobs time steps lowers prediction accuracy but reflects late arrivals that frequently occur in robot navigation.The authors include these cases, unlike the original Social LSTM evaluation.Source: C. Quantitative Results · p. 5

Deployment

  • Accounting for all crowd agents increases computational complexity, but GPU-parallelized inference achieves real-time performance at 10Hz.Source: C. Quantitative Results · p. 5

D. Qualitative Results

Qualitative analysis visualizes Social Attention’s learned relative importance assignments and shows both successful non-local attention patterns and systematic failures. The qualitative results complement the quantitative evaluation by illustrating when motion and distance shape attention.

Visualization

  • Figure 4 visualizes predicted attention weights around surrounding pedestrians, with larger circles indicating greater relative attention.The red trajectory is the target pedestrian; solid dots show past locations and blue diamonds show current locations.Source: D. Qualitative Results · pp. 5–6

Successful cases

  • The model gives higher weight to a nearby dynamic pedestrian than to distant pedestrians in one scenario.Source: D. Qualitative Results · p. 6
  • The model assigns lower importance to stationary nearby pedestrians than to a dynamic pedestrian farther away.This illustrates attention that is not determined solely by spatial proximity.Source: D. Qualitative Results · p. 6
  • The model assigns equal importance to dynamic pedestrians when they are all too far away to influence the target pedestrian.Source: D. Qualitative Results · p. 6

Failure cases

  • In failure cases, the model can over-attend to distant, non-influential pedestrians while ignoring nearby pedestrians, or assign equal weights when one pedestrian matters more.The authors leave investigation of these prediction failures to future work.Source: D. Qualitative Results · p. 6

Conclusion

  • The paper concludes that Social Attention learns relative pedestrian influence and improves trajectory prediction while providing qualitative insight into crowd attention.The model is evaluated on two publicly available datasets.Source: §VI · p. 6

Related papers

Loading 1710.04689v2…