PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene-LSTM: A Model for Human Trajectory Prediction

Huynh Manh, Gita Alaghband

arXiv:1808.04018v2cs.CV

TL;DR

Predicting pedestrian trajectories is difficult because social interactions and scene structures shape possible future paths. The paper couples pedestrian LSTMs with grid-cell Scene-LSTMs and a filter for selecting relevant scene information. Its models outperform existing methods in the reported comparisons, including substantial error reductions against social-interaction methods.

  • Problem

    Future pedestrian paths are difficult to predict because social interactions and scene structures can shape movement, including nonlinear turns.

    Source: §1 · p. 1

  • Method

    The method trains pedestrian LSTMs together with grid-cell Scene-LSTMs that learn common paths, while a scene data filter selects information relevant to each target’s state.

    Source: §1 · p. 2

  • Results

    Scene-LSTM-a and Scene-LSTM-n outperform the comparison methods on all three metrics on most video sequences, with Scene-LSTM-a reducing prediction error by 68% from Social-LSTM and 83% from Attention-LSTM.

    Source: §4.4 · p. 7

  • Takeaways & Limitations

    The results support that scene information encoded in grid-cell memories can help predict human movement in the tested video sequences.

    Source: §5 · p. 8

Abstract

from arXiv · show

We develop a human movement trajectory prediction system that incorporates the scene information (Scene-LSTM) as well as human movement trajectories (Pedestrian movement LSTM) in the prediction process within static crowded scenes. We superimpose a two-level grid structure (scene is divided into grid cells each modeled by a scene-LSTM, which are further divided into smaller sub-grids for finer spatial granularity) and explore common human trajectories occurring in the grid cell (e.g., making a right or left turn onto sidewalks coming out of an alley; or standing still at bus/train stops). Two coupled LSTM networks, Pedestrian movement LSTMs (one per target) and the corresponding Scene-LSTMs (one per grid-cell) are trained simultaneously to predict the next movements. We show that such common path information greatly influences prediction of future movement. We further design a scene data filter that holds important non-linear movement information. The scene data filter allows us to select the relevant parts of the information from the grid cell's memory relative to a target's state. We evaluate and compare two versions of our method with the Linear and several existing LSTM-based methods on five crowded video sequences from the UCY [1] and ETH [2] datasets. The results show that our method reduces the location displacement errors compared to related methods and specifically about 80% reduction compared to social interaction methods.

1. Introduction

Predicting pedestrian trajectories is difficult because social interactions and scene structures can shape future movement, including nonlinear turns. Scene-LSTM learns common paths within grid cells alongside pedestrian movement and uses filtered scene information to predict target locations.

1. Introduction

  • Scene structures can shape nonlinear movement: people leaving an alley may turn onto sidewalks, while social interactions can also affect their next movements.Open areas permit many possible paths and direction changes.Source: §1 · p. 1
  • Scene-LSTM and pedestrian LSTMs learn together, encoding common movements in scene-grid memories that help predict future locations, including turns from alleys onto sidewalks.The scene is divided into grid cells and finer sub-grids; trained grid-cell memories inform predictions at test time.Source: §1 · pp. 1–2
  • The scene data filter uses a hard filter to retain nonlinear movement information and a soft filter to select scene data relevant to each target’s state.The selected scene data is combined with the target’s movement to predict its next location.Source: §1 · p. 2
  • The authors model each target’s walking behavior with location offsets rather than the absolute locations used in prior research -.The paper identifies this as a contribution to human trajectory prediction.Source: §1 · p. 2

1.1 Problem Definition

  • Given observed pedestrian trajectories, the task is to predict each target’s locations over future frames in a static crowded scene.The paper frames the task as predicting continuous two-dimensional locations.Source: §1.1 · p. 2
  • Figure 1 contrasts a prediction using trained grid cells with one that does not; the caption reports greater accuracy with the trained cells.The trained cells are shown in pink, and the two predicted trajectories in red and black.Source: Figure 1 · p. 2

2. Related Works

Related work is divided into social-interaction models and social-scene models. The latter combine scene features with social interaction, while this paper focuses on relationships between past trajectories and movements in specific scene areas.

Social Interaction Models

  • Social-interaction approaches model pedestrian relations such as grouping and avoidance, using energy minimization, Gaussian processes, or LSTMs to predict movement.Social-LSTM uses nearby pedestrians’ states; Social-Attention [8] represents scene-wide social influence with an attention vector.Source: §2 · pp. 2–3

Social-Scene Models

  • Social-scene models combine scene features with social interactions, often using CNN-extracted scene layouts as inputs to target LSTMs or measuring obstacle distances.The cited approaches include, [13], and [22].Source: §2 · p. 3
  • Prior social-scene methods do not characterize how past trajectories relate to a target’s current movements in a specific scene area; this work learns those relationships in grid-cell memories.The paper compares its method with social-interaction methods, [8] and social-scene method.Source: §2 · p. 3

3. System Design

The system-design section introduces LSTM networks as the basis for modeling pedestrian movement. It describes how memory and gates process past and current information, while noting that movement histories alone do not capture scene-dependent future behavior.

3.1 LSTM: A Review

  • An LSTM contains a memory cell and input, output, and forget gates that control how time-series information is stored and used.The output gate selects memory information used to calculate the hidden state.Source: §3.1 · p. 3
  • For pedestrian prediction, an LSTM uses a target’s hidden state and current location to calculate its predicted location.The output layer maps the hidden state to the predicted location through learned weights and a bias.Source: §3.1 · p. 3
  • The authors argue that independently modeling a target’s movement is insufficient because future behavior may differ from past behavior and depends on scene context.The section then points to Scene-LSTM as a way to model that context.Source: §3.1 · p. 3

3.2 The overview of our model:

The system combines pedestrian movement LSTMs with scene-model and scene-data-filter components. The pedestrian model predicts movement from location offsets, using filtered scene information to inform its next-location prediction.

3.2 The overview of our model:

  • Scene LSTMs, pedestrian movement LSTMs, and a scene data filter form the system’s three main components.The overview places these components in a pipeline that first preprocesses video images.Source: §3.2 · p. 3

3.3 Pedestrian’s movement

  • Each target’s pedestrian LSTM predicts its next location from its current memory state, with filtered scene data supplied as an input.The model predicts relative location offsets rather than absolute locations; the offsets are less tied to a particular scene layout.Source: §3.3 · pp. 3–4
  • The pedestrian LSTM uses an embedded location offset and its prior hidden state to predict movement, then combines the predicted offset with the previous location.The output also includes a predicted displacement distribution and correlation coefficient.Source: §3.3 · pp. 3–4

3.4 Scene Model

The Scene Model trains grid-cell memories to encode common movement patterns. A target’s location within a cell and its prior hidden state contribute to the cell’s state, which is passed to the scene data filter.

3.4 Scene Model

  • The Scene Model trains grid-cell memories to characterize common trajectories within the cells.Each grid cell has a hidden state that is updated during training and supplied to the scene data filter.Source: §3.4 · p. 4
  • A target’s position within its grid cell is represented as a one-hot vector over 16 sub-grids.The location vector is concatenated with the target’s previous hidden state as input to that grid cell’s LSTM.Source: §3.4 · p. 4
  • Trained grid-cell states encode common paths relative to scene structures or obstacles, including buildings, snowy pavement, and trees.The paper states that these histories help predict trajectories more accurately.Source: §3.4 · p. 4

3.5 Scene Data Filter (SDF)

The Scene Data Filter limits which grid-cell information can influence a target’s predicted movement. It first gates cells containing nonlinear trajectories, then selects scene information relevant to the target’s state.

3.5 Scene Data Filter (SDF)

  • The filter addresses cells whose memories may be unhelpful when many distinct trajectories share an unconstrained area.The paper describes such memories as potentially chaotic and not helpful for predicting or adjusting a target’s trajectory.Source: §3.5 · p. 5
  • The hard filter passes scene data only from grid cells containing nonlinear trajectories observed during training.A trajectory is classified as nonlinear when its Φ_i exceeds 0.2 in the experiments.Source: §3.5 · p. 5
  • The soft filter selects the target-relevant portion of the hard-filtered scene data to produce the final filtered input.That filtered data is combined with the target’s embedded vector for the pedestrian movement model.Source: §3.5 · p. 5
  • Training minimizes negative log-likelihood, and testing uses observed trajectories to update grid-cell memories before forecasting future locations.The forecast proceeds from the last observed frame through the predicted frames.Source: §3.5 · p. 5

4.1 Datasets and Metrics

Evaluation uses five ETH and UCY video sequences, with three displacement-error metrics. The paper compares two Scene-LSTM variants against linear and LSTM-based methods.

4.1 Datasets and Metrics

  • The evaluation covers five sequences and 1,536 pedestrians across ETH and UCY, with UCY control-point locations linearly interpolated between annotations.The sequences are ETH-Hotel, ETH-Univ, UCY-Univ, ZARA-01, and ZARA-02.Source: §4.1 · p. 5
  • The metrics are ADE over predicted trajectory locations, NDE over nonlinear predicted trajectories, and FDE at each trajectory’s final predicted location.The passage defines ADE and NDE as mean-square errors and FDE as mean-square error at the final location.Source: §4.1 · pp. 5–6
  • The comparison includes Linear, LSTM, Social-LSTM, Social-Attention, SS-LSTM, and Scene-LSTM-a and Scene-LSTM-n.Scene-LSTM-a uses scene data from all grid cells, while Scene-LSTM-n uses only non-linear grid cells; some methods are excluded because code or comparable metrics are unavailable.Source: §4.1 · p. 6

4.2 Implementation details:

Training uses a leave-one-out, two-stage procedure, with a separate adaptation and test split for each held-out video. The models use fixed grid, memory, and optimization settings.

4.2 Implementation details:

  • Each model uses 128-dimensional memory and hidden states, an 8×8 scene grid with 4×4 sub-grids, Adam at 0.003, dropout 0.2, and gradient clipping at 10.Training uses PyTorch on a Tesla P100-SXM2 GPU.Source: §4.2 · p. 6
  • Stage 1 trains and validates on four videos, selects the lowest-ADE weights, and repeats this leave-one-out process across all five held-out videos.For each permutation, four videos are split 80%/20% for training and validation, and trained for 100 epochs.Source: §4.2 · p. 6
  • Stage 2 adapts the selected weights on 50% of the held-out video for 10 epochs, then tests on its remaining 50%.Testing observes eight frames (3.2 seconds) and predicts the next 12 frames (4.8 seconds) in a sliding window.Source: §4.2 · pp. 6–7
  • Figure 4 plots ADE against stage-2 training-data amount for each video, separating Scene-LSTM-a and Scene-LSTM-n before comparing them.The comparison panel uses a different scale.Source: Figure 4 · p. 6

4.3 The impacts of training data amount in stage-2

More stage-2 training data lowers prediction error for both Scene-LSTM variants, but the amount needed varies with the video sequence. Similar scene layouts can benefit from movement patterns learned during stage 1.

4.3 The impacts of training data amount in stage-2

  • At 50% rather than 0% stage-2 training data, ADE falls 89% for Scene-LSTM-a and 68% for Scene-LSTM-n.Testing is performed on the last 50% of each video.Source: §4.3 · p. 7
  • UCY-Zara01 and UCY-Zara02 reach good results with little scene information, whereas ETH-Hotel, ETH-Univ, and UCY-Univ require more.The latter videos have scene structures distinct from one another and from the shared Zara layout, whose common movements can be learned in stage 1.Source: §4.3 · p. 7

4.4 Comparison with related methods

Across five video sequences, the Scene-LSTM variants outperform related methods on all three metrics on most sequences, with reported gains over social-interaction methods. The nonlinear-grid variant is stronger on sequences with nonlinear movement, while qualitative examples illustrate scene-specific turns, slowing, and stops.

4.4 Comparison with related methods

  • Scene-LSTM-a reduces prediction error 68% from Social-LSTM and 83% from Attention-LSTM; both variants outperform others on all three metrics on most sequences.The methods predict 12 frames from 8 observed frames; Scene-LSTM variants have better FDE on all sequences than SS-LSTM.Source: Table 1 · p. 7, §4.4 · p. 7
  • Qualitative examples show Scene-LSTM-n capturing alley turns, sidewalk entry, and stopping or slowing patterns more accurately than the other methods.Figure 5 includes UCY-Zara01, ETH-Univ, and ETH-Hotel examples.Source: Figure 5 · p. 8, §4.4 · p. 8
  • Scene-LSTM-n performs better on every sequence with observed nonlinear movement, except ETH-Hotel, where most people move linearly.The authors report that selecting nonlinear grid cells yields more accurate trajectories in nonlinear scenes than Scene-LSTM-a's all-grid approach.Source: §4.4 · p. 8

5. Conclusion

The conclusion presents Scene-LSTM as a way to use learned scene information alongside pedestrian trajectories for prediction. The authors identify combining scene and social models, broadening interaction modeling, and applying the method to tracking as future work.

5. Conclusion

  • Scene-LSTM assigns a learned memory cell to each scene grid cell to retain past local movement information for predicting future trajectories.The authors report that their methods outperform existing methods.Source: §5 · p. 8
  • In qualitative examples, the models predict movement toward building shade and an alley-exit turn, while other methods sometimes predict paths into obstacles or straight into the street.The examples attribute these predictions to similar paths observed in the grid cells during training.Source: §5 · p. 8
  • Future work will combine scene and social models, study interactions with people and objects, and apply trajectory prediction to multi-target tracking.Source: §5 · p. 8
Loading 1808.04018v2…