PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

Bingbin Liu, Ehsan Adeli, Zhangjie Cao, Kuan-Hui Lee, Abhijeet Shenoi, Adrien Gaidon, Juan Carlos Niebles

arXiv:2002.08945v1cs.CV

TL;DR

Pedestrian intent prediction needs spatiotemporal scene reasoning because conventional approaches do not capture relationships among objects and context. The paper builds graph-based models from segmented instances using pedestrian-centric and location-centric views, achieving reported intent-prediction accuracy on STIP and JAAD about one second before crossing.

  • Problem

    Existing prediction models lack the ability to reason over spatiotemporal data and object relationships, while pedestrian intent is important for autonomous vehicles' navigation.

  • Method

    The paper builds spatiotemporal graphs over segmented scene instances and applies graph convolution in pedestrian-centric and location-centric settings.

  • Results

    The framework predicts pedestrian crossing intent with accuracy over 80% on STIP and slightly under 80% on JAAD about one second ahead.

  • Takeaways & Limitations

    Spatiotemporal relationship reasoning supports pedestrian-intent prediction in dense driving scenes using pedestrian-centric and location-centric views.

Abstract

from arXiv · show

Reasoning over visual data is a desirable capability for robotics and vision-based applications. Such reasoning enables forecasting of the next events or actions in videos. In recent years, various models have been developed based on convolution operations for prediction or forecasting, but they lack the ability to reason over spatiotemporal data and infer the relationships of different objects in the scene. In this paper, we present a framework based on graph convolution to uncover the spatiotemporal relationships in the scene for reasoning about pedestrian intent. A scene graph is built on top of segmented object instances within and across video frames. Pedestrian intent, defined as the future action of crossing or not-crossing the street, is a very crucial piece of information for autonomous vehicles to navigate safely and more smoothly. We approach the problem of intent prediction from two different perspectives and anticipate the intention-to-cross within both pedestrian-centric and location-centric scenarios. In addition, we introduce a new dataset designed specifically for autonomous-driving scenarios in areas with dense pedestrian populations: the Stanford-TRI Intent Prediction (STIP) dataset. Our experiments on STIP and another benchmark dataset show that our graph modeling framework is able to predict the intention-to-cross of the pedestrians with an accuracy of 79.10% on STIP and 79.28% on \rev{Joint Attention for Autonomous Driving (JAAD) dataset up to one second earlier than when the actual crossing happens. These results outperform the baseline and previous work. Please refer to http://stip.stanford.edu/ for the dataset and code.

I. INTRODUCTION

Pedestrian intent prediction requires reasoning about visual relationships and context beyond pedestrian-specific cues or trajectories. The paper proposes spatiotemporal graph reasoning with pedestrian-centric and location-centric perspectives for autonomous-driving intent prediction.

  • Pedestrian intent, defined as whether a person will cross the street, provides a high-level cue for safer and smoother autonomous driving.
  • Trajectory prediction is challenging because subtle and changing factors can produce many possible pedestrian trajectories.
  • Existing approaches commonly focus on pedestrian-specific features such as location, velocity, and pose, while ignoring interactions with surrounding scene elements.
  • The proposed approach builds a pedestrian-centric dynamic scene graph from segmented instances and applies graph convolution to reason about relationships between instances.
  • The work models intent from pedestrian-centric and location-centric perspectives and introduces a dataset designed for vehicle-centric scenes.

II. RELATED WORK

Prior work spans pedestrian detection, tracking, pose, trajectory prediction, and action anticipation, but intent prediction remains distinct because trajectories are complex and contextual reasoning may require additional modalities. The proposed framework instead uses raw video and spatiotemporal graph reasoning to model scene relationships.

  • Pedestrian detection and tracking methods provide foundational estimates of people in scenes, which this work takes as given while focusing on visual reasoning.
  • Trajectory prediction models future pedestrian positions, but trajectories are difficult to define and require more annotations and supervision.
  • Some prior methods incorporate context through additional modalities or constraints, including depth and bird’s-eye-view information.
  • Action anticipation research includes sequence-based forecasting, spatiotemporal graphs, object affordances, and reinforcement learning.
  • The paper uses raw video frames and constructs pedestrian-centric spatiotemporal graphs whose features are processed over time.

A. Scene Parsing

The model parses video frames into pedestrians and contextual objects, then represents their relationships with a pedestrian-centered graph. Graph edges encode spatial and appearance-based relevance for reasoning about crossing intent.

  • Scene Parsing: Input frames are parsed into pedestrian bounding boxes and binary masks for objects of interest.Ground-truth pedestrian boxes and off-the-shelf instance segmentation provide the inputs; appearance information is discarded to preserve relative location and movement cues.
  • Graph Convolution: Graph convolution propagates node features through an adjacency matrix using trainable weights.X contains node feature vectors, A is the adjacency matrix, and W is the trainable graph-convolution weight.
  • Graph Convolution: Each pedestrian is modeled as the center of a star graph connected to surrounding objects.The graph uses separate nodes for the pedestrian and contextual objects, with edges representing relationship strength.
  • Prediction: The resulting representation captures relationships between the pedestrian and surrounding scene elements for future crossing prediction.The model observes crossing or non-crossing frames and predicts crossing likelihood over a future temporal horizon.
  • Graph Convolution: Edge weights combine pedestrian appearance with spatial relations represented by an eight-dimensional bounding-box vector.The spatial vector includes union-box dimensions and relative corner and center offsets; pedestrian appearance can indicate attention or gaze direction.

2) Location-Centric Prediction:

The location-centric setting predicts whether any pedestrian will enter the area the ego-car is expected to cover. It uses an ego-centered star graph to focus computation on scene elements relevant to near-future driving behavior.

  • Location-Centric Prediction: Future crossing is mapped to whether a pedestrian enters a designated trapezoidal area in the near future.The area represents the road region the current car is expected to cover.
  • Location-Centric Prediction: Focusing on potentially relevant pedestrians simplifies computation and is more directly aligned with vehicle control.The setting is especially useful in busy scenes containing numerous pedestrians.
  • Location-Centric Prediction: The location-centric graph centers on the ego-car viewpoint and connects it to pedestrians, vehicles, riders, and traffic signs.This star graph replaces a separate pedestrian-centered graph for each pedestrian.
  • Location-Centric Prediction: Context objects and the egocentric scene are embedded in a common space, with sigmoid-constrained inner products defining edge weights.This replaces pairwise relative spatial location as the basis for edge weighting.

C. Temporal Connection

Temporal connections extend the per-frame pedestrian-centric graphs with information from the observed frame sequence. GRUs encode pedestrian history and aggregate frame-level graph representations.

  • Temporal Connection: Pedestrian nodes across frames are connected with a GRU to encode temporal relations.Contextual objects are not explicitly associated across frames because graph convolution can propagate their information through the frame-level representations.

D. Prediction GRU

After graph convolution on each observed frame, the model aggregates temporal representations and uses a prediction GRU to anticipate future crossing behavior. The evaluation examines datasets, baselines, ablations, feature choices, and prediction horizons.

  • D. Prediction GRU: Two graph-convolution layers refine pedestrian and context features on every observed frame.The refined features are concatenated at each frame before temporal aggregation.
  • D. Prediction GRU: An additional GRU aggregates frame-level representations, and its last hidden state feeds a designated prediction GRU.The prediction GRU produces the future crossing-behavior forecast.
  • Evaluation: The experiments compare the method with broad baselines and evaluate structure ablations, feature choices, and expanded temporal horizons.The evaluation uses two datasets and tests how far into the future the model can predict.
  • JAAD Dataset: JAAD contains 346 videos and 82,032 frames captured under varied urban scenes, weather, and lighting conditions.Its labels include pedestrian locations, nine actions, and crossing behavior.
  • STIP Dataset: STIP contains 923.48 minutes and 1,108,176 frames from dense urban areas, with over 350,000 manually annotated pedestrian boxes.The dataset spans eight cities, three simultaneous camera views, and varied weather conditions.

2) Stanford-TRI Intent Prediction (STIP) Dataset:

The STIP dataset includes multi-view driving scenes with pedestrian annotations, including cross/not-cross labels. Its test material includes 23.43 minutes of video.

  • STIP provides left, front, and right camera views of sample scenes, with pedestrians annotated by bounding boxes and cross/not-cross labels.
  • The dataset includes 23.43 minutes of videos for testing.

B. Baseline Models

The study compares the proposed graph model with pose, trajectory, action-recognition, and concatenation alternatives, then evaluates graph structure, temporal connectivity, and feature choices. The proposed model outperforms the listed baselines, while temporal history and contextual features contribute to performance.

  • Baseline Models: The model predicts per-frame crossing probabilities for up to K future frames from T observed frames.Performance is reported both averaged over future frames and on the Kth frame.
  • Baseline Models: The proposed model outperforms baselines based on trajectory prediction, pose, action anticipation, and early action recognition.
  • Results on JAAD Dataset: The pose-based model performs below expectations, possibly because pose estimates are low quality on the more challenging current JAAD version.
  • Ablation Study: Removing temporal connections among pedestrian nodes hurts performance, indicating that pedestrian nodes benefit from maintaining temporal history.
  • Ablation Study: The ablations compare graph depth, weight sharing, graph connectivity, temporal relations, and concatenation against a 30-frame prediction setting.
  • Ablation Study: Adding contextual features is useful, while relative-position changes may provide object-type information without semantic labels.

2) Extending the Temporal Horizon:

The temporal-horizon analysis evaluates predictions up to 1, 2, and 3 seconds ahead. More observations improve performance, but accuracy and confidence decline farther into the predicted future, and confidence does not always track accuracy.

  • Extending the Temporal Horizon: The experiments extend prediction from 30 frames to 60 and 90 frames, corresponding to 1, 2, and 3 seconds into the future.
  • Extending the Temporal Horizon: Accuracy improves as more observations arrive, but both accuracy and confidence decrease as the prediction horizon grows.
  • Extending the Temporal Horizon: The 30-frame model performs best over the first 30 future frames despite being the least confident overall.
  • Extending the Temporal Horizon: The 60- and 90-frame models infer intent more confidently after receiving more data and supervision.

3) Results of Location-Centric Prediction:

The location-centric experiments use a lightweight version of the framework on a restructured dataset and examine prediction across increasing horizons. The graph model is initialized from a concatenation model when longer-horizon training creates memory difficulties.

  • Results of Location-Centric Prediction: The restructured dataset contains 32 video clips and 72,882 frames, with 14,808 frames containing crossing behavior.
  • Results of Location-Centric Prediction: Prediction horizons are evaluated over 30, 60, and 90 frames, equivalent to 1, 2, and 3 seconds.
  • Results of Location-Centric Prediction: The location-centric setting uses the same framework as the pedestrian-centric setting as a lightweight learning scheme.
  • Results of Location-Centric Prediction: The experiments compare graph modeling with a concatenation baseline and initialize the graph model from pretrained concatenation features.

D. Results on STIP Dataset

Experiments on STIP evaluate graph-based intent prediction across observation and prediction horizons, including front-camera and three-camera settings. Longer-horizon confidence declines over time, while three-camera context improves performance across most settings.

  • Prediction-length analysis: The baseline concat model encountered memory issues with longer time horizons, so the graph model was initialized from its shorter-horizon weights and fine-tuned for longer predictions.
  • Prediction-length analysis: The STIP evaluation compares 2- or 4-second observations with predictions extending 1, 2, or 3 seconds into the future.The front-camera graph and the combined left, front, and right camera graph are evaluated separately.
  • Prediction-length analysis: Prediction confidence decreases monotonically at each future step, although predicting farther into the future achieved slightly better accuracy than predicting for 2 seconds.
  • Multi-camera reasoning: With all three cameras combined into one graph, 2 seconds of observation predicts no worse than 4 seconds of observation.Side-camera object locations are shifted horizontally according to their views before graph construction.
  • Overall result: The conclusion reports accuracy above 80% on STIP and slightly below 80% on JAAD about one second before actual crossing.The study also evaluates pedestrian-centric and location-centric graph settings.
Loading 2002.08945v1…