Source-linked AI summary
Peeking into the Future: Predicting Future Person Activities and Locations in Videos
Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander Hauptmann, Li Fei-Fei
TL;DR
The paper addresses future pedestrian path prediction while accounting for activities and intentions that may influence movement. It introduces Next, a multi-task model using rich visual features and auxiliary activity-location prediction, and reports strong trajectory results alongside meaningful activity predictions. The authors also identify scope and fairness boundaries for deployment.
Problem
Future pedestrian path prediction is important for video understanding and applications, but existing work mostly neglects the intentions and activities guiding navigation.
Method
Next jointly predicts future paths and activities using rich behavioral and interaction features, with auxiliary prediction of the activity location through a discretized Manhattan Grid.
Results
The model achieves the best-published performance on two public benchmarks and produces meaningful future activity predictions, while joint learning improves future path prediction.
Takeaways & Limitations
Joint modeling of paths and activities can benefit future path prediction while providing an additional prediction of what activity a person will perform.
Takeaways & Limitations
The method has not been tested across different populations and is restricted to predefined activities, so further evaluation is needed before real-world use.
Abstract
from arXiv · showhide
Deciphering human behaviors to predict their future paths/trajectories and what they would do from videos is important in many applications. Motivated by this idea, this paper studies predicting a pedestrian's future path jointly with future activities. We propose an end-to-end, multi-task learning system utilizing rich visual features about human behavioral information and interaction with their surroundings. To facilitate the training, the network is learned with an auxiliary task of predicting future location in which the activity will happen. Experimental results demonstrate our state-of-the-art performance over two public benchmarks on future trajectory prediction. Moreover, our method is able to produce meaningful future activity prediction in addition to the path. The result provides the first empirical evidence that joint modeling of paths and activities benefits future path prediction.
1. Introduction
The paper frames future pedestrian path prediction as important for video understanding and applications, while arguing that activities and intentions can shape the path. It proposes jointly predicting paths and activities with rich visual cues and an auxiliary activity-location task.
- Motivation: Future pedestrian trajectory prediction uses past video information to forecast movement for applications including self-driving cars and socially aware robots.The paper describes trajectory prediction as an essential building block in video understanding.
- Motivation: Existing work mostly neglects the specific purposes or intentions that guide how people navigate public spaces.The paper contrasts simple goals such as entering a room with more complicated activities such as putting things into a car.
- Approach: The paper models intention through a predefined set of 29 NIST activities, including “loading” and “object transfer”.These activities correspond to different possible future paths in the motivating example.
- Approach: Next jointly predicts future paths and activities using rich features of appearance, body movement, behavior, and interactions with surroundings.The model is designed to capture visual cues that people use when anticipating others’ movements.
- Approach: An auxiliary task predicts the future activity location using a discretized Manhattan Grid to facilitate training.The paper reports that this auxiliary task improves future path prediction accuracy.
- Results: Experiments on ETH & UCY and ActEV/VIRAT report state-of-the-art trajectory results and meaningful future activity predictions.The authors describe the work as the first empirical demonstration that joint modeling can considerably improve future path prediction.
2. Related Work
Prior trajectory work models social interactions, physical scenes, or limited visual cues, while this paper combines rich person semantics and surroundings for joint path and activity prediction.
- Person trajectory prediction: Earlier trajectory models primarily represent people as points while modeling social interactions or scene effects.Related approaches include social pooling, adversarial training, inverse reinforcement learning, and scene-based spatial modeling.
- Person visual features: Visual-feature-based trajectory methods use faces or keypoints, whereas this work considers richer visual semantics for future prediction.The cited prior methods focus on awareness from faces or keypoint features in specific video settings.
- Activity prediction: Activity anticipation has used recurrent models for early action recognition, but this work jointly predicts activity and trajectory using behavior and interaction modeling.The paper distinguishes its setting by combining both prediction targets and both types of person-centered modeling.
- Proposed approach: The Next architecture encodes appearance, body movement, person-scene relations, and person-object relations before generating trajectories and activities.Separate behavior and interaction modules provide the visual representations used by the prediction system.
- Proposed approach: Focal attention summarizes correlations among encoded visual features for trajectory decoding, while grid classification and regression predict activity locations.The activity-location auxiliary task uses multi-scale Manhattan Grid outputs for destination probabilities and coordinate deviations.
- Proposed approach: The network is trained end-to-end with a multi-task objective combining trajectory loss with activity-location classification and regression losses.The primary trajectory term is an L2 loss summed over people and prediction times.
4. Experiments
Experiments evaluate Next on ActEV/VIRAT and ETH/UCY for future trajectory prediction, including quantitative comparisons, qualitative analyses, and ablations. The results report favorable trajectory performance, activity predictions, and benefits from the model’s features and multi-task design.
- Experimental setup: Next is evaluated on ActEV/VIRAT and ETH/UCY using trajectory prediction benchmarks and, on ActEV/VIRAT, future activity prediction.ActEV/VIRAT uses official training and validation splits; ETH/UCY uses leave-one-scene-out evaluation across five scenes.
- Experimental setup: The ActEV/VIRAT setup observes 3.2 seconds and predicts 4.8 seconds of person trajectory, using ADE and FDE for trajectories and mAP for activities.Trajectory errors are measured in pixel space on ActEV/VIRAT and in meters on ETH and UCY.
- ActEV/VIRAT results: 10 points: Next outperforms Social-LSTM and Social-GAN on the “move FDE” metric and reports state-of-the-art future trajectory prediction on ActEV/VIRAT.The comparison includes single-model results and best results from 20 model outputs.
- ActEV/VIRAT results: Qualitative examples show correct trajectory and activity predictions, an imperfect activity prediction with a mostly correct path, and a failure to capture two-person interaction.The failure case predicts that two people will separate although they actually stop and talk.
- Ablation analysis: Ablations show that person behavior and interaction features support trajectory prediction, behavior features are more essential for activity prediction, and removing focal attention or multi-task tasks hurts performance.The multi-task ablations remove activity-label prediction, activity-location prediction, or both.
- ETH & UCY results: On ETH/UCY, Next obtains the best-published single-model result on ETH and best average performance, with especially strong results on HOTEL and ZARA2.The authors attribute these scene-specific advantages to attention selecting visual signals for small movements while remaining competitive on large movements.
5. Conclusion
The paper presents Next, a neural network that jointly predicts human trajectories and future activities using rich visual features and auxiliary activity-location prediction. Experiments show meaningful activity predictions and effective trajectory prediction on multiple benchmarks, while the activity set and evaluation scope remain bounded.
- Next jointly predicts human trajectories and future activities using rich visual features of behavior and surrounding interactions.The model also uses an auxiliary task predicting activity locations to facilitate joint training.
- The model was evaluated on popular and recent large-scale video benchmarks for person trajectory prediction.
- Next successfully predicts meaningful future activities both quantitatively and qualitatively.
- The system targets a predefined set of 30 NIST activities and may not work beyond those activities.
- The method has not been tested across different populations, leaving privacy, safety, and fairness issues for further evaluation before real-world deployment.
Appendix
The appendix provides additional experimental details and analysis for the ActEV/VIRAT and ETH & UCY benchmarks, including statistical comparisons between the datasets.
- The appendix presents further experiment details and analysis for the ActEV/VIRAT and ETH & UCY benchmarks.
- It also provides statistical comparisons of the two datasets.
5.1. ActEV/VIRAT Details
The ActEV/VIRAT details distinguish static and moving trajectories, examine prediction difficulty, and document experimental variability and feature ablations.
- Trajectory Type: ActEV/VIRAT trajectories are labeled moving when the observed activity is Walk, Run, or Ride Bike; otherwise they are labeled static.
- Trajectory Type: The mean displacement between the last observed point and predicted trajectory points differs substantially between static and moving trajectories.
- Nearest Neighbor Experiment: Nearest-neighbor future prediction shows that human trajectories remain non-trivial to predict because people navigate differently even within the same scene.
- Single Output Experiment: Twenty identical Next models with different initializations were trained, with mean and standard deviation reported in Table 3.
- Feature Ablation: Ablation experiments separately remove person-object, person-scene, person-keypoint, and person-appearance features.
5.2. Activity Detection Experiment
The activity detection experiments test whether current activity labels suffice for future prediction and examine quantitative and qualitative successes and failures on ActEV/VIRAT.
- Activity Detection Experiment: The drop from 0.192 to 0.155 mAP indicates that predicting activity 4.8 seconds into the future is not trivial.
- More Qualitative Analysis: In successful qualitative cases, trajectory and future activity predictions are both correct, including walking past a car and gesturing near its trunk.
- More Qualitative Analysis: Some cases have correct activity predictions but incorrect trajectories, such as choosing the wrong direction or car door.
- More Qualitative Analysis: Other cases contain errors in both trajectory and activity prediction, including missing a bike rider’s turn to avoid an incoming car.
- Benchmark Comparison: ActEV/VIRAT is larger than ETH & UCY and includes bounding-box and activity annotations for multi-task learning.
5.3. ETH & UCY Details
The ETH and UCY evaluation uses a dataset configuration affected by unavailable videos, with fixed-size person boxes derived from point annotations. Results for four other splits remain comparable to prior methods.
- Dataset differences: Unavailable videos make the UNIV test set smaller and the training set about 34% smaller than in previous methods.The unavailable videos are two from UNIV and one from ZARA3.
- Dataset differences: Test sets for the other four splits are unchanged, so their reported numbers are comparable.
- Preprocessing: The method expands each annotated person point into a fixed 50-by-80-pixel bounding box for appearance and person-scene feature pooling.The point is placed at the center of the bounding box’s bottom line, and videos are resized to 720x576.