Human Trajectory Prediction via Counterfactual Analysis

Guangyi Chen; Junlong Li; Jiwen Lu; Jie Zhou

Source-linked AI summary

Human Trajectory Prediction via Counterfactual Analysis

Guangyi Chen, Junlong Li, Jiwen Lu, Jie Zhou

arXiv:2107.14202v1Jul 29, 2021cs.CV

TL;DR

Human trajectory prediction uses past motion and environmental interactions, but bias between training and deployment environments is often overlooked. The paper applies causal graphs and counterfactual interventions to trajectory clues, then reports consistent improvements across baseline types and state-of-the-art results on public benchmarks.

Problem
Trajectory prediction methods use history and environment-interaction clues, but may overlook bias between training and deployment environments.
Source: §1 · pp. 1–2
Method
The method builds a causal graph and intervenes on history-trajectory features, subtracting counterfactual predictions from factual predictions.
Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4
Results
The method consistently improves RNN- and CNN-based predictors and achieves state-of-the-art results on public pedestrian trajectory datasets.
Source: §1 · p. 2
Takeaways & Limitations
The plug-and-play module is intended to reduce environment-bias effects and emphasize trajectory clues across baseline prediction methods.
Source: §1 · p. 2, §5 · p. 9
The method runs more slowly than baselines because counterfactual analysis adds computation, though the authors say the extra cost is not heavy.
Source: §4.2 · p. 8

Abstract

from arXiv · show

Forecasting human trajectories in complex dynamic environments plays a critical role in autonomous vehicles and intelligent robots. Most existing methods learn to predict future trajectories by behavior clues from history trajectories and interaction clues from environments. However, the inherent bias between training and deployment environments is ignored. Hence, we propose a counterfactual analysis method for human trajectory prediction to investigate the causality between the predicted trajectories and input clues and alleviate the negative effects brought by environment bias. We first build a causal graph for trajectory forecasting with history trajectory, future trajectory, and the environment interactions. Then, we cut off the inference from environment to trajectory by constructing the counterfactual intervention on the trajectory itself. Finally, we compare the factual and counterfactual trajectory clues to alleviate the effects of environment bias and highlight the trajectory clues. Our counterfactual analysis is a plug-and-play module that can be applied to any baseline prediction methods including RNN- and CNN-based ones. We show that our method achieves consistent improvement for different baselines and obtains the state-of-the-art results on public pedestrian trajectory forecasting benchmarks.

1. Introduction

Counterfactual analysis addresses train–deployment environment bias in trajectory prediction by contrasting factual and counterfactual predictions. The method is plug-and-play and is reported to improve baseline predictors consistently and achieve state-of-the-art results on public datasets.

Trajectory prediction forecasts pedestrians’ future paths from past positions and supports autonomous-vehicle planning, robot navigation, and crowd surveillance.Social and physical interactions make prediction challenging, including interactions with other pedestrians and scene context.Source: §1 · pp. 1–2
Training–deployment environment bias can make interaction modules overfit and misattribute trajectory choices to scene context, while unpredictable deployment makes transfer learning difficult.The introduction illustrates the risk with left turns wrongly attributed to crossroads.Source: §1 · p. 2
Counterfactual intervention replaces history-trajectory features, compares factual and counterfactual predictions, and subtracts the latter to reduce biased-environment effects.Suggested counterfactual trajectories include uniform rectilinear motion, mean trajectories, or random trajectories.Source: §1 · p. 2
The method consistently improves RNN- and CNN-based predictors and achieves state-of-the-art results on public pedestrian trajectory datasets.It is presented as a plug-and-play module for baseline prediction methods.Source: §1 · p. 2

2. Related Work

Related work models social and physical interactions, uses sequence and generative predictors, and applies causal inference in other fields. The paper targets overlooked environment bias and opaque clue-to-prediction relationships in trajectory forecasting.

Trajectory methods model social interactions with pooling, attention, and graph models, and may incorporate physical scene context, but generally overlook train–deployment environment bias.The paper links this overlooked bias to overfitting in environment-interaction modules.Source: §2 · p. 2
RNN and temporal-CNN predictors combine motion and interaction clues, yet their likelihood-based predictions leave causal relations between observed clues and outputs nontransparent.The paper proposes intervention to examine the effects of history and environment clues.Source: §2 · p. 3
Generative approaches model behavioral indeterminacy with GANs, VAEs, bivariate Gaussian predictions, or Transformers.Source: §2 · p. 3
Causal-inference research spans other fields; this paper applies counterfactual analysis to trajectory prediction to examine clue effects and reduce biased-data effects.Its training process replaces past-trajectory features with counterfactual alternatives and subtracts the counterfactual prediction from the original.Source: §2 · p. 3, Figure 2 · p. 3

3. Approach

The approach frames prediction around history trajectories and environment interactions, then uses a causal graph and counterfactual interventions to isolate trajectory contributions. It extends this analysis to generative prediction and implements it in recurrent and convolutional baselines.

3.1. Problem Definition

The prediction task takes each pedestrian’s history trajectory and environment interactions as inputs and predicts its future trajectory, with model parameters optimized using an L2 loss.The environment interactions are extracted from other pedestrians’ trajectories and physical scene context.Source: §3.1 · pp. 3–4

3.2. Causal Graph

The causal graph represents history trajectory, environment interactions, and future trajectory, with environment interactions treated as a confounder of history and future.The graph encodes that the environment influences both observed trajectories and predictions, so a prediction may wrongly attribute a left turn to a crossroads.Source: §3.2 · p. 4

3.3. Counterfactual Analysis

Counterfactual analysis replaces history-trajectory features with imagined values, compares the resulting prediction with the factual prediction, and trains the network on that difference.Candidate interventions include a zero vector, the mean of history trajectories, or random trajectories; the method describes the resulting causal prediction as more reliable by avoiding environment-confounder bias.Source: §3.3 · p. 4

3.4. Generative Model

For generative predictors, the method adds a noise latent variable to produce stochastic future trajectories and applies the causal-prediction approach to the generative model.The paper says the analysis remains applicable across predictor changes and whether generative modeling is used.Source: §3.4 · pp. 4–5

3.5. Implementation Details

Counterfactual analysis is added to STGAT and Social-STGCNN by comparing predictions from factual and counterfactual trajectory clues.The implementations replace the history feature vector in STGAT or the graph nodes in Social-STGCNN while maintaining interaction clues.Source: §3.5 · p. 5

4. Experiments

Experiments on ETH and UCY evaluate counterfactual analysis with RNN- and CNN-based predictors, comparing benchmark performance, intervention choices, deployment cost, and qualitative trajectories.

4.1. Datasets and Experimental Settings

Across five ETH and UCY scenes, evaluation observes 8 frames (3.2 seconds) and predicts 12 frames (4.8 seconds), using ADE and FDE with the closest of 20 generated samples.Training and validation use four scene sets, with the remaining set for testing.Source: §4.1 · p. 5

4.2. Quantitative Evaluation

Causal methods have the best average error among the compared RNN- and CNN-based models; Causal-STGAT reports +0.07/+0.16 ADE/FDE improvement over STGAT, and Causal-STGCNN over +0.02/+0.09 over reproduced Social-STGCNN.The paper also reports slight improvement over IDL and Social-STGCNN.Source: Table 1 · p. 6, §4.2 · p. 6
Zero, mean, and random-vector counterfactual interventions all improve baseline performance; their results are close, while zero is slightly better.For random intervention, training uses a uniform draw from [−0.1, 0.1] and testing uses its expectation, the zero vector.Source: §4.2 · pp. 7–8
Counterfactual analysis adds no parameters but slows inference because it requires extra computation.The authors say the speed cost is not heavy because the analysis uses only part of the network.Source: §4.2 · p. 8

4.3. Qualitative Evaluation

In the gathering scenario, conventional predictors may forecast pedestrians meeting due to training-environment bias, while counterfactual analysis removes that negative effect and emphasizes their tendency to move together.The authors also report more reliable trajectories for both CNN- and RNN-based methods in a second ETH scene.Source: §4.3 · pp. 8–9

5. Conclusion

The method compares factual predictions with predictions under counterfactual trajectory features to investigate clue–trajectory causality. The authors report that this comparison encourages learning trajectory-related causes while alleviating training–deployment bias.

5. Conclusion

Replacing trajectory-clue features with counterfactual features and subtracting the resulting prediction from the original defines the method’s comparison.The comparison is used to investigate causalities between observed clues and predicted trajectories.Source: §5 · p. 9
The authors use the factual–counterfactual prediction difference to encourage learning trajectory-related causes and alleviate training–deployment environment bias.Source: §5 · p. 9
The plug-and-play module is presented for off-the-shelf trajectory predictors, with experiments covering scenes, counterfactual implementations, and baseline generalization.Source: §5 · p. 9

A. Physical Environment

The authors apply their causal model to a Social-STGCNN baseline with visual scene features to evaluate physical-environment prediction. They report very significant improvement when addressing physical and social environment bias.

A. Physical Environment

The physical-environment evaluation adds a visual-feature branch to Social-STGCNN, extracting scene features with ResNet34 and joining them with node features and position embeddings after an MLP.Source: A. Physical Environment · p. 11
The causal model is applied to mitigate bias in both physical and social environment information.Source: A. Physical Environment · p. 11
The authors report very significant improvement from applying the causal model to physical and social environment features.Table 4 is identified as the physical-environment evaluation.Source: A. Physical Environment · p. 11, Table 4 · p. 11

B. Experiment on SDD

On SDD, the authors apply their causal model to PECNet and report improvement over the PECNet state-of-the-art baseline. The experiments use the standard SDD train–test split.

B. Experiment on SDD

The SDD benchmark contains 20 scenes and over 11,000 unique pedestrians, and the experiments use its standard train–test split.Source: B. Experiment on SDD · p. 11
For Causal-PECNet, the past-trajectory feature is replaced with a zero-vector intervention, and the difference between factual and counterfactual outputs forms the causal pooled feature.Source: B. Experiment on SDD · p. 11
Causal-PECNet improves over the SOTA PECNet baseline on SDD.Table 5 compares several state-of-the-art models on SDD; the passage also distinguishes reported PECNet performance from reproduced PECNet results.Source: B. Experiment on SDD · p. 11, Table 5 · p. 11