Source-linked AI summary
Injecting Knowledge in Data-driven Vehicle Trajectory Predictors
Mohammadhossein Bahari, Ismail Nejjar, Alexandre Alahi
TL;DR
Knowledge-driven trajectory predictors can be feasible but limited, while data-driven predictors model interactions yet may produce unrealistic predictions and generalize poorly. RRB adds learned, physically constrained residuals to a knowledge-driven trajectory and uses MPC for kinematic constraints. Experiments report improved accuracy and generalization to new scenes relative to prior works.
Problem
Knowledge-driven predictors have limited performance on complex interactions, while data-driven predictors can be unrealistic and generalize poorly to unseen data.
Source: §1 · p. 1Method
Results
Experiments report that RRB combines the generalization strengths of knowledge-driven models with data-driven accuracy and outperforms prior works in accuracy and generalization to new scenes.
Source: §4.4 · p. 9, §5 · p. 10Takeaways & Limitations
The proposed approach demonstrates generalization to new scenes and the multimodal case while combining knowledge-driven and data-driven trajectory prediction.
Source: §1 · p. 2The IVW formulation assumes temporally independent trajectory points, although the knowledge-driven prediction and residual are not independent.
Source: §3.3.2 · pp. 5–6
Abstract
from arXiv · showhide
Vehicle trajectory prediction tasks have been commonly tackled from two distinct perspectives: either with knowledge-driven methods or more recently with data-driven ones. On the one hand, we can explicitly implement domain-knowledge or physical priors such as anticipating that vehicles will follow the middle of the roads. While this perspective leads to feasible outputs, it has limited performance due to the difficulty to hand-craft complex interactions in urban environments. On the other hand, recent works use data-driven approaches which can learn complex interactions from the data leading to superior performance. However, generalization, \textit{i.e.}, having accurate predictions on unseen data, is an issue leading to unrealistic outputs. In this paper, we propose to learn a "Realistic Residual Block" (RRB), which effectively connects these two perspectives. Our RRB takes any off-the-shelf knowledge-driven model and finds the required residuals to add to the knowledge-aware trajectory. Our proposed method outputs realistic predictions by confining the residual range and taking into account its uncertainty. We also constrain our output with Model Predictive Control (MPC) to satisfy kinematic constraints. Using a publicly available dataset, we show that our method outperforms previous works in terms of accuracy and generalization to new scenes. We will release our code and data split here: https://github.com/vita-epfl/RRB.
1 Introduction
Knowledge-driven predictors produce feasible but limited predictions, while data-driven methods model interactions but can be unrealistic and generalize poorly. The paper proposes RRB to combine both, with physically constrained residuals and MPC for kinematic feasibility.
1 Introduction
- Knowledge-driven methods yield feasible predictions but miss non-average behavior and complex interactions, whereas data-driven methods can be unrealistic and overfit or perform poorly out of distribution.The paper argues that combining domain knowledge with data can retain the strengths of both approaches.Source: §1 · p. 1
- The paper frames driving as combining physical domain knowledge, such as road constraints and collision avoidance, with experience of social conventions between drivers.Source: §1 · pp. 1–2
- A combined model must remain differentiable for training while preserving realistic outputs despite potentially unrealistic neural-network predictions.Source: §1 · p. 2
- RRB adds trainable residuals to a knowledge-driven trajectory, using the base prediction for coarse behavior and residuals for social interactions and long-tail behavior.Residuals are physically constrained and combined with the knowledge-driven trajectory using inverse-variance weighting; MPC then enforces kinematic feasibility.Source: §1 · p. 2
- The stated contributions include using MPC for kinematic constraints and demonstrating generalization to new scenes and the multimodal case.Source: §1 · p. 2
2 Related Work
Prior work injects knowledge through constraints, specialized architectures, learning procedures, training data, post-processing, or residual corrections. This paper highlights limits in preserving physical feasibility with learned residuals and in applying kinematic layers to coordinate predictions.
2 Related Work
- The related-work overview contrasts knowledge-driven trajectory methods with data-driven interaction and multimodal-prediction methods, while noting that these approaches rely on either knowledge or data exclusively.Source: §2 · p. 3
- Prior approaches inject knowledge through feasible-output constraints, specialized network structures, learning algorithms, training data, or post-processing; other work uses neural networks to correct knowledge-model errors.Source: §2 · p. 3
- Neural residuals can make physics-based constraints hard to preserve; this work confines residual values and uses variance-based integration instead of naive addition.Source: §2 · p. 3
- Road-penalty methods can require expensive occupancy heatmaps or reduce ℓ2 performance, while direct nonlinear constraint optimization can lead to sub-optimal solutions.Source: §2 · p. 3
- Kinematic layers convert predicted control commands into feasible trajectories, but can be incompatible with coordinate-output models and limit later knowledge injection.The paper instead uses MPC with coordinate outputs and latent control commands.Source: §2 · p. 3
3 Proposed Method
The method combines a knowledge-driven trajectory with learned residuals, then uses uncertainty-aware weighting and Model Predictive Control to produce constrained predictions. It also supports multimodal prediction through a closest-mode loss.
3 Proposed Method
- The method combines a knowledge-driven trajectory with conditioned data-driven residuals, merges them, and applies Model Predictive Control to satisfy kinematic constraints.The residuals address fine-grained behaviors absent from the knowledge-driven trajectory.Source: §3 · p. 4
3.1 Problem Formulation
- The model predicts future vehicle positions from scene imagery and ego-vehicle and surrounding-vehicle histories, representing each future position sequence as a Gaussian distribution.The ego-vehicle is the one whose future is being predicted.Source: §3.1 · p. 4
3.2 Knowledge-aware Prediction
- The knowledge-aware trajectory uses road lanes to represent scene-compliant average behavior and can be generated by any off-the-shelf knowledge-driven predictor.Its Gaussian mean is the KD trajectory, with fixed variances approximated from training-data statistics.Source: §3.2 · p. 4
3.3 Realistic Residual Block
- The RRB confines learned residuals before merging them with the KD prediction through an IVW-addition block.The figure describes three encoders for history, interactions, and KD trajectory, followed by a residual decoder that estimates a residual distribution.Source: §3.3 · pp. 4–5, Figure 3 · p. 5
3.3.1 Residual Estimator
- The residual estimator encodes agent history, nearby interactions, and the KD trajectory, then predicts Gaussian residual parameters with a mean scaled by C.It excludes agents behind the vehicle and retains only the closest vehicles; the mean is bounded using tanh, with C chosen as half the minimum road width in each scene.Source: §3.3.1 · p. 5
3.3.2 IVW-addition
- IVW-addition weights the KD and residual-adjusted trajectories to minimize the merged trajectory’s variance, accounting for dependence because the KD trajectory is a prior for the residual.The weighting solution is obtained using Lagrangian multipliers.Source: §3.3.2 · pp. 5–6
3.4 Multimodal Prediction and Loss Function
- For multimodal prediction, the RRB adjusts multiple KD trajectories, and Winner-Takes-All loss evaluates the mode closest to ground truth.The loss also applies to the single-mode case with M = 1.Source: §3.4 · p. 6
3.5 Model Predictive Control
- MPC predicts positions under bicycle-model dynamics, initialized from the current state and bounded by feasible control limits.The state comprises position, orientation, and speed; controls are acceleration and steering angle.Source: §3.5 · p. 6
4 Experiments
Across scene-generalization and scene-overfitting evaluations, RRB combines scene-aware predictions with learned interaction residuals, while the ablations examine transfer across knowledge-driven models and the components supporting realistic outputs.
4 Experiments
- The experiments test RRB against knowledge-driven, data-driven, and mixed baselines, including variants with two modes and MPC.The stated hypotheses include whether RRB improves performance and generalization, whether MPC satisfies kinematic constraints, and whether residual confinement and IVW-addition support realistic outputs.Source: §4 · p. 6, §4.3 · pp. 7–8
4.1 Dataset
- With three scenes held out for testing, the Interaction dataset evaluation uses 5 observed frames (2.5 s), predicts 10 frames (5 s), and averages performance across three scene categories.The held-out scenes are DR_USA_Intersection_MA, DR_USA_Roundabout_SR, and DR_CHN_Merging_ZS.Source: §4.1 · p. 7
4.2 Implementation Details
- Models were trained for 50 epochs with batch size 32 using Adam, an initial learning rate of 0.001 halved every 10 epochs, and MLP building blocks.The paper reports hidden-layer sizes separately for the history and interaction encoders, KD encoder, and decoder.Source: §4.2 · p. 7
4.3 Metrics and Baselines
- ADE/FDE measure displacement, RV measures the percentage of predicted points in off-road areas, and CT measures spatial endpoint error after retiming to ground-truth speed.For multimodal predictions, ADE/FDE use the mode closest to ground truth, while RV averages modes weighted by probability.Source: §4.3 · p. 7
4.4 Results
- In scene-generalization, knowledge-driven models outperform others on RV and CT, data-driven models have better ADE/FDE, and RRB outperforms knowledge-driven and mixed baselines with zero RV.The paper also reports RRB_M outperforming prior works, especially S-WaGDAT_M, in scene-overfitting; Figure 4 qualitatively presents RRB as realistic and interaction-aware.Source: §4.4 · pp. 8–9, Table 1 · p. 8, Table 2 · p. 8
4.5 Ablation Study
- Without retraining, RRB improves Lin and KD2 across all metrics; non-confined residuals improve ADE/FDE but have larger RV, while simple addition increases RV and CT.The authors attribute the latter increase to using uncertain residuals without uncertainty weighting.Source: §4.5 · p. 10, Table 3 · p. 10
5 Conclusions and Future Work
The authors conclude that RRB combines knowledge-driven priors with learned residuals and outperforms other approaches in accuracy and generalizability. They propose extending it to pedestrians and mixed-agent settings.
5 Conclusions and Future Work
- RRB outperforms the other counterparts in both accuracy and generalizability.The experiments report that knowledge-driven models generalize to new scenes but have inferior performance, while data-driven models are more accurate but tend to make unrealistic predictions in new scenes.Source: §5 · p. 10
- RRB adds learned residuals to knowledge-driven predictions, using physically constrained inverse-variance weighting and MPC to support realistic outputs.The residuals model human stochastic behavior, and MPC adds kinematic constraints to the final prediction.Source: §5 · p. 10
- The authors propose testing RRB with more complex knowledge-driven models and extending it to pedestrians and mixed vehicle, pedestrian, and cyclist environments.They identify pedestrian social rules and agent-specific constraints and dynamics as relevant considerations.Source: §5 · p. 10