Source-linked AI summary
Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty
Apratim Bhattacharyya, Mario Fritz, Bernt Schiele
TL;DR
Reliable inner-city driving requires anticipating people in dynamic scenes from a moving vehicle for at least one second. The paper introduces a two-stream Bayesian sequence model that jointly predicts pedestrian trajectories and vehicle odometry with uncertainty estimates, and reports that odometry prediction, sequence modeling, and uncertainty modeling support long-horizon prediction.
Problem
Inner-city traffic requires predicting dynamic agents over at least one second from an on-board camera, where future motion is non-deterministic and timely anticipation matters for safe driving.
Method
A two-stream Bayesian LSTM encoder-decoder predicts pedestrian bounding-box sequences conditioned on predicted vehicle odometry while modeling aleatoric and epistemic uncertainty.
Results
The experiments show that one-second people-trajectory prediction is possible, uncertainty correlates with squared prediction error, and jointly predicting odometry significantly improves performance.
Takeaways & Limitations
Long-term sequence modeling and predicted odometry are essential components for accurate on-board people prediction under uncertainty.
Takeaways & Limitations
The social-pooling version of the comparison model did not converge on the authors’ dataset.
Abstract
from arXiv · showhide
Progress towards advanced systems for assisted and autonomous driving is leveraging recent advances in recognition and segmentation methods. Yet, we are still facing challenges in bringing reliable driving to inner cities, as those are composed of highly dynamic scenes observed from a moving platform at considerable speeds. Anticipation becomes a key element in order to react timely and prevent accidents. In this paper we argue that it is necessary to predict at least 1 second and we thus propose a new model that jointly predicts ego motion and people trajectories over such large time horizons. We pay particular attention to modeling the uncertainty of our estimates arising from the non-deterministic nature of natural traffic scenes. Our experimental results show that it is indeed possible to predict people trajectories at the desired time horizons and that our uncertainty estimates are informative of the prediction error. We also show that both sequence modeling of trajectories as well as our novel method of long term odometry prediction are essential for best performance.
1. Introduction
Reliable inner-city driving requires anticipating dynamic agents over at least one second, because that horizon is comparable to braking distance at conservative residential speeds. The paper proposes an on-board, uncertainty-aware model for long-term pedestrian trajectory prediction that jointly accounts for people and vehicle motion.
- Motivation: At 25 miles/hour (∼40 km/hour), one second of travel roughly equals the braking distance, motivating at least one-second traffic anticipation.Anticipation is also described as supporting collision prevention, practical safety distances, and decision making around pedestrians and cars.
- Contribution: The paper proposes the first approach to predict pedestrian and cyclist trajectories from on-board cameras over long horizons with uncertainty estimates.The predictions are made relative to a moving vehicle, rather than a stationary viewpoint.
- Contribution: Its two-stream formulation jointly predicts long-term person bounding boxes and vehicle ego motion while modeling aleatoric and epistemic uncertainty.The contributions include a sequence-to-sequence model and evaluation of alternative architectures, including the importance of the two-stream design.
2. Related work
Prior trajectory-prediction work commonly relies on 3D coordinates or models social interactions among pedestrians. This paper instead emphasizes single-camera on-board prediction, where vehicle ego-motion dominates social aspects.
- Human Trajectory Prediction: 3D pedestrian trajectory methods require stereo cameras and/or LIDAR, while the proposed method uses one camera and avoids unreliable depth coordinates.The paper notes that noisy depth is especially problematic in unconstrained or unknown environments.
- Human Trajectory Prediction: Social trajectory models estimate pedestrian interactions and joint trajectories, but on-board prediction is characterized as being dominated by vehicle ego-motion.The related work contrasts social-context modeling with the moving-vehicle setting addressed here.
- Uncertainty Modeling: Bayesian recurrent models provide precedent for modeling uncertainty, including frameworks that jointly estimate aleatoric and epistemic uncertainty.The paper situates its uncertainty modeling among Bayesian RNN and Bayesian deep-learning approaches.
3. On-board Pedestrian Prediction under Uncertainty
The model predicts future pedestrian bounding-box sequences from on-board cameras by conditioning a Bayesian sequence model on past pedestrian motion and vehicle odometry, including predicted future odometry. It estimates uncertainty by combining Bayesian model uncertainty with observation noise while using a dedicated odometry stream for long-term ego-motion prediction.
- Model overview: The two-stream model predicts pedestrian bounding boxes and vehicle odometry jointly from on-board visual observations and trajectory history.The odometry stream predicts future ego motion, while the bounding-box stream uses pedestrian and odometry sequences for prediction.
- Model overview: Pedestrian predictions are conditioned on past bounding boxes, past odometry, and predicted future odometry.Vehicle odometry is represented by speed and steering angle, and future odometry is predicted from past odometry and on-board visual observations.
- Bounding-box prediction: The decoder predicts future bounding boxes from an encoded summary of past pedestrian motion and odometry concatenated with future odometry.The sequence-to-sequence architecture contains embedding layers, an encoder RNN, and a decoder RNN; predicted Gaussian parameters are sampled for inference.
- Bayesian uncertainty: The Bayesian RNN encoder-decoder represents epistemic uncertainty by learning a posterior distribution over RNN models and marginalizing over their parameters.The model uses weight matrices as variational variables and approximates posterior integration with a Bernoulli variational distribution and Monte Carlo sampling.
- Bayesian uncertainty: Aleatoric uncertainty is modeled with a Gaussian observation distribution at each prediction time step, while predictive variance measures total uncertainty.The predictive distribution is Gaussian at every time step, and uncertainty is decomposed into epistemic and aleatoric components before averaging across time.
- Odometry prediction: The odometry stream predicts a mean future ego-motion sequence using past odometry and visual features, with past odometry supporting short-term prediction and images supporting longer-term cues.The stream combines odometry and visual summary vectors in an LSTM decoder to predict future speed and steering-angle sequences.
4. Experiments
Experiments on Cityscapes evaluate one-second-ahead pedestrian and odometry prediction, comparing Bayesian, sequence-modeling, Kalman, and two-stream approaches. Results show that vehicle odometry, trajectory sequence modeling, and uncertainty estimates are important for prediction quality and decision support.
- Dataset and evaluation: Cityscapes contains 2,975 training, 500 validation, and 1,525 test sequences recorded from vehicles in inner-city scenes.Each sequence lasts 1.8 seconds and includes associated odometry information.
- Dataset and evaluation: The evaluation predicts 15 future time-steps using pixel-space MSE and negative log-likelihood, with likelihood measuring probability assigned to the true sequence.Bayesian models use dropout rate 0.35, λ = 10^-4, and 50 Monte-Carlo samples.
- Bounding-box prediction: The homoscedastic LSTM outperforms the linear Kalman filter, while the aleatoric LSTM improves negative log-likelihood over the basic LSTM.These comparisons indicate that bounding-box sequences can exhibit complex motion and that modeling aleatoric uncertainty improves probabilistic predictions.
- Bounding-box prediction: Longer conditioning improves performance until |Bp| = 8, while the odometry oracle substantially outperforms the Bayesian LSTM, showing the importance of vehicle-motion information.The Bayesian LSTM also outperforms the compared vanilla LSTM on recursive bounding-box center prediction.
- Two-stream model: Jointly predicting odometry and pedestrian bounding boxes significantly improves performance and recovers a substantial fraction of the odometry-oracle performance.Higher error remains when odometry is uncertain, such as at T-intersections, and detector noise or tracker failures contribute residual error.
- Uncertainty analysis: Predicted uncertainty correlates with squared error and upper-bounds the mean prediction error, supporting its use for expressing trust in predictions and informing decisions.Sequences with multiple likely futures receive higher predictive variance when the mean estimate has higher error.
5. Conclusion
The paper presents one-second pedestrian prediction from on-board observations, enriched with uncertainty estimates and supported by Bayesian modeling and long-term odometry prediction.
- The model predicts pedestrians over a 1-second horizon from on-board observations with theoretically grounded uncertainty estimates.
- A Bayesian approach and long-term odometry prediction are identified as key to the model’s success.
- The study compares multiple architecture choices and selects a novel two-stream Bayesian LSTM encoder-decoder.
A. Additional Details of Training Objective
The supplementary training details formulate the model in Bayesian terms, optimize it using Monte-Carlo integration, and specify separate bounding-box and odometry streams.
- Training objective: The training inputs are past bounding boxes, past odometry, and future vehicle odometry, while the target is the future bounding-box sequence.
- Training objective: The recurrent model predicts each future bounding box conditioned on the preceding prediction, the past bounding-box sequence, and vehicle odometry.
- Training objective: Monte-Carlo integration with N samples estimates the training objective’s integral.
- Training objective: Replacing the log-probability term with an exponential squared-error term and adding regularization yields the reported training objective.
- Two-stream model: The bounding-box stream samples all layer weights from the approximate posterior q(ω), while the odometry stream is specified separately.
C. Database Statistics
The Cityscapes database contains pedestrian tracks ranging from 6 to 30 frames, with similar length distributions in training and test data and many tracks spanning the full sequence.
- Pedestrian track lengths range from 6 to 30 frames.
- The track-length distribution is consistent across the training and test sets.
- Many pedestrian tracks extend across the full sequence length of 30 frames.
D. Evaluation with Varying Size of LSTM
The supplementary evaluation examines LSTM sizes and reports that increasing the homoscedastic model from 128 to 512 neurons worsens performance, whereas Bayesian versions remain comparable.
- The evaluation varies LSTM size while using past bounding-box sequences with |Bp| = 8.
- The 512-neuron homoscedastic LSTM performs worse than the 128-neuron version because the larger model overfits bounding-box estimation noise.
- Bayesian LSTM versions have comparable performance across sizes, attributed to dropout preventing overfitting.
F. Additional Evaluation of our Two-stream Model
The Bayesian Two-stream LSTM outperforms the homoscedastic and aleatoric-only variants, while the comparisons show distinct trade-offs between uncertainty modeling and prediction error.
- The Bayesian Two-stream LSTM outperforms all other evaluated methods.The compared models share the same odometry prediction stream, isolating differences in bounding-box prediction and uncertainty modeling.
- The heteroscedastic LSTM outperforms the homoscedastic LSTM on the L metric.This indicates that modeling observation-dependent uncertainty assigns higher probability to the true bounding-box sequence.
- Aleatoric-only modeling yields poorer MSE when it compensates for unmodeled epistemic uncertainty.The result distinguishes uncertainty calibration from mean prediction accuracy.
G. Additional Analysis of the Quality of our Uncertainty Metric
The uncertainty analysis relates predictive uncertainty to squared prediction error and compares uncertainty visualizations across odometry and trajectory predictions.
- Odometry predictions are visualized as 15 time-step points overlaid on the last visual observation.Blue denotes ground truth, red the Kalman Filter, yellow the LSTM without visual input, and green the LSTM with visual input.
- Predictions from the Two-stream model remain accurate over 15 future time-steps.Point estimates are also compared with Kalman Filter and One-stream baselines.
- The analysis examines uncertainty versus squared error and maximum observed squared error.These plots assess whether larger uncertainty corresponds to larger prediction errors.
- A robust error bound requires modeling both epistemic and aleatoric uncertainty.The analysis therefore evaluates uncertainty quality beyond observation noise alone.
H. Additional Video Results
Additional video results include examples of both point estimates and predictive distributions.
- The supplementary video contains examples of point estimates and predictive distributions.