Source-linked AI summary
Uncertainty-Aware Trajectory Forecasting from Imperfect Tracking
Stephane Da Silva Martins, Victor Petrovic, Emanuel Aldea, Sylvie Le Hégarat-Mascle
TL;DR
Practical trajectory forecasting must handle tracker outputs containing localization noise, missed detections, confidence fluctuations, and identity switches. The paper propagates these reliability cues as Gaussian state uncertainty and trains with temporally correlated noise plus clean-teacher distillation, improving displacement metrics and probabilistic reliability across evaluated protocols.
Problem
Most forecasting methods assume clean, temporally consistent observed histories, although operational trackers produce noisy, incomplete, and ambiguous trajectories whose reliability signals may be informative.
Method
The framework represents observations and predictions as Gaussian states, combines localization and association uncertainty through total variance, and trains with OU noise and response-based distillation from a clean teacher.
Results
Experiments on Oxford Town Centre, VIRAT, and ETH/UCY pseudo-detections show improvements in displacement error metrics and probabilistic reliability.
Takeaways & Limitations
Tracker-derived reliability can be propagated to forecasting models rather than removed through denoising, supporting predictions that reflect noisy-input reliability.
Takeaways & Limitations
The method assumes each observed state comes from an upstream tracker with observation reliability available for propagation.
Abstract
from arXiv · showhide
Most trajectory forecasting models are trained on clean annotated histories, and are often evaluated under the same idealized assumption, although practical deployments rely on trajectories produced by imperfect multi-object trackers. The real-world observations exhibit localization jitter, missed or unstable detections, and data-association ambiguity, which are usually either ignored or removed through denoising. This paper instead treats tracking-derived reliability cues as an informative signal to be propagated to the predictor. We propose a plug-in uncertainty-aware formulation in which each observed state is encoded as an uncertain state representation, modeled by a Gaussian distribution whose covariance combines detection-level localization uncertainty and association-level ambiguity through the law of total variance. Existing backbones are adapted with minimal architectural changes: input trajectories are represented as Gaussian observations, and predicted trajectories are produced as Gaussian forecasts rather than deterministic coordinates. To train predictors that remain robust under structured observation noise, we combine temporally correlated Ornstein-Uhlenbeck perturbations with response-based knowledge distillation from a teacher trained on clean trajectories. Experiments on Oxford Town Centre and VIRAT using real tracker outputs, together with a complementary ETH/UCY pseudo-detection protocol, show that the proposed formulation improves displacement accuracy and the reliability-sharpness trade-off of probabilistic forecasts.
1 Introduction
Trajectory forecasting often assumes clean observed histories, but deployed trackers produce noisy, incomplete, and ambiguous trajectories whose reliability cues can inform prediction. The paper propagates this uncertainty through Gaussian state representations and trains predictors for robust probabilistic forecasting.
- Most trajectory forecasting methods assume that observed past trajectories are clean, temporally consistent, and directly available at inference.
- Tracker outputs contain localization noise, confidence fluctuations, missed detections, and identity switches, while also indicating challenging conditions such as occlusion or crowding.
- Each observation is represented as a Gaussian state whose covariance separates localization uncertainty from association uncertainty and combines them through the law of total variance.
- The plug-in formulation embeds Gaussian state parameters and predicts Gaussian position moments with a negative log-likelihood objective while preserving the core predictor.
- The training strategy combines temporally correlated noise injection with response-based distillation from a teacher trained on clean trajectories.
- Experiments evaluate displacement robustness and probabilistic reliability on Oxford Town Centre, VIRAT, and an ETH/UCY pseudo-detection protocol.
2 Related Work
Prior trajectory forecasting models social interactions, scene context, multimodality, and longer horizons, but commonly assumes clean observations. Robust forecasting addresses imperfect perception through video or detection inputs, denoising, or uncertainty-aware representations.
- Trajectory forecasting research has progressed from recurrent and pooling models toward graph-based, attention-based, and transformer-based architectures for social and contextual reasoning.
- Generative approaches such as GANs, VAEs, diffusion models, and flow matching represent multiple plausible futures but generally assume clean and complete observed histories.
- Robust forecasting methods reduce dependence on explicit tracking by predicting from raw videos, detections, or multi-hypothesis tracking structures.
- Other methods denoise or disentangle corrupted histories, whereas this work treats tracking errors as uncertainty information to expose to the predictor.
- Tracking cues such as localization covariance, detector confidence, and association ambiguity can be combined into Gaussian observation representations through moment matching.
3 Methodology
The methodology propagates tracker-derived uncertainty into trajectory forecasting by representing observations and predictions probabilistically, while training with structured noise and clean-teacher distillation. It combines detection uncertainty, association ambiguity, Gaussian backbone adaptations, and reliability-aware objectives with minimal architectural changes.
- Motivation: Tracker-produced histories may contain localization noise, confidence fluctuations, missed detections, and identity switches, while these imperfections also provide reliability information.The framework treats tracker outputs as uncertain observations rather than deterministic coordinates.
- Model Adaptation: Existing predictors receive Gaussian observation statistics through an adapted embedding and produce Gaussian future-position parameters instead of deterministic coordinates.The covariance is encoded using marginal standard deviations and a correlation coefficient, preserving the backbone while adding uncertainty awareness.
- Training Strategy: Training combines temporally correlated tracking-like noise with response-based distillation from a clean-trajectory teacher, matching the teacher mean while allowing the student covariance to reflect noisy inputs.Mean-only distillation avoids forcing the student to copy the teacher’s clean-history covariance.
- Probabilistic Modeling: Each observation is modeled as a Gaussian state whose covariance combines localization uncertainty from detections with association uncertainty from plausible matching candidates.The two components are combined through the law of total variance and represented as a unified covariance interface.
- Probabilistic Modeling: Association ambiguity is estimated from soft candidate weights, while localization uncertainty uses detection position, bounding-box geometry, and confidence-dependent spatial variance.The resulting Gaussian mixture is reduced to a single Gaussian by matching its first two moments; spatially dispersed candidates increase association uncertainty.
- Objective: The Gaussian outputs are optimized with negative log-likelihood, training the model to reduce displacement error while learning predictive covariance evaluated through coverage and area metrics.The same covariance interface expresses both inference-time tracking ambiguity and training-time perturbations.
4 Experiments
The experiments evaluate uncertainty-aware forecasting across real tracker outputs and a complementary pseudo-detection protocol, testing robustness across backbones, training configurations, uncertainty choices, and noise objectives.
- Datasets and protocols: Experiments use Oxford Town Centre and VIRAT with tracker-derived uncertainty, plus a complementary ETH/UCY pseudo-detection protocol rather than a standard benchmark comparison.OTC and VIRAT use chronological 80/20 video splits; ETH/UCY uses detector adaptation before Gaussian observation conversion.
- Evaluation design: Three trajectory-prediction backbones are evaluated under five training configurations with identical probabilistic input embeddings and NLL losses.The configurations include clean-data baseline evaluation, Kalman filtering, tracker-output training, and tracking-noise-aware distillation.
- Main results: Distillation yields lower ADE/FDE in most cases and remains stable across datasets and backbones, while probabilistic inputs improve forecast reliability under noisy observations.The student improves the observed trade-off among NLL, coverage, area, and displacement accuracy; the fully probabilistic variant performs best overall in the input-output ablation.
- Ablations: Gaussian augmentation improves over no augmentation, Ornstein-Uhlenbeck noise further reduces errors, and NLL-based distillation outperforms KL for all backbones.NLL-based distillation does not force the student covariance to match the clean teacher’s covariance.
- Comparisons and robustness: On OTC, the NATRA-inspired reimplementation is weaker for VISTA, MART, and SingularTrajectory on reported minADE/minFDE/AUC comparisons.The comparison uses a reimplementation because official NATRA code is unavailable.
- Comparisons and robustness: On OTC, IoU-based association proxies produce lower NLL than BoT-SORT internal costs for all three evaluated backbones, without establishing universal IoU superiority.The reported comparisons are VISTA (2.30 vs. 1.34 and 1.24), MART (−15.56 vs. −23.49 and −22.20), and SingularTrajectory (−10.99 vs. −14.62 and −15.04).
5 Conclusion
The paper forecasts from imperfect tracker outputs by passing Gaussian observations with localization and association uncertainty to the predictor. OU noise and a clean teacher train the student on corrupted inputs, and experiments show improvements in displacement accuracy and probabilistic reliability.
- Conclusion: The method represents each tracker observation as a Gaussian state whose covariance combines localization and association uncertainty.The predictor receives both estimated position and an explicit reliability descriptor.
- Conclusion: The student is trained with temporally correlated OU noise and a clean teacher while operating on corrupted probabilistic observations.The clean teacher supplies a motion target during training, while deployment uses imperfect tracker outputs.
- Conclusion: Experiments on Oxford Town Centre, VIRAT, and the ETH/UCY pseudo-detection protocol show improvements in displacement error metrics and probabilistic reliability.The ETH/UCY protocol is complementary to the real-tracker evaluations.