Source-linked AI summary
Probabilistic 3D Multi-Object Tracking for Autonomous Driving
Hsu-kuang Chiu, Antonio Prioletti, Jie Li, Jeannette Bohg
TL;DR
3D multi-object tracking must estimate dynamic object states reliably for autonomous-driving planning. The paper uses a Kalman Filter with training-set covariance initialization and Mahalanobis-distance association, and reports a large AMOTA improvement over AB3DMOT, including first place in the NuScenes Tracking Challenge.
Problem
3D multi-object tracking is needed to provide reliable object trajectories and dynamic scene information for autonomous-driving planning.
Method
The method uses a Kalman Filter, initializes state and noise covariances from training-set statistics, and associates predictions with detections using Mahalanobis distance.
Results
The method outperforms AB3DMOT by 38.2% in overall AMOTA on the NuScenes validation set and achieves first place in the NuScenes Tracking Challenge.
Takeaways & Limitations
Uncertainty-aware association remains effective when predicted objects and detections do not overlap, including for small objects and sudden accelerations.
Takeaways & Limitations
The prediction model assumes constant linear and angular velocity and constant object dimensions during prediction.
Abstract
from arXiv · showhide
3D multi-object tracking is a key module in autonomous driving applications that provides a reliable dynamic representation of the world to the planning module. In this paper, we present our on-line tracking method, which made the first place in the NuScenes Tracking Challenge, held at the AI Driving Olympics Workshop at NeurIPS 2019. Our method estimates the object states by adopting a Kalman Filter. We initialize the state covariance as well as the process and observation noise covariance with statistics from the training set. We also use the stochastic information from the Kalman Filter in the data association step by measuring the Mahalanobis distance between the predicted object states and current object detections. Our experimental results on the NuScenes validation and test set show that our method outperforms the AB3DMOT baseline method by a large margin in the Average Multi-Object Tracking Accuracy (AMOTA) metric.
1. Introduction
The paper presents an online 3D multi-object tracker that uses a Kalman Filter, uncertainty-aware association, and training-set statistics for covariance initialization. It reports substantially better NuScenes tracking performance than the AB3DMOT baseline.
- 3D multi-object tracking estimates object locations, orientations, and scales over time to support robust trajectories for autonomous-driving planning.Temporal information can filter detector outliers and improve robustness to partial or full occlusions.
- The tracker models each object with a Kalman Filter and uses linear and angular velocity in its state and process model.Object detections provide measurements for the filter update.
- Mahalanobis distance performs outlier detection and data association using uncertainty in predicted object states.Unlike 3D-IOU, it can measure distance when predictions and detections do not overlap.
- The method estimates initial-state, process-noise, and observation-noise covariance matrices from training-data statistics.This supports filter convergence and avoids using future or ground-truth information in validation and test experiments.
- The method outperforms the AB3DMOT baseline by a large margin in AMOTA and takes first place in the NuScenes Tracking Challenge.The evaluation uses MEGVII detection results as measurements.
2. Related Work
The paper builds on online 3D tracking and object-detection systems by using uncertainty-aware association and training-data covariance estimates. Its approach extends Kalman-filter tracking with Mahalanobis affinity between predictions and detections, including non-overlapping cases.
- 3D object detectors provide frame-by-frame bounding boxes as measurements for multi-object tracking systems.The paper uses the voxel-based MEGVII detector's results as tracker measurements.
- Prior 3D tracking methods include AB3DMOT, LSTM-based motion estimation with image features, and map-assisted tracking.The paper uses AB3DMOT as a comparison baseline.
- The proposed method measures affinity between predictions and detections with or without direct overlap using the Mahalanobis distance.This distance incorporates prediction uncertainty and is used for outlier detection in filtering.
- The method estimates state and noise covariance matrices from training-data statistics and includes angular velocity in the state.The paper reports quantitatively improved tracking and qualitatively more accurate trajectories, especially for object orientation.
3. A Kalman Filter for Multi-Object Tracking
The tracker uses a Kalman Filter to predict and update 3D object states from frame-by-frame detections, with uncertainty-aware association through Mahalanobis distance. Training-set statistics initialize covariance matrices, while greedy matching and track memory complete the online pipeline.
- State Representation: Each tracked object has an 11-variable state containing position, orientation, dimensions, and frame-to-frame motion variables.The state includes 3D center position, z-axis orientation, bounding-box length, width, height, and four velocity-related variables.
- Prediction Model: The process model assumes constant linear and angular velocity and constant object dimensions during prediction, with Gaussian acceleration noise.Acceleration noise affects position and velocity predictions, while dimensions remain unchanged until the update step.
- Data Association: Mahalanobis distance associates predicted states with detections by weighting residual differences with predicted uncertainty, including cases where boxes do not overlap.Unlike 3D-IOU, it uses the Kalman Filter covariance and can produce a nonzero distance without geometric overlap.
- Data Association: The tracker uses greedy bipartite matching with an upper-bound distance threshold, then updates matched state means and covariances through the Kalman Filter.The implementation sorts candidate pairs by distance and accepts unmatched pairs only below the threshold.
- Covariance Estimation: Initial state, process-noise, and observation-noise covariance matrices are estimated from training-set statistics rather than heuristic identity-based choices.Validation and test data are excluded from this estimation to avoid future or ground-truth information during evaluation.
- Covariance Estimation: Acceleration noise is included in both position and velocity prediction equations because omitting it from position would underestimate next-position uncertainty and weaken association robustness.The paper explicitly connects this uncertainty modeling choice to more robust data association.
4. Experiment Results
On the NuScenes validation set, the proposed tracker substantially outperforms AB3DMOT in AMOTA, while ablations and visualizations show benefits from Mahalanobis association, data-driven covariance estimation, and angular velocity.
- Quantitative results: AMOTA improves by 38.2% over the official AB3DMOT baseline and by 5.2% over the authors’ AB3DMOT implementation on MEGVII detections.The evaluation uses overall and per-category AMOTA on the NuScenes validation set.
- Ablations: Mahalanobis-distance association outperforms 3D-IOU methods, especially for small objects whose boxes may not overlap despite close predictions and detections.Unlike 3D-IOU, Mahalanobis distance incorporates predicted-state uncertainty and remains informative when boxes do not overlap.
- Ablations: The greedy association algorithm performs better than the Hungarian algorithm, and data-driven covariance estimation outperforms heuristic covariance choices.These findings come from replacing components of the proposed method with corresponding AB3DMOT components.
- Ablations: Excluding angular velocity does not reduce AMOTA, because NuScenes matching uses 2D center distance and ignores rotation accuracy.Including angular velocity nevertheless produces more realistic qualitative trajectories and better orientation during turns.
- Qualitative results: Bird’s-eye-view visualizations show that the proposed tracker continues tracking cars through sharp turns and pedestrians through noisy detections where AB3DMOT loses tracks.Angular velocity improves estimated car orientation during turning, while both proposed variants track pedestrian location and orientation in the shown example.
5. Conclusion
The paper presents an online 3D multi-object tracking method using Mahalanobis distance for data association and training-set statistics to initialize Kalman Filter covariance matrices. It outperforms the 3D-IOU-based AB3DMOT baseline by a large margin in AMOTA.
- The method uses Mahalanobis distance for data association and training-set statistics to initialize the Kalman Filter covariance matrices.
- Figure 3 visualizes pedestrian tracking results in bird’s-eye view.
- The method outperforms the 3D-IOU-based AB3DMOT baseline by a large margin in the AMOTA evaluation metric.