PaperlayerView exact revision on arXiv

Source-linked AI summary

Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving

Hsu-kuang Chiu, Jie Li, Rares Ambrus, Jeannette Bohg

arXiv:2012.13755v2cs.CVcs.RO

TL;DR

3D multi-object tracking must associate detections with existing tracks and manage track lifecycles for autonomous driving. This paper proposes a probabilistic multimodal system that learns feature fusion, distance combination, and track initialization, and reports stronger tracking than same-detector baselines on NuScenes and KITTI.

  • Problem

    Existing tracking-by-detection systems face challenges in data association and track life-cycle management, while prior association metrics and initialization rules have limited use of appearance and geometry.

  • Method

    The system fuses 2D image and 3D LiDAR features, learns to combine deep feature and Mahalanobis distances, and learns whether unmatched detections should initialize tracks.

  • Results

    The method outperforms tracking baselines using the same object detectors on NuScenes and KITTI, with multimodal fusion producing a 2.8 performance gain over CenterPoint.

  • Takeaways & Limitations

    Learned multimodal association and track initialization provide robust, data-driven tracking results with fewer false-positive tracks in qualitative evaluation.

Abstract

from arXiv · show

Multi-object tracking is an important ability for an autonomous vehicle to safely navigate a traffic scene. Current state-of-the-art follows the tracking-by-detection paradigm where existing tracks are associated with detected objects through some distance metric. The key challenges to increase tracking accuracy lie in data association and track life cycle management. We propose a probabilistic, multi-modal, multi-object tracking system consisting of different trainable modules to provide robust and data-driven tracking results. First, we learn how to fuse features from 2D images and 3D LiDAR point clouds to capture the appearance and geometric information of an object. Second, we propose to learn a metric that combines the Mahalanobis and feature distances when comparing a track and a new detection in data association. And third, we propose to learn when to initialize a track from an unmatched object detection. Through extensive quantitative and qualitative results, we show that when using the same object detectors our method outperforms state-of-the-art approaches on the NuScenes and KITTI datasets.

I. INTRODUCTION

3D multi-object tracking supports safer autonomous driving by estimating traffic participants over time, but existing association metrics overlook appearance and geometry. The paper addresses this gap with learned multimodal association and track-initialization decisions.

  • 3D multi-object tracking estimates each traffic participant’s location, orientation, and scale over time for autonomous driving.Temporal information can filter detection outliers, improve robustness to occlusions, and support motion forecasting.
  • Existing tracking-by-detection methods commonly associate detections and tracks using center or Mahalanobis distance.These metrics account for object distance and bounding-box state differences.
  • Center and Mahalanobis metrics ignore objects’ geometric and appearance features, making data association a central challenge.Prior learned association methods still failed to outperform a simpler center-distance method, highlighting the difficulty of effective neural association.
  • The proposed system learns to combine Mahalanobis and geometric-appearance distances and to initialize tracks from unmatched detections.The features come from 3D LiDAR point clouds and 2D camera images, while tracking remains within a Kalman Filter framework.
  • Track life-cycle management affects false positives and identity switches but has received limited research attention.Prior systems either initialize tracks for every unmatched detection or promote temporary tracks after consecutive matches.
  • The system uses three trainable modules and outperforms reported tracking methods on NuScenes and KITTI with the same object detectors.Feature fusion further improves performance, while qualitative results show significantly fewer false-positive tracks.

A. 3D Object Detection

The tracking system operates on 3D detector outputs within a tracking-by-detection framework. Its novelty is learning multimodal association and data-driven track life-cycle management rather than relying only on conventional distance metrics.

  • Most 3D multi-object tracking systems take 3D detector bounding boxes as input, so detector choice influences tracking performance.Detectors may use camera images, LiDAR point clouds, or combined inputs.
  • The paper uses CenterPoint to provide 3D detections but states that its tracking method is detector-agnostic.CenterPoint quantizes LiDAR point clouds, builds a PointNet feature map, and predicts object centers, sizes, and orientations.
  • Tracking-by-detection methods associate detections with existing tracks using metrics such as 3D IoU, Mahalanobis distance, or center distance.These methods differ in whether they use overlap, uncertainty-aware state distance, or object-center distance.
  • Several multimodal or end-to-end methods combine trajectory, geometric, and appearance information but reportedly do not outperform CenterPoint on NuScenes.The paper therefore targets association while retaining a probabilistic tracking framework.
  • The proposed system learns to combine Mahalanobis and deep feature distances and manages track life cycles in a data-driven way.Its three components fuse features, learn association distances, and decide whether unmatched detections initialize tracks.

A. Kalman Filters

The tracker uses a Kalman Filter to represent and predict object states from detector observations. Its motion model assumes constant velocity and constant object dimensions during prediction.

  • Each object state contains 11 variables covering 3D position, orientation, dimensions, and frame-to-frame changes in position and orientation.The state is represented as st = (x, y, z, a, l, w, h, dx, dy, dz, da)^T.
  • The motion model assumes constant linear and angular velocity and constant object dimensions during prediction.This is implemented as a linear motion model within the standard Kalman Filter formulation.
  • The Kalman Filter predicts the next state mean and covariance using the transition matrix and process-noise covariance.The predicted covariance propagates prior uncertainty through the dynamics model and adds process noise.
  • CenterPoint supplies per-frame 3D bounding-box observations to the Kalman Filter.Each observation includes position, orientation, dimensions, and estimated x- and y-velocity differences.
  • The observation model is linear with additive zero-mean Gaussian noise, whose covariance is estimated from training-set statistics.The process and observation covariance matrices are denoted Q and R.

B. Fusion of 2D and 3D features

The method fuses 2D camera and 3D LiDAR features for each detection, then uses the fused representations in learned association and track-initialization modules. Association combines feature dissimilarity with Mahalanobis distance.

  • The Feature Fusion Module combines per-detection features from 2D camera images and 3D LiDAR point clouds.The fused features feed both the Distance Combination Module and the Track Initialization Module.
  • The fusion process extracts a LiDAR feature map region and a projected image-region feature for each detected object.The image representation includes a Mask R-CNN RoIAlign vector and a camera-plane indicator.
  • Ffused = G1(F2D) + F3D combines the transformed 2D feature with the 3D feature.G1 is an MLP followed by reshaping, producing a fused feature tensor for N detections.
  • The learned association metric combines state-estimate information with appearance and geometric feature information for N detections and M tracks.It is designed as a linear combination of Mahalanobis and deep feature distances.
  • D = DMah + α ⊙(Dfeat −(0.5 + β)) combines Mahalanobis distance with feature distance using learned coefficient matrices.The constant 0.5 serves as an initial bias term, and the Mahalanobis distance compares detections with predicted track states.
  • The association network is trained in two stages: first learning deep feature distance, then learning coefficients for the final combined metric.The distance map is computed from fused features of detections and tracks.

1) Deep Feature Distance:

The method learns a deep feature distance by treating track-detection matching as binary classification. Binary cross-entropy supervises the distance using a matching-indicator matrix.

  • 1) Deep Feature Distance:: The network learns deep feature distance Dfeat by framing feature-distance learning as binary classification.The approach uses a neural network to distinguish matched from unmatched track-detection feature pairs.
  • 1) Deep Feature Distance:: Binary cross-entropy trains Dfeat against matching-indicator matrix K, where 0 denotes matched and 1 denotes unmatched feature pairs.Pairs are labeled using ground-truth object identities across adjacent frames and a 2D center-distance condition.

2) Combining Coefficients:

The Distance Combination Module learns coefficient matrices that combine deep feature distances with Mahalanobis distance for association and outlier rejection. Its training losses encourage positives to fall below, and negatives above, a rejection threshold.

  • 2) Combining Coefficients:: The module fixes Dfeat and learns coefficient matrices α and β to adjust the final distance D according to deep-feature importance.The combined distance is used to compare tracks and detections.
  • 2) Combining Coefficients:: A two-stage training procedure first learns Dfeat and then learns α and β to produce the final combined distance metric D.The coefficient-learning stage follows feature-distance learning.
  • 2) Combining Coefficients:: The contrastive loss encourages every positive distance di to be smaller than every negative distance dj by adjusting α and β.Positive and negative track-detection pairs are separated through a max-margin objective.
  • 2) Combining Coefficients:: Additional max-margin losses train positive distances below threshold T and negative distances above T for unmatched-outlier rejection.T is the inference threshold used to reject unmatched outliers.
  • 2) Combining Coefficients:: At test time, the combined distance is used for data association with the greedy matching algorithm from ProbabilisticTracking.The implementation uses the learned distance within the association stage.

D. Track Initialization Module

The Track Initialization Module replaces heuristic initialization rules with binary classification of unmatched detections. It uses fused features to predict whether each unmatched detection should start a new track.

  • D. Track Initialization Module: Prior methods either initialize every unmatched detection or wait for a fixed number of consecutive matches before promoting temporary tracks.These are the heuristic baselines addressed by the proposed module.
  • D. Track Initialization Module: The Track Initialization Module treats initialization as binary classification and takes each unmatched detection’s fused feature as input.It outputs a confidence score for creating a new track.
  • D. Track Initialization Module: The module computes confidence score P with P = G4(Ffused), where G4 contains convolutional, MLP, and Sigmoid layers.G4 is trained as a binary classifier with cross-entropy loss.
  • D. Track Initialization Module: The target label is 1 when a ground-truth object is close to the detection and 0 otherwise.This defines the binary supervision for track initialization.
  • D. Track Initialization Module: At inference, an unmatched detection initializes a new track when Pn exceeds 0.5, reducing false-positive tracks.The module applies a confidence threshold to initialization decisions.

IV. EXPERIMENTAL RESULTS

The method is evaluated on NuScenes and KITTI using standard multi-object tracking metrics and benchmark-specific detector settings. The reported tables compare overall and category-level performance against baseline methods.

  • IV. EXPERIMENTAL RESULTS: The evaluation uses NuScenes and KITTI validation data, with 700 NuScenes training sequences and 150 validation sequences.The KITTI split contains 10 training sequences and 11 validation sequences.
  • IV. EXPERIMENTAL RESULTS: AMOTA is the main NuScenes Tracking Challenge metric and averages tracking accuracy across recall thresholds.The metric is reported for overall tracking and individual object categories.
  • IV. EXPERIMENTAL RESULTS: Table I compares overall AMOTA and category-specific AMOTA on the NuScenes validation set, marking the best result in each column.The table includes baseline methods and an implementation using ProbabilisticTracking code with CenterPoint detections.
  • IV. EXPERIMENTAL RESULTS: Table II reports overall AMOTA and car AMOTA on NuScenes while noting that different 3D detectors can significantly affect tracking accuracy.GNN3DMOT reports overall AMOTA only, whereas PnPNet reports car AMOTA only.
  • IV. EXPERIMENTAL RESULTS: Table III reports AMOTA and MOTA for cars on KITTI using Point R-CNN and the same train-validation split as GNN3DMOT.MOTA is reported alongside AMOTA for the KITTI comparison.

C. Quantitative Results

The method outperforms competing trackers under matched detector inputs on NuScenes and KITTI, with further gains from combining LiDAR and image features. Input detection quality remains important to final tracking performance.

  • Using exactly the same 3D LiDAR input, the method outperforms CenterPoint and ProbabilisticTracking.
  • Fusing LiDAR and image features produces a 2.8 performance gain in overall AMOTA over the previous state-of-the-art CenterPoint.
  • The method is compared with GNN3DMOT and PnPNet on NuScenes and KITTI.

D. Ablation Study

Ablations show consistent gains from learned distance combination, track initialization, and multimodal feature fusion. Qualitative examples connect these components to fewer false positives and adaptive use of appearance information.

  • Ablation Study: Both the Distance Combination and Track Initialization Modules consistently improve performance, with the highest results when both are enabled.
  • Ablation Study: Fusing 2D and 3D features consistently increases performance, indicating that the model leverages appearance and geometric information.
  • Qualitative Results: The method achieves an 11.0 AMOTA increase on motorcycles versus CenterPoint, alongside a substantial reduction in false-positive tracks.
  • Qualitative Results: In bird’s-eye-view visualizations, the proposed tracks contain fewer false positives and are closer to ground truth than CenterPoint’s tracks.
  • Qualitative Results: The Distance Combination Module predicts α = 2.594 for large, clear motorcycles and α = 1.802 for small, blurred objects.
  • Qualitative Results: The Track Initialization Module rejects false-positive detections in Sequence 1 Frame 28 instead of creating new tracks.

V. CONCLUSION

The paper presents an online probabilistic multimodal tracker that learns feature fusion, association weighting, and track life-cycle decisions. It outperforms same-detector state-of-the-art baselines quantitatively and qualitatively, while future work targets additional modalities, detectors, motion models, and differentiable filtering.

  • The proposed tracker fuses 2D camera and 3D LiDAR features for data association and data-driven track life-cycle management.
  • On NuScenes and KITTI, the method outperforms state-of-the-art baselines using the same object detector in quantitative and qualitative evaluations.
  • Future work includes additional modalities, novel object detectors, category-specific motion models, and end-to-end differentiable filtering.
Loading 2012.13755v2…