PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Multi-Object Tracking: A Baseline and New Evaluation Metrics

Xinshuo Weng, Jianren Wang, David Held, Kris Kitani

arXiv:1907.03961v5cs.CVcs.RO

TL;DR

3D MOT needs systems that are accurate without sacrificing computational efficiency and simplicity, and it also lacks fair standardized evaluation directly in 3D. This work combines LiDAR 3D detections with a 3D Kalman filter and Hungarian association, adds a 3D evaluation tool with three integral metrics, and achieves state-of-the-art benchmark performance with real-time speed.

Abstract

from arXiv · show

3D multi-object tracking (MOT) is an essential component for many applications such as autonomous driving and assistive robotics. Recent work on 3D MOT focuses on developing accurate systems giving less attention to practical considerations such as computational cost and system complexity. In contrast, this work proposes a simple real-time 3D MOT system. Our system first obtains 3D detections from a LiDAR point cloud. Then, a straightforward combination of a 3D Kalman filter and the Hungarian algorithm is used for state estimation and data association. Additionally, 3D MOT datasets such as KITTI evaluate MOT methods in the 2D space and standardized 3D MOT evaluation tools are missing for a fair comparison of 3D MOT methods. Therefore, we propose a new 3D MOT evaluation tool along with three new metrics to comprehensively evaluate 3D MOT methods. We show that, although our system employs a combination of classical MOT modules, we achieve state-of-the-art 3D MOT performance on two 3D MOT benchmarks (KITTI and nuScenes). Surprisingly, although our system does not use any 2D data as inputs, we achieve competitive performance on the KITTI 2D MOT leaderboard. Our proposed system runs at a rate of $207.4$ FPS on the KITTI dataset, achieving the fastest speed among all modern MOT systems. To encourage standardized 3D MOT evaluation, our system and evaluation code are made publicly available at https://github.com/xinshuoweng/AB3DMOT.

I. INTRODUCTION

The paper addresses the practical cost and complexity of modern 3D MOT while proposing a simple baseline and standardized 3D evaluation. Despite using classical modules, the system achieves strong 3D and competitive 2D tracking performance.

I. INTRODUCTION

  • 3D MOT research has prioritized accuracy while often imposing high computational cost and complex pipelines that make real-time deployment and modular comparison difficult.Leading systems can have substantially different pipelines despite only minor performance differences.Source: §I · p. 1
  • 207.4 FPS on KITTI makes the system the fastest among modern MOT systems while achieving competitive KITTI 2D MOT performance.The system uses no 2D input; the authors hypothesize that 3D tracking resolves depth ambiguities and reduces mismatches.Source: §I · p. 1, Figure 1 · p. 1
  • State-of-the-art 3D MOT performance is achieved on both KITTI and nuScenes using a straightforward combination of classical modules.The baseline combines a 3D Kalman filter with the Hungarian algorithm and uses an off-the-shelf 3D detector.Source: §I · p. 1
  • KITTI’s image-plane evaluation does not fairly compare 3D MOT methods because stronger 3D tracking need not produce higher 2D performance.The paper proposes direct 3D evaluation to address this mismatch.Source: §I · pp. 1–2
  • The work proposes a 3D MOT evaluation tool and three metrics that evaluate systems across confidence thresholds rather than relying on one manually selected threshold.This is intended to expose performance across operating points and standardize 3D MOT evaluation.Source: §I · p. 2

III. APPROACH

The online system processes current-frame LiDAR detections with a 3D Kalman filter, then maintains trajectories through prediction, association, updating, and birth/death handling.

III. APPROACH

  • Each frame begins with 3D detections from a LiDAR point cloud, followed by prediction, association, Kalman updating, and trajectory birth/death management.The pipeline uses current-frame detections and trajectories associated from previous frames.Source: §III · p. 2, Figure 2 · p. 3
  • The detector represents each object by 3D center location, 3D size, heading angle, and confidence score.The detection tuple is (x, y, z, θ, l, w, h, s).Source: A. 3D Object Detection · p. 2

B. 3D Kalman Filter: State Prediction

  • Trajectory prediction uses a constant-velocity model in 3D and does not explicitly estimate camera ego-motion.The 11-dimensional trajectory state includes position, orientation, size, confidence, and linear velocity.Source: B. 3D Kalman Filter: State Prediction · pp. 2–3

C. Data Association

Data association matches predicted 3D trajectories with current detections using an affinity matrix and the Hungarian algorithm, rejecting pairs below geometric thresholds.

C. Data Association

  • The system builds an m_t−1×n_t affinity matrix from 3D IoU or negative center distance between predicted trajectories and detections.Each trajectory-detection pair contributes one affinity value.Source: C. Data Association · p. 3
  • The Hungarian algorithm solves the resulting bipartite matching problem in polynomial time.This produces matched and unmatched trajectories and detections.Source: C. Data Association · p. 3
  • Matches are rejected when 3D IoU falls below IoU_min or center distance exceeds dist_max.The threshold depends on which affinity definition is used.Source: C. Data Association · p. 3
  • The association output separates matched trajectories and detections from unmatched trajectories and detections for subsequent updating and memory management.T_match and D_match contain the matches, while T_unmatch and D_unmatch are their complementary sets.Source: C. Data Association · p. 3

D. 3D Kalman Filter: State Update

The Kalman update combines matched trajectory and detection states while correcting orientation ambiguity that can otherwise produce implausible intermediate headings and poor 3D overlap.

D. 3D Kalman Filter: State Update

  • Matched trajectory states are updated from corresponding detections using a weighted average determined by state uncertainty.The update follows the Kalman filter’s Bayes-rule formulation.Source: D. 3D Kalman Filter: State Update · p. 3
  • Directly applying the Bayes update to orientation can fail when detection and trajectory headings differ by nearly π.Prediction or detection errors can create this situation even though objects should move smoothly.Source: D. 3D Kalman Filter: State Update · p. 3
  • The orientation correction adds π to the trajectory heading when the orientation difference exceeds π/2, keeping the two headings roughly consistent.This prevents the updated orientation from falling midway between opposite headings and producing low 3D IoU.Source: D. 3D Kalman Filter: State Update · p. 3

E. Birth and Death Memory

The tracker manages object births by delaying new-track creation until unmatched detections persist, and manages deaths by retaining unmatched tracks temporarily before deletion.

E. Birth and Death Memory

  • Unmatched detections are treated as potential new objects, but new trajectories are created only after continued matching across the required confirmation period.New trajectories are initialized from the most recent detection with zero velocity in vx, vy, and vz.Source: E. Birth and Death Memory · p. 3
  • Unmatched trajectories are retained for Agemax frames to avoid deleting true objects during temporary missed detections, then deleted when confirmed disappeared.This retention allows true-positive trajectories with missing detections to be interpolated rather than immediately removed.Source: E. Birth and Death Memory · pp. 3–4

IV. NEW 3D MOT EVALUATION TOOL

The proposed evaluation tool evaluates 3D tracking directly in 3D space instead of projecting results onto the image plane, enabling fairer comparison of 3D MOT systems.

IV. NEW 3D MOT EVALUATION TOOL

  • KITTI provides 3D trajectories but conventionally evaluates them after projection to the image plane, while no direct 3D MOT evaluation tool was available.Image-plane evaluation may not fairly compare systems whose 3D tracking quality differs.Source: §IV · p. 4
  • The tool replaces 2D IoU matching with 3D IoU or center-distance matching between tracked objects and 3D ground truth.A match succeeds when 3D IoU exceeds IoUthres or center distance is below Distthres.Source: §IV · p. 4

V. NEW MOT EVALUATION METRICS

The paper identifies threshold dependence as a limitation of CLEAR metrics and proposes integral metrics that summarize tracking performance across operating points.

A. Limitation of the CLEAR Metrics

  • CLEAR metrics ignore tracked-object confidence scores, forcing users to manually choose a threshold to filter low-confidence trajectories.This manual choice is needed because low-confidence false-positive trajectories can otherwise reduce MOTA.Source: A. Limitation of the CLEAR Metrics · p. 4
  • Confidence thresholds can vary with detections and datasets, requiring extensive validation experiments and obscuring performance across operating points.The KITTI analysis evaluates 40 thresholds defined by recall values from 0% to 100%, excluding 0%.Source: A. Limitation of the CLEAR Metrics · p. 4
  • A single-threshold leaderboard can favor systems that perform well at one operating point but poorly elsewhere, whereas robust systems should maintain high MOTA across thresholds.The paper contrasts this with evaluating a MOTA-over-recall curve and summarizing it using integral metrics.Source: A. Limitation of the CLEAR Metrics · p. 4

B. Integral Metrics: AMOTA and AMOTP

  • AMOTA and AMOTP integrate MOTA and MOTP across recall values, approximating the integrals with summations over discrete confidence thresholds.The confidence score of a trajectory is defined as the average of its frame-level confidence scores.Source: B. Integral Metrics: AMOTA and AMOTP · pp. 4–5
  • MOTA can have a recall-dependent upper bound below 100%, motivating scaled accuracy so that sMOTA and sAMOTA remain upper-bounded by 100%.The scale adjustment accounts for the number of ground-truth objects available at each recall value.Source: Figure 3 · p. 5
  • The evaluation uses L=40 recall values to balance integration accuracy and evaluation speed, but assigns zero beyond a system's maximum recall and therefore favors high-recall systems.The authors justify this bias because high recall is crucial for preventing collisions in autonomous systems.Source: B. Integral Metrics: AMOTA and AMOTP · p. 5

C. Scaled Accuracy Metric: sAMOTA

The section explains why standard MOTA-based integral evaluation can be bounded below 100% and introduces scaled metrics to restore an interpretable 0–100% range.

C. Scaled Accuracy Metric: sAMOTA

  • MOTA may have a strict upper bound below 100% at many recall values, making conventional integral percentages difficult to interpret.This conflicts with the usual 0%–100% interpretation of integral metrics such as average precision.Source: C. Scaled Accuracy Metric: sAMOTA · p. 5
  • AMOTA can be upper bounded by 50% because the MOTA-versus-recall curve has an upper-bound triangle.The bound follows from unavoidable false negatives at recall r and non-negative false positives and identity switches.Source: C. Scaled Accuracy Metric: sAMOTA · p. 5
  • sMOTA and sAMOTA rescale MOTA and AMOTA so their values can range from 0% to 100%.The scaling compensates for the recall-dependent upper bound caused by unavoidable false negatives.Source: C. Scaled Accuracy Metric: sAMOTA · p. 5

VI. EXPERIMENT

The experiments evaluate the proposed metrics and standard CLEAR metrics on KITTI and nuScenes using established splits, categories, and matching criteria, with comparisons against modern 3D MOT baselines.

A. Settings

  • The study reports AMOTA, AMOTP, sAMOTA, and standard CLEAR metrics including MOTA, MOTP, IDS, FRAG, and FPS.These metrics combine the proposed confidence-aware evaluation with conventional tracking measures.Source: A. Settings · pp. 5–6
  • The evaluation covers KITTI and nuScenes 3D MOT datasets, which provide LiDAR point clouds and 3D bounding-box trajectories.KITTI 3D MOT evaluation uses the validation set because test ground truth is unavailable to users.Source: A. Settings · p. 6
  • Results are organized across KITTI Car, Pedestrian, and Cyclist categories and nuScenes performance averaged over seven categories.KITTI uses category-specific results, while nuScenes aggregates Car, Truck, Trailer, Pedestrian, Bicycle, Motorcycle, and Bus.Source: A. Settings · p. 6
  • The method is compared with open-sourced 3D MOT systems FANTrack and mmMOT using the same 3D detections where applicable.KITTI uses PointRCNN detections and nuScenes uses Megvii detections; FANTrack additionally receives projected 2D detections.Source: A. Settings · p. 6
  • The reported best results use a 3D Kalman-filter state with position, orientation, dimensions, confidence, and linear velocity, plus dataset-specific association thresholds.The implementation uses Fmin=3, Agemin=2, and different matching thresholds across KITTI categories and nuScenes.Source: A. Settings · p. 6

B. Experimental Results

The system achieves strong 3D MOT performance across KITTI and nuScenes, including challenging pedestrian and cyclist tracking, while running at 207.4 FPS without GPUs for tracking.

KITTI results

  • The system establishes state-of-the-art 3D MOT performance on KITTI Cars, consistently outperforming modern systems across matching criteria and achieving zero identity switches.The comparison uses 3D IoU thresholds of 0.25, 0.5, and 0.7.Source: B. Experimental Results · p. 6
  • Strong performance extends to KITTI pedestrians and cyclists, although these objects are more challenging to track because of their small size.Source: B. Experimental Results · p. 6

nuScenes results

  • nuScenes provides a more challenging evaluation setting because sparse LiDAR, complex scenes, and low frame rate produce lower-quality 3D detections.Source: B. Experimental Results · p. 6

Inference time

  • 207.4 FPS is achieved on KITTI validation data without GPUs, making the tracking system the fastest among compared 3D MOT systems.This excludes the 3D detector component.Source: B. Experimental Results · p. 7

Qualitative comparison

C. Ablation Study

The ablations show that 3D detection quality, 3D filtering, orientation correction, and selected trajectory thresholds materially affect performance, while angular velocity has little consistent effect.

Effect of 3D Detection Quality

  • Lower-quality monocular 3D detections produce worse tracking than LiDAR-based detections, showing that 3D detection quality is crucial to MOT performance.Source: C. Ablation Study · p. 7

3D v.s. 2D Kalman Filter

  • Replacing the 3D Kalman filter with a 2D filter increases identity switches from 0 to 7 and fragmentation from 15 to 43, while reducing key accuracy metrics by 3% to 4%.The authors attribute this difference to 3D tracking resolving depth ambiguity in the image plane.Source: C. Ablation Study · p. 7

Effect of Angular Velocity vθ

  • Adding angular velocity improves sAMOTA and AMOTA by 0.01% but decreases AMOTP and MOTA by up to 0.08%, so the final system omits it for simplicity.Source: C. Ablation Study · p. 8

Effect of Orientation Correction

  • Orientation correction improves performance across all metrics, supporting its inclusion in the final system.Source: C. Ablation Study · p. 8

Effect of Threshold IoUmin

Trajectory thresholds

  • Birmin=3 performs better than Birmin=1 or 5, while Agemax=2 outperforms both lower and higher alternatives on the reported accuracy metrics.Birmin controls when unmatched detections create trajectories, and Agemax controls how long unmatched trajectories continue being tracked.Source: C. Ablation Study · p. 8
Loading 1907.03961v5…