Source-linked AI summary
EagerMOT: 3D Multi-Object Tracking via Sensor Fusion
Aleksandr Kim, Aljoša Ošep, Laura Leal-Taixé
TL;DR
Mobile-object tracking must combine depth sensors’ precise but short-range 3D measurements with cameras’ ability to localize distant objects in images. EagerMOT fuses both modalities and updates tracks with full or partial evidence, achieving state-of-the-art results across the reported KITTI and NuScenes MOT evaluations.
Problem
Depth sensors provide precise 3D localization only within limited sensing ranges, whereas cameras can localize distant objects only in the image domain.
Method
EagerMOT fuses 2D and 3D detections and uses two-stage association to update tracks even when evidence from only one modality is available.
Results
EagerMOT achieves state-of-the-art results across multiple MOT tasks and benchmarks, including KITTI and NuScenes.
Takeaways & Limitations
The framework supports varied LiDAR-camera configurations and detector combinations without additional detector-specific fine-tuning.
Abstract
from arXiv · showhide
Multi-object tracking (MOT) enables mobile robots to perform well-informed motion planning and navigation by localizing surrounding objects in 3D space and time. Existing methods rely on depth sensors (e.g., LiDAR) to detect and track targets in 3D space, but only up to a limited sensing range due to the sparsity of the signal. On the other hand, cameras provide a dense and rich visual signal that helps to localize even distant objects, but only in the image domain. In this paper, we propose EagerMOT, a simple tracking formulation that eagerly integrates all available object observations from both sensor modalities to obtain a well-informed interpretation of the scene dynamics. Using images, we can identify distant incoming objects, while depth estimates allow for precise trajectory localization as soon as objects are within the depth-sensing range. With EagerMOT, we achieve state-of-the-art results across several MOT tasks on the KITTI and NuScenes datasets. Our code is available at https://github.com/aleksandrkim61/EagerMOT.
I. INTRODUCTION
EagerMOT addresses the complementary limitations of depth- and image-based sensing by fusing available 2D and 3D observations. It tracks distant objects in images and localizes them precisely in 3D once they enter depth-sensing range.
- Depth-based methods provide 3D tracking but have limited range and sensitivity to reflective or low-albedo surfaces.
- Image-based methods localize distant or partially occluded objects precisely in the image domain but do not provide reliable 3D localization.
- EagerMOT fuses observations from 2D and 3D object detectors to produce a more complete interpretation of scene dynamics.
- The framework updates tracks using partial image-based or LiDAR-based evidence, enabling track initialization before objects enter depth-sensing range.
- EagerMOT supports multiple LiDAR-camera configurations, several MOT tasks, and deployment without additional detector-specific training.
II. RELATED WORK
Related work spans image-domain, LiDAR or stereo-based, and sensor-fusion tracking methods. EagerMOT differs by independently tracking in 2D and 3D while relying on complementary bounding-box detections across sensor configurations.
- 2D MOT: Vision-based trackers detect and associate objects in the image domain using detector extensions, regression, offsets, or re-identification cues.
- 3D MOT: LiDAR-based tracking-by-detection has grown with advances in point-cloud learning and 3D detection, but strong reliance on 3D detections creates vulnerabilities.
- Fusion-based methods: Fusion-based tracking remains under-explored, with prior methods combining 2D and 3D states, monocular SLAM, optical flow, scene flow, stereo depth, or learned models.
- Fusion-based methods: Unlike jointly coupled-state approaches, EagerMOT tracks targets simultaneously in 2D and 3D without explicitly coupling their states.
III. METHOD
EagerMOT combines 2D and 3D detections through observation fusion and two-stage temporal association. Its track updates remain possible when only one sensor modality supplies evidence.
- At each frame, EagerMOT receives 3D bounding-box detections and 2D detections from LiDAR and image streams.
- An observation-fusion module associates 2D and 3D detections from the same objects before temporal tracking.
- The two-stage association module matches detections across time and updates tracks using full or partial observation information.
- The formulation can recover from short occlusions, maintain approximate 3D location when one detector fails, and track far-away objects before 3D sensing begins.
A. Fusion
The fusion module greedily matches projected 3D detections with 2D detections using image-plane overlap, creating fused or modality-specific partial instances. The procedure also accommodates multiple cameras.
- The module takes 3D bounding-box detections from LiDAR and rectangular 2D detections from images, then establishes cross-modal matches.
- Candidate 3D–2D pairings are sorted by descending image-plane overlap and greedily combined when overlap exceeds θ_fusion and neither detection is already matched.
- Fused instances retain precise 3D location and a 2D bounding box, with optional additional information such as a segmentation mask.
- Unmatched detections become partial observations containing information from only one modality.
- For multiple cameras, fusion is performed independently in each image plane, and one candidate view is selected when a 3D detection matches multiple 2D detections.
- The framework then supports two association stages, first matching instances with 3D information and then processing unmatched tracks using remaining evidence.
B. Matching
EagerMOT updates tracks through two matching stages, prioritizing 3D evidence before associating remaining instances in the 2D image domain. Its association metric incorporates both box distance and orientation, while partial observations can still update track states.
- First-stage association: 3D detections are first greedily matched to existing tracks using a scaled distance between observed and predicted oriented bounding boxes.The distance combines Euclidean box distance with normalized cosine distance between orientation vectors.
- First-stage association: Orientation-aware association was more robust than 3D IoU and Mahalanobis distance, especially at NuScenes’ low frame rate.The comparison is reported experimentally for the proposed association criterion.
- Second-stage association: After 3D matching, unmatched 2D-capable instances are greedily associated with remaining tracks using 2D bounding-box IoU.Tracks use projected predicted 3D boxes when available, otherwise their last observed 2D boxes.
- Second-stage association: Tracks outside reliable LiDAR coverage can rely on observed 2D boxes because insufficient 3D evidence prevents reliable motion-model initialization.The authors identify learned appearance or 2D-state prediction models as possible future improvements.
- State update: Matched instances update tracks with new 2D and/or 3D state information, while 3D states are filtered with a constant-velocity linear Kalman filter.Track states can be updated from only 3D information, only 2D information, or both.
C. Track lifecycle
EagerMOT manages trajectories with rules for discarding, confirming, and initializing tracks based on recent multimodal updates.
- Track lifecycle: A track is discarded when it has not received any 2D or 3D instance update during the last Agemax frames.The lifecycle rule treats either modality as a valid update.
- Track lifecycle: A track is confirmed when it is associated in the current frame and has received 2D information within the last Age2d frames.This reflects the stated lower precision of 3D object detectors relative to image-based detectors.
IV. EXPERIMENTAL EVALUATION
The evaluation covers KITTI and NuScenes across 3D MOT, 2D MOT, and MOTS settings, using benchmark-specific metrics and detector configurations. Comparisons include published methods and detector-dependent baselines.
- Evaluation scope: EagerMOT is evaluated on NuScenes 3D MOT, KITTI 3D MOT, KITTI 2D MOT, and KITTI MOTS.The study uses two datasets and official benchmarks where available.
- Benchmark results: Table II reports KITTI 3D MOT validation results under the protocol of [35], with detector-specific markers and baselines from [36].Several methods report results only for the car class.
- Evaluation metrics: The evaluation focuses on MOTA, with AMOTA and AMOTP for KITTI 3D MOT and MOTSA and MOTSP for MOTS.KITTI 2D MOT benchmark results use HOTA under its updated evaluation protocol.
- Detector configurations: NuScenes experiments use CenterPoint 3D detections and Cascade R-CNN 2D detections, while KITTI experiments use Point-GNN or Point R-CNN and specified 2D detectors.The method accepts detector outputs without pre-filtering the KITTI 3D detections.
- Benchmark results: Table III reports KITTI 2D MOT results while noting that methods use different object detectors.EagerMOT uses RRC for cars, matching the detector used by MOTSFusion and BeyondPixels.
A. Ablation studies
Ablations show that 2D detections and the proposed association function materially affect tracking performance, while detector quality also influences results. The multimodal design improves recall but can reduce localization precision when relying on predictions.
- Data association: 0.712 AMOTA for “Full” versus 0.651 AMOTA for “No 2D info” demonstrates the impact of 2D detections on NuScenes validation performance.The full model improves recall by +0.054 while AMOTP decreases by −0.018.
- Data association: +0.004 AMOTA over 2D distance and +0.036 AMOTA over 3D IoU show greater robustness for the proposed association function.The paper attributes 3D IoU’s weaker suitability to NuScenes’ significantly lower scan rate than KITTI’s.
- Detection sources: Better object detectors lead to better tracking performance, while EagerMOT can use off-the-shelf detectors without expensive tracker-specific training.The ablation explicitly studies the effect of detection quality on overall performance.
B. Benchmark results
EagerMOT performs across NuScenes and KITTI MOT benchmarks by combining complementary 2D and 3D evidence, improving association and maintaining tracks when one modality lacks detections.
- NuScenes is evaluated as a challenging 3D MOT benchmark with diverse object classes and a 2 FPS frame rate.
- +5.15 and +7.79 sAMOTA on KITTI 3D MOT for car and pedestrian classes, respectively, demonstrate gains over AB3DMOT using the same 3D detections.
- On KITTI 2D MOT, EagerMOT achieves state-of-the-art HOTA despite tracking in 3D and using 2D detections only as a secondary cue.Image detections update tracks when 3D detections fail because of signal sparsity or occlusion; car association accuracy reaches 74.16 AssA.
- MOTS adapts the approach by passing segmentation masks from associated instances to tracks after data association.
- For KITTI MOTS, EagerMOT improves over MOTSFusion by +1.03 for cars and +3.61 for pedestrians using the same 2D detections and segmentation masks.It reports HOTA of 74.66 for cars and 57.65 for pedestrians, while additionally using LiDAR-based 3D detections.
C. Runtime discussion
Runtime varies with sensor configuration: EagerMOT runs at 4 FPS on multi-camera NuScenes and 90 FPS on single-camera KITTI, excluding detection and ego-motion estimation.
- 4 FPS is achieved on NuScenes by the Python implementation, excluding object detection and ego-motion estimation.This is slower but more accurate than LiDAR-only StanfordIPRL-TRI and AB3DMOT, which run at 10 FPS.
- 90 FPS is achieved on KITTI because the setup uses one camera and avoids multi-camera association.This exceeds reported rates for GNN3DMOT at 5 FPS, mmMOT at 4 FPS, and FANTrack at 25 FPS.
- KITTI configuration uses a single camera with fixed fusion, 3D, and 2D thresholds and track-age limits for both classes.
- NuScenes uses class-dependent 3D thresholds and 2D age limits, reflecting its seven-class multi-camera configuration.
- For 3D MOT, confirmed tracks retain original confidence scores, while unconfirmed-track scores are halved for each frame without 2D updates.
V. CONCLUSION
The paper concludes that EagerMOT consistently supports sensor-fusion tracking across datasets, tasks, detector choices, and sensory configurations. The authors position it as a baseline for future sensor-fusion MOT research.
- EagerMOT leverages detections from varying sensor modalities through a two-stage association procedure.
- The method performs consistently well across datasets and tracking tasks and works with varied object detectors without detector-specific fine-tuning.
- The authors intend the framework to serve as a baseline for future research in sensor-fusion-based multi-object tracking.