Source-linked AI summary
DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association
Xiyang Wang, Chunyun Fu, Zhankun Li, Ying Lai, Jiawei He
TL;DR
The paper addresses data-association challenges and the accuracy-speed trade-off in 3D MOT. It uses camera-LiDAR fusion with deep association to track objects across 2D and 3D, and reports strong tracking results on KITTI and nuScenes.
Problem
3D MOT still faces false negatives and false positives from occlusions, while existing methods can trade tracking accuracy against computation speed and struggle to use both sensors fully.
Source: §I · p. 1, B. Problem Statement · p. 2Method
Results
On nuScenes, the method achieves the highest AMOTA (63.5%), MOTA (53.3%) and Recall (69.6%); on KITTI, it achieves the highest HOTA (75.46%), highest AssA (80.06%) and least IDSW (84).
Source: B. Experimental Results · p. 5Takeaways & Limitations
The authors report that motion-only association remains highly robust, works with a variety of object detectors, and delivers fast tracking speed.
Source: §V · p. 7
Abstract
from arXiv · showhide
In the recent literature, on the one hand, many 3D multi-object tracking (MOT) works have focused on tracking accuracy and neglected computation speed, commonly by designing rather complex cost functions and feature extractors. On the other hand, some methods have focused too much on computation speed at the expense of tracking accuracy. In view of these issues, this paper proposes a robust and fast camera-LiDAR fusion-based MOT method that achieves a good trade-off between accuracy and speed. Relying on the characteristics of camera and LiDAR sensors, an effective deep association mechanism is designed and embedded in the proposed MOT method. This association mechanism realizes tracking of an object in a 2D domain when the object is far away and only detected by the camera, and updating of the 2D trajectory with 3D information obtained when the object appears in the LiDAR field of view to achieve a smooth fusion of 2D and 3D trajectories. Extensive experiments based on the typical datasets indicate that our proposed method presents obvious advantages over the state-of-the-art MOT methods in terms of both tracking accuracy and processing speed. Our code is made publicly available for the benefit of the community.
I. INTRODUCTION
The introduction highlights unresolved data-association challenges and the complementary sensing limits of cameras and LiDAR. It presents a deep-association camera-LiDAR method that tracks distant objects in 2D and fuses 3D information when available.
I. INTRODUCTION
- False negatives and false positives caused by occlusions remain data-association challenges for multi-object tracking.Source: §I · p. 1
- Existing work has largely focused on camera-based 2D/3D tracking or LiDAR-based 3D tracking, with few methods fusing both sensors for 3D MOT.Source: §I · p. 1
- Camera-LiDAR fusion tracks objects in 2D when only the camera detects them, then uses LiDAR information to update their trajectories in 3D.The method is described as achieving a smooth fusion of 2D and 3D trajectories.Source: §I · p. 1
- Figure 1 compares KITTI 2D MOT trackers on HOTA and speed; the proposed method is described as optimal in HOTA and very fast.Source: Figure 1 · p. 1, §I · p. 1
A. Literature Review
Camera-based MOT commonly tracks in 2D using image features, while LiDAR-based MOT provides depth for 3D tracking. The review notes limitations of each sensor and the complexity of existing fusion approaches.
Camera-based Multi-Object Tracking
- Camera-based tracking commonly uses image appearance and motion features for 2D association, while stereo cameras can support 3D tracking using depth.Many early methods use detection, trajectory prediction, cost-matrix construction, and Hungarian matching.Source: A. Literature Review · p. 2
- Camera-only methods usually lack depth, and stereo depth is less accurate than LiDAR depth while typically requiring substantial computation.Source: A. Literature Review · p. 2
LiDAR-based Multi-Object Tracking
- LiDAR-based methods provide depth for 3D tracking, but LiDAR lacks pixel information for rich appearance features and has difficulty detecting distant objects.Source: A. Literature Review · p. 2
Fusion-based Multi-Object Tracking
- Prior camera-LiDAR fusion methods combine image and point-cloud information, but one reviewed approach does not properly handle 2D trajectories or adequately update them with 3D information.The review describes methods using mixed-integer programming, deep structured models, and two-stage association.Source: A. Literature Review · p. 2
- Existing fusion methods often use complex feature-extraction networks and require powerful GPUs, making real-time implementation difficult.Source: A. Literature Review · p. 2
B. Problem Statement
The problem statement identifies gaps in depth, distant-object tracking, real-time computation, and sensor-data fusion. The contributions are a fast camera-LiDAR tracking framework with a lightweight deep-association mechanism and detector flexibility.
B. Problem Statement
- Camera-based MOT commonly lacks depth for 3D tracking, while LiDAR-based tracking struggles with distant objects and fusion methods can be too computationally complex for real-time use.Source: B. Problem Statement · p. 2
- Projecting LiDAR detections onto images can discard pixel information for image objects that the LiDAR does not detect.Source: B. Problem Statement · p. 3
C. Original Contributions
- The proposed camera-LiDAR 3D tracking framework is designed for fast, robust operation and superior MOT performance on typical tracking datasets.Source: C. Original Contributions · p. 3
- Its deep-association mechanism fuses 2D and 3D trajectories without complex cost functions or feature-extraction networks.Source: C. Original Contributions · p. 3
- The framework is described as suitable for real-time use and compatible with arbitrary 2D and 3D detectors without additional training.Source: C. Original Contributions · p. 3
D. Outline of the Paper
The paper outlines its progression from system architecture and method details to experiments and conclusions. Its architecture takes camera and LiDAR detections through sensor fusion and four-level association to produce 2D and 3D trajectories.
D. Outline of the Paper
- The paper proceeds from the framework architecture to method details, comparisons with state-of-the-art trackers, and conclusions.Source: D. Outline of the Paper · p. 3
II. SYSTEM ARCHITECTURE
- The architecture has three parts: inputs, deep association, and outputs, using camera-based 2D and LiDAR-based 3D detections.Source: §II · p. 3
- LiDAR 3D boxes are projected into image coordinates, and their IoU with camera detections is thresholded to fuse sensor information.Source: §II · p. 3
- Four association levels prioritize detections from both sensors, then handle unmatched 3D tracks, camera-only detections with 2D tracks, and finally fusion of projected 3D tracks with 2D tracks.Source: §II · p. 3
- The framework outputs both 3D and 2D trajectories for tracked objects.Source: §II · p. 3
III. PROPOSED METHOD
The method takes camera and LiDAR detections as input, projects 3D boxes into the image domain, and uses 2D-box overlap to distinguish fused detections from sensor-only detections.
A. Camera-LiDAR Fusion
- A 2D detector extracts objects from images, while a 3D detector extracts objects from point clouds.The resulting 2D and 3D bounding-box information is passed to the tracking framework.Source: A. Camera-LiDAR Fusion · p. 3
- The architecture diagram depicts the proposed camera-LiDAR fusion-based method.The method uses the camera and LiDAR detections as inputs to its tracking framework.Source: Figure 2 · p. 3, A. Camera-LiDAR Fusion · p. 3
- The method projects each 3D box into the image domain and compares it with the 2D box using IoU.IoU below a threshold yields camera-only and LiDAR-only objects; IoU above it yields detections present in both domains.Source: A. Camera-LiDAR Fusion · p. 4
B. Deep Association
The four-level association mechanism combines camera and LiDAR detections to handle association challenges, including reappearing objects and distant objects not yet detected by LiDAR. Experiments on KITTI and nuScenes report low ID switching and fast computation speed.
B. Deep Association
- The proposed deep association mechanism targets ID switches after occlusion, detector-related false negatives and positives, and inaccurate tracking of distant objects with LiDAR alone.The mechanism comprises four levels of data association.Source: B. Deep Association · p. 4
- At level one, fused detections are matched to existing 3D trajectories using a cost that combines 3D IoU and normalized Euclidean distance.The distance term addresses cases where fast motion after a brief disappearance makes IoU zero.Source: B. Deep Association · p. 4
- At level two, unmatched 3D trajectories are matched with LiDAR-only detections, and unmatched detections become tentative trajectories.A tentative trajectory is confirmed only after successful association for three consecutive frames.Source: B. Deep Association · p. 4
- At level three, a separate image-domain tracker associates 2D trajectories with camera-only detections of distant objects.When an object enters LiDAR's field of view, its 2D trajectory is updated using 3D detector information for 3D tracking.Source: B. Deep Association · p. 4
- At level four, unmatched 3D trajectories are associated with image-domain trajectories using projected-box IoU, then fused into new 3D trajectories.The fused trajectory takes the 2D trajectory's ID, appearance-frame count, and state.Source: B. Deep Association · p. 4
- Experiments on KITTI and nuScenes report superior tracking performance, including low ID switching and fast computation speed.The reported comparison concerns the proposed deep association mechanism.Source: B. Deep Association · pp. 4–5
C. Track Management
The track-management approach adds a reappeared state to the adopted trajectory-management scheme.
C. Track Management
- Trajectories can be dead, tentative, confirmed, or reappeared.The paper adopts the trajectory-management approach in and adds the reappeared state.Source: C. Track Management · p. 5
- A confirmed trajectory becomes reappeared after occlusion leaves it unassociated for several frames.The state distinguishes an occluded track from one that has become dead.Source: C. Track Management · p. 5
- A reappeared trajectory becomes dead if it remains unassociated for consecutive frames beyond a threshold.The paper describes this as disappearance from the sensor field of view.Source: C. Track Management · p. 5
IV. EXPERIMENTS
The section presents comparative experimental results to illustrate the effectiveness of the proposed 3D tracking framework.
IV. EXPERIMENTS
- The section presents comparative experimental results for the proposed 3D tracking framework.The passage identifies comparison as the form of the experiments.Source: §IV · p. 5
- The stated purpose of the experiments is to illustrate the framework's effectiveness.No specific outcome is given in this passage.Source: §IV · p. 5
- The experiments concern the proposed 3D tracking framework.The passage does not report particular metrics or comparisons.Source: §IV · p. 5
A. Experimental Setup
Experiments use KITTI and nuScenes with established detectors and tracking metrics to evaluate the proposed method against state-of-the-art trackers.
A. Experimental Setup
- The method is tested on KITTI and nuScenes, with KITTI results reported for training and testing sets and nuScenes results described as good.The comparison includes state-of-the-art MOT methods.Source: A. Experimental Setup · p. 5
- KITTI experiments use RRC for 2D detection and PointRCNN for 3D detection; nuScenes experiments use Cascade RCNN and CenterPoint, respectively.These detector choices are used for comparison with existing literature.Source: A. Experimental Setup · p. 5
- Evaluation uses CLEAR and HOTA metrics for 2D MOT and AMOTA and sAMOTA for nuScenes 3D tracking.CLEAR includes MOTA, MOTP, and IDSW; HOTA includes Detection Accuracy and Association Accuracy.Source: A. Experimental Setup · p. 5
B. Experimental Results
The method achieves leading results on nuScenes and KITTI benchmarks, while qualitative examples illustrate early camera-based tracking and ID continuity after occlusion. Ablations report a speed–accuracy trade-off for appearance features and generally stable tracking across detector choices.
B. Experimental Results
- The benchmark comparison covers KITTI Car and seven nuScenes categories, using leaderboard results collected on the dates specified in the paper.Table II methods marked with ‘#’ use the proposed method’s RRC 2D detector and/or PointRCNN 3D detector.Source: B. Experimental Results · p. 5, Table II · p. 6
- 63.5% AMOTA, 53.3% MOTA, and 69.6% Recall are the highest on nuScenes; KITTI results lead in HOTA (75.46%), AssA (80.06%), and IDSW (84).On KITTI, HOTA improves by 5.65% over AB3DMOT and by 1.07% over EagerMOT; AssA improves by 5.90% over EagerMOT, whose IDSW is 239.Source: Table I · p. 5, B. Experimental Results · p. 5, Table II · p. 6
- Tracking starts at frame 9 using the proposed method, versus frame 35 for AB3DMOT, after 2D detection begins at frame 7 and 3D detection at frame 33.The 2D track is updated with 3D information once the vehicle enters LiDAR detection range.Source: B. Experimental Results · p. 6
- After six consecutive occluded frames, the vehicle with ID 8214 is re-detected at frame 71 and tracking resumes without an ID switch.The example is from KITTI testing sequence 0010.Source: B. Experimental Results · p. 6, Figure 4 · p. 7
- Adding appearance information slightly improves HOTA and MOTA but greatly reduces computing speed, so the current method excludes appearance features.The authors attribute modest tracking gains partly to the point-cloud basis and the relatively simple appearance extractor.Source: B. Experimental Results · p. 7, Table III · p. 7
- Tracking performance stays generally stable across detector choices, although Point-GNN with Yolov3 performs worst and PointRCNN with Yolov3 is fastest.The authors report that detector choices affect results without causing significant overall variation.Source: B. Experimental Results · p. 7, Table IV · p. 7
V. CONCLUSION AND FUTURE WORK
The framework combines camera and LiDAR through motion-only deep association, achieving strong tracking results and a balance of speed and accuracy. Future work will add appearance and ego-motion information, refine detections, and improve trajectory management.
V. CONCLUSION AND FUTURE WORK
- On the tested dataset, the method achieves the highest HOTA, the highest AssA, and the least IDSW among the compared state-of-the-art MOT methods.Source: §V · p. 7
- Motion-only association remains robust in ablations, works with varied object detectors, handles occlusions, and supports fusion of 2D and 3D trajectories at fast tracking speed.The framework is described as balancing computing speed and tracking accuracy for real-time autonomous-driving MOT.Source: §V · p. 7
- Future work will add LiDAR and visual appearance features to motion-based association, incorporate GPS/IMU data to compensate for ego-motion prediction errors, refine detections, and investigate more efficient trajectory management.Source: §V · p. 7