Source-linked AI summary
GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning
Xinshuo Weng, Yongxin Wang, Yunze Man, Kris Kitani
TL;DR
Online MOT must learn discriminative features for affinity-based matching while handling complementary appearance and motion information across 2D and 3D spaces. The paper introduces GNN feature interaction, a joint four-branch extractor, and ensemble training, achieving state-of-the-art 3D MOT performance on KITTI and nuScenes. A reported scope limitation is that the method’s hyperparameters were tuned for 3D MOT rather than 2D MOT.
Problem
Online MOT needs discriminative object features for affinity computation and Hungarian data association.
Method
The method combines GNN feature interaction with a joint extractor for 2D and 3D appearance and motion, using ensemble training across four branches.
Results
State-of-the-art 3D MOT performance is reported on both KITTI and nuScenes.
Takeaways & Limitations
Joint multi-modal features and feature interaction improve the discriminative feature learning targeted by the method.
Takeaways & Limitations
The method’s hyperparameters and ablations were tuned for 3D MOT, while its 2D evaluation uses lower-quality projected detections unless replaced.
Abstract
from arXiv · showhide
3D Multi-object tracking (MOT) is crucial to autonomous systems. Recent work uses a standard tracking-by-detection pipeline, where feature extraction is first performed independently for each object in order to compute an affinity matrix. Then the affinity matrix is passed to the Hungarian algorithm for data association. A key process of this standard pipeline is to learn discriminative features for different objects in order to reduce confusion during data association. In this work, we propose two techniques to improve the discriminative feature learning for MOT: (1) instead of obtaining features for each object independently, we propose a novel feature interaction mechanism by introducing the Graph Neural Network. As a result, the feature of one object is informed of the features of other objects so that the object feature can lean towards the object with similar feature (i.e., object probably with a same ID) and deviate from objects with dissimilar features (i.e., object probably with different IDs), leading to a more discriminative feature for each object; (2) instead of obtaining the feature from either 2D or 3D space in prior work, we propose a novel joint feature extractor to learn appearance and motion features from 2D and 3D space simultaneously. As features from different modalities often have complementary information, the joint feature can be more discriminate than feature from each individual modality. To ensure that the joint feature extractor does not heavily rely on one modality, we also propose an ensemble training paradigm. Through extensive evaluation, our proposed method achieves state-of-the-art performance on KITTI and nuScenes 3D MOT benchmarks. Our code will be made available at https://github.com/xinshuoweng/GNN3DMOT
1. Introduction
The paper targets discriminative feature learning in online MOT by replacing isolated object features with graph-based interaction and jointly learning complementary 2D and 3D appearance and motion features.
- Motivation: Online MOT extracts object features independently, computes pairwise similarities, and uses the resulting affinity matrix for Hungarian data association.The pipeline’s key challenge is learning discriminative features for objects with different identities.
- Motivation: Independent feature extraction is sub-optimal because pairwise similarities should be interdependent when each current-frame object matches at most one previous-frame object.Increasing similarity between a likely matching pair should reduce similarity with other objects to avoid association confusion.
- Feature interaction: GNN-based feature interaction updates each object node by aggregating features from other nodes, making features responsive to the surrounding objects.The authors observe that the affinity matrix becomes more discriminative after several GNN layers.
- Joint feature learning: The joint extractor learns appearance and motion features simultaneously from both 2D and 3D spaces because these modalities provide complementary information.The motivation includes cases where objects are close in images but separated in 3D because of depth differences.
- Joint feature learning: Four branches model 2D appearance, 2D motion, 3D appearance, and 3D motion; ensemble training randomly turns off branches to reduce reliance on one modality.The branches are fused before GNN feature interaction, and the network is end-to-end trainable.
- Results: The method combines GNN-based feature interaction with a joint 2D–3D extractor and reports state-of-the-art performance on KITTI and nuScenes 3D MOT benchmarks.The contribution summary also reports competitive performance on corresponding 2D MOT benchmarks.
2. Related Work
Related work frames online MOT as detection, discriminative feature learning, and bipartite matching, then positions this paper’s joint multi-feature and GNN approach against prior methods.
- Online Multi-Object Tracking: Online MOT performance is mainly affected by detection quality and discriminative feature learning before affinity-based Hungarian matching.Prior comparisons often use identical detection results to isolate feature-learning performance.
- Feature selection: Prior methods use motion and appearance features, including handcrafted distances, IoU, color histograms, CNN appearance features, and learned motion features.The related work describes both filter-based and deep-learning-based approaches for motion representation.
- Feature selection: The closest feature-selection work jointly learns 2D and 3D appearance features but omits motion cues, ensemble training, and feature interaction.The paper contrasts its four-branch, ensemble-trained extractor and GNN mechanism with that prior approach.
- Graph Neural Networks: GNNs process graph-structured data through node feature aggregation and have succeeded in segmentation, recognition, tracking, re-identification, and point-cloud tasks.The paper introduces this aggregation mechanism as the basis for feature interaction in MOT.
- Graph Neural Networks: The authors report no prior GNN application to MOT and distinguish their generic spatial-temporal, multi-modal interaction framework from relation and temporal trajectory networks.Those earlier networks model detection context or past trajectories rather than interactions among object features across modalities.
3. Approach
The approach combines joint 2D/3D appearance and motion features with GNN-based interaction for online data association. Fused object features form a sparse cross-frame graph whose node updates and edge regression produce affinities for matching.
- Feature Fusion: Features from the four branches are fused before graph construction, with concatenation used in the final network.The evaluated fusion operators are concatenation and addition; the final model uses concatenation.
- Ensemble Training Paradigm: During training, the ensemble paradigm randomly drops one to three branches while retaining at least one, reducing reliance on any single feature branch.A random branch-drop combination is sampled at each training iteration.
- Graph Construction: The fused features form a graph with M+N nodes, and edges connect nodes across frames when their 3D and 2D centers fall within distance thresholds.This produces sparse cross-frame connections rather than a fully connected graph.
- Graph Neural Network for Data Association: The GNN iteratively aggregates neighboring node features and uses edge regression to compute the M × N affinity matrix for matching.The final edge-regression metric is a two-layer MLP that maps differences between node features to similarity scores between 0 and 1.
4. Experiments
Experiments evaluate GNN3DMOT on KITTI and nuScenes using standard 3D MOT metrics, implementation settings, benchmark comparisons, and ablations. The method achieves state-of-the-art 3D MOT results, while ablations examine multimodal features, fusion, edge regression, aggregation, ensemble training, and GNN depth.
- Evaluation Setup: KITTI and nuScenes provide 2D and 3D data, enabling evaluation of 3D MOT and reference 2D MOT performance.KITTI results use the car subset, while nuScenes results average across all categories.
- Evaluation Setup: The evaluation uses CLEAR metrics, including MOTA, MOTP, IDS, FRAG, and FPS, together with sAMOTA, AMOTA, and AMOTP.3D MOT evaluations use the tool from [48], with validation sets used where test ground truth is unavailable.
- Benchmark Results: GNN3DMOT consistently outperforms modern 3D MOT systems on KITTI, while its KITTI 2D MOT result is 80.40 2D MOTA.The compared 3D MOT systems include FANTrack, mmMOT, and AB3DMOT.
- Benchmark Results: GNN3DMOT achieves state-of-the-art 3D MOT performance on nuScenes and runs at 5.2 FPS on the KITTI test set using one 1080Ti GPU.nuScenes 3D MOT performance is lower than KITTI because nuScenes 3D detection performance is less mature.
- Ablation Study: Combining appearance and motion features improves performance, concatenate fusion outperforms add fusion, and a two-layer MLP outperforms conventional similarity metrics for edge regression.These findings come from KITTI-Car validation ablations using the 3D MOT evaluation tool.
- Ablation Study: Type 4 node aggregation performs best; ensemble training is strongest at r=0.5; and three GNN layers produce the best performance.Feature interaction also improves results when using outputs from later GNN layers, while deeper GNNs tend to overfit.
5. Conclusion
The paper proposes a 3D MOT method combining joint 2D–3D feature extraction with GNN-based feature interaction, and reports state-of-the-art performance on KITTI and nuScenes.
- The method combines a joint 2D–3D feature extractor with GNN-based feature interaction for 3D MOT.
- The proposed approach is evaluated through extensive experiments on KITTI and nuScenes datasets.
- The paper reports state-of-the-art 3D MOT performance on both datasets.