Source-linked AI summary
Towards Real-Time Multi-Object Tracking
Zhongdao Wang, Liang Zheng, Yixuan Liu, Yali Li, Shengjin Wang
TL;DR
MOT systems that separately detect targets and compute appearance embeddings can be inefficient, while prior real-time efforts mainly accelerate association. This paper introduces JDE, which jointly learns detection and embeddings in a single-shot network; it reaches near-real-time speed with tracking accuracy comparable to SDE systems.
Problem
Separate detection and embedding models make MOT inference inefficient, while existing real-time research largely addresses association rather than the complete MOT system.
Method
JDE jointly learns target detection and appearance embeddings in a shared single-shot network, using shared features and multiscale prediction architecture.
Results
20.2 FPS with MOTA=64.4% was achieved on MOT-16, compared with <6 FPS and MOTA=66.1% for Faster R-CNN + QAN embedding.
Takeaways & Limitations
JDE provides a near-real-time MOT baseline whose tracking accuracy is comparable to state-of-the-art online MOT methods.
Takeaways & Limitations
Inaccurate detections under large pedestrian overlap remain a major source of ID switches and lower IDF1 scores.
Abstract
from arXiv · showhide
Modern multiple object tracking (MOT) systems usually follow the \emph{tracking-by-detection} paradigm. It has 1) a detection model for target localization and 2) an appearance embedding model for data association. Having the two models separately executed might lead to efficiency problems, as the running time is simply a sum of the two steps without investigating potential structures that can be shared between them. Existing research efforts on real-time MOT usually focus on the association step, so they are essentially real-time association methods but not real-time MOT system. In this paper, we propose an MOT system that allows target detection and appearance embedding to be learned in a shared model. Specifically, we incorporate the appearance embedding model into a single-shot detector, such that the model can simultaneously output detections and the corresponding embeddings. We further propose a simple and fast association method that works in conjunction with the joint model. In both components the computation cost is significantly reduced compared with former MOT systems, resulting in a neat and fast baseline for future follow-ups on real-time MOT algorithm design. To our knowledge, this work reports the first (near) real-time MOT system, with a running speed of 22 to 40 FPS depending on the input resolution. Meanwhile, its tracking accuracy is comparable to the state-of-the-art trackers embodying separate detection and embedding (SDE) learning ($64.4\%$ MOTA \vs $66.1\%$ MOTA on MOT-16 challenge). Code and models are available at \url{https://github.com/Zhongdao/Towards-Realtime-MOT}.
1 Introduction
Modern MOT commonly separates detection and appearance embedding, creating a computational bottleneck for real-time systems. JDE addresses this by jointly learning both outputs in a single-shot network while retaining comparable tracking accuracy.
- Motivation: Tracking-by-detection separates target localization from association, requiring distinct detection and embedding components.The overall inference time is roughly the sum of these components.
- Motivation: Joint detection and embedding can share low-level features and avoid recomputation.A two-stage alternative integrates the tasks through Faster R-CNN, while JDE uses a single-shot network.
- Proposed Approach: JDE simultaneously outputs detection results and corresponding appearance embeddings from one network.The proposed framework is designed to improve MOT efficiency while remaining nearly as accurate as SDE methods.
- Results: 20.2 FPS with MOTA=64.4% was achieved on the MOT-16 test set, compared with under 6 FPS and MOTA=66.1% for Faster R-CNN + QAN embedding.These figures illustrate the speed–accuracy trade-off between JDE and a separate detection-and-embedding system.
- Results: JDE performs as well as strong SDE model combinations while achieving the fastest speed in experiments using the same training data.The experiments analyze training data, architecture, learning objectives, optimization, and validation metrics.
2 Related Work
Related MOT work largely improves data association while taking detection results and appearance embeddings as inputs. Other approaches explore alternative tracking paradigms, but their speed–accuracy trade-off remains a challenge without an additional embedding model.
- Related Work: Recent MOT methods primarily model association through graph optimization, end-to-end neural networks, or alternative tracking paradigms.The first two categories were prevailing solutions during the past decade.
- Association Modeling: Graph-based association represents detections as nodes and possible linkages as edges, then minimizes fixed or learned costs.More complex formulations include multi-cuts, minimum cliques, and graph networks.
- Alternative Paradigms: Alternative tracking paradigms such as spatial-offset prediction simplify tracking but provide unsatisfying accuracy unless an embedding model is added.The resulting performance–speed trade-off still needs improvement.
- Related Learning Strategies: Jointly learning auxiliary associative embeddings with a main task has also performed well in person search, human pose estimation, and point-based object detection.This connects the paper's joint-learning strategy to related vision tasks.
3 Joint Learning of Detection and Embedding
JDE jointly learns target detection and appearance embeddings in a single-shot network, using shared FPN features and multi-task objectives. Its embedding learning and online association are designed to support efficient tracking across scales.
- Problem formulation: JDE simultaneously predicts bounding boxes and appearance embeddings in one forward pass.The model outputs predicted boxes and embeddings for detected targets rather than sending boxes or feature maps to a separate re-ID model.
- Architecture overview: The FPN backbone produces predictions at three scales, with each head outputting classification, box regression, and dense embedding maps.The dense prediction map has size (6A + D) × H × W and is divided into the three task-specific outputs.
- Appearance embeddings: The appearance objective learns an embedding space where same-identity observations are closer than different-identity observations.The paper considers Euclidean or cosine distance and notes that satisfying the detection and embedding objectives enables simple association.
- Appearance embeddings: Cross-entropy is selected for embedding learning because it pulls an anchor away from all negative classes, unlike the sampled negatives used by the upper-bound loss.The loss comparison motivates using LCE as the embedding objective in JDE.
- Multi-task optimization: Automatic uncertainty-based loss balancing is adopted to weight detection and embedding tasks without restricting the search to a reduced set of loss weights.The paper states that reduced-space search can yield decent results but may produce weights far from optimal.
- Online association: Online association combines cosine appearance affinity and Mahalanobis motion affinity in a Hungarian assignment procedure.The association method maintains a tracklet pool and uses a combined cost matrix for matching observations to tracklets.
4 Experiments
Experiments evaluate joint training data, embedding losses, loss weighting, and tracking performance. JDE offers competitive accuracy with substantially higher speed than strong SDE combinations, while inaccurate boxes remain a limitation in crowded scenes.
- Datasets and evaluation: Six publicly available datasets are combined into a large-scale training set containing pedestrian bounding-box annotations and partial identity annotations.The datasets include pedestrian detection, MOT, and person-search data.
- Datasets and evaluation: The experiments evaluate detection accuracy, embedding discrimination, and complete MOT tracking performance using separate validation protocols.Detection uses AP at IoU 0.5; embedding evaluation uses ground-truth-box embeddings across multiple validation sets.
- Embedding losses: The cross-entropy embedding loss outperforms both triplet-loss variants by +46.0/+43.9 TAR@FAR=0.1.The authors attribute this to cross-entropy enforcing comparison against all negative classes rather than only sampled mini-batch negatives.
- Loss weighting: The weight-by-uncertainty strategy outperforms the other evaluated loss-weighting strategies, while Loss.Norm improves on Uniform but remains inferior to App.Opt.Uniform weighting lets the larger embedding loss dominate training and produces poor detection and tracking results.
- Tracking results: 22 FPS and 65.8% MOTA make JDE-DN53 nearly as accurate as Cascade-R101 + PCB at 6 FPS and 66.2% MOTA.JDE achieves a stronger speed-accuracy trade-off than strong SDE combinations, including in low- and high-density settings.
- Analysis and limitations: In crowded scenes, inaccurate single-stage detections drift under pedestrian overlap, causing embedding ambiguity, more ID switches, and lower IDF-1.Replacing the joint embedding with a separately learned embedding leaves IDF-1 and ID switches almost unchanged, pointing to detection accuracy as the major issue.
- Tracking results: 30.3 FPS is achieved at 864 × 408 with only a minor performance drop of Δ = -2.6% MOTA.At 1088 × 608, JDE reaches 22.2 FPS and is estimated to run at least 2 ∼3× faster than existing methods.
- Embedding analysis: Dense embeddings produce better query-target correspondence than detection feature maps in the retrieval visualization.Similarity maps are computed using pixel-wise cosine similarity.
5 Conclusion
JDE shares detection and appearance-feature learning in one MOT model, significantly reducing runtime while maintaining tracking accuracy comparable to state-of-the-art online methods.
- JDE learns target detection and appearance features in a shared model.
- The shared design significantly reduces MOT runtime, enabling near-real-time operation.
- JDE achieves tracking accuracy comparable to state-of-the-art online MOT methods.