Source-linked AI summary
Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification
Long Chen, Haizhou Ai, Zijie Zhuang, Chong Shang
TL;DR
The paper targets unreliable detection and difficult data association in online multi-object tracking. It combines detection and track candidates, scores them with an efficient fully convolutional network, and uses learned ReID features; experiments report real-time, state-of-the-art MOT16 performance.
Problem
Tracking-by-detection must associate unreliable detections with existing tracks despite occlusion, missing detections, false positives, and inaccurate bounding boxes.
Method
The framework selects candidates from detection and tracking outputs using an efficient fully convolutional scoring network and learned ReID appearance representations for association.
Results
The proposed tracker achieves real-time and state-of-the-art performance on the MOT16 benchmark, with the combined scoring function and ReID features best across all reported validation metrics.
Takeaways & Limitations
Combining detection and track candidates addresses missing detections, while learned ReID features enhance identification under intra-category occlusion.
Takeaways & Limitations
Track candidates are used only after at least two observed detections are available to construct a reasonable motion model.
Abstract
from arXiv · showhide
Online multi-object tracking is a fundamental problem in time-critical video analysis applications. A major challenge in the popular tracking-by-detection framework is how to associate unreliable detection results with existing tracks. In this paper, we propose to handle unreliable detection by collecting candidates from outputs of both detection and tracking. The intuition behind generating redundant candidates is that detection and tracks can complement each other in different scenarios. Detection results of high confidence prevent tracking drifts in the long term, and predictions of tracks can handle noisy detection caused by occlusion. In order to apply optimal selection from a considerable amount of candidates in real-time, we present a novel scoring function based on a fully convolutional neural network, that shares most computations on the entire image. Moreover, we adopt a deeply learned appearance representation, which is trained on large-scale person re-identification datasets, to improve the identification ability of our tracker. Extensive experiments show that our tracker achieves real-time and state-of-the-art performance on a widely used people tracking benchmark.
1. INTRODUCTION
People tracking is challenged by occlusion and unreliable detections in tracking-by-detection systems. The paper addresses these issues online by selecting candidates from both detections and tracks with deep scoring and learned appearance features.
- Motivation: Tracking-by-detection associates detections across frames, but occlusion, missing detections, false positives, and inaccurate bounding boxes create data-association ambiguities.Similar object appearances and intra-category occlusion further complicate association.
- Approach: The framework selects candidates from both detection and tracking outputs in each frame to handle unreliable detection online.This produces redundant candidates that can complement one another across scenarios.
- Approach: Reliable track predictions support short-term association when detections are missing or inaccurate, while confident detections help prevent long-term tracking drift.The complementary roles motivate retaining both candidate sources.
- Approach: A fully convolutional neural network provides deep candidate scoring while sharing computations across the image for real-time selection.The contribution is framed as selecting optimal candidates from a considerable candidate set with deep neural networks.
- Contributions: The tracker combines hierarchical association using spatial information and deeply learned person re-identification features to improve identification under intra-category occlusion.The paper reports real-time and state-of-the-art performance on a widely used people-tracking benchmark.
2. RELATED WORK
Prior work uses tracking-by-detection, batch optimization, multiple detectors, and person re-identification, while this framework emphasizes an online ReID-based approach.
- Tracking-by-Detection: Tracking-by-detection methods commonly associate tracklets with detections, while multiple-detector over-detection can improve tracking but is inefficient for real-time applications.The proposed framework instead uses one detector and candidates from existing tracks.
- Batch Methods: Batch tracking methods use future frames or backward tracklets to stabilize similarity measurements and reduce noisy-detection ambiguities.Person re-identification has also been explored within global optimization methods.
- Person Re-Identification: This framework leverages deeply learned ReID features online to improve identification when objects with similar category appearances are occluded.Its use of ReID is positioned within an online rather than global-optimization setting.
3. PROPOSED METHOD
The framework selects candidates from both detections and track predictions, then hierarchically associates selected candidates using spatial and deeply learned appearance information. Its fully convolutional classifier shares image-wide computation while position-sensitive pooling preserves object structure.
- Candidate selection and association: Candidates come from both detections and existing tracks, forming the inputs to sequential candidate selection and data association.This redundancy lets detection and tracking outputs complement each other across unreliable-detection scenarios.
- Real-time object classification: The R-FCN classifier predicts image-wide score maps with a lightweight encoder and upsampling decoder for efficient candidate classification.Shared computation avoids separately classifying heavily overlapping candidate image patches.
- Real-time object classification: Position-sensitive RoI pooling divides each candidate into k × k bins and extracts one corresponding response from each of k^2 score maps.This preserves spatial information that simple averaging of foreground probabilities inside a region can lose.
- Tracklet confidence and scoring: Tracklet confidence decreases with the number of post-detection track predictions and requires at least two associated detections before a track becomes a candidate.The confidence reflects Kalman-filter reliability using temporal tracklet information.
- Tracklet confidence and scoring: A unified score fuses classification probability with tracklet confidence, after which non-maximal suppression removes redundant candidates using score and IoU thresholds.Detection candidates and track candidates are distinguished as Cdet and Ctrk, while strk penalizes uncertain tracks.
- Appearance representation and association: The appearance model extracts feature vectors with a deep network trained on large-scale person re-identification datasets and compares candidates using Euclidean distance.Hierarchical association first uses appearance for detection candidates, then IoU for remaining candidates; track appearance is updated only with associated detections.
4. EXPERIMENTS
Experiments on MOT16 evaluate the framework’s components, appearance representations, efficiency, and test-set performance. The results support unified candidate scoring with ReID features as a real-time approach.
- 4.1. Experiment Setup: The validation study uses five MOT16 training sequences, while final evaluation compares the tracker with state-of-the-art methods on the test set.MOT16 contains separate training and test sets, each with seven challenging unconstrained video sequences.
- 4.2. Analysis on Validation Set: Adding classification probability for candidates from detection and tracks improves MOTA by 4.6% over the baseline.The baseline uses Kalman-filter prediction and IoU-based association with detections.
- 4.2. Analysis on Validation Set: Combining tracklet confidence with classification probability further improves MOTA and reduces false positives.The confidence term penalizes candidates from uncertain tracks.
- 4.2. Analysis on Validation Set: ReID appearance representations significantly improve identification performance, evaluated using IDF1 and IDS, and the combined system performs best across all metrics.The appearance representations are compared with hand-crafted features such as color histograms and HOG.
- 4.2. Analysis on Validation Set: The R-FCN-based approach is more time efficient than patch classification because it shares computations across the entire image.Figure 3 measures average per-frame time on MOT16-03, which contains over 50 people per frame.
- 4.3. Evaluation on Test Set: On the MOT16 test set, the tracker runs in real time, outperforms existing online trackers on most metrics, and is about 5~20 times faster than most existing methods.The strongest reported improvements include IDF1, IDR, MT, and ML; all trackers use the benchmark’s detections.
5. CONCLUSION
The paper concludes with an online people-tracking framework that selects candidates from detection and track outputs using an efficient R-FCN scoring function. ReID features improve identification under intra-category occlusion, yielding real-time, state-of-the-art MOT16 performance.
- 5. CONCLUSION: The framework addresses unreliable detection by selecting candidates from both detection and tracking outputs.This supplies redundant candidates that can complement one another across tracking scenarios.
- 5. CONCLUSION: An efficient R-FCN shares computations across the image to formulate the candidate-selection scoring function.The method is designed to support real-time candidate selection.
- 5. CONCLUSION: ReID features improve identification during intra-category occlusion, and the learned representations outperform traditional hand-crafted features by a large margin.The features are trained using a data-driven approach on person re-identification data.
- 5. CONCLUSION: The proposed tracker achieves real-time and state-of-the-art performance on the MOT16 benchmark.The conclusion states this as the overall benchmark outcome.