Source-linked AI summary
Tracking without bells and whistles
Philipp Bergmann, Tim Meinhardt, Laura Leal-Taixe
TL;DR
Multi-object tracking must handle re-identification, motion, occlusions, and difficult detections, yet dedicated methods have yielded limited gains. Tracktor instead converts an object detector into an online tracker through bounding-box regression, then adds simple re-identification and motion extensions. The resulting system achieves state-of-the-art benchmark performance, while the analysis identifies complex tracking cases that remain unresolved.
Problem
Data association in tracking-by-detection must handle missing and spurious detections, occlusions, and target interactions, while recent models have improved MOT16 accuracy by only 2.4% in two years.
Method
Tracktor converts an object detector into an online tracker by exploiting its bounding-box regression, with re-identification and motion-model extensions.
Results
Tracktor achieves state-of-the-art performance on three challenging multi-object tracking benchmarks, and dedicated tracking methods are not substantially better than the regression approach in the analysis.
Takeaways & Limitations
The findings motivate a tracking paradigm that uses detector capabilities and focuses research on remaining complex tracking challenges.
Takeaways & Limitations
The core regression step assumes targets move only slightly between frames, an assumption usually ensured by high frame rates.
Abstract
from arXiv · showhide
The problem of tracking multiple objects in a video sequence poses several challenging tasks. For tracking-by-detection, these include object re-identification, motion prediction and dealing with occlusions. We present a tracker (without bells and whistles) that accomplishes tracking without specifically targeting any of these tasks, in particular, we perform no training or optimization on tracking data. To this end, we exploit the bounding box regression of an object detector to predict the position of an object in the next frame, thereby converting a detector into a Tracktor. We demonstrate the potential of Tracktor and provide a new state-of-the-art on three multi-object tracking benchmarks by extending it with a straightforward re-identification and camera motion compensation. We then perform an analysis on the performance and failure cases of several state-of-the-art tracking methods in comparison to our Tracktor. Surprisingly, none of the dedicated tracking methods are considerably better in dealing with complex tracking scenarios, namely, small and occluded objects or missing detections. However, our approach tackles most of the easy tracking scenarios. Therefore, we motivate our approach as a new tracking paradigm and point out promising future research directions. Overall, Tracktor yields superior tracking performance than any current tracking method and our analysis exposes remaining and unsolved tracking challenges to inspire future research directions.
1. Introduction
Multi-object tracking remains difficult because data association must handle missing and spurious detections, occlusions, and interactions, while recent complexity has produced only marginal gains. Tracktor challenges this paradigm by using an object detector’s regression head for tracking and extending it with simple components to achieve strong benchmark performance.
- Tracking-by-detection separates independent per-frame detection from linking corresponding detections across time.
- 2.4% improvement in multiple object tracking accuracy over two years on MOT16 illustrates the marginal gains of increasingly complex models.
- Tracktor uses an object detector’s bounding-box regression to perform temporal realignment of existing tracks.
- A re-identification Siamese network and motion model extend Tracktor, producing state-of-the-art performance on three challenging benchmarks.
- The analysis reports that dedicated tracking methods do not perform substantially better than the regression-based approach across challenging scenarios.
- The paper proposes Tracktor as a tracking paradigm that exploits detector capabilities while directing research toward unresolved complex tracking challenges.
2. A detector is all you need
Tracktor converts a regression-based object detector into an online multi-object tracker without tracking-specific training or complex test-time optimization. It propagates tracks through bounding-box regression, initializes new tracks from detections, and adds motion and re-identification extensions for harder conditions.
- Tracktor exploits a detector’s bounding-box regression to perform multi-object tracking without tracking-specific training or complex test-time optimization.
- A regression-based detector refines proposal locations through classification and bounding-box regression heads.
- The tracker defines trajectories as ordered bounding boxes for objects across video frames.
- Tracktor: For each frame, bounding-box regression extends active trajectories by regressing previous boxes on current-frame features and transfers their identities.
- Tracktor: Tracktor assumes targets move only slightly between frames, an assumption usually supported by high frame rates.
- Tracktor: Tracks are deactivated when classification scores fall below a threshold or when nonmaximum suppression identifies overlapping object tracks.
- Tracktor: New trajectories begin only for detections with insufficient overlap with active trajectories.
- Tracking extensions: Motion models and short-term Siamese-network re-identification extend Tracktor to address large camera motion, low frame rates, and identity preservation.
3. Experiments
The experiments evaluate Tracktor and Tracktor++ on pedestrian-tracking benchmarks, including ablations, benchmark comparisons, and analyses of tracker characteristics. Tracktor++ achieves state-of-the-art MOTA on all three MOTChallenge benchmarks without training or optimization on tracking data.
- Tracktor and Tracktor++ are evaluated on several pedestrian-tracking datasets, including challenging MOTChallenge sequences with frequent occlusions and crowded scenes.
- Ablation study: The ablation study examines detector-based tracking, improved detection, and extensions with re-identification and camera motion compensation.Combining the extensions yields Tracktor++.
- Benchmark evaluation: Tracktor++ achieves a new state-of-the-art in MOTA for public detections on all three MOTChallenge benchmarks.
- Benchmark evaluation: The benchmark evaluation uses no training or optimization on the tracking training set and applies the same Tracktor++ tracker across benchmarks.
- Benchmark evaluation: On MOT16, Tracktor outperforms other trackers by a large margin in false negatives and IDF1, while also generalizing to 2D MOT 2015 without its training sequences.
- Failure-case analysis: Visibility analysis compares the ratio of tracked objects across visibility levels against the ground-truth visibility distribution.
4. Analysis
The analysis examines where Tracktor and other trackers succeed or fail, using challenging scenarios and oracle variants to identify future directions. Tracktor is robust to occlusions and long detection gaps, while oracle results expose detector and extension bottlenecks.
- Object visibility: Tracktor achieves superior performance for partially occluded bounding boxes, even at visibility as low as 0.3.Dedicated identity-preserving and offline interpolation methods do not successfully tackle highly occluded objects.
- Detection gaps: Tracktor achieves far superior coverage of detection gaps, especially gaps longer than 15 frames.Offline interpolation and motion prediction improve gap coverage for other trackers, but neither matches frame-by-frame regression robustness.
- Identity preservation: Tracktor++ has identity-preservation performance comparable with dedicated trackers without offline graph optimization.It also reduces false positives in MOT17 and MOT16 and can incorporate additional identity-preserving extensions.
- Detector oracles: Ground-truth regression and killing substantially improve MOTA and false positives, while ground-truth killing can deteriorate identity preservation.The detector therefore remains a major source of Tracktor’s performance limitations.
- Extension oracles: 27? Oracle-MM and Oracle-reID substantially improve IDF1 and identity switches, while Oracle-ALL reaches 72.2% MOTA with virtually no false positives.Oracle-ALL is evaluated as the absolute upper bound for Tracktor with the specified detector.
- Towards a new tracking paradigm: The proposed future directions are Tracktor with tracking-specific extensions and detector-generated tracklets for subsequent tracking methods.A hallucinating motion model is identified as promising for large gaps, occlusions, and camera movement.
5. Conclusions
The conclusion argues that a detector’s bounding-box regressor can solve most benchmark tracking scenarios without tracking-specific training. Tracktor remains extendable and motivates a tracking paradigm centered on exploiting detector capabilities.
- Conclusions: A trained Faster-RCNN bounding-box regressor solves most tracking scenarios present in current benchmarks.Tracktor requires no specific training on tracking ground-truth data and operates online.
- Conclusions: Tracktor++ achieves substantial new state-of-the-art performance on MOTChallenge after adding re-identification and camera-motion compensation.The conclusion describes these as extensions of the detector-based tracker.
- Conclusions: Dedicated tracking methods do not yield substantially better performance than the regression-based Tracktor in the analyzed challenging scenarios.The analysis is presented as motivation for a new tracking paradigm using the detector’s full capabilities.
Abstract
The supplementary material supports reproduction by adding pseudocode, implementation and training details, and expanded experimental results for Tracktor++.
- Abstract: The supplementary material provides pseudocode and implementation and training details for Tracktor and its tracking extensions.It also expands the experimental analysis with sequence-level MOTChallenge results for Tracktor++.
- Abstract: Additional supplementary material details the experiments and analysis underlying Tracktor++ benchmark results.The results cover each sequence and set of public detections.
- Abstract: The supplementary material complements the main work with further details on the object detector and tracking extensions.These materials are intended to provide a more complete account of the method and evaluation.
A. Implementation
The implementation section supplies additional details and references to facilitate reproduction of Tracktor and its extensions.
- A. Implementation: Additional implementation details and references are provided for reproducing Tracktor and its extensions.The stated purpose is completeness and facilitation of reproduction.
- A. Implementation: The supplementary implementation material covers both Tracktor and its extensions.It is presented as a complement to the main method description.
- A. Implementation: The implementation section prioritizes reproducibility through supplementary technical information and references.No additional algorithmic result is stated in the supplied passage.
A.1. Tracktor
Tracktor converts an object detector into a tracker by using bounding-box regression for temporal realignment, without dedicated tracking-data training or optimization. Tracktor++ adds re-identification and motion-model extensions to improve multi-object tracking.
- Tracktor: The detector requires no dedicated training or optimization on tracking ground-truth data.The detector is trained for multi-object detection on MOT17Det.
- Tracking extensions: Tracktor++ extends the vanilla tracker with a motion model and re-identification.The extensions target camera motion compensation and identity comparison between bounding-box features.
- Tracking extensions: Camera motion compensation uses image registration with Euclidean or affine alignment depending on the camera movement.Euclidean alignment is used for rotating motion, while affine alignment also handles camera translation.
- Tracking extensions: The CVA motion model moves a bounding-box center according to the difference between its two previous centers and is applied only to AVG-TownCentre.It predicts motion in the direction of the previous velocity.
- Tracking extensions: The re-identification component uses a Siamese network to compare bounding-box features and estimate identity similarity.It trains a TriNet architecture with triplet loss and batch hard sampling.
B. Experiments
The experiments report official and published MOTChallenge results for Tracktor++ and direct readers to the official benchmark site for sequence-level results from other trackers.
- Experiments: Tracktor++ results are summarized in Table 3 using official and published MOTChallenge benchmark evaluations.The reported summary concerns the Tracktor++ tracker.
- Experiments: Sequence-level results for the other trackers are available through the official MOTChallenge website.The reference covers each sequence and detection set mentioned in the work.
- Experiments: The experiments distinguish the benchmark summary for Tracktor++ from the detailed external results for comparison trackers.This distinction follows the reporting structure described for Table 3 and the official website.
B.1. Evaluation metrics
The evaluation reports tracking accuracy, identity preservation, false positives, and false negatives, while also specifying the public-detection comparison setting.
- Evaluation metrics: MOTA and IDF1 measure overall tracking performance and track-identity preservation, respectively.They are identified as the principal tracking metrics discussed in the evaluation.
- Evaluation metrics: False positives count predicted boxes without matching ground truth, whereas false negatives count uncovered ground-truth boxes.These metrics quantify detection errors in the tracking output.
- Public detections: The evaluation reclassifies raw public detections and applies its own nonmaximum-suppression procedure before comparison.The authors describe this processing as making comparison with public trackers fair.
- Public detections: Table 2 compares trackers on the MOT17 test set using Faster R-CNN public detections.The Tracktor-no-FPN++ variant applies vanilla Faster R-CNN.
B.3. Evaluation on public detections
The public-detection evaluation reduces the role of public detections to track initialization and tests fixed thresholds without sequence-specific fine-tuning.
- Public detections: Tracktor reclassifies and regresses public detections, reducing their equalizing effect to the initialization of new tracks.The authors also evaluate the tracker without relying on the advantage of reclassification and regression.
- Public detections: The evaluation uses fixed σactive = 0.5, λactive = 0.6, and λnew = 0.3 across all benchmarks.These values were selected as optimal for the MOT17 training dataset.
- Public detections: A higher λactive than λnew makes tracking more stable by reducing active-track termination and new-track initialization.The passage connects the threshold relationship with NMS behavior.
B.5. Tracktor video frame rate robustness
Tracktor’s bounding-box regression requires sufficiently high frame rates, or equivalently small frame-by-frame object displacements. The analysis evaluates robustness under reduced frame rates and compares Tracktor variants in this setting.
- Tracktor bounding-box regression depends on sufficiently high video frame rates and small frame-by-frame object displacements.
- The robustness evaluation reduces originally 30 FPS MOT17 training sequences by removing frames from the data and ground truth.
- Figure 1 reports tracking performance for Tracktor and Tracktor++ on low-frame-rate MOT17-FRCNN sequences.