Tracking without bells and whistles

1. Introduction

• MOT16 multiple-object tracking accuracy improved only 2.4% in two years, while missing and spurious detections, occlusions, and target interactions complicate data association [45].The introduction frames these difficulties as challenges in crowded environments.Source: §1 · p. 1
• Tracktor uses an object detector’s regression head to realign existing track boxes over time, showing detection-trained networks can construct trajectories in challenging scenarios.The paper uses this approach to ask when dedicated tracking algorithms are necessary.Source: §1 · p. 1
• Tracktor’s re-identification Siamese network and motion-model extensions yield state-of-the-art performance on three challenging multi-object tracking benchmarks.The base method exploits a detector’s regression head to temporally realign object bounding boxes.Source: §1 · p. 1
• The authors report that dedicated trackers perform no substantially better than their regression approach in challenging scenarios, and identify unresolved challenges as targets for future research.They propose detector-based tracking as a new paradigm and analyze failure cases and promising research directions.Source: §1 · p. 1

1.1. Related work

• Related work covers graph-based detection association, appearance and re-identification models, motion prediction, and video-detection methods; Tracktor instead preserves identity while aligning detections and is reported to better handle occlusions.The cited discussion contrasts Tracktor’s online approach with more computationally costly graph methods and prior video-detection approaches.Source: §1.1 · pp. 2–3

2. A detector is all you need

• Tracktor exploits detector bounding-box regression to track objects without tracking-specific training or complex test-time optimization, and is reported to achieve state-of-the-art performance on challenging tracking scenarios.The approach can transfer to tracking datasets or scenarios where only detection data is available.Source: §2 · p. 3, §2.2 · p. 4

2.1. Object detector

• The implementation uses Faster R-CNN with ResNet-101 and FPN trained on MOT17Det; proposal classification and regression refine pedestrian boxes before non-maximum suppression.Source: §2.1 · p. 3

2.2. Tracktor

• Each frame, Tracktor regresses active boxes on current-frame features and transfers their identities, assuming limited inter-frame motion; it deactivates weak or overlapping tracks and initializes unmatched detections.The method relies on high frame rates to help ensure that the target remains near its previous box.Source: §2.2 · pp. 3–4

2.3. Tracking extensions

• Camera-motion compensation aligns frames with ECC, constant-velocity prediction addresses low frame rates, and short-term Siamese re-identification reconnects deactivated tracks.The re-identification network is trained on tracking ground truth and limits candidate pairs using an IoU threshold.Source: §2.3 · p. 4

3. Experiments

• For fair comparison, all methods process the same precomputed MOTChallenge public detections, while MOTA and IDF1 measure object coverage and identity, respectively.The benchmarks include sequences with frequent occlusions and crowded scenes, varying in viewpoint, object size, camera motion, and frame rate.Source: §3 · p. 4

3.1. Ablation study

• On MOT17 training-set ablations, reID and CMC significantly reduce identity switches and increase IDF1; online regression-based tracking also outperforms D&T despite its tracking-data training and offline track generation.The ablation evaluates extensions on Faster R-CNN public detections; Tracktor-no-FPN also illustrates the potential of improved detection methods.Source: §3.1 · p. 5, Table 1 · p. 5

3.2. Benchmark evaluation

• Tracktor++ achieves new state-of-the-art MOTA on all three MOTChallenge benchmarks using the same MOT17Det-trained tracker, without tracking-train-set training or optimization.All sequences use camera motion compensation and re-identification; constant velocity is also used for the low-frame-rate AVG-TownCentre sequence.Source: Table 2 · p. 5, §3.2 · p. 5
• On MOT16, Tracktor++ outperforms other trackers by a large margin on false negatives and IDF1, and it achieves a new state-of-the-art on 2D MOT 2015 without using that benchmark’s training sequences.The results indicate generalization across datasets with different scene characteristics and robustness to comparatively lower-performing detections.Source: §3.2 · p. 5
• On Faster R-CNN public detections, none of the presented sophisticated methods tracks a higher ratio of visible objects than Tracktor, especially for highly occluded boxes.Figure 2 compares tracked-object ratios by visibility and shows the ground-truth visibility distribution.Source: Figure 2 · p. 6

4.1. Tracking challenges

• Tracktor outperforms other methods on partially occluded boxes with visibility as low as 0.3, while identity-preservation and offline-interpolation methods do not appear to solve heavy occlusion.The paper attributes high MOTA values largely to the unbalanced distribution of ground-truth visibilities; Tracktor++ adds only minor gains over vanilla Tracktor.Source: §4.1 · p. 6
• Trackability decreases with object size across detection sets, but Tracktor compensates for inferior DPM and Faster R-CNN detections; methods perform similarly above 250 pixels in height.SDP’s additional small detections benefit all methods except MOTDT17; the paper links that exception to MOTDT17’s appearance model and re-identification vulnerability to small detections.Source: §4.1 · p. 6
• Tracktor’s frame-by-frame regression achieves far better detection-gap coverage than dedicated trackers, especially for gaps longer than 15 frames; MOTDT17 covers long gaps better than jCC.Offline methods benefit most from SDP detections, while neither Tracktor nor MOTDT17 shows notable robustness to gap length in that setting.Source: Figure 3 · p. 7, §4.1 · p. 7
• Tracktor++ achieves identity-preservation performance comparable to dedicated trackers without offline graph optimization, and Tracktor reduces false positives in MOT17 and MOT16.MOTDT17, despite its sophisticated appearance model and re-identification, is not substantially superior to Tracktor and its extensions.Source: §4.1 · p. 7

4.2. Oracle trackers

• Oracle-ALL reaches 72.2% MOTA, with high IDF1 and virtually no false positives, marking the upper bound for Tracktor with a Faster R-CNN and FPN detector.Oracle-ALL uses ground-truth killing, regression, and re-identification.Source: Table 4 · p. 8, §4.2 · p. 8
• Ground-truth killing and regression oracles substantially improve MOTA and false positives, but ground-truth killing worsens identity preservation because regression struggles with unseen boxes.Oracle-Kill uses ground truth instead of NMS or classification score; Oracle-REG places boxes at ground-truth positions instead of using regression.Source: §4.2 · p. 8
• The extension oracles improve IDF1 and identity switches, and their results indicate that a predictive motion model could hallucinate object positions through long occlusions.The proposed motion model could avoid offline post-processing and false positives from incorrect linear occlusion paths during long gaps and camera movement.Source: §4.2 · p. 8

4.3. Towards a new tracking paradigm

• The paper proposes using Tracktor with tracking-specific extensions and generating frame-to-frame tracklets from the detector, leaving remaining complex cases to a subsequent tracker.Bounding-box regression would cover large, highly visible objects, while a hallucinating motion model could handle the rest; detector-generated tracklets could provide richer inputs for later motion or appearance models.Source: §4.3 · p. 8

5. Conclusions

• Tracktor extended with re-identification and camera motion compensation provides a substantial new state-of-the-art on MOTChallenge.The approach converts a trained Faster-RCNN detector into an online tracker without training on tracking ground-truth data.Source: §5 · p. 8
• A trained Faster-RCNN detector’s bounding-box regressor is enough to solve most tracking scenarios present in current benchmarks.The detector converted to Tracktor works online and requires no specific training on tracking ground-truth data.Source: §5 · p. 8
• None of the analyzed dedicated tracking methods performed substantially better than regression-based Tracktor on challenging tracking scenarios.Source: §5 · p. 8
• The authors propose that Tracktor establishes a new tracking paradigm by utilizing the object detector’s full capabilities.Source: §5 · p. 8

• The supplement includes Tracktor pseudocode, implementation and detector-training details, plus expanded experiments, analysis, and Tracktor++ MOTChallenge results for each sequence and public-detection set.Source: Abstract · p. 9

A. Implementation

• The authors provide additional implementation details and references for Tracktor and its extensions to facilitate reproduction of their results.Source: A. Implementation · p. 9

A.1. Tracktor

• Tracktor uses Faster R-CNN with FPN trained on MOT17Det, requiring no dedicated training or optimization on tracking ground truth.The detector uses ResNet-101; FPN and crop-and-resize pooling are intended to improve results for comparatively small objects.Source: §A.1 · p. 9
• On MOT17Det, vanilla FRCNN is on par with the official implementation, while adding FPN yields detection performance close to SDP.The compared detectors rank DPM, FRCNN, then SDP in ascending detection performance.Source: Table 1 · p. 9, §A.1 · p. 9

A.2. Tracking extensions

• Tracktor++ extends Tracktor with two multi-pedestrian tracking additions: a motion model and re-identification.These extensions are described as tracking-specific additions to the base tracker.Source: §A.2 · p. 9
• Camera-motion compensation uses ECC image registration, with Euclidean alignment for rotating movement and affine alignment when translation is present.The method treats all MOT17 sequences as rotating because camera translation there is comparatively slow; autonomous-driving sequences in 2D MOT 2015 use affine alignment.Source: §A.2 · p. 9
• The constant-velocity model shifts bounding boxes along their previous velocity before regression, and is applied only to AVG-TownCentre in 2D MOT 2015.The model is intended to facilitate regression when low frame rates produce large object displacements between frames.Source: §A.2 · pp. 9–10
• Short-term re-identification compares bounding-box features with a Siamese network trained as TriNet using triplet loss and batch hard strategy.TriNet is based on ResNet-50 and trained on MOT17 tracking identities; batches contain 18 identities with four samples each, discarding identities with fewer than four samples.Source: §A.2 · p. 10

• Table 3 summarizes Tracktor++’s official and published MOTChallenge results; sequence- and detection-specific results for other trackers are available on the official challenge website.Source: B. Experiments · p. 10

B.1. Evaluation metrics

• MOTA and IDF1 combine basic tracking metrics, and all metrics use the official MOTChallenge evaluation code.Source: §B.1 · p. 10
• False positives count boxes covering no ground truth, while false negatives count ground-truth boxes uncovered by any detection.Source: §B.1 · p. 10
• Identity switches count boxes that cover a ground-truth object from a different track than in the previous frame.Source: §B.1 · p. 10
• Mostly tracked and mostly lost classify ground-truth tracks covered by boxes for at least 80% or at most 20% of their duration, respectively.Source: §B.1 · p. 10

B.2. Raw DPM detections

• Because public DPM detections use classification scores and an unknown IoU threshold for NMS, the authors rescored raw detections and applied their own NMS.Source: §B.2 · p. 10
• The authors also processed raw DPM detections for MOT17 and considered comparison with public trackers fair because several trackers use raw detections with their own scores and NMS.Source: §B.2 · p. 10

B.3. Evaluation on public detections

• On the MOT17 test set, evaluation using only Faster R-CNN public detections supports the paper’s overall conclusions about Tracktor.The passage does not provide Table 2’s values, so it does not establish specific rankings or metric differences.Source: §B.3 · p. 10
• Tracktor reclassifies and regresses public detections with a private detector, limiting their equalizing effect to initialization of new tracks.Source: §B.3 · p. 10
• The authors emphasize Tracktor’s potential relative to other state-of-the-art trackers even without reclassification and regression advantages.Source: §B.3 · p. 10
• Tracktor-no-FPN++ uses vanilla Faster R-CNN for reclassification and regression, effectively leaving public detections unchanged.It omits Feature Pyramid Networks.Source: §B.3 · p. 10

B.4. Tracktor thresholds

• Threshold experiments used σactive = 0.5, λactive = 0.6, and λnew = 0.3 on all benchmarks, without sequence- or detection-specific fine-tuning.These thresholds were chosen to be optimal for the MOT17 training dataset.Source: §B.4 · p. 10
• A higher λactive than λnew makes the tracker more stable by reducing active-track removals by NMS and new-track initializations.Source: §B.4 · p. 10
• A comparatively higher λactive relaxes potential object-object occlusions and implies confidence in regression performance.Source: §B.4 · p. 10

B.5. Tracktor video frame rate robustness

• Tracktor box regression depends on high frame rates and small frame-by-frame displacements; MOT16 and MOT17 mostly contain benign frame rates and slowly moving pedestrians.The passage identifies a powerful motion model as a possible way to address this limitation and notes camera motion compensation as a rudimentary model.Source: §B.5 · pp. 10–11

C. Oracle trackers

• Oracle-reID uses Hungarian matching between new detections and ground truth, with ground-truth identity matches yielding perfect re-identification.Source: §B.5 · p. 11

B.5. Tracktor video frame rate robustness

• Tracktor and Tracktor++ show fairly robust MOTA and IDF1 down to 5 FPS, while very low frame rates particularly hurt identity preservation.The evaluation uses MOT17 training sequences originally recorded at 30 FPS, with frames and ground truth removed to reduce frame rates.Source: Figure 1 · p. 11, §B.5 · p. 11

C. Oracle trackers

• The oracle analysis substitutes selected aspects of vanilla Tracktor with ground-truth information to simulate perfect behavior and highlight future research directions.The paper says these oracles support its analysis of challenging tracking scenarios.Source: C. Oracle trackers · p. 11
• Oracle-Kill retains tracks when their IoU with the corresponding ground-truth box is at least 0.5 and uses ground-truth matching to resolve occlusions above 0.8 IoU.Predicted and ground-truth tracks are matched with the Hungarian algorithm.Source: C. Oracle trackers · p. 11
• Oracle-REG simulates perfect regression by matching tracks to ground truth at frame t −1 with an IoU threshold of 0.5, then setting boxes to ground truth at frame t.Source: C. Oracle trackers · p. 11
• Oracle-MM moves the previous box center to the ground-truth center at frame t, while regression still determines box height and width.Source: C. Oracle trackers · p. 11