Source-linked AI summary
Tracking Objects as Points
Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl
TL;DR
Tracking-by-detection separates object detection from temporal association, while CenterTrack jointly detects and tracks using two frames and prior tracklet centers. Its local offset-based tracker achieves state-of-the-art results across 2D and monocular 3D benchmarks, with real-time operation.
Problem
Tracking-by-detection dominates modern tracking but separates recognition from association, motivating a simpler joint detection-and-tracking approach.
Method
CenterTrack uses current and previous frames plus a prior tracklet heatmap to detect object centers and predict offsets for greedy adjacent-frame association.
Results
CenterTrack sets a new state of the art on MOT17, KITTI, and nuScenes 3D tracking benchmarks while running online and in real time.
Takeaways & Limitations
The point-based framework provides simple local association and extends to monocular 3D tracking by regressing additional 3D attributes.
Takeaways & Limitations
CenterTrack is local: objects that leave or are occluded and later reappear receive new identities, and its local model is weaker than offline methods on IDF1 and identity switches.
Abstract
from arXiv · showhide
Tracking has traditionally been the art of following interest points through space and time. This changed with the rise of powerful deep networks. Nowadays, tracking is dominated by pipelines that perform object detection followed by temporal association, also known as tracking-by-detection. In this paper, we present a simultaneous detection and tracking algorithm that is simpler, faster, and more accurate than the state of the art. Our tracker, CenterTrack, applies a detection model to a pair of images and detections from the prior frame. Given this minimal input, CenterTrack localizes objects and predicts their associations with the previous frame. That's it. CenterTrack is simple, online (no peeking into the future), and real-time. It achieves 67.3% MOTA on the MOT17 challenge at 22 FPS and 89.4% MOTA on the KITTI tracking benchmark at 15 FPS, setting a new state of the art on both datasets. CenterTrack is easily extended to monocular 3D tracking by regressing additional 3D attributes. Using monocular video input, it achieves 28.3% AMOTA@0.2 on the newly released nuScenes 3D tracking benchmark, substantially outperforming the monocular baseline on this benchmark while running at 28 FPS.
1 Introduction
CenterTrack replaces separate detection and association with a point-based framework that jointly detects and tracks objects. It uses prior tracklet information and predicted center offsets to achieve simple, online tracking with strong benchmark performance.
- CenterTrack represents each object by its bounding-box center and tracks that point through time.The framework learns a 2D offset between adjacent frames and associates objects using center distance.
- The detector receives two consecutive frames and a heatmap of prior tracklets, then predicts object centers and offsets to previous-frame centers.The offset is learned as an additional center-point attribute with little additional computational cost.
- Tracking objects as points simplifies both tracking-conditioned detection and temporal association.Prior points form a heatmap for joint reasoning, while displacement prediction provides a sparse-optical-flow-like link across frames.
- Aggressive data augmentation enables CenterTrack to learn tracking from static image datasets with hallucinated motion and no real video input.The augmentation prevents the model from merely repeating prior-frame predictions.
- 28.3% AMOTA@0.2 on nuScenes outperforms the monocular baseline by a factor of 3 while running at 22 FPS.The method is purely local and does not reconnect long-range tracks.
2 Related work
Related tracking systems commonly separate object detection from temporal association, introducing computational or temporal-overlap limitations. CenterTrack instead jointly learns association with detection and uses prior track information as an input cue.
- Tracking-by-detection first detects objects independently in each frame and then solves bounding-box association across time.SORT uses Kalman-filtered boxes and bipartite matching, while related methods extend this paradigm with additional features.
- Existing association methods either discard image appearance features or require computationally expensive feature extractors.CenterTrack learns association jointly with detection instead of treating it as a separate stage.
- Region-proposal approaches assume substantial box overlap between frames, which fails in low-framerate regimes.Tracktor consequently requires a motion model for low-framerate sequences.
- Prior keypoint heatmaps provide tracking information without slowing the detector and can help reason about occluded objects.The detector can retain detections from the prior frame when objects are no longer visible.
3 Preliminaries
CenterTrack builds on CenterNet, which detects objects as heatmap peaks representing centers and regresses their box sizes and refined locations. The tracking framework extends this point-based representation with temporal inputs and offset outputs.
- CenterNet takes a single image and produces class-specific detections represented by center points, confidence scores, and bounding-box sizes.Object size is regressed as height and width from the detected center.
- CenterNet detects object centers as local maxima in a low-resolution heatmap with downsampling factor R = 4.A peak is the strongest response in a 3 × 3 neighborhood.
- The CenterNet training objective combines focal loss, size regression, and refined center-location regression.Ground-truth centers are rendered as Gaussian-shaped peaks, with kernel width determined by object size.
- CenterTrack extends the CenterNet-style framework to use current and prior-frame information and produce tracking offsets.Its framework takes a prior-center heatmap as an input and extracts sizes and offsets from current heatmap peaks.
4 Tracking objects as points
CenterTrack addresses detection and temporal association with one end-to-end network operating on adjacent frames and prior tracklet centers. It predicts center displacements for greedy local matching and uses augmentation to tolerate noisy tracking inputs.
- Tracking objects as points: CenterTrack treats tracking as propagating detection identities across consecutive frames rather than reconnecting tracks across temporal gaps.Objects that leave or are occluded and later reappear receive new identities.
- Tracking objects as points: The network receives current and previous frames plus prior tracked objects, and jointly detects current objects and assigns consistent identities.Each tracked object is represented by center, size, confidence, and identity.
- Tracking-conditioned detection: A prior-frame center heatmap supplies temporal context for current-frame detection and can help recover occluded objects.Only prior detections above confidence threshold τ are rendered to reduce false-positive propagation.
- Association through offsets: CenterTrack predicts a 2D displacement from each current center to its previous-frame center and learns it with a regression objective.The displacement is an additional output that captures inter-frame object motion.
- Association through offsets: Greedy matching links each detection to the closest unmatched prior detection within a size-based radius, spawning a new tracklet otherwise.The method avoids a complicated distance metric or graph matching.
- Training on video data: Training simulates jitter, false positives, and false negatives in prior tracklets to produce a robust tracking-conditioned detector.The augmentation models errors that can occur in test-time tracklet heatmaps.
- Training on video data: The prior frame may be sampled from a nearby frame rather than exactly t − 1, with experiments using |k − t| < 3.This sampling is used to avoid overfitting to the video framerate.
- Training on static images: Without labeled video, tracking can be simulated by randomly scaling and translating a single image to create a synthetic previous frame.The paper reports this training strategy as effective on standard detection benchmarks.
5 Experiments
CenterTrack is evaluated on 2D MOT17 and KITTI tracking, monocular 3D nuScenes tracking, and component ablations. It achieves strong benchmark results while revealing trade-offs between local association, tracking-conditioned detection, motion modeling, and long-range identity maintenance.
- Datasets and metrics: The experiments cover 2D multi-object tracking on MOT17 and KITTI and monocular 3D tracking on nuScenes.MOT17 and KITTI evaluate 2D tracking, while nuScenes evaluates monocular 3D tracking.
- Main results: CenterTrack improves MOT17 MOTA by 5 points over Tracktor v2 in the public-detection setting.This corresponds to an 8.6% relative improvement, with methods using the same underlying detector.
- Main results: 67.3% MOTA is achieved on MOT17 with private detection at 17 FPS end-to-end.The private configuration simultaneously detects and tracks objects.
- Limitations: The tracker is local: it does not reconnect lost long-range tracks, and its IDF1 and identity-switch performance is weaker than offline methods.The authors identify combining local tracking with stronger offline long-range models as future work.
- Main results: 89.44% MOTA is achieved on KITTI with flip testing at 82ms runtime.Without flip testing, the model runs at 45ms and obtains 88.7% validation MOTA.
- Main results: 28.3% AMOTA@0.2 and 4.6% AMOTA@1 are achieved on nuScenes, outperforming the monocular baseline.The authors attribute the result to a stronger 3D detector and learned motion that is more stable at low framerates than hand-crafted rules.
- Ablation studies: Tracking-conditioned detection improves MOTA by approximately 2% on MOT and 3% on KITTI, while reducing identity switches.The heatmap produces more false positives but fewer false negatives, and its overall effect improves MOTA.
- Ablation studies: Offset prediction strongly benefits nuScenes and consistently reduces identity switches on MOT and KITTI, especially when inter-frame motion is large.On high-framerate MOT17, even no motion modeling performs competitively; hand-crafted Kalman motion performs worse on KITTI and nuScenes.
6 Conclusion
CenterTrack is an end-to-end framework that jointly detects and tracks objects using greedy association. It applies the same simple association algorithm to both 2D and 3D tracking.
- CenterTrack performs simultaneous object detection and tracking in an end-to-end framework.
- The tracker uses a simple greedy identity-association algorithm based on center distance.The same algorithm is used for both 2D and 3D tracking.
- In public detection, tracks are initialized only when detections are near provided bounding boxes.
B Results on MOT16
CenterTrack ranks highly on MOT16 while using a single online network pass, contrasting with slower matching-heavy competitors.
- CenterTrack ranks 2nd among all published MOT16 leaderboard entries.
- CenterTrack operates online at 17 FPS with a single pass through the network.
- LMP requires O(n^2) deep-network forward passes to compute person-reidentification features for all bounding-box pairs.
- The evaluation covers MOT16 test sets under private detection.
C 3D detection
The 3D detection formulation adds an offset for projected 3D box centers and evaluates CenterNet-based performance against established detectors.
- CenterTrack additionally regresses the offset from the 2D center to the projected 3D bounding-box center.
- The offset loss uses L1 regression between ground-truth offsets and predicted offsets sampled at object-center locations.
- CenterNet with offset prediction has 3D detection performance on-par with Mappilary and PointPillars, but below LiDAR-based Megvii.
- The 4-dimensional box formulation regresses distances to the top, left, bottom, and right bounding-box borders.
E CrowdHuman dataset
CrowdHuman is a dense, heavily occluded human dataset with both visible and amodal bounding-box annotations.
- CrowdHuman contains 15k training images with common pose annotations.
- The dataset features high density and large occlusion.
- CrowdHuman annotates both visible and amodal bounding boxes.
F Pretraining experiments
CrowdHuman pretraining improves MOT performance, while the model still achieves measurable tracking accuracy without seeing MOT training data.
- 52.2 MOTA is achieved on the MOT dataset after training on CrowdHuman without seeing any MOT data.
- Without CrowdHuman pretraining, validation performance drops to 60.7% MOTA.
- Pretraining improves detection quality by decreasing false negatives.
- Most MOT challenge entries use external pretraining data, and some use private data.
G Additional experiments on KITTI
Additional KITTI experiments reproduce the main conclusions observed on MOT. Video training, heatmap-noise simulation, and matching strategy each affect or preserve tracking performance in distinct ways.
- Training on static images performs slightly worse than training on video on KITTI, largely because KITTI has greater inter-frame motion than MOT.
- Removing random heatmap noise substantially worsens performance and produces a high false-negative rate.
- Hungarian matching performs equivalently to greedy matching in the KITTI experiments.
H Output and rendering threshold
Tracking evaluation requires carefully chosen output and rendering thresholds because MOTA ignores prediction confidence. The experiments search these thresholds to balance false negatives and false positives.
- Because MOTA ignores prediction confidence, selecting an output threshold is essential for tracking algorithms.
- Increasing both the output and rendering thresholds yields fewer predictions, increasing false negatives while decreasing false positives.