Source-linked AI summary
Global Tracking Transformers
Xingyi Zhou, Tianwei Yin, Vladlen Koltun, Philipp Krähenbühl
TL;DR
Multi-object tracking traditionally relies on local pairwise association or offline combinatorial optimization, motivating a direct global formulation. GTR encodes detections across a temporal window and uses detection-derived trajectory queries to produce trajectories jointly with detection. It achieves 20.1 tracking mAP on TAO test and competitive 75.3 MOTA and 59.1 HOTA on MOT17, while improving over a same-detector QDTrack comparison.
Problem
Existing trackers primarily use greedy pairwise association or offline combinatorial optimization, leaving a need for direct global trajectory modeling.
Method
GTR encodes object features from multiple frames and uses detection features from one frame as trajectory queries that assign detections into trajectories within an end-to-end trainable temporal window.
Results
20.1 tracking mAP on TAO test and 75.3 MOTA and 59.1 HOTA on MOT17 are reported, with a 4.3 mAP and 1.9 AssA gain over QDTrack using the same detector.
Takeaways & Limitations
The framework integrates global tracking into state-of-the-art detectors and directly outputs consistent long-term trajectories without combinatorial optimization.
Takeaways & Limitations
The 32-frame temporal window cannot recover from missing detections or occlusions larger than 32 frames, and TAO training uses static images because multi-class multi-object tracking training sets are unavailable.
Abstract
from arXiv · showhide
We present a novel transformer-based architecture for global multi-object tracking. Our network takes a short sequence of frames as input and produces global trajectories for all objects. The core component is a global tracking transformer that operates on objects from all frames in the sequence. The transformer encodes object features from all frames, and uses trajectory queries to group them into trajectories. The trajectory queries are object features from a single frame and naturally produce unique trajectories. Our global tracking transformer does not require intermediate pairwise grouping or combinatorial association, and can be jointly trained with an object detector. It achieves competitive performance on the popular MOT17 benchmark, with 75.3 MOTA and 59.1 HOTA. More importantly, our framework seamlessly integrates into state-of-the-art large-vocabulary detectors to track any objects. Experiments on the challenging TAO dataset show that our framework consistently improves upon baselines that are based on pairwise association, outperforming published works by a significant 7.7 tracking mAP. Code is available at https://github.com/xingyizhou/GTR.
1. Introduction
Global tracking addresses the limits of frame-by-frame association and offline graph optimization by using a transformer to group detections from multiple frames directly into trajectories. The framework jointly integrates detection and tracking and performs competitively on MOT17 and TAO.
- Motivation: Tracking-by-detection separates independent per-frame detection from temporal association, while local trackers primarily use greedy pairwise links.Local association is efficient but lacks an explicit whole-trajectory model and can struggle with heavy occlusion or strong appearance change.
- Motivation: Global trackers improve robustness through offline graph-based optimization, but can be slow and are usually detached from the detector.
- Approach: GTR encodes detections from multiple consecutive frames and uses trajectory queries from one frame to assign detections into global trajectories.Each query produces one trajectory by assigning a detection from each frame through a softmax distribution.
- Approach: The framework replaces pairwise association and graph optimization with a few transformer layers that directly output detections and their temporal associations.During inference, GTR operates in sliding windows with a temporal size of 32 frames and links trajectories between windows online.
- Results: 20.1 tracking mAP on TAO test and 75.3 MOTA and 59.1 HOTA on MOT17 demonstrate the framework's reported benchmark performance.On TAO, the result significantly exceeds published work at 12.4 tracking mAP.
2. Related work
Prior tracking methods either associate detections locally through pairwise matching or solve global associations with graph optimization. Transformer-based video methods provide temporal context, while GTR explicitly supervises object identity associations for tracking.
- Local multi-object tracking: Local trackers maintain confirmed tracks and link newly detected objects using pairwise object-track distance metrics.Examples include Kalman-filter-based location or feature updates, detector-proposal propagation, and predicted-location association.
- Global tracking: Traditional global trackers formulate association across all frames as combinatorial optimization over detection graphs.Min-cost maxflow and neural graph solvers select globally consistent assignments.
- Transformers in tracking: Transformer trackers use object or historical-track queries, but the cited approaches propagate identities or associate through updated bounding-box locations.
- Video object detection: Video object detectors use temporal attention to provide global context, but they do not explicitly learn object identity associations for tracking.GTR differs by explicitly supervising object association across frames.
3. Preliminaries
Tracking-by-detection decomposes multi-object tracking into detecting candidate objects in each frame and associating them across time. GTR instead performs joint detection and global association within one differentiable network pass over a 32-frame clip.
- Object detection: An object detector maps each image to candidate objects with locations, and multi-class detection additionally produces class scores from object features.
- Tracking by detection: Tracking-by-detection links existing trajectories to current detections using an object indicator at each frame.The indicator can denote no association or one of the detected objects.
- Global association: Most prior methods use greedy pairwise matches or offline combinatorial optimization, whereas this work learns global tracking within a 32-frame video clip in one forward pass.The formulation uses a probabilistic association model instantiated in a transformer architecture.
4. Global tracking transformers
GTR formulates global tracking as differentiable trajectory prediction over object features from multiple frames, using trajectory queries to assign detections without intermediate pairwise association. It trains jointly with detection and performs online inference with sliding windows and track linking.
- Tracking formulation: GTR links detections to trajectory queries by predicting association scores over objects from all frames.The resulting per-frame associations combine into trajectories, with an explicit no-association output.
- Tracking formulation: The tracking transformer consumes all detection features across the temporal sequence and produces trajectory-specific association scores.The total feature matrix contains N detections across T frames, with each feature represented in D dimensions.
- Training: Training maximizes trajectory-assignment likelihood, uses IoU-based matching to assign detections, and combines association loss with standard detection losses.The association objective includes empty trajectories for unassociated features and is jointly trained with classification and bounding-box regression losses.
- Trajectory queries: Object features serve as trajectory queries, while unmatched features become background queries and inference uses one frame’s features to avoid duplicate trajectories.After detection NMS, distinct object features from the selected frame produce different trajectories.
- Online inference: Online inference processes 32-frame sliding windows, uses current-frame features as queries, and links predicted trajectories to existing tracks with Hungarian matching.The first frame initializes detections as trajectories; subsequent windows use average assignment likelihood to match current and prior tracks.
- Architecture: GTR’s lightweight transformer uses a one-layer encoder and decoder, and its global reasoning avoids the separate cosine-distance grouping step used by ReID-based trackers.The architecture takes object features as encoder input and queries as decoder input, producing an association matrix.
5. Experiments
Experiments evaluate GTR on TAO and MOT17 using official metrics, comparisons with local and state-of-the-art trackers, and design ablations. Results show gains from global temporal association, strong benchmark performance, and selective benefits from architectural choices.
- Evaluation setup: Experiments use TAO's large-vocabulary tracking mAP@0.5 and MOT17's MOTA, with HOTA and association accuracy also reported.TAO covers 488 classes in a long-tail setting, while MOT17 contains pedestrian crowd scenes and uses the private detection protocol.
- Global versus local association: Increasing GTR's temporal window consistently improves association accuracy, with T = 32 outperforming FairMOT by 1.8 AssA and 1.7 IDF1 on MOT17.At T = 2, GTR behaves as a local tracker and cannot recover from occlusions or missing detections; TAO performance saturates at T = 16.
- Comparison to the state-of-the-art: 74.1 MOTA, 71.1 IDF1, and 59.0 HOTA place GTR above most concurrent transformer trackers on the MOT17 test set.GTR trails TransMOT by 1.4 MOTA but runs 2× faster, under differing data, detector, and training setups.
- Design choice experiments: Removing self-attention and cross-attention considerably decreases AssA, whereas learned positional or temporal embeddings do not improve association accuracy.The final model therefore retains encoder attention without positional embeddings.
- Design choice experiments: One encoder and one decoder attention layer suffice, and location-based inference improves MOT17 AssA by 3 but is unnecessary on low-frame-rate TAO.The authors suggest detected object features make the task easier than approaches that use pixel features.
6. Conclusion
The paper presents a joint detection-and-tracking framework centered on a global tracking transformer that groups object features across a temporal window into trajectories. Its main scope boundaries are the 32-frame window, dependence on sliding-window identity aggregation, and static-image-only TAO training.
- 6. Conclusion: The global tracking transformer takes object features from all frames within a temporal window and groups them into trajectories.The framework is presented as a joint object detection and tracking system.
- 6. Conclusion: The model performs competitively on the MOT17 and TAO benchmarks.
- 6. Conclusion: GPU memory limits the temporal window to 32 frames, requiring sliding-window inference to aggregate identities across larger temporal extents.
- 6. Conclusion: The model cannot recover from missing detections or occlusions lasting longer than 32 frames.
- 6. Conclusion: The TAO model is trained only on static images because publicly available multi-class multi-object tracking training sets are lacking.