Source-linked AI summary
TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking
Peng Chu, Jiang Wang, Quanzeng You, Haibin Ling, Zicheng Liu
TL;DR
MOT needs robust association of detections across time while modeling interactions among many objects efficiently. TransMOT uses sparse weighted trajectory graphs with spatial-temporal graph Transformers and a cascade association framework, achieving state-of-the-art performance across four MOT benchmarks.
Problem
Robust MOT association must model target history, appearance, and spatial-temporal relationships, while general Transformers are ineffective for videos containing many objects.
Method
TransMOT represents tracked trajectories as sparse weighted graphs, applies spatial and temporal Transformer layers, and adds cascade association for low-score detections and long-term occlusions.
Results
The framework achieves the best overall performance and new state-of-the-art results on MOT15, MOT16, MOT17, and MOT20; on MOT17, it improves over TransTrack by 10.0% in IDF1 and 3.0% in MOTA.
Takeaways & Limitations
Sparse graph Transformers provide an efficient and effective way to model spatial-temporal relationships for multi-object tracking.
Abstract
from arXiv · showhide
Tracking multiple objects in videos relies on modeling the spatial-temporal interactions of the objects. In this paper, we propose a solution named TransMOT, which leverages powerful graph transformers to efficiently model the spatial and temporal interactions among the objects. TransMOT effectively models the interactions of a large number of objects by arranging the trajectories of the tracked objects as a set of sparse weighted graphs, and constructing a spatial graph transformer encoder layer, a temporal transformer encoder layer, and a spatial graph transformer decoder layer based on the graphs. TransMOT is not only more computationally efficient than the traditional Transformer, but it also achieves better tracking accuracy. To further improve the tracking speed and accuracy, we propose a cascade association framework to handle low-score detections and long-term occlusions that require large computational resources to model in TransMOT. The proposed method is evaluated on multiple benchmark datasets including MOT15, MOT16, MOT17, and MOT20, and it achieves state-of-the-art performance on all the datasets.
1. Introduction
TransMOT addresses robust target association in tracking-by-detection MOT by modeling object spatial-temporal relationships with sparse graph Transformers. It combines this model with cascade association to handle low-score detections and long-term occlusions, achieving strong benchmark performance.
- Robust MOT association requires modeling target history, appearance, and spatial-temporal relationships alongside accurate detection.
- General Transformers are inefficient for videos containing many objects, motivating sparse graph representations for spatial-temporal modeling.
- TransMOT arranges tracked trajectories as sparse weighted graphs and uses spatial graph encoder, temporal encoder, and spatial decoder layers.
- TransMOT achieves the best overall performance and establishes new state-of-the-art results on MOT15, MOT16, MOT17, and MOT20.
- The cascade association framework handles low-score detections and long-term occlusions without modeling all such cases directly in TransMOT.
2. Related Works
Prior MOT association methods use matching, graph optimization, recurrent, differentiable, and Transformer-based approaches. TransMOT instead uses sparse spatial graphs with Transformer networking to jointly model spatial and temporal relationships efficiently.
- Tracking-by-detection MOT commonly associates detections across frames using Hungarian matching, network flow, or multiple hypotheses tracking.
- Graph-based association methods can require computationally expensive global optimization on large graphs.
- Deep learning association methods have used motion-only RNNs, tensor-approximation layers, differentiable Hungarian networks, and joint detection-association models.
- TransMOT constructs within-frame spatial graphs and uses Transformer architecture to jointly learn spatial and temporal relationships for efficient association.
3. Overview
The online TransMOT framework receives tracklet histories and current-frame detections, represents them with spatial graphs, and predicts constrained associations while modeling object events such as entry, exit, and occlusion.
- Each tracklet maintains state information, including past locations over the previous T image frames.
- At each new frame, online tracking removes exited tracklets, determines occlusions, updates existing locations, and creates tracklets for entering objects.
- Detection and feature-extraction subnetworks produce current-frame candidate detections and visual features for the association subnetwork.
- The spatial-temporal graph Transformer selects candidate proposals for tracklets and models entering, exiting, and occlusion events.
- Association is formulated as constrained optimization using a scoring function that computes affinity between each tracklet state and candidate.
- TransMOT builds a weighted current-frame candidate graph and weighted tracklet graphs from previous T frames, then generates an assignment matrix.
4. TransMOT
TransMOT factorizes spatial and temporal modeling over sparse weighted graphs, using graph-transformer encoder and decoder layers to produce object assignments. A three-stage cascade extends association to low-score detections and long-term occlusions while improving speed and accuracy.
- Spatial Graph Transformer: TransMOT uses graph multi-head attention to model spatial relationships among tracklets and candidates through self-attention.This mechanism is used in both the spatial graph transformer encoder and decoder.
- Spatial-Temporal Graph Encoder: The encoder processes tracklet states from the past T frames using a sequence of spatial graphs and temporal feature encoding.Spatial graph attention is computed independently at each timestamp, followed by a temporal Transformer over each tracklet.
- Spatial Graph Construction: Tracklet graphs connect nodes when bounding-box IoU is positive, with a sparse weight matrix storing edge weights and zeros for disconnected pairs.Each node represents a tracklet state, and node features are embedded into a feature tensor before graph attention.
- Spatial-Temporal Graph Encoder: Sparse graph weights restrict attention to tracklets with spatial interactions, making the encoder more effective and faster than attending broadly.Graph convolution also aggregates information from neighboring nodes before feed-forward projection and normalization.
- Spatial Graph Transformer Decoder: The decoder applies graph attention to the current candidate graph and uses virtual source and sink nodes for new-tracklet, exit, or occlusion events.Candidate nodes connect when their bounding-box IoU is positive, with edge weights set to IoU.
- Cascade Association Framework: The cascade framework improves inference speed and tracking accuracy by handling low-score detections and long-term occlusions outside the core TransMOT modeling window.For occluded tracklets, stored visual features and latest visible bounding boxes define an association cost using feature distance and normalized top distance; duplicate candidates are then removed using box overlap.
5. Experiments
Experiments evaluate TransMOT across four MOT challenge datasets, detection settings, and ablations, showing strong tracking performance and trade-offs among association modules, temporal history, resolution, and visual features.
- Datasets and metrics: Four standard pedestrian-tracking benchmarks—MOT15, MOT16, MOT17, and MOT20—are evaluated using public and private detection tracks.
- Datasets and metrics: The experiments report ID score metrics, CLEAR MOT metrics, mostly tracked targets, and mostly lost targets.
- Evaluation results: MOT16 private-detector results outperform all published trackers in both IDF1 and MOTA, while MOT17 results achieve the best IDF1 and MOTA metrics among compared methods.
- Evaluation results: 10.0% and 3.0% gains in IDF1 and MOTA, respectively, are reported for TransMOT-D over TransTrack on MOT17.
- Ablation study: Removing Match with Motion lowers MOTA and slows inference, whereas removing Long-Term Occlusion Handling fragments tracklets, reduces IDF1, and increases inference speed.
- Ablation study: Temporal history beyond one frame improves association, but increasing T beyond 5 adds complexity without performance gain and slows inference; lower resolution misses distant small targets.
6. Conclusion
TransMOT models tracklets and candidate detections as weighted graphs to explicitly leverage spatial-temporal relationships. It improves tracking accuracy and computational efficiency, with cascade association further optimizing speed and accuracy.
- TransMOT formulates tracklets and candidate detections as weighted graphs for explicit spatial-temporal modeling.
- The approach achieves higher tracking accuracy and greater computational efficiency than traditional Transformer-based methods.
- Cascade association filters low-score candidates, recovers long-term occlusions, and removes duplicate detections.
- TransMOT achieves state-of-the-art performance on MOT15, MOT16, MOT17, and MOT20 benchmark datasets.