TransTrack: Multiple Object Tracking with Transformer

• Existing MOT methods suffer from model complexity and computational cost because detection and association use multi-stage pipelines.Source: §1 · p. 1
• Separating detection and re-identification prevents them from benefiting from shared knowledge, motivating joint detection and tracking.Source: §1 · p. 1
• Directly transferring the SOT query-key mechanism causes false negatives because newly appearing objects lack corresponding previous-frame features.Source: §1 · p. 2
• TransTrack uses learned object queries to detect new objects and previous-frame track queries to maintain consistent tracklets.Source: §1 · p. 2
• 74.5 MOTA on MOT17 and 64.5 MOTA on MOT20 demonstrate competitive benchmark performance.Source: §1 · p. 2

Transformer in vision tasks

• Transformer uses attention-based query-key processing and has been applied successfully to detection, segmentation, 3D processing, and visual backbones.Source: §2 · p. 2
• Before TransTrack, published transformer-based solutions for object tracking were not yet available, motivating a transformer model for MOT.Source: §2 · p. 3

Tracking-by-detection

• Tracking-by-detection first detects objects and then associates them using re-identification features, IoU, or other information.Source: §2 · p. 3

Joint-detection-and-tracking

• Joint-detection-and-tracking methods aim to perform detection and tracking simultaneously in a single stage.Source: §2 · p. 3

• TransTrack’s architecture uses a CNN, an encoder, two query-specific decoders, and IoU matching for final association.Source: Figure 3 · p. 3

• TransTrack forms detection queries from learned parameters and track queries from previously detected object features.Source: §3 · p. 3
• The two query sources generate detection boxes and tracking boxes in parallel, then IoU matching produces the final ordered box set.Source: §3 · p. 4
• A transformer encoder-decoder architecture uses shared feature maps as keys for parallel detection and propagation decoders.Source: §3.1 · p. 4

Object Detection

• Learned object queries look up objects in global image features and output detection boxes.Source: §3.1 · p. 4

Object Propagation

• Previous-frame object features convey appearance and location information for locating corresponding objects in the current frame.Source: §3.1 · p. 4

Box Association

• Unmatched detection boxes are retained to create new tracklets after Kuhn-Munkres IoU matching.Source: §3.1 · p. 4

• The same training loss supervises detection and tracking boxes, while inference propagates and associates objects across following frames.Source: §3.2 · p. 4, §3.3 · p. 4

Evaluation setup

• MOTA is the primary metric used to measure overall multiple object tracking accuracy.The evaluation uses the widely adopted MOT metrics set.Source: §4 · p. 5
• The ablation protocol splits MOT17 training data into training and validation portions, while test-set models use combined CrowdHuman and MOT data.Models are first pretrained on CrowdHuman and then fine-tuned on MOT during ablation.Source: §4 · p. 5, §4.1 · p. 5

MOT17 results

• TransTrack achieves comparable results with state-of-the-art methods on MOT17 under the private-detector setting.Its MOTP and FN performance is especially competitive.Source: §4.2 · p. 5
• TransTrack precisely localizes objects and successfully detects most objects, while its ID-switch score remains inferior to state-of-the-art methods.The paper links MOTP to localization precision and FN to successful detection, while describing ID-switch performance as promising but not state of the art.Source: §4.2 · p. 5

Benchmark characteristics

• MOT20 contains more crowded scenes, more severe object occlusion, and smaller objects than MOT17.These characteristics make both detection and tracking more challenging.Source: §4.3 · p. 6
• All methods achieve lower performance on MOT20 than on MOT17 under the private-detector setting.The passage attributes the lower performance to MOT20's greater difficulty.Source: §4.3 · p. 6

Results

• TransTrack achieves comparable results with current state-of-the-art methods on MOT20 for detection and association metrics.The comparison is reported under the private-detector setting.Source: §4.3 · p. 6

4.4.1 Transformer Architecture

• 55.4 MOTA is achieved by the original Transformer, while Transformer-DC5 improves by 3.6 MOTA through higher feature resolution.The higher-resolution DC5 variant also incurs the memory cost associated with dilation convolution.Source: §4.4.1 · p. 6
• 65.0 MOTA is achieved by Deformable Transformer, which fuses multiple-scale features and becomes TransTrack's default architecture.Transformer-P3 produces performance close to Transformer-DC5 without further gain from higher resolution.Source: §4.4.1 · p. 6, Table 2 · p. 6

4.4.2 Query in Decoder

• 58.3 MOTA is obtained using only learned object queries, but index-based association produces wrong matches when objects move long distances.The learned-query-only setting can detect objects but has limited association performance.Source: §4.4.2 · p. 6, Table 3 · p. 6
• Using only previous-frame track queries handles large object motion but cannot detect objects that appear after the first frame.The resulting limitation is numerous false negatives for new-coming objects.Source: §4.4.2 · p. 6, Table 3 · p. 6
• Combining learned object queries with previous-frame object-feature queries yields the best detection and tracking performance.The visualization associates single-query settings with ID switches or missing objects, whereas the two-query setting avoids these cases more effectively.Source: §4.4.2 · p. 6, Table 3 · p. 6, Figure 4 · p. 7

4.4.3 Matching strategy of tracking boxes

• 0.6% versus 0.4% ID-switch performance shows that matching with previous-frame objects does not improve ID-switch avoidance over current-frame bipartite matching.The result supports the query-key mechanism's ability to locate corresponding objects in the current frame.Source: §4.4.3 · p. 7

4.4.4 Box association

• Hungarian matching and NMS merging have equivalent effects in the box-association stage.Both strategies merge track boxes and detection boxes into the desired ordered object set.Source: Table 5 · p. 7, §4.4.4 · p. 8

Comparisons with other trackers

• TransTrack is compared with detector-plus-motion and detector-plus-Re-ID trackers to assess gains beyond its detector.The comparison uses Kalman filtering for motion and alternative Re-ID feature pathways.Source: §4.5 · p. 8, §4.5.1 · p. 8, §4.5.2 · p. 8

Motion model

• 0.5% versus 1.0% improvement appears when sampling one frame every four frames, where larger motion makes feature-query association more beneficial.On the high-frame-rate MOT17 videos, adjacent-frame motion is minor and association methods show little difference.Source: §4.5.1 · p. 8
• Kalman filtering and TransTrack provide similar ID performance on MOT17 because the dataset's adjacent-frame object motion is small.The reported explanation attributes the limited difference to MOT17's 14–30 FPS frame rate.Source: §4.5.1 · p. 8

Re-ID features

• Independent Re-ID cross-attention improves Re-ID features over a shared layer but degrades the detector, so overall MOTA does not surpass default TransTrack.The independent pathway separates Re-ID and detection features for their respective supervision sources.Source: Table 7 · p. 8, §4.5.2 · p. 8, Figure 5 · p. 8

Conclusion

• TransTrack uses learned object queries for detection and previous-frame track queries to propagate previously detected objects.The framework performs detection and tracking jointly within a transformer-based pipeline.Source: §5 · p. 8
• 74.5 MOTA on MOT17 and 64.5 MOTA on MOT20 demonstrate competitive benchmark performance.The paper presents TransTrack as a novel perspective for multiple object tracking.Source: §5 · p. 8

Training Data

• 61.6 MOTA using only MOT17 training data rises to 64.8 MOTA after CrowdHuman pre-training followed by MOT17 fine-tuning.The comparison evaluates the effect of external pre-training data on the MOT17 validation set.Source: A. Training Data · p. 9, Table 8 · p. 9
• Fine-tuning on the combination of CrowdHuman and MOT data performs better than fine-tuning on MOT data alone.The ablation evaluates models on the test set of each MOT benchmark.Source: A. Training Data · p. 9, Table 9 · p. 9

Accuracy vs. Speed

• 15 FPS decreases to 10 FPS when the decoder count increases from 1 to 6, while MOTA improves significantly.The authors select 6 decoders as the default setting.Source: B. Accuracy vs. Speed · p. 9, Table 10 · p. 9
• MOTA saturates when the input image short side reaches 800 pixels.The authors use 800 pixels as the default input-size setting.Source: B. Accuracy vs. Speed · p. 9, Table 11 · p. 9
• The number of decoders and input image size jointly expose a trade-off between inference speed and tracking accuracy.Decoder increases affect FPS and MOTA, whereas increasing resolution eventually stops improving MOTA.Source: B. Accuracy vs. Speed · p. 9, Table 10 · p. 9, Table 11 · p. 9