PaperlayerView exact revision on arXiv

Source-linked AI summary

TransTrack: Multiple Object Tracking with Transformer

Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, Ping Luo

arXiv:2012.15460v2cs.CV

TL;DR

Multiple object tracking requires both discovering newly appearing objects and associating previously tracked objects without a complex multi-stage pipeline. TransTrack uses transformer queries for joint detection and tracking, achieving 74.5 MOTA on MOT17 and 64.5 MOTA on MOT20.

  • Problem

    Existing MOT pipelines are complex, and directly transferring SOT query-key mechanisms misses newly appearing objects because they lack previous-frame features.

  • Method

    TransTrack uses learned object queries for detection and previous-frame object features as track queries, producing detection and tracking boxes in a joint transformer framework.

  • Results

    74.5 MOTA on MOT17 and 64.5 MOTA on MOT20 are reported on the benchmark test sets.

  • Takeaways & Limitations

    TransTrack provides a joint-detection-and-tracking perspective and a competitive baseline for multiple object tracking.

Abstract

from arXiv · show

In this work, we propose TransTrack, a simple but efficient scheme to solve the multiple object tracking problems. TransTrack leverages the transformer architecture, which is an attention-based query-key mechanism. It applies object features from the previous frame as a query of the current frame and introduces a set of learned object queries to enable detecting new-coming objects. It builds up a novel joint-detection-and-tracking paradigm by accomplishing object detection and object association in a single shot, simplifying complicated multi-step settings in tracking-by-detection methods. On MOT17 and MOT20 benchmark, TransTrack achieves 74.5\% and 64.5\% MOTA, respectively, competitive to the state-of-the-art methods. We expect TransTrack to provide a novel perspective for multiple object tracking. The code is available at: \url{https://github.com/PeizeSun/TransTrack}.

1. Introduction

TransTrack addresses the complexity of tracking-by-detection and the failure of directly transferring SOT query-key mechanisms to MOT. It combines learned queries for new objects with previous-frame features for association in a joint framework.

  • Existing MOT methods suffer from model complexity and computational cost because detection and association use multi-stage pipelines.
  • Separating detection and re-identification prevents them from benefiting from shared knowledge, motivating joint detection and tracking.
  • Directly transferring the SOT query-key mechanism causes false negatives because newly appearing objects lack corresponding previous-frame features.
  • TransTrack uses learned object queries to detect new objects and previous-frame track queries to maintain consistent tracklets.
  • 74.5 MOTA on MOT17 and 64.5 MOTA on MOT20 demonstrate competitive benchmark performance.

2. Related Work

Prior MOT work is dominated by tracking-by-detection, while joint-detection-and-tracking seeks single-stage processing. TransTrack applies the transformer query-key mechanism to MOT.

  • Transformer in vision tasks: Transformer uses attention-based query-key processing and has been applied successfully to detection, segmentation, 3D processing, and visual backbones.
  • Transformer in vision tasks: Before TransTrack, published transformer-based solutions for object tracking were not yet available, motivating a transformer model for MOT.
  • Tracking-by-detection: Tracking-by-detection first detects objects and then associates them using re-identification features, IoU, or other information.
  • Joint-detection-and-tracking: Joint-detection-and-tracking methods aim to perform detection and tracking simultaneously in a single stage.
  • TransTrack’s architecture uses a CNN, an encoder, two query-specific decoders, and IoU matching for final association.

3. TransTrack

TransTrack uses learned object queries and previous-frame track queries with shared transformer features to produce detection and tracking boxes. IoU matching combines them into final tracks, which are trained and propagated across frames.

  • TransTrack forms detection queries from learned parameters and track queries from previously detected object features.
  • The two query sources generate detection boxes and tracking boxes in parallel, then IoU matching produces the final ordered box set.
  • A transformer encoder-decoder architecture uses shared feature maps as keys for parallel detection and propagation decoders.
  • Object Detection: Learned object queries look up objects in global image features and output detection boxes.
  • Object Propagation: Previous-frame object features convey appearance and location information for locating corresponding objects in the current frame.
  • Box Association: Unmatched detection boxes are retained to create new tracklets after Kuhn-Munkres IoU matching.
  • The same training loss supervises detection and tracking boxes, while inference propagates and associates objects across following frames.

4. Experiments

TransTrack is evaluated on MOT17 using standard multiple-object-tracking metrics, with training and evaluation procedures described for the benchmark. Under the private-detector setting, it achieves competitive performance and precise object localization.

  • Evaluation setup: MOTA is the primary metric used to measure overall multiple object tracking accuracy.The evaluation uses the widely adopted MOT metrics set.
  • Evaluation setup: The ablation protocol splits MOT17 training data into training and validation portions, while test-set models use combined CrowdHuman and MOT data.Models are first pretrained on CrowdHuman and then fine-tuned on MOT during ablation.
  • MOT17 results: TransTrack achieves comparable results with state-of-the-art methods on MOT17 under the private-detector setting.Its MOTP and FN performance is especially competitive.
  • MOT17 results: TransTrack precisely localizes objects and successfully detects most objects, while its ID-switch score remains inferior to state-of-the-art methods.The paper links MOTP to localization precision and FN to successful detection, while describing ID-switch performance as promising but not state of the art.

4.3. MOT20 benchmark

MOT20 presents a more difficult benchmark than MOT17 because of crowded scenes, severe occlusion, and smaller objects. TransTrack nevertheless remains comparable to current state-of-the-art methods across detection and association metrics.

  • Benchmark characteristics: MOT20 contains more crowded scenes, more severe object occlusion, and smaller objects than MOT17.These characteristics make both detection and tracking more challenging.
  • Benchmark characteristics: All methods achieve lower performance on MOT20 than on MOT17 under the private-detector setting.The passage attributes the lower performance to MOT20's greater difficulty.
  • Results: TransTrack achieves comparable results with current state-of-the-art methods on MOT20 for detection and association metrics.The comparison is reported under the private-detector setting.

4.4. Ablation study

The ablations examine transformer architecture, decoder queries, matching strategies, and box association. They support using deformable multi-scale features and combining learned object queries with previous-frame track queries.

  • 4.4.1 Transformer Architecture: 55.4 MOTA is achieved by the original Transformer, while Transformer-DC5 improves by 3.6 MOTA through higher feature resolution.The higher-resolution DC5 variant also incurs the memory cost associated with dilation convolution.
  • 4.4.1 Transformer Architecture: 65.0 MOTA is achieved by Deformable Transformer, which fuses multiple-scale features and becomes TransTrack's default architecture.Transformer-P3 produces performance close to Transformer-DC5 without further gain from higher resolution.
  • 4.4.2 Query in Decoder: 58.3 MOTA is obtained using only learned object queries, but index-based association produces wrong matches when objects move long distances.The learned-query-only setting can detect objects but has limited association performance.
  • 4.4.2 Query in Decoder: Using only previous-frame track queries handles large object motion but cannot detect objects that appear after the first frame.The resulting limitation is numerous false negatives for new-coming objects.
  • 4.4.2 Query in Decoder: Combining learned object queries with previous-frame object-feature queries yields the best detection and tracking performance.The visualization associates single-query settings with ID switches or missing objects, whereas the two-query setting avoids these cases more effectively.
  • 4.4.3 Matching strategy of tracking boxes: 0.6% versus 0.4% ID-switch performance shows that matching with previous-frame objects does not improve ID-switch avoidance over current-frame bipartite matching.The result supports the query-key mechanism's ability to locate corresponding objects in the current frame.
  • 4.4.4 Box association: Hungarian matching and NMS merging have equivalent effects in the box-association stage.Both strategies merge track boxes and detection boxes into the desired ordered object set.

4.5. Comparisons with other trackers

TransTrack is compared with motion-model and Re-ID-based trackers to isolate the gains from its joint design. Independent Re-ID features improve Re-ID performance but degrade detection, while motion-based differences are most visible with larger inter-frame motion.

  • Comparisons with other trackers: TransTrack is compared with detector-plus-motion and detector-plus-Re-ID trackers to assess gains beyond its detector.The comparison uses Kalman filtering for motion and alternative Re-ID feature pathways.
  • Motion model: 0.5% versus 1.0% improvement appears when sampling one frame every four frames, where larger motion makes feature-query association more beneficial.On the high-frame-rate MOT17 videos, adjacent-frame motion is minor and association methods show little difference.
  • Motion model: Kalman filtering and TransTrack provide similar ID performance on MOT17 because the dataset's adjacent-frame object motion is small.The reported explanation attributes the limited difference to MOT17's 14–30 FPS frame rate.
  • Re-ID features: Independent Re-ID cross-attention improves Re-ID features over a shared layer but degrades the detector, so overall MOTA does not surpass default TransTrack.The independent pathway separates Re-ID and detection features for their respective supervision sources.

5. Conclusion

TransTrack establishes a transformer-based joint-detection-and-tracking pipeline that detects new objects and propagates previously detected ones. It achieves competitive MOTA on MOT17 and the more challenging MOT20 benchmark.

  • Conclusion: TransTrack uses learned object queries for detection and previous-frame track queries to propagate previously detected objects.The framework performs detection and tracking jointly within a transformer-based pipeline.
  • Conclusion: 74.5 MOTA on MOT17 and 64.5 MOTA on MOT20 demonstrate competitive benchmark performance.The paper presents TransTrack as a novel perspective for multiple object tracking.

A. Training Data

Training-data experiments show that external CrowdHuman data and combined fine-tuning data improve TransTrack's MOT17 performance. The reported comparisons isolate both pre-training and fine-tuning choices.

  • Training Data: 61.6 MOTA using only MOT17 training data rises to 64.8 MOTA after CrowdHuman pre-training followed by MOT17 fine-tuning.The comparison evaluates the effect of external pre-training data on the MOT17 validation set.
  • Training Data: Fine-tuning on the combination of CrowdHuman and MOT data performs better than fine-tuning on MOT data alone.The ablation evaluates models on the test set of each MOT benchmark.

B. Accuracy vs. Speed

TransTrack's accuracy–speed trade-off depends on decoder count and input resolution. More decoders improve MOTA but reduce FPS, while performance saturates at an 800-pixel short side.

  • Accuracy vs. Speed: 15 FPS decreases to 10 FPS when the decoder count increases from 1 to 6, while MOTA improves significantly.The authors select 6 decoders as the default setting.
  • Accuracy vs. Speed: MOTA saturates when the input image short side reaches 800 pixels.The authors use 800 pixels as the default input-size setting.
  • Accuracy vs. Speed: The number of decoders and input image size jointly expose a trade-off between inference speed and tracking accuracy.Decoder increases affect FPS and MOTA, whereas increasing resolution eventually stops improving MOTA.
Loading 2012.15460v2…