PaperlayerView exact revision on arXiv

Source-linked AI summary

TrackFormer: Multi-Object Tracking with Transformers

Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, Christoph Feichtenhofer

arXiv:2101.02702v3cs.CV

TL;DR

Multi-object tracking must initialize tracks, preserve identities, and form trajectories over time, while existing approaches often depend on separate association or auxiliary models. TrackFormer formulates MOT as frame-to-frame set prediction with an encoder-decoder Transformer and query-based attention, achieving state-of-the-art results on MOT17, MOT20, and MOTS20. Its public-detection evaluation also requires track-initialization filtering, and its spatial track-query representation is limited for long-term occlusions with large movement.

  • Problem

    Multi-object tracking requires simultaneous reasoning about track initialization, identity preservation, and spatio-temporal trajectories.

  • Method

    TrackFormer jointly performs detection and tracking by evolving static object queries and autoregressive identity-preserving track queries with Transformer attention.

  • Results

    TrackFormer achieves state-of-the-art performance on MOT17 and MOT20 tracking and MOTS20 segmentation benchmarks.

  • Takeaways & Limitations

    The tracking-by-attention paradigm handles initialization, identity, and trajectory formation without additional matching, graph optimization, or motion and appearance modeling.

Abstract

from arXiv · show

The challenging task of multi-object tracking (MOT) requires simultaneous reasoning about track initialization, identity, and spatio-temporal trajectories. We formulate this task as a frame-to-frame set prediction problem and introduce TrackFormer, an end-to-end trainable MOT approach based on an encoder-decoder Transformer architecture. Our model achieves data association between frames via attention by evolving a set of track predictions through a video sequence. The Transformer decoder initializes new tracks from static object queries and autoregressively follows existing tracks in space and time with the conceptually new and identity preserving track queries. Both query types benefit from self- and encoder-decoder attention on global frame-level features, thereby omitting any additional graph optimization or modeling of motion and/or appearance. TrackFormer introduces a new tracking-by-attention paradigm and while simple in its design is able to achieve state-of-the-art performance on the task of multi-object tracking (MOT17 and MOT20) and segmentation (MOTS20). The code is available at https://github.com/timmeinhardt/trackformer .

1. Introduction

TrackFormer addresses multi-object tracking by jointly performing detection and data association through attention. It introduces autoregressive track queries and reports state-of-the-art results across MOT17, MOT20, and MOTS20.

  • MOT requires following multiple object trajectories while preserving identities as objects move through video.
  • Tracking-by-regression implicitly associates tracks but typically relies on graph optimization or motion and appearance models.Its isolated local bounding-box regression lacks object identity and global communication between tracks.
  • TrackFormer introduces tracking-by-attention, jointly performing detection and data association while evolving tracks across frames.The approach forms trajectories by evolving a set of tracks from frame to frame.
  • State-of-the-art performance is reported on MOT17 and MOT20 tracking benchmarks and the MOTS20 segmentation benchmark.
  • Autoregressive track queries embed an object’s spatial position and identity to follow it through space and time.Static object queries detect new objects and subsequently transform into track queries.

2. Related work

Prior MOT methods organize tracking around detection association, regression, appearance, motion, or segmentation. TrackFormer instead casts the entire objective as set prediction and jointly reasons about initialization, identity, and trajectories with attention.

  • Tracking-by-detection forms trajectories by associating frame-wise detections over time.Existing approaches use graph optimization or learned matching models for association.
  • Appearance-based methods use similarity or re-identification features but struggle with object-object occlusions in crowded scenes.
  • Motion-based models predict trajectories using assumptions or learned dynamics, while projected nonlinear 3D motion remains challenging in 2D images.
  • Tracking-by-regression replaces detection association with regression of previous object locations to current positions but often adds re-identification, motion, or graph models.
  • Prior MOT attention methods associate detections without jointly solving detection and tracking.TrackFormer extends attention to the entire tracking objective as a single set prediction problem.

3. TrackFormer

TrackFormer formulates online MOT as frame-to-frame set prediction in a Transformer encoder-decoder. Static object queries initialize tracks, while autoregressive track queries carry identity and location across frames and support short-term recovery.

  • Architecture: TrackFormer extracts frame features with a CNN, encodes them with a Transformer encoder, and decodes queries using self- and encoder-decoder attention.The decoder outputs embeddings mapped to bounding boxes and class predictions through MLPs.
  • Query types: Static object queries initialize new tracks, while autoregressive track queries track existing objects across frames.The two query types are jointly decoded to support unified detection and tracking.
  • Track queries: Track queries carry identity information while adapting to changing object positions autoregressively.Self-attention over object and track queries enables new-object detection while avoiding re-detection of existing tracks.
  • Frame processing: At frame t = 0, valid object-query detections spawn track queries; later frames decode the joint set of Nobject + Ntrack queries.Tracks can be removed when classification confidence falls below σtrack or through NMS with threshold σNMS.
  • Re-identification: Short-term track recovery keeps removed track queries active for up to Ttrack-reid frames without dedicated re-identification training.The spatial information in track queries limits recovery during long-term occlusions with large object movement.
  • Training: Training uses two adjacent frames and optimizes set prediction over all N = Nobject + Ntrack outputs against ground-truth classes and boxes.Ground-truth objects are assigned to predictions through track identity or costs based on box similarity and class.

4. Experiments

TrackFormer is evaluated on MOT17, MOT20, and MOTS20 using private and public detection settings, segmentation metrics, and component ablations. It achieves state-of-the-art benchmark results while using attention-based association and shows that training augmentations and joint segmentation training affect tracking performance.

  • Benchmarks: TrackFormer is evaluated on MOT17 and MOT20 for tracking and on MOTS20 for tracking and segmentation.MOT17 and MOT20 include private and public detection settings, while MOTS20 evaluates instance-level segmentation.
  • Benchmark results: Public-detection TrackFormer performs on-par in MOTA with state-of-the-art methods without CrowdHuman pretraining, while its identity preservation is surpassed only by one method and offline approaches.Public detection evaluation uses minimum-IoU track-initialization filtering.
  • Benchmark results: MOTA (+5.0) and IDF1 (1.7) establish new state-of-the-art results on private MOT17 detections among methods trained only on CrowdHuman.Methods pretrained on six additional tracking datasets surpass these results.
  • Benchmark results: TrackFormer surpasses or matches several methods on MOT20 while using only CrowdHuman pretraining, despite those methods using six additional tracking datasets.The paper presents this as the first MOT20 method pretrained only on CrowdHuman.
  • Analysis and ablations: Global attention provides top performance without additional motion or appearance models, while track queries avoid heuristic greedy matching and graph optimization.The approach jointly addresses track initialization and association within one formulation.
  • Benchmark results: MOTSA and IDF1 achieve state-of-the-art results on MOTS20, without a dedicated mask-tracking formulation.TrackFormer predicts instance-level masks with a segmentation head and is evaluated in a private setting.
  • Analysis and ablations: Removing pretraining reduces MOTA by 2.0 and IDF1 by 1.6, while removing false-positive and frame-range augmentations reduces them by 5.2 and 11.2 points.False-negative augmentation is retained because it is described as indispensable for joint object- and track-query training.
  • Analysis and ablations: Joint mask training improves tracking IDF1 by 1.2 without improving MOTA, helping resolve ambiguous occlusion scenarios during training.The ablation evaluates bounding-box tracking performance.

5. Conclusion

The paper presents tracking-by-attention as a unified Transformer approach for detection and multi-object tracking. TrackFormer jointly handles initialization, identity, and trajectory formation without additional matching, graph optimization, or motion/appearance modeling, achieving state-of-the-art results for tracking and segmentation.

  • TrackFormer applies autoregressive track-query embeddings to follow objects across video sequences.Each query represents an object and evolves through the sequence.
  • The Transformer encoder-decoder jointly addresses track initialization, identity, and trajectory formation without additional matching or motion/appearance models.
  • The approach achieves state-of-the-art results for multi-object tracking and segmentation.

A.1. Backbone and training

The implementation uses a six-layer, eight-head Deformable DETR encoder and decoder and follows Deformable DETR training hyperparameters. It avoids the DC5 backbone because of its larger memory requirement.

  • The Deformable DETR encoder and decoder each use 6 layers with 8 attention heads.
  • The DC5 backbone is omitted because its higher final-stage resolution would require substantially more memory.
  • Training follows Deformable DETR, with λcls = 2, λℓ1 = 5, and λiou = 2.
  • Track augmentation uses pFN = 0.4 and pFP = 0.1, while MOT17 frames receive 1% image-size jitter.

A.2. Dataset splits

Dataset-split experiments use the same private training pipeline for each split. MOT17 ablations separate sequences and train on earlier frames, while MOTS20 validation averages metrics across splits.

  • All ablation studies and MOTS20 training-set experiments apply the same private training pipeline to each split.
  • For MOT17, seven sequences are divided into two splits, training on the first 50% and evaluating on the last 50% of frames.
  • MOTS20 validation metrics are averaged over all splits, with results reported from a single epoch producing the best mean MOTA / MO.

A.3. Transformer encoder-decoder architecture

The architecture visualization emphasizes track identities through color-coded decoder queries and distinguishes shared-dimensional frame features from the query types. Track initialization, removal, and re-identification are controlled by tuned thresholds and patience settings.

  • The visualization uses color-coded queries to indicate track identities instead of spatial encoding.
  • CNN-extracted frame features have the same number of channels as both query types.
  • Grid search sets detection and track thresholds to σdetection = 0.4 and σtrack = 0.4.
  • Track re-identification uses an inactive patience of Ttrack-reid = 5 and score threshold σtrack-reid = 0.4.

B.1. Public detections and track filtering

TrackFormer requires filtering track initializations to compare with public-detection methods, but MOT17 lacks a standardized filtering protocol. The authors favor IoU filtering because it better reflects detector-quality differences and yields a more challenging comparison.

  • Public detection comparability: MOT17 lacks a standardized protocol for filtering TrackFormer’s track initializations by public detections.TrackFormer cannot directly produce tracking outputs from detection inputs, unlike classic tracking-by-detection methods.
  • Center-distance filtering: Center-distance filtering initializes a track when a public detection’s center falls inside the corresponding track bounding box.
  • Center-distance filtering: Center-distance filtering produces only small differences between DPM and SDP results despite their expected detector-quality gap.The authors report that DPM and SDP should be worst and best, respectively, but their filtered results differ little.
  • IoU filtering: IoU filtering initializes tracks only when public detections overlap the corresponding box by more than 0.5.This protocol is intended to provide a fairer comparison with methods that directly process public detections.
  • IoU filtering: IoU filtering is more challenging than center-distance filtering and produces detector differences that correlate more meaningfully with detection performance.The authors therefore use IoU-based filtering for the main comparison with CenterTrack.
  • Comparison with CenterTrack: TrackFormer’s IDF1 gain over CenterTrack is significantly larger under IoU filtering, while its high IDF1 accuracy does not rely on the filtering protocol.

B.2. MOT17, MOT20 and MOTS20 results

The supplementary results cover MOT17, MOT20, and MOTS20, reporting per-sequence tracking, segmentation, and standard CLEAR MOT evaluation measures. The tables organize results across private and public-detection settings.

  • Evaluation metrics: The evaluation reports MOTA and IDF1 alongside CLEAR MOT metrics including MT, ML, FP, FN, and ID switches.For segmentation, sMOTSA counts true positives rather than only masks with IoU larger than 0.5.
  • MOT17 results: TrackFormer results are reported per sequence for private and public detection filtering on the MOT17 test set.Table A.2 covers private results, while Table A.3 applies minimum IoU filtering to public detections.
  • MOT20 results: TrackFormer private-detection results are reported per sequence on the MOT20 test set.The corresponding results are provided in Table A.4.
  • MOTS20 results: TrackFormer tracking and segmentation results are reported per sequence on the MOTS20 test set under private detections.The corresponding results are provided in Table A.5.
  • Comparison results: Table A.6 gives CenterTrack’s original per-sequence MOT17 results with center-distance public-detection filtering.The table notes that these results do not reflect varying DPM, FRCNN, and SDP detection quality.
Loading 2101.02702v3…