PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Transformer Tracker for Object Tracking

Fan Ma, Mike Zheng Shou, Linchao Zhu, Haoqi Fan, Yilei Xu, Yi Yang, Zhicheng Yan

arXiv:2203.15175v2cs.CV

TL;DR

Current tracking methods generally train separately for SOT and MOT, despite the practical value of a system that can switch between them. UTT uses a shared transformer tracker trained on both tasks. The authors report comparable performance to state-of-the-art methods on SOT and MOT benchmarks, while noting limits in online MOT tracking and MOT category coverage.

  • Problem

    Current tracking methods train separately for SOT and MOT, whose datasets and tracking objects differ, while maintaining separate systems can be expensive and inefficient.

    Source: §1 · pp. 1–2

  • Method

    UTT uses one track transformer to correlate target and tracking-frame features for localization, and is end-to-end trained on datasets from both SOT and MOT.

    Source: §1 · p. 2

  • Results

    UTT achieves comparable performance to state-of-the-art algorithms on LaSOT, TrackingNet, and GOT-10k in SOT and MOT16 in MOT.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A single model is enough to address both SOT and MOT tasks, according to the authors’ conclusion.

    Source: §5 · p. 8

    The tracker cannot currently perform online tracking on MOT and is trained for pedestrians only in MOT.

    Source: D. Limitations · p. 12

Abstract

from arXiv · show

As an important area in computer vision, object tracking has formed two separate communities that respectively study Single Object Tracking (SOT) and Multiple Object Tracking (MOT). However, current methods in one tracking scenario are not easily adapted to the other due to the divergent training datasets and tracking objects of both tasks. Although UniTrack \cite{wang2021different} demonstrates that a shared appearance model with multiple heads can be used to tackle individual tracking tasks, it fails to exploit the large-scale tracking datasets for training and performs poorly on single object tracking. In this work, we present the Unified Transformer Tracker (UTT) to address tracking problems in different scenarios with one paradigm. A track transformer is developed in our UTT to track the target in both SOT and MOT. The correlation between the target and tracking frame features is exploited to localize the target. We demonstrate that both SOT and MOT tasks can be solved within this framework. The model can be simultaneously end-to-end trained by alternatively optimizing the SOT and MOT objectives on the datasets of individual tasks. Extensive experiments are conducted on several benchmarks with a unified model trained on SOT and MOT datasets. Code will be available at https://github.com/Flowerfan/Trackron.

1. Introduction

UTT uses one tracking model for SOT and MOT, where current methods generally train separately for the two tasks. It is end-to-end trained on both, and the authors report comparable performance to state-of-the-art methods across SOT and MOT benchmarks.

1. Introduction

  • Current tracking methods generally train separately on individual datasets for either SOT or MOT.SOT tracks an annotated target of any category from the first frame, while MOT estimates boxes and identities for objects from known categories.Source: §1 · p. 1
  • Prior Siamese trackers have not addressed both tasks with a unified paradigm, while UniTrack’s shared backbone and multiple heads limit its use of large-scale training data.The introduction notes that UniTrack’s tracking capacity is limited and it can fail in complex scenarios.Source: §1 · pp. 1–2
  • A unified tracker could switch between tracking specific or unseen instances and perceiving general classes, avoiding two separate systems in applications such as AR/VR.The paper describes maintaining separate systems as expensive and inefficient.Source: §1 · p. 2
  • UTT correlates target features with tracking-frame features and repeatedly updates localization using a new search proposal cropped from the previous localization.The paper presents this as a shared design for SOT and MOT and alternately trains on datasets from each task.Source: §1 · p. 2
  • The authors present UTT as an end-to-end tracking model trained on both SOT and MOT tasks.They identify this as the first such model to their knowledge.Source: §1 · p. 2
  • UTT achieves comparable performance to state-of-the-art algorithms on LaSOT, TrackingNet, and GOT-10k for SOT and on MOT16 for MOT.The authors evaluate a unified model on benchmarks for both tracking settings.Source: §1 · p. 2

2. Related Work

Prior work studies SOT and MOT with different tracking approaches, while unified tracking has sought a shared model across the tasks. UTT uses a shared tracking head and transformer-based feature correlation to address both settings.

2.1. Single Object Tracking

  • SOT methods include correlation-filter and Siamese trackers, with transformer approaches using target representations; UTT instead uses correlation attention between target and search features.The paper notes that cropping targets and tracking frames for every object becomes inefficient when multiple objects are specified.Source: §2.1 · p. 2

2.2. Multiple Object Tracking

  • MOT commonly links detections across frames; transformer-based methods propagate boxes, while UTT applies self-attention to multiple target features and correlation attention to update them with search features.The UTT framework uses a target decoder and proposal decoder to produce target features and candidate search areas.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

2.3. Unified Tracking

  • UniTrack shares a backbone but uses task-specific heads; UTT instead shares one tracking head and can be trained on both SOT and MOT datasets.The paper says UniTrack’s SOT performance is not competitive with current state-of-the-art methods.Source: §2.3 · p. 3

3. Method

UTT uses a shared track-transformer framework for SOT and MOT, treating reference boxes and frame features as inputs to localization. Its design combines target-context encoding, proposal-based search, and alternating task-specific training.

3.1. Overview

  • The framework treats the initial SOT frame or previous MOT frame as the reference, using its boxes and features with tracking-frame features to predict target locations.SOT reference boxes are initial annotations; MOT reference boxes are detections.Source: §3.1 · p. 3

3.2. Track Transformer

  • The track transformer processes target features and candidate search areas to predict target localization in the tracking frame.Its inputs are tracking-frame features, reference features, and reference boxes.Source: §3.2 · p. 4

3.2.1 Target Decoder

  • Multi-head cross-attention adds tracking-frame context to RoIAlign target features, which are then refined by a feed-forward network.The target feature is the query, while tracking-frame features provide the keys and values.Source: §3.2.1 · p. 4

3.2.2 Proposal Decoder

  • A proposal decoder uses target-to-frame correlation to generate heat maps for target corners and form candidate search areas.The proposals reduce search area; relying only on the previous box can fail when it contains background or the target reappears elsewhere.Source: §3.2.2 · p. 4

3.2.3 Target Transformer

  • The target transformer crops a fixed-size search region per object, updates target features through correlation, and predicts localization as a delta to the proposal.For N objects and an H × W frame feature map, its stated complexity is O(K^2NC), versus O(HWNC) for cross-attention.Source: §3.2.3 · pp. 4–5

3.3. Training

  • UTT alternates SOT and MOT training iterations, using task-specific objectives and data loaders within one model.MOT training uses noisy ground-truth boxes as initial proposals and includes a detection head; SOT uses the proposal decoder and also penalizes the initial proposal.Source: §3.3 · p. 5
  • UTT outperforms UniTrack by a large margin on the SOT datasets in comparisons covering TrackingNet, LaSOT, and GOT-10k.The supplied caption reports no individual metric values.Source: Table 1 · p. 6

4. Experiment

UTT is trained by alternating SOT and MOT objectives across six datasets, then evaluated on benchmarks for both tasks. It is competitive with leading SOT trackers and reports the highest MOTP among MOT methods without person re-identification features.

4.1. Implementation Details

  • Training alternates SOT and MOT batches across COCO, TrackingNet, LaSOT, GOT10k, MOT16, and CrowdHuman.The model uses a ResNet backbone with ImageNet-pretrained weights and trains for 800K iterations.Source: §4.1 · p. 6

4.2. Evaluations in SOT

  • On LaSOT, UTT outperforms UniTrack by almost 30 points on Success and is competitive with TransT.It also outperforms most SOT trackers, including SiamRPN++ and KYS, on all reported datasets.Source: §4.2 · p. 6
  • On TrackingNet, UTT obtains 25.8% higher Success and 33.5% higher Precision than UniTrack, and is competitive with TransT.TransT crops both the target and tracking frame and cannot handle multiple object tracking.Source: §4.2 · p. 6
  • On GOT10k, UTT achieves the second-highest SR0.75 among all methods and outperforms most Siamese trackers.The comparison includes SiamRPN++, SiamFC++, and SiamAttn.Source: §4.2 · p. 6

4.3. Evaluations in MOT

  • UTT achieves the highest MOTP score, 81.4, among methods without person re-identification features, while its MOTA is slightly below FairMOT.UTT-DFDETR also outperforms CTracker on MOTA; the reported MOTP result is for UTT using FairMOT detections.Source: Table 2 · p. 7, §4.3 · p. 7

4.4. Ablative Studies

The ablations examine decoder and correlation design, transformer settings, and training modes; visualizations show the unified model tracking a specified volleyball alongside detected people.

4.4.1 Target- and Proposal Decoder

  • Adding the proposal decoder largely improves LaSOT tracking, and adding the target decoder further improves Success by 0.6.The authors link the proposal decoder’s benefit to correcting false proposals from previous-frame localization; the target decoder updates target features through interaction with the tracking frame.Source: §4.4.1 · p. 7, Table 3 · p. 7

4.4.2 Target Transformer

  • The proposed correlation improves Success by 1.8% on SOT and MOTA by 1.0% on MOT over UTT-MCA.The comparison replaces the correlation and self-attention modules with multi-head cross attention and also tests removing multi-head self-attention.Source: §4.4.2 · p. 7, Table 4 · p. 7
  • A search-feature pool size of 7 gives the highest performance on both tasks, while more target-transformer iterations improve performance but slow training and inference.The search feature is cropped using the target proposal and interacts with the target feature to update localization.Source: §4.4.2 · p. 8, Figure 3 · p. 8

4.4.3 Training Recipe

  • Unified training achieves higher MOTA but lower IDF1 than MOT-Only, while the SOT-only model cannot track MOT objects and the MOT-trained model performs poorly on SOT.The paper attributes the lower IDF1 to more ID switches and the higher MOTA to fewer false positives and false negatives.Source: §4.4.3 · p. 8

4.4.4 Visualization

  • In the unified tracking mode, the model tracks a specified volleyball alongside multiple people detected by the deformable DETR head, including a person occluded by others.The visualization supports concurrent SOT and MOT testing with one model.Source: §4.4.4 · p. 8, Figure 4 · p. 8

5. Conclusion

The conclusion argues that one end-to-end-trained model can address SOT and MOT. UTT uses a track transformer for different tracking scenarios.

5. Conclusion

  • The authors conclude that one UTT model, end-to-end trained on SOT and MOT datasets, is enough to address both tasks.They suggest this may encourage unified tracking algorithms for more varied tracking scenarios.Source: §5 · p. 8

A. MOT Detection & Tracking

The MOT system combines detections with track-transformer localization to match objects across frames and maintain identities.

A. MOT Detection & Tracking

  • The MOT setup uses deformable DETR as its detection head and trains the backbone, detection head, and track transformer together.The tracking frame receives detected boxes from the reference frame as target proposals; the tracker outputs tracked boxes while the detector also produces current-frame detections.Source: A. MOT Detection & Tracking · p. 11
  • The system matches tracked and detected boxes using an IoU threshold of 0.9, marks unmatched tracks as lost, and assigns new IDs to unmatched detections.Matched objects retain their identities across frames.Source: A. MOT Detection & Tracking · p. 11

B. Unified Training

UTT training alternates updates from SOT and MOT data loaders, applying task-specific tracking objectives and including detection losses for MOT.

B. Unified Training

  • SOT updates the backbone and track transformer using tracking box loss, with target and proposal decoders producing target features and candidate search areas.The SOT iteration predicts localization in reference and tracking frames and calculates the SOT loss.Source: B. Unified Training · p. 11
  • Each unified-training iteration updates the model first with SOT data and then with MOT data.The procedure uses separate SOT and MOT data loaders.Source: B. Unified Training · p. 11
  • MOT training uses detection and tracking losses to update the backbone, detection head, and track transformer.The detection head predicts box localization and classification; noisy ground-truth proposals have minimum IoU 0.1 with the ground-truth boxes to ensure target inclusion.Source: B. Unified Training · pp. 11–12

C. Detailed Results on MOT16

MOT16 tracking results are evaluated through an online server, with per-video scores reported in Tables 6 and 7 and two model variants described. [p6]

C. Detailed Results on MOT16

D. Limitations

The authors identify online MOT tracking and training beyond pedestrians as limitations of the current tracker. [p7]

D. Limitations

  • The tracker does not yet support online tracking on MOT, although the target speed is ideally over 30 FPS.Source: D. Limitations · p. 12
  • The MOT task is trained only on pedestrians, while broader categories such as vehicles remain to be considered.The authors note that adding categories is more related to the detection head.Source: D. Limitations · p. 12
  • The manuscript focuses mainly on the unified transformer tracker rather than the detection head.Source: D. Limitations · p. 12

E. Code

The implementation is based mainly on Detectron2 and Pytracking and supports distributed training and testing. [p9]

E. Code

  • The implementation is based mainly on Detectron2 and Pytracking.Source: E. Code · p. 12
  • The implementation supports distributed training and testing.Source: E. Code · p. 12
  • The code is described as making previous methods, especially Siamese pipelines, easy to reproduce.Source: E. Code · p. 12

Related papers

Loading 2203.15175v2…