PaperlayerView exact revision on arXiv

Source-linked AI summary

Transforming Model Prediction for Tracking

Christoph Mayer, Martin Danelljan, Goutam Bhat, Matthieu Paul, Danda Pani Paudel, Fisher Yu, Luc Van Gool

arXiv:2203.11192v1cs.CV

TL;DR

Optimization-based tracking provides useful global reasoning but limits target-model expressivity and use of current-frame information. The paper introduces ToMP, a Transformer-based predictor that generates localization and bounding-box-regression weights, achieving state-of-the-art results on three benchmarks.

  • Problem

    Optimization-based model prediction limits target-model flexibility by relying on previous frames and preventing learned priors or current test-frame information from contributing to weight prediction.

  • Method

    ToMP uses a Transformer-based model predictor to estimate weights for a compact target model and a second set of weights for precise bounding-box regression, using target location and size encodings.

  • Results

    ToMP sets a new state of the art on three tracking benchmarks and achieves 68.5% AUC on LaSOT.

  • Takeaways & Limitations

    The framework bridges DCF-style compact target models with Transformer-based prediction and supports transductive estimation of tracking weights from training and test-frame information.

  • Takeaways & Limitations

    Transformer self-attention creates a large memory footprint affecting training and inference runtime, while distractors during target occlusion remain a typical failure scenario.

Abstract

from arXiv · show

Optimization based tracking methods have been widely successful by integrating a target model prediction module, providing effective global reasoning by minimizing an objective function. While this inductive bias integrates valuable domain knowledge, it limits the expressivity of the tracking network. In this work, we therefore propose a tracker architecture employing a Transformer-based model prediction module. Transformers capture global relations with little inductive bias, allowing it to learn the prediction of more powerful target models. We further extend the model predictor to estimate a second set of weights that are applied for accurate bounding box regression. The resulting tracker relies on training and on test frame information in order to predict all weights transductively. We train the proposed tracker end-to-end and validate its performance by conducting comprehensive experiments on multiple tracking datasets. Our tracker sets a new state of the art on three benchmarks, achieving an AUC of 68.5% on the challenging LaSOT dataset.

1. Introduction

Optimization-based trackers provide global reasoning but impose inductive bias that limits target-model flexibility. ToMP replaces this prediction mechanism with a Transformer-based predictor and extends it to bounding-box regression.

  • DCF trackers learn target models by minimizing discriminative objectives over previous frames, integrating foreground and background information for global reasoning.
  • Figure 1 evaluates stepwise changes using success AUC on NFS and LaSOT, including separate ToMP-50 and ToMP-101 backbone variants.
  • Optimization-based prediction limits flexibility because it cannot incorporate learned priors or current test-frame information when computing target-model weights.
  • ToMP uses a Transformer-based model predictor to generate weights for a compact target model, retaining DCF-style localization while learning more powerful models.
  • The predictor also generates a second set of weights for a bounding-box regressor conditioned on the current target.

2. Related Work

Related trackers combine discriminative model prediction, feature fusion, and Transformer architectures in different ways. These approaches vary in how they generate target representations and produce localization or bounding-box predictions.

  • Earlier DCF methods learn target models by explicitly optimizing discriminative objectives, with later work enabling end-to-end training by unrolling iterative optimization.
  • DTT uses two networks to predict target location and bounding box from extracted features.
  • TransT fuses features through self- and cross-attention modules before separate target-classification and bounding-box-regression heads.
  • TrDiMP combines Transformer Encoder features with the DiMP model predictor and uses a probabilistic IoUNet for bounding-box regression.
  • STARK jointly processes stacked training and test features with a DETR-style Transformer and directly predicts the target bounding box from decoder-related features.

3. Method

ToMP replaces optimization-based target model prediction with a Transformer that jointly reasons over training and test frames, predicts target-model weights, and extends prediction to bounding-box regression.

  • 3.1. Background: Optimization-based trackers derive target-model weights by minimizing an objective over annotated and pseudo-labeled training frames.The objective combines prediction residuals with regularization, while the training set includes the annotated first frame and previously tracked frames.
  • 3.1. Background: These methods impose inductive bias, cannot integrate learned priors or current test-frame information transductively, and require optimizer hyper-parameters.They may also overfit or underfit the training samples and use features without target-state enhancement.
  • 3.2. Transformer-based Target Model Prediction: ToMP replaces the model optimizer with a Transformer-based predictor that directly learns target-model weights from data through end-to-end training.The architecture retains a compact DCF target model while allowing the predictor to integrate target-specific priors and current test-frame features.
  • 3.2. Transformer-based Target Model Prediction: Target-state encodings combine foreground embeddings with Gaussian target-location maps, while a test encoding marks features belonging to the test frame.These encodings are fused with deep image features before joint Transformer processing.
  • 3.2. Transformer-based Target Model Prediction: A Transformer Encoder jointly processes reshaped training and test features, and its outputs are used by a Decoder to predict target-model weights.Self-attention supports global reasoning across full frames and across multiple training and test frames; the predicted filter is applied to encoded test features for localization.
  • 3.3. Joint Localization and Box Regression: ToMP jointly predicts target-localization and bounding-box-regression models because the related tasks can benefit from one another.Bounding-box supervision uses an ltrb representation with normalized distances from feature-map locations to the four box sides.

4. Experiments

Experiments evaluate ToMP through ablations and comparisons across seven tracking benchmarks. The results show that its encoding choices, two-stage model prediction, and Transformer-based prediction yield strong performance, including state-of-the-art results, while memory use and distractors remain limitations.

  • Experimental Setup: ToMP-101 and ToMP-50 achieve 19.6 and 24.8 FPS, respectively, using ResNet-101 and ResNet-50 backbones.The tracker is implemented in PyTorch within PyTracking and evaluated on a single Nvidia RTX 2080Ti GPU.
  • Ablation Study: Using only foreground and test embeddings gives the best target-state encoding performance, while adding a learned background embedding decreases tracking performance.Removing the foreground embedding also reduces performance, and omitting target extent information causes a significant drop.
  • Ablation Study: A separate Transformer query for bounding-box regression decreases performance, supporting a shared query conditioned on the foreground embedding.The model predictor estimates weights for both the target model and bounding-box regressor, but decoupling the query produces a significant performance drop.
  • Ablation Study: Disabling two-stage model prediction decreases performance by 5.6 AUC on LaSOT.Including previous tracking results improves performance over using only the initial frame, while inaccurate predicted boxes can harm bounding-box regression.
  • Comparison to the State of the Art: On LaSOT, both ToMP variants outperform STARK, TransT, TrDiMP, and DTT in AUC, with ToMP-50 exceeding STARK-ST101 at 67.6 versus 67.1.ToMP also outperforms TransT and TrDiMP by more than one percentage point on every annotated LaSOT attribute.
  • Comparison to the State of the Art: ToMP exceeds KeepTrack on UAV123 by +0.5% and +0.3%, while ToMP-101 achieves the best overall VOT2020 performance with the highest robustness.On TrackingNet, ToMP-101 achieves the second-best AUC behind STARK and outperforms other Transformer-based trackers such as TransT and TrDiMP.
  • Limitations: Self-attention creates a large memory footprint affecting training and inference runtime, and distractors during target occlusion are a typical failure scenario.The latter reflects ToMP's lack of explicit distractor handling as used by KeepTrack.

5. Conclusion

ToMP combines a Transformer-based model predictor with compact target localization and precise bounding box regression, achieving state-of-the-art performance on three challenging datasets.

  • ToMP uses a Transformer-based model predictor to estimate weights for a compact target model that localizes the target in the test frame.
  • The predictor also produces a second set of weights for precise bounding box regression.
  • Two new modules encode target location and bounding box information in the training features.
  • ToMP receives comprehensive experimental validation on several challenging datasets and sets a new state of the art on three.

Appendices

The supplementary material expands the paper with implementation details, architecture and inference descriptions, visual comparisons, failure cases, and detailed experimental results.

  • Supplementary Section A provides details about training, model architecture, and inference.
  • Supplementary Section B reports visual comparisons with state-of-the-art trackers, different model predictors, and failure cases.
  • Supplementary Section C provides more detailed results for the experiments presented in the main paper.

A. Training, Architecture and Inference

This appendix organizes additional material around training, architecture, and inference, with each topic described in sequence.

  • The appendix first provides additional training details.
  • It then gives a detailed description of the employed architectures.
  • Finally, it provides further inference details.

A.1. Training and Architecture Details

The appendix includes training settings, feature-processing details, and supplementary visual and tabular analyses of model predictors and inference settings.

  • Training: Training target states use a Gaussian with standard deviation 1/4 relative to the base target size, with τ = 0.05 separating foreground and background regions.
  • Visual analysis: Figure 6 visually compares target score maps produced by different model predictors.
  • Inference analysis: Table 9 analyzes inference settings and their impact on tracking performance using success-curve AUC.
  • Architecture: Model-predictor features come from the third ResNet block with stride 16, while a convolution reduces 1024 channels to the Transformer's 256-channel dimension.

A.2. Inference Details

Inference combines confidence-based sample selection with a Transformer-predicted target model and bounding-box regression design. Experiments also examine training-frame count and centerness components.

  • Inference Details: A confidence threshold η selects previous tracking results for training, with threshold 0.9 yielding high performance on LaSOT, NFS, and OTB-100.The decision uses the maximal target score-map value.
  • Training Samples: The supplementary experiments compare using one initial and one recent frame against larger sets of training frames.Additional initial frames use vertical flipping and random translation for bounding-box augmentation.
  • Bounding Box Regression: The tracker uses a compact target-model branch together with a bounding-box regression component inspired by FCOS.The classification branch directly localizes the object center, while bounding-box regression estimates accurate target extents.
  • Centerness Ablation: Omitting centerness during training and inference achieves the best tracking performance.Adding centerness can reduce performance because it may fail to identify the target among distractors.

B. Visual Results

Visual analyses compare ToMP with optimization-based and Transformer trackers, showing stronger localization in distractor-heavy visible-target scenes but identifying recovery failures under occlusion and close interactions.

  • Visual Comparisons: ToMP-101 generally produces more robust and sometimes more accurate bounding boxes than SuperDiMP and STARK-ST101 across LaSOT sequences.Its robustness is especially evident when distractors are present while the target remains partly visible without full occlusion.
  • Target Model Prediction: The Transformer model predictor produces cleaner, less ambiguous target score maps than DiMP-based optimization prediction.It suppresses distractor-induced local maxima, although this can make recovery less likely after the target is lost.
  • Failure Cases: ToMP typically fails when distractors coincide with target occlusion, approach, or overlap.When objects merge, the tracker may detect one object and later fail to distinguish the target from the distractor.
  • Benchmark Results: On NFS, ToMP is almost as robust as KeepTrack while achieving superior accuracy and a new state of the art.The broader benchmark comparison covers overall AUC on OTB-100, NFS, and UAV123.

C. Experiments

Additional experiments evaluate ToMP with a segmentation-refinement add-on and provide LaSOT and LaSOTExtSub precision analyses against competing methods.

  • VOT2020ST: ToMP produces bounding boxes, so AlphaRefine is added to generate segmentation masks for comparison on the VOT2020 short-term challenge.This aligns ToMP’s output with methods evaluated using segmentation masks.
  • VOT2020ST: ToMP-101 achieves competitive VOT2020ST performance when combined with AlphaRefine.The passage specifically reports that ToMP-101 achieves the same EAO as a comparison method, but the remainder of the value is truncated.

C.2. UAV123, OTB-100 and NFS

Results on UAV123, OTB-100, and NFS show a recurring accuracy advantage for ToMP, with the clearest state-of-the-art result on NFS and broader attribute-level comparisons on LaSOT.

  • UAV123 and OTB-100: On UAV123 and OTB-100, ToMP reaches the highest accuracy at T > 0.7 despite lower robustness than some competing trackers at T < 0.6.KeepTrack and PrDiMP50 show higher robustness in the cited UAV123 comparison, with similar conclusions reported for OTB-100.
  • NFS: On NFS, ToMP is almost as robust as KeepTrack but achieves superior accuracy, leading to a new state of the art.The comparison is reported using success-plot behavior and overall AUC.
  • Normalized Precision: Normalized precision evaluates center-location accuracy relative to target size, with tracker ranking determined by AUC.The supplied plots cover LaSOT and LaSOTExtSub test sets.
  • LaSOT Attribute Analysis: The LaSOT attribute analysis reports ToMP-101 as best on all but three attributes.It ranks second for Motion Blur and third for Full Occlusion, with ToMP-50 also ahead for Full Occlusion.
Loading 2203.11192v1…