PaperlayerView exact revision on arXiv

Source-linked AI summary

MixFormerV2: Efficient Fully Transformer Tracking

Yutao Cui, Tianhui Song, Gangshan Wu, Limin Wang

arXiv:2305.15896v2cs.CV

TL;DR

Transformer trackers offer strong accuracy but remain inefficient for practical GPU and CPU deployment. MixFormerV2 removes dense convolutional and complex score-prediction modules with prediction tokens, simple MLP heads, and distillation-based model reduction, achieving strong accuracy–speed trade-offs across platforms.

  • Problem

    Transformer-based trackers achieve strong benchmark accuracy, but their inference efficiency remains an obstacle to practical deployment on GPU and CPU platforms.

  • Method

    MixFormerV2 applies a unified transformer to template, search, and four prediction tokens, uses simple MLP heads for box and confidence prediction, and reduces models through dense-to-sparse and deep-to-shallow distillation.

  • Results

    MixFormerV2 obtains a favorable accuracy–speed trade-off, with MixFormerV2-B reaching 70.6% AUC on LaSOT at 165 FPS and MixFormerV2-S surpassing FEAR-L by 2.7% AUC on LaSOT at real-time CPU speed.

  • Takeaways & Limitations

    The framework provides a unified fully transformer tracker with high-performance tracking on both GPU and CPU platforms.

  • Takeaways & Limitations

    Training MixFormerV2-S has substantial overhead because multiple distillation and pruning stages each require 500 epochs.

Abstract

from arXiv · show

Transformer-based trackers have achieved strong accuracy on the standard benchmarks. However, their efficiency remains an obstacle to practical deployment on both GPU and CPU platforms. In this paper, to overcome this issue, we propose a fully transformer tracking framework, coined as \emph{MixFormerV2}, without any dense convolutional operation and complex score prediction module. Our key design is to introduce four special prediction tokens and concatenate them with the tokens from target template and search areas. Then, we apply the unified transformer backbone on these mixed token sequence. These prediction tokens are able to capture the complex correlation between target template and search area via mixed attentions. Based on them, we can easily predict the tracking box and estimate its confidence score through simple MLP heads. To further improve the efficiency of MixFormerV2, we present a new distillation-based model reduction paradigm, including dense-to-sparse distillation and deep-to-shallow distillation. The former one aims to transfer knowledge from the dense-head based MixViT to our fully transformer tracker, while the latter one is used to prune some layers of the backbone. We instantiate two types of MixForemrV2, where the MixFormerV2-B achieves an AUC of 70.6\% on LaSOT and an AUC of 57.4\% on TNL2k with a high GPU speed of 165 FPS, and the MixFormerV2-S surpasses FEAR-L by 2.7\% AUC on LaSOT with a real-time CPU speed.

1 Introduction

Efficient tracking requires maintaining high accuracy while reducing the computational cost of transformer-based trackers, especially for deployment on CPUs. MixFormerV2 addresses this challenge with a fully transformer architecture and distillation-based reduction, achieving strong accuracy–speed trade-offs on GPU and CPU platforms.

  • Motivation: Transformer trackers improve accuracy through unified feature extraction and target integration, but inference efficiency remains a deployment obstacle, particularly on CPUs.Existing pipelines may combine a transformer backbone with dense corner prediction and complex score prediction modules.
  • Fully Transformer Tracking: MixFormerV2 concatenates four learnable prediction tokens with template and search tokens, then uses simple MLP heads to predict box coordinates and confidence.The prediction tokens encode template–search relations, while the MLP heads directly regress coordinate probability distributions.
  • Distillation-Based Reduction: MixFormerV2 uses dense-to-sparse and deep-to-shallow distillation to transfer knowledge from dense-head trackers and prune backbone layers.An intermediate teacher further bridges large and small models for real-time CPU tracking.
  • Results: 70.6% AUC on LaSOT at 165 FPS is achieved by MixFormerV2-B, while MixFormerV2-S surpasses FEAR-L by 2.7% AUC on LaSOT at real-time CPU speed.The two variants target efficient deployment across GPU and CPU platforms.

2 Related Work

Related work spans efficient visual tracking, knowledge distillation, and vision-transformer compression. These lines of research provide the context for MixFormerV2’s efficiency-oriented architecture and model-reduction strategy.

  • Efficient Visual Object Tracking: Efficient visual tracking research includes Siamese-based, online, and transformer-based architectures aimed at practical tracking.Recent transformer trackers increasingly use attention-based structures while pursuing improved efficiency and effectiveness.
  • Knowledge Distillation: Knowledge distillation transfers information from teacher models to student models, using prediction or feature supervision for classification and regression tasks.Prior regression approaches include feature mimicking and logits distillation for bounding-box locations.
  • Vision Transformer Compression: Vision-transformer compression uses quantization, distillation, pruning, and neural architecture search, including methods that prune tokens or optimize ViT architectures.These approaches target faster inference while preserving useful model capacity.

3 Method

MixFormerV2 is a fully transformer tracker that predicts boxes and confidence from mixed template, search, and learnable prediction tokens through simple MLP heads. Its distillation-based reduction combines dense-to-sparse knowledge transfer with progressive deep-to-shallow pruning to improve efficiency.

  • Fully Transformer Tracking: MixFormerV2 applies a plain vision transformer to template, search, and learnable prediction tokens, replacing convolutional tracking heads with simple token-based MLP heads.The prediction tokens support direct end-to-end bounding-box and quality-score prediction.
  • Fully Transformer Tracking: Learnable prediction tokens capture template–search correlations through mixed attention and provide compact target-aware representations for regression and classification.They are trained to integrate template and search-area information progressively.
  • Token-Based Prediction: The tracker predicts probability distributions for the four box coordinates and estimates confidence by averaging the prediction tokens before a two-layer Score Head.A shared MLP processes the four coordinate tokens, while the Score Head predicts a real-valued confidence score.
  • Distillation-Based Model Reduction: Dense-to-sparse distillation transfers localization knowledge from MixViT’s dense corner heads to MixFormerV2’s sparse token-based regression heads.MixViT’s two-dimensional corner distributions are converted into one-dimensional coordinate distributions and used as soft labels with KL-divergence supervision.
  • Distillation-Based Model Reduction: Deep-to-shallow distillation progressively prunes transformer layers by decaying selected block weights toward zero while supervising matched student and teacher representations.The eliminated layers become identity transformations, allowing the remaining blocks to form the pruned student model.
  • Distillation-Based Model Reduction: MLP hidden dimensions are additionally reduced using feature mimicking and logits distillation to address CPU inference latency bottlenecks.The reduction targets the hidden feature dimension of MLPs in transformer blocks, identified as a key CPU-latency factor.

4 Experiments

Experiments show that MixFormerV2’s token-based heads and distillation strategies improve tracking accuracy while preserving high inference speed on GPU and CPU platforms.

  • Experimental setup: The experiments evaluate MixFormerV2 components, distillation strategies, and comparisons with state-of-the-art trackers across multiple tracking benchmarks.The study includes ablations on LaSOT and evaluations on LaSOT, LaSOText, TrackingNet, UAV123, TNL2K, and VOT2022.
  • MixFormerV2 framework: 70.6% AUC at 165 FPS is achieved by MixFormerV2-B with token-based quality score prediction, improving the baseline by 1.7% AUC with almost no latency increase.MixViT-B’s score prediction module reduces running speed by 13.0% relative to the token-based design.
  • MixFormerV2 framework: Token-based distribution regression improves accuracy over direct absolute-position prediction and enables dense-to-sparse distillation.The design uses four prediction tokens to estimate coordinate distributions rather than absolute target positions.
  • Distillation ablations: 2.2% AUC improvement is obtained when MixViT-L is used as the dense-to-sparse distillation teacher.Using MixViT-B as teacher increases the baseline by 1.4% AUC, while the larger teacher provides the stronger gain.
  • Distillation ablations: 1.7% AUC is gained from logits distillation and an additional 0.4% AUC from feature mimicking.The results support using both prediction-level and feature-level supervision for tracking distillation.
  • State-of-the-art comparison: MixFormerV2-B surpasses OSTrack by 1.5% AUC on LaSOT and 2.4% AUC on TNL2K while running at 165 FPS versus 105 FPS.MixFormerV2-S also surpasses FEAR-L by 2.7% AUC on LaSOT with real-time CPU speed.

5 Conclusion

The paper concludes that MixFormerV2 combines a fully transformer architecture with simple MLP heads and distillation-based reduction to achieve an accuracy–speed trade-off on GPU and CPU platforms.

  • Conclusion: MixFormerV2 uses standard ViT backbones on mixed token sequences and simple MLP heads for box regression and quality score estimation.The framework removes dense convolutional heads and complex score prediction modules.
  • Conclusion: The distillation-based model reduction paradigm further improves MixFormerV2’s efficiency.The paper presents dense-to-sparse and deep-to-shallow distillation as the reduction strategy.
  • Conclusion: MixFormerV2 achieves a good trade-off between tracking accuracy and speed on both GPU and CPU platforms.

S.1 Broader Impact

The paper identifies both beneficial applications and risks associated with deploying MixFormerV2 for tracking in real-world settings.

  • Potential applications: MixFormerV2 could support human-machine interaction, visual surveillance, and unmanned vehicles while maintaining high running speed.
  • Risks: Use of the tracker for illegal surveillance is a specific concern raised by the paper.
  • Risks: Unmanned-vehicle deployment may face challenges in complex real-world scenarios.The paper encourages researchers to consider the impacts of using trackers in particular real-world settings.

S.2 Limitations

The main limitation of MixFormerV2-S is the substantial training overhead caused by its multiple distillation and pruning stages.

  • Training overhead: Each model-reduction step for MixFormerV2-S requires 500 training epochs, making the full training process time-consuming.The pipeline successively distills and prunes models from 12 layers to 8 layers, 4 layers, and finally the CPU-oriented variant.
  • Training overhead: MixFormerV2-S requires multiple dense-to-sparse and deep-to-shallow pruning stages before reaching real-time CPU tracking.

S.3 Details of Training Time

Training uses eight Nvidia RTX8000 GPUs, with the two distillation stages requiring substantial multi-hour runtimes.

  • 43 hours are required for the dense-to-sparse distillation stage on eight Nvidia RTX8000 GPUs.
  • The deep-to-shallow stage takes 42 hours for 12-to-8-layer reduction and 35 hours for 8-to-4-layer reduction.

S.4 More Results on VOT2020 and GOT10k

MixFormerV2 performs competitively on VOT2020 and GOT10k, combining strong benchmark scores with high or real-time inference speed.

  • VOT2020: 0.322 EAO on VOT2020 lets MixFormerV2-B surpass CSWinTT by 1.8%.
  • VOT2020: 0.258 EAO makes MixFormerV2-S higher-scoring than LightTrack while maintaining real-time CPU speed.
  • GOT10k: 0.7% and 1.6% AO improvements over MixFormer and TransT, respectively, are achieved by MixFormerV2-B on GOT10k at 165 FPS.
  • Benchmark comparison: Table 7 reports state-of-the-art comparisons on VOT2020 and GOT10k, with best results shown in bold.

S.5 More Ablation Studies

The ablations examine prediction-token designs, progressive depth pruning, localization-head computation, and attention or probability visualizations.

  • Prediction-token design: Separate prediction tokens for the four box coordinates with a shared MLP head retain the best accuracy among tested designs.All designs estimate probability distributions for the four bounding-box coordinates.
  • Progressive model depth pruning: PMDP surpasses Tea-skip4 by 1.0% on LaSOT_ext, supporting the effectiveness of the proposed progressive pruning method.Tea-skip4 initializes the student from the teacher’s skipped layers and represents an extreme PMDP case.
  • Localization-head computation: A linear layer has O(CinCout) complexity, whereas a convolutional layer has O(CinCoutHoutWoutK^2) complexity.Cin and Cout denote input and output dimensions; Hout and Wout denote output feature-map shape; K is kernel size.
  • Localization-head computation: The Py-Corner head requires about 3.9B operations and has a computational load hundreds of times larger than T4.The comparison excludes some bias and normalization operations and notes additional interpolation operations for Py-Corner.
  • Attention visualization: Prediction tokens attend to target-relevant regions in both search and template features, extracting partial target information to relate them.Figures 5 and 6 visualize prediction-token-to-search and prediction-token-to-template attention maps.
  • Probability-distribution visualization: Probability distributions can handle occlusion and ambiguous boundaries, but strong occlusion and similar objects still cause distribution shifts.Figure 7 visualizes predicted box distributions and backbone attention weights on LaSOT examples.
Loading 2305.15896v2…